Today's Black Gold Swan - Presenting The Reason Why The CME's Crude Market Was Halted For Over One Hour

Tyler Durden's picture

Earlier today, we reported on the extended halt of the CME Globex crude market, which following an errant trading pattern, did not quite crash, but did the next best thing - go offline for a full 75 minutes. Why did this happen? Our initial speculation was that this "may have been an algo gone berserk in advance of what may or may not have been a block order.... Someone take quote stuffing a little too far today?" It turns out we were not too far off. Below is Nanex visualization of just what occurred in those seconds between 13:59:57 and 14:04:55 when "a blast of quotes corrupted a memory queue causing the software to believe the queue was full all the time." In other words just under two years after the May 2010 flash crash, another algo may have been the reason for the halt in one of the world's most important markets. At least this time there was no 10% "correction." How long until there is, and when it does happen again, will it be limited to just 10%? Oh, and whatever you do, most certainly don't expect this little incident to be brought up ever again by those in control, for any precautionary measure to be taken, or for the SEC to ever get involved. Any of those three would immediately imply something is very wrong with the market. And that's simply not allowed.

From Nanex:

NYMEX Black Swan ~ The March 2012 Crude Oil Futures Quote Loop

On February 13, 2012, starting 13:59:57, quotes for crude oil began queuing. At 14:00:35, all of the queued quotes were sent at once. Again at 14:01:08 the same 38 second block of quotes sent earlier was sent again -- old timestamps and all plus a few new quotes. Again at 14:01:18, all quotes since 13:59:57 were sent again. This repeated 12 times.

From a programmers perspective, it looks like a system problem caused a blast of quotes that corrupted a memory queue causing the software to believe the queue was full all the time.

 

Tick chart of bid prices (red) along with quote age (blue).

Note that as the cycle repeats, it includes a few more quotes (the new quotes + those since 13:59:57). There are 500 quotes between time axis labels.

 

500 millisecond chart of ETF U.S. Oil Fund (USO) showing massive quote traffic as it reacts to stale futures quotes.

 

500 millisecond chart of ETF U.S. Oil Fund (USO) showing massive trade executions in reaction to stale futures quotes.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
bob_dabolina's picture

Crude got the ol' DDoS

Fuckin' broken markets. 

nope-1004's picture

don't expect this little incident to be brought up ever again by those in control, for any precautionary measure to be taken, or for the SEC to ever get involved

 

Are you that confident they'd know what to look for.  LOL.

 

cn21713's picture

 

(1) How many times has this happened before but gone un-noticed?  Maybe this time the corruption eventually took the system off line, but overflowing a buffer and writing into areas of memory being used by other parts of the program can have totaly unpredictable results.

(2) It looks like they need to do more HFT stress tests with their software.   (But I'm sure they outsourced that to India where they are getting top notch Software Quality Assurance.)

(3) I saw someone mention something about Java and garbagecollection.  This is probably high performance code.  It might not even be written in C, it could be machine language.  Machine language is still used where super high performance is needed (like certain parts of device drivesrs).

 

bob_dabolina's picture

Remember that guy Sergey Aleynikov? The guy accused of "stealing" the code GS uses (used to) trade with (even though he designed it)? 

http://www.bloomberg.com/news/2010-12-10/ex-goldman-programmer-aleynikov-found-guilty-of-stealing-software-secrets.html

The same code Goldman Sachs admitted The bank has raised the possibility that there is a danger that somebody who knew how to use this program could use it to manipulate markets in unfair ways,” Facciponti said

 http://articles.businessinsider.com/2009-07-07/wall_street/29963107_1_stock-market-worth-millions-code#ixzz1mJbbYhCz

That story died fast and we wonder why stuff like this happens in important markets like Crude Oil.

How about the Chinese man who stole Fed code? 

http://www.reuters.com/article/2012/01/19/us-nyfed-theft-idUSTRE80H27L20120119

That's just the stuff that has made it to the press. 

ozziindaus's picture

 

 

 

 

The peoples in terror, the leaders made the error
And now they can’t even look in the mirror
‘Cause we gotta suffer while things get rougher
And that’s the reason why we got to get tougher
To learn from the past and work for the future
And don’t be a slave to no computer

TruthInSunshine's picture

The next FUBAR day - which we have only had one of since 2010, is going to cause galacticically large damage to many, many participants in these hopelessly rigged, broken markets.

I literally can't wait until the faux bulls of the sell-side propaganda get a chance to appear on financial specials and explain to the investing public why they have nothing to fear, should remain calm, and that all we be well over the intermediate rather than near term.

The amount of leverage in these insanely low volume markets (especially equity markets), coupled with the quality of the assets and competence of the managers of those asset portfolios underlying that leverage is going to create another one of those days that lives in infamy shortly.

Oh, and the bondholders; how could I forget about holders of historically low yielding bonds. They had better hope that about 24 nations and 500 publicly traded companies can keep yields on new issuances of their debt at or below the levels they're enjoying for a period that's quite close to their maturation.

Anyone participating in these markets with their own money is a complete and utter imbecile. Anyone participating in these markets with other people's money is....well...participating in these markets with other people's money.

Ghordius's picture

excellent comment, it all comes back to "participating in these markets with other people's money."

though this applies to the majority of traders - look at the leverages

Max Fischer's picture

 

 

Hey Bob...

How's your APPL short working?  Right before earnings, you said you were going short.  Let me guess... had to hit the "reset" button on that one?  That reset button sure is handy. 

Max Fischer

 

bob_dabolina's picture

I did short aapl....last year and with success. You can find the posts somewhere on my website as I have not the time nor the inclination to sort through them myself. 

tradeonfire.blogspot.com

Max Fischer's picture

 

 

No. No. No. Bob.  I'm not going to spend a second going through your stupid blog. 

I'm referring to your AAPL short from a few weeks back, right before earnings. You posted on ZH that you were going short, and since then, its catapulted from ~$425 to ~$500.  

Of all the stocks to short last year, AAPL was NOT one of them.  In the past year, AAPL has moved from ~$300 to ~$500, so if you've shorted that stock with any success, I'd recommend you change your account from a fake one with a "reset" button to a real one.  Quite frankly, I think you're a liar. 

 

 

bob_dabolina's picture

You must have me confused with another Bob. There are many Bobs floating around, if you can get the comment you're referring to I'd like that. I stand up to the positions I make, people who know me here would tell you that. 

I am looking at shorting aapl and I made this post just for you Max: 

http://tradeonfire.blogspot.com/2012/02/aapl-without-comment.html

You a buyer here or what?

...I just may short the FUCK out of AAPL but if I do, my blog would be the first place to check. It's sooooooooooooooo tempting looking at that chart.

tulip bulbs

fuu's picture

"Quite frankly, I think you're a liar."

 

That's quite the slice of ironic pie there bubba.

Bringin It's picture

The reality here is that Bob once not too long ago, pantsed himself.  Pantsed is a word.  You can look it up.  Max remembers.

In Bob's defence, I will say that he comes up with some good posts some times, like one above on this thread.

QuasiYoda's picture

Max Fischer you are a destructive personality for a trading chat site if this post is any indication of your work.  You must be losing money as witnessed by your need to attack other traders rather than give positive counsel. Nobody makes perfect calls, as such by all seasoned traders it is expected to take small losses on the road to large gains.  You however are badgering another trader for making a call against a stock in a parabolic move.  That is a high risk high reward play but one that one should Not become ego identified with at all.  It is this type of badgering that you are pushing that can trick a trader into making bad moves due to ego identifying with the trade.   Currently you appear to be nothing but a spoiled little boy striking out at others for petty reasons but with dangerous pot'l consequences for all.  Grow up or go hire a shrink to help you with your issues.

 

AllWorkedUp's picture

Vegas is so much safer and much better odds. No wonder the volume is drying up.

Henry Hub's picture

You're right, there is no way HFT is written in Java. It's waaay too slow. This has to be written in Assembly Language. It will probably turn out to be a buffer overflow problem. HAHAHA!

I used to program the IBM 360 in Assembly Language back in the 1960's. Often times the bug was buffer overflow. I couldn't believe some asshole didn't code this properly back then. But it still goes on today.

Andre's picture

I strongly doubt assembly language was used. Not that many people remember how to do complex apps in it any more. You, me a few others. Most folks these days have a hard time figuring out how to use it even in drivers.

I'm thinking C, full optimization, a Linux distro stripped of a lot of bloat, on a VERY non-standard machine with very non-standard internet bandwidth. DNS cache locks as well, so there is no lookup time. Crays use(d) Unix, so it can be made to run pretty fast.

For a real trip down memory lane-

"There are two things a man must do before his life is done-

Write two lines in APL, and make the buggers run."

trav7777's picture

never know; could be proprietary.  Some shit is still in AL.  Linux is what the big supercomps are using now.

bag holder's picture

Check yourself before you wreck yourself

GFKjunior's picture

Most quant/htf firm job ad's want extensive C++ background, but more importantly intense maths, more than one job posting I've seen said they run Red Hat.

Rynak's picture

Maybe, but the software in question was NOT hft algos (i.e. the clients), but the server software of the exchanges, that let HFTs send orders to them.

This is a one-to-many scenario..... the HFT algos could be written in python, and still be fast enough for HFT-resolution.... the server on the other hand, it has be really fast at handling incoming requests.

Noteworthy is that in this case, the server doesn't need to do much computation.... it mostly just has to store incoming data, and send the very same data out. Thus, the requirements are mostly one of fast data transport and storage: You don't need ASM for that - C should work fine.... you do however need really fast datastructs (fast methods to organize data), fast memory and most importantly fast interconnects to the clients on which the HFT-algos run.

Oliver Jones's picture

So they probably use something like memcached for the server.

Non Passaran's picture

And "probably" it's MS DOS 6.0! As others have said, the client side compute stuff is a different system from the system that places orders and ticker farms that receive data feeds. The Internet is full of detailed information of how that works. And probably there's no memcached in there. As TFA says, they overwhelmed the queue. I'm not a fucking expert either, but if you design the system to be able to handle X trades or bids/asks per second, then you allocate that much resources (or maye 100 or 200% more, to be able to deal with irregularities). But then you get these maniacs who send 1000% more requests and of course the system choked. Regarding other comments whether HFT should be banned, I disagree. IIRC last year one of ZH articles (or a comment) said exchanges can (could) limit per-participant data inflow and price it accordingly, similar to traffic congestion charges. In this particular case the system couldn't do that so they allowed to be overloaded. They should spend more money to build a better and bigger system, or introduce congestion charges, or kick this HFT shop from their exchange. There's no need for the government to regulate this as the only result would be HFTs would invent some other new trick (or switch to dark pools) and because of compliance and other overheads transaction costs would increase for regular traders. Screw that. As for claims of price manipulation, I tend to believe they do that, but that too is OK as long as we know what they're up to. You change your trading style (or maybe stop day trading) and that's it. Or a year or two later Amazon comes with a $1/day HFT platform for the average Joe and individual traders could write their own code/algos, use Amazon's templates or use/modify open source HFT alogos.

The government needs to ensure transparency and disclosure and that's about it (and since they can't even do that properly, why would we want them to try to do even more? The more they get involved, the worse it gets!). It makes more sense to push for complete transparency and disclosure and ask them to bugger off when it comes to the rest (including insider trading).

Archon7's picture

If it reaches metabolic states while conjoined without standard flux, can beam width social loci ever seem hardly satisficed?  Haha!!!  of course, jam discordant complexity with retrofitted clarity!  

non_anon's picture

ha ha, how about cobalt

non_anon's picture

cool, is google fucking with anybody else, it keeps crashing my ie explorer, or am I drunk off my ass?

Rynak's picture

Probably either some IE plugin (BHO), or google once again toying with too much new webtech shit. Do they even remember anymore, how they as a search engine went to #1?

Everybodys All American's picture

I'm confident they know what to look for. I'm also confident they know who is behind this manipulation/front running as well. I'm pretty sure these fellas are the same guys who buy our treasuries and who have gunned the futures nearly every day since the US debt was downgraded last year.

TheMerryPrankster's picture

Ray Charles isn't the only one who enjoyed playing with black boxes. Wall street is fascinated by the magic of investing in things opaque.

Why not just burn your money in the fireplace? You enjoy it twice, watching it burn and keeping you warm. And then you stay warm knowing you kept it out of the clutches of bankers.

Algos- its for people that think the lottery has too good of odds.

non_anon's picture

$10 a gallon regular unleaded by summer, mark my words

Non Passaran's picture

Thanks for this extremely insightful and reliable tip!
I'm going to invest all my savings in oil futures!

SHEEPFUKKER's picture

The bitchez got the glitchez.

lineskis's picture

All your memory are belong to us!

TheMerryPrankster's picture

and de bankerz gots all de richezzz.

Eireann go Brach's picture

This looks like a Cat Scan of the Greek finance minister's whacked fucking brain! It is full of goobly gook, dead brain cells and makes absolutely no sense!

max2205's picture

But 'they' sure made sure USO gap'd up after the halt. Unblvabl

rubyruffruff's picture

never a broken market if u run a fresh mkt manipulating algo in the last 30min's of trade.....just a coincidence.......ask infinium capital (a bunch of great lads who were fined 1mil for a small coincidence prob.).......was watching the episode unfold in carbon form this afternoon......very funny.......the algo boys need 2 sharpen the spears......sloppy and obvious.....not the hint of a great mind in the mix....this should b an easy one for cftc if anyone is awake.....xoxoxoxo

 

MarketWatchTerrorist's picture

They work hard to convince us all that the markets obey rules.  They've got millions studying market charts like astrologists study the astrological signs.  They tell everyone that the markets return X(XX?)% annually since 1945, and that equities are the safest high yield investment around.

 

But it's all bull shit.  The markets are an artificial construct that they control.  And when the market does something they don't like, they change it.  After all, it's their game, they make the rules.  And when they feel like it they break those same rules.