Presenting The Reason For The Thanksgiving After Hours Melt Down: Another Electronic "Glitch"

Tyler Durden's picture

As readers will recall, just after the abbreviated Thanksgiving session, there were some pretty dramatic afterhours fireworks, both in stocks, and in a variety of volatility indices, that of gold (^GVZ)most notably. As the charts below capture, the drop in the futures had offset basically the entire day's upside in the span of milliseconds, leaving many wondering just what had caused this. Luckily, courtesy of the Tabb Group's Paul Rowady we now know that this was yet another glitch borne out of the hyper-technological sophistication of the current marketplace, in which the smallest error can and will propagate through the system uncontrolled resulting in major losses for those who are aligned on the same side as the ponzi. In other words: it was yet another flash crash which luckily did not have a major impact as virtually no volume was being transacted in the market. All this merely means that Ben Bernanke, who is doing everything in his power to boost asset values, has increasingly more variables working against him as the system continues being pushed ever further away from its natural equilibrium, until one day it all just burns down.

These were the charts we posted on November 26:



and Gold:

And here is the explanation for what happened:

The Tale of the Buried Headline

On Friday, Nov. 26, a technical oversight allowed stock and options exchanges operated by BATS Global Markets to extend regular trading beyond the abbreviated holiday session, a session that was officially scheduled to close at 1:00 p.m. EST.

The resulting cascade of problems this oversight caused is both fascinating – for its symbolism of how oddly precarious some of our most mature markets have become – and cautionary for its addition to the growing list of precedents that could be interpreted to foreshadow things to come.

Allowing the regular trading session to continue beyond the scheduled close, BATS inadvertently marked after-hours trades as normal-session trades, which ultimately influenced the closing prices of equities and indexes on that date. That in turn caused options that would have previously expired worthless to be marked in-the-money and therefore exercised.

Since no other exchanges were distributing market data marked as “regular session” after 1 p.m. on Nov. 26, the BATS’ market data was used by market data vendors like Thomson Reuters, Bloomberg and IDC for that day’s closing prices, and the Options Clearing Corporation (OCC) used that data to determine which options were in- and out-of-the-money for the month-end exercise.

For example, while the closing price at 1 p.m. for the SPDR S&P 500 ETF (SPY) was greater than $119, the BATS prices ultimately used by the OCC ended up at less than $119. The impact of this (which was further inadvertently exacerbated by the OCC’s auto-exercise service) was for SPY Nov. 119 calls to expire worthless and the SPY Nov. 119 puts to be in-the-money – when the opposite should have been the case. Eventually a few P&Ls on Monday morning were askew.

Clearly, the scenario of a holiday-abbreviated session falling on the third Friday – expiration Friday – of a month had not properly been considered in all the hours of development for these particular exchanges. And, clearly, market data vendors had insufficient data scrubbing routines in place to test the validity of certain raw market data.

Finally, it is clear that the OCC did not have its own procedures in place to vet certain market data, even if it was coming from the most venerable sources. In short, this was not a data problem. This was a processing problem in which a bunch of folks dropped the ball.

While all the yammering and chattering around this episode suggests that relatively little damage was done, it speaks of the wild world of uncontrollable complexity risk.

So fasten your seat belts because it could get worse before it gets better. The good news: what doesn’t kill us makes us stronger.
Back in the day when I was the CIO for a high-turnover statistical arbitrage fund, I used to feel a perverse sense of good fortune when my team would experience unforeseen “glitches.” Since there was simply no way to anticipate every possible permutation of challenges “in the laboratory,” the only way to come anywhere close to bullet-proofing our systems and processes was to keep our jalopy on the proverbial track for as long as possible and overcome whatever technical or market-oriented challenge came our way – thereby ultimately (and hopefully) evolving our platform to a finely-tuned performance machine.

The same logic can be applied to the aforementioned BATS event as well as the recent Flash Crash. (In the case of the Flash Crash, the regulators bear some responsibility for not “federating” the rule book for an increasingly fragmented marketplace.)

Confidence issues and other grumblings aside, these problems – now that they’re in the rear-view mirror – actually make markets better, stronger healthier and more fault tolerant.

Sure, on one level, any kind of glitch on financial exchanges spooks the bejesus out of an already-traumatized public and serves as catnip for a mainstream media machine conditioned to inflate any and all imperfections in the financial firmament. However, the fact that these events have expanded our collective library of possible scenarios is a net positive for everyone (assuming we learn from them).

Here’s the buried headline: While we should always expect and prepare for the unexpected, the capital markets’ growing dependence on technology to do more, faster and with fewer people represents a recipe for an increasing frequency of surprises.

The quality control requirements for this level of unprecedented complexity defy comprehension, particularly within the largest financial intermediaries and the most fragmented markets. It’s a wonder that more glitches don’t find their way to the light of day.

To begin with, regulators around the world could play a major role in combating complexity risk by simplifying the rule book.
But the truth of the matter is that we simply cannot understand the full spectrum of what could go wrong.

One of our industry’s greatest ironies may be that the most challenging aspect of combating complexity risk is in finding ways to simplify systems, processes and operational infrastructure.

An increasing reliance on technology and automation creates intangible costs (and risks) that are not properly appreciated in our business. Excessive cutting of people in favor technology – or over-automation – incrementally exacerbates these risks because people are the only defense against “what we don’t know.”

Exhibit A: State Street just announced that it will cut 5 percent of its workforce through the end of 2011 as part of an “information technology transformation.”

Mark another point in the win column for complexity risk.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
themosmitsos's picture


I do NOT buy it!!

Don't fucking tell me somebody fucked up and "Accidentally left the markets trading AH" !!

Fuck that!!


Sudden Debt's picture


You want the blue or red once today?

Water or Vodka to wash them down?

the not so mighty maximiza's picture

a slow gin fiz with a chaser of prun juice.

Kobe Beef's picture

I, for one, welcome our new robot masters...

cossack55's picture

Anyone need to borrow a match?

TWORIVER's picture

For the Gold bugs. You are in good shape as Head and Shoulders is invalidated and a rally to 1500 is likely. The caveat is if we break 1385 first, That would negate any bullish potential. I think you guys will get your rally, just be mindful of reality below 1385. Have a good one.

Sudden Debt's picture

I guess will first see some attempts to get that down. But the more they try, the worse it will get for them :)

TWORIVER's picture

don't disagree, but the bulls should be able to get it over 1450 without much trouble. Gold corrrected from 1424 to 1310, so the upside projection would be 1485 - 1535. Just being mindful of 1385 in the meantime.



jus_lite_reading's picture

Something smells downtown, and it ain't the Fulton fish market!

Sudden Debt's picture


StychoKiller's picture

Aye, we're doin' the best we can Captain, but these new pennies don'ave enou' copper in'em!

doggis's picture

how truly broken are these markets when moves reflects insider information or flash crashes. what is wrong with market participants, why is this 'normalized' ???? bull market my ass!

Oh regional Indian's picture

Flash Crash my Ash!
No accidents, per se, here. The system is fairly well controlled. The timing alone makes it suspicious. It was another test of one sort or another.

The vicious down move, when it does come, will be a controlled or timed crash.


Sudden Debt's picture

anything is possible.

It could well have been that Benny B. took his playstation controllers instead of his markets controllers and that the ship went down because nobody was pressing the buttons.


FranSix's picture

How might one go about pulling the plug, for example?  You set the market up for hyperinflationary expectations and currency collapse while you work towards negative interest rates in the short term market.

Not that anyone really has any control over the whole issue.

Oh regional Indian's picture

Hey SD, your comments are always, interesting, funny, appropos, etc. But your Avatar drives me nuts, I have to scroll away!


Just letting you know.

And yes, so many possibilities. 

Market makers and market "breakers", co-located and co-controlled. Not a great idea, eh?


A_MacLaren's picture

How can it be that the cause/blame is not directed Waddell and Reed?

Aren't they always to blame, scapegoat and all... ?

Crispy's picture

Of course we could take all the volume back to the floors. But then there would still be complaints of robbery.


trav7777's picture

look; the reality here is that you have potentially thousands of algorithms competing.  There has been no integration testing of these things.  It's like a digital version of Robot Wars, you don't know what the other guy's fielding until you see it.  A lot of times shit just happens that's unanticipated and the speed of computers means it runs away from you faster than Chernobyl going suddenly critical.

Imagine if you installed 500 programs onto Windoze which were programmed to terminate the other running programs and hog system resources.  That is the modern market.

mikla's picture


It is a very simple engineering concept that "summing" noise curves results in *increased* rare events of dramatic amplification.

The casual observer might expect that over-all, summing noise curves results in a more *stable* system -- the noise curves average out each other.  And, that is true.  Most of the time.  Until it's not.

In effect, there's a rare chance that all the curves you are summing are "accidentally" ALL at their peak, yielding a HUGE net peak, or ALL at their trough, yielding a HUGE net trough.  That's Taleb's Black Swan.  That *will* happen, because it can't not happen.  And, with HFT, it will happen at increased frequency and increased severity.

Making it even *worse* (we were screwed even before this next assertion), these market systems engage in *intentional feedback loops*.  When one predatory algorithm sees a trend, it "piles onto" that trend.  That means that literally, we have a system ENGINEERED to create Armageddon.  (No, "breakers" won't fix this, as seen in continual breach of lock-limits on commodities futures markets, and as seen in dramatic "drift" from "true" price discovery.)

Taleb's complaints were that we *could* engineer for "stability" (yes, at the possible expense of "perceived short-term efficiency"), but instead, we engineered for brittleness and a system intentionally designed to destroy the world market system.

Thus, it is not a question of "if" but "when".


If you're a visual person, check out the "red spike" in the graph below:  Things like that shouldn't occur, but will, and do, at increasing rate and amplification based on the above discussion.

mbeebe's picture

This did not occur on the third Friday or typical monthly was the 4th Friday. Nonetheless, there were some weekly options that were exercised that should not have been.

SwingForce's picture

The 26th was not the 3rd Friday, only Weekly Options expired that day, not monthly.

Bearster's picture

Just think for a moment, if it was the Internet itself that he was describing.  Words like "fragmented", "unprecedented complexity", "dependence on technology", "being forced to do more with fewer people", would not be used.  The Internet is extremely robust, as those who are following the failed attempts to censor wikileaks can see.

What's the difference?  The Internet is not micromanaged (mismanaged) by regulators.

CitizenPete's picture

    << be leave >>


 !piD gnikcuF ehT yuB

spekulatn's picture

Markets should never close. Trade 24 hours a day. 7 days a week. Even on holidayz.



TraderTimm's picture

This is as 'accidental' as CME's order-crossing from QA to Production on Globex. How they still have any traders confident in their technology is utterly unfathomable.