Explaining The Knightmare

Tyler Durden's picture

On the day Knight blew up, and its stock tumbled initially to the $7 range, when the market speculated the loss may be "only" as large as $150-$250MM, we calculated courtesy of a Nanex analysis which suggested the modus operandi of the "berserk" algo, that the finaly loss would be far greater. This was confirmed a day later when it was made public that the final loss KCG experienced in just 45 minutes of trading was at least $440 million, and will be far greater when the losses associated with all the external trading reroutes are calculated. Nonetheless, with the SEC still completely mum on the whole issue (for one simple reason: it has no idea what happened, and is quiet not out of malice, but sheer incompetence), there is still an open question of just what happened. Here, once again from Nanex, is the complete post-mortem of a firm that was almost fully mortem, explaining everything that happened.

From Nanex:

The Knightmare Explained

We believe Knight accidentally released the test software they used to verify that their market making software functioned properly, into NYSE's live system.

In the safety of Knight's test laboratory, this test software (we'll call it, the Tester) sends patterns of buy and sell orders to its new Retail Liquidity Provider (RLP) Market Making software, and the resulting mock executions are recorded. This is how they could ensure their new market making software worked properly before deploying to the NYSE live system.

When the time comes to deploy the new market making software, which is likely handled by a different group, the Tester is accidentally included in the release package and started on NYSE's live system. On the morning of August 1st, the Tester is ready to do its job: test market making software. Except this time it's no longer in the lab, it's running on NYSE's live system. And it's about to test any market making software running, not just Knights. With real orders and real dollars. And it won't tell anyone about it, because that's not its function.

For stocks where Knight is the only one running market making software as a RLP, and the Tester is the only algo trading that's crossing the bid/ask spread, then we'll see consistent buy and sell patterns of trade executions, all marked regular, and all from the NYSE, and all occurring at prices just above the bid or just below the ask. Examples include EXC and NOK and you can see these patterns in charts here. The Tester is functioning just as it did in the lab, and Knight's market making software is intercepting these orders and executing them. Knight won't lose any money on these trades, but they will be generating a lot of wash sales.

For stocks where Knight is not the only market maker, or when there are other algos actively trading (and crossing the bid/ask spread), then some, or all of the orders sent by the Tester will be executed by someone other than Knight, and Knight will now have a position in the stock. Meaning it could be making or losing money. The patterns generated for these stocks will depend greatly on the activity of the other players.

Because the Tester indiscriminately buys at the ask and sells at the bid, and because the bid/ask spreads are very wide during the open, we now understand why many stocks moved violently at that time. The Tester was simply hitting the bid or offer, and the side it hit first, determined whether the stock opened sharply up or down.

Since the Tester doesn't think it's dealing with real dollars, it doesn't have to keep track of its net position. It's job is to send buy and sell orders in test pattern waves. This explains why Knight didn't know right away that it was losing a lot of money. They didn't even know the Tester was running. When they realized they had a problem, the first likely suspect would be the new market making software. We think the two periods of time when there was a sudden drop in trading (9:48 and 9:52) are when they restarted the system. Once it came back, the Tester, being part of the package, fired up too and proceeded to continue testing. Finally, just moments before an economic news release at 10am, someone found and killed the Tester.

We can fully appreciate the nightmare their team must have experienced that morning.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
Id fight Gandhi's picture

Skynet has become self-aware

sunaJ's picture

You have a firm believer here that one does not have to look for malicious intent as an explanation of events when sheer incompetence will suffice.

NotApplicable's picture

Given the last two days EVERYTHING in my office that uses java database connectors is going haywire... yeah, I can see this.

Thankfully I don't have a single app using them (yay!)

*meanwhile I'm testing my code where? the production server, baby!*

LongSoupLine's picture

"We meet again...Mr. Anderson."

NotApplicable's picture

How can it really be a true test otherwise?

Dr. Kenneth Noisewater's picture

There's a certain class of bug in model or order flow code that is only revealed when actual money is on the line.

calltoaccount's picture

Except where the SEC is concerned, incompetence is the plausible "cover story" for massive long term gross criminal connivance and enablement.

Various sections of the US Securities & Exchange Acts expressly oblige the SEC to refrain from and prohibit any acts that harm the investing public — yet they have regularly, consistently, almost addictively ignored the statutes’ mandate– to instead permit and encourage explicitly proscribed conduct that unfairly and illegally steals from the public to enrich hedge funds, investment banks, broker-wheeler-dealers and other market insiders– all to the extreme prejudice of the investing public and the global credibility of US capital markets.

The crisis now decapitating the world’s economies didn’t have to happen– would not in fact have happened if those charged with protecting the financial system and the investing public— namely Congress in general and the Securities and Exchange Commission in particular— had simply done their jobs, upheld their oaths, and seen to the enforcement of laws already on the books, instead of consistently looking the other way; or worse still, directly aiding their campaign contributors, benefactors, patrons, employers and future employers by contemptible acts of public disdain such as repealing Glass Steagel, allowing Reg Sho’s grandfathering of billions in counterfeit shares, scuttling the systemic market protections provided by the uptick rule, and encouraging countless Trillions in credit default swaps.

Kiwi Pete's picture

How is this not theft? I think it's called skimming or front-running here and is totally illegal. Straight to jail for anyone who does it. What has America become? At least you can land a buggy on Mars. Well done Nasa!

andrewp111's picture

The SEC allowed Madoff to operate for 30 years. They looked the other way at Stanford. They are as incompetent as it gets.

Zero_Sum's picture

This wasn't SkyNet. It wasn't self-aware. Knight accidentally released financial ebola into the NYSE well water.

MsCreant's picture

ebola that only made them ill...

Thomas's picture

That's not true. There were all sorts of winners and losers. Some HFTs were certainly winners. Others, by definition, lost. This system is simply all fucked up. I have one thing to say to these algo traders: Just keep crashin' and burnin' boys and girls. You will eventually all commit mass suicide. 

"Look what they've done to my song, Ma."

--Melanie--

MassDecep's picture

This video is a must see about Knight and them crossing TPTB.

http://youtu.be/9QAdyVZG5io

DanP1966's picture

This still makes no sense.

A) Why would a test system have any connection to anything other than another test system & QA?

 

B) Why would you configure your test system to do what this article is claiming? It would have no test value even for negative testing.

 

C) It still does not explain why a major upgrade would be done mid-week.

 

D) It still does not explain how a test system, even if redirected to point to the exchanges production environement should be able to execute trades.

 

E) Assuming that the test system did have the flaw described (nonsensicle on its own) it still does not  explain HOW a test system gets pointed to the exchange production environement or how the exchange could allow it to happen. There HAS to be some form of certification on the exchange side when new code is linked to it.

 

F) How would this have gotten past even a basic QA?

 

This still smells to me like somebody did something intentionally.

 

Is it plausible that it was all an accident that resulted from a cascade of best practice failures? Ok...sure...it is plausible but so unlikely as to be nearly rediculous.

 

BUT....then.....given the number of rediculous actions that violate best practices that we have seen from Wall Street I suppose I would not be that shocked to know that their IT operations are slipshod.

Bazinga's picture

I can fully appreciate the nightmare their team experienced but I DON'T CARE. Companies that F-up need to go out of business if the F-up is big enough to bury them. If you can't stand the heat, get the hell out of Skynet's kitchen bitchez.

Karl von Bahnhof's picture

Damn! And I hoped it was china!

MachoMan's picture

I haven't looked at anything more than headlines, but if I was the head of a now failed entity, I'd be apologizing for my complete and total idiocy...  it just seems like there is more of an apology for risk taking behavior biting you rather than your stewardship failing...  it's basically a deflection of the real problem...  come out, admit failure, and never manage money again...  spend the rest of your life on the beach with the spoils you made from a non productive activity.  Kudos.

You Didn't Build That's picture

Luckily, Knight didn't build that.

RingToneDeaf's picture

It is all on the level

slewie the pi-rat's picture

kinda like a digital chernobyl?

btw i finally understood something from nanex, BiCheZ!   YaY! 

slewie the pi-rat's picture

hiFreakTesting101

new media:  new non-atomic record destruction of digits belonging to self & others

can we get a nurse in here with a shot of thorazine, please?  stat!

"intelligent design" has met the twilightZone where a lifetime of robo_T trades were condensed into a corporate nightmare on elmo street

video at 5!

bigwavedave's picture

Goldman bought the entire book at a discount. Squiddly

Jupiter's picture

 

Could have happened to any firm, not evidence of any exploitative HFT, etc.

 

larz's picture

except that the tester is supposed to be us traders out here and our 401k's etc

MachoMan's picture

I suppose it's inevitable...  however, just because an activity was bound to happen, doesn't mean it was bound to happen here and now.  It certainly is a plausible explanation, but it still leaves a lot to the imagination...  such as the degree of competency of the HFT implementers, their ties with competitors, how deep their pockets are, etc. 

The other thing is, was this their first rodeo?  Did they finally get a PC about the time that the content on the internet started to suck?  If they're just venturing into the world of HFT, then they're fucking dinosaurs (only a matter of time before the cigarettes kill them all, right kids?). 

Praetorian Guard's picture

However, I wonder to what extent someone actually profited from this "mistake"? maybe it was an inside job? If another firm, ie GS knew this was going to be launched at a certain time, they could manipulate the market to force this algo to work in their favor and make tons of $$$$$$. Hmmm....

Dr. Engali's picture

It's my belief that there are no accidents. I am willing to bet there is more to this story.

CvlDobd's picture

Word!

I think the best we can hope for is the Knight CEO selling the story to Hollywood for millions and digging for tidbits of truth in John Malkovich's character.

magpie's picture

...and i guess India struck back at Reuters.

financial apocalyptic contagion's picture

yea the explanation that "the tester" was randomly included in the market making software is hard to believe.

It's more likely Ethan Hunt/ James Bond were behind this

NotApplicable's picture

Well, it wouldn't have been "random," but rather, sloppy code updates where someone either just copied the whole directory structure (which assumes it was segregated), or forgot to set the test flag to false somewhere.

CheapBastard's picture

Despite losing $440 million, the silver lining is they probably saved $2,456 on hiring inexperienced DBAs and software engineers.

StandardDeviant's picture

Being a software developer myself, I don't find it at all hard to believe.

You can see on the Nanex charts (Chart 6 at http://www.nanex.net/aqck2/3522.html) that there were two points where volume dropped suddenly, then picked up again, suggesting that the Knight staff thought their new MM software was at fault and bounced it, unaware that the tester was also part of the software release and was being restarted right along with it.

(Just before 10:00, someone seems to have figured out what was going on, and did a quick "kill -9".  Imagine the look on his face...)

mac768's picture

common sense is not common

 

LongSoupLine's picture

nor is it a requirement of employment within the financial sector...whether it be operations or enforcement.

Meesohaawnee's picture

im not joking when i joke. bullish!! seems to me the more fraud the worse earnings data are the bigger the vapor melt up to cover up how bad things are for obama. its payback for wall street for obama not locking any up and letting ben keep the bonuses flying.

css1971's picture

If so, this'd be an example of cutting great big pages never mind corners in their testing environment.

FinalCollapse's picture

If Knight released faulty software into NYSE then both parties are equally guilty. NYSE failed to perform the due diligence.

KidHorn's picture

You don't understand. Night doesn't release software into NYSE. They have software that communicates with the NYSE. The NYSE had nothing to do with this.

It would be analogous to blaming nasdaq if you fat fingered a MSFT trade and bought 10,000 shares instead of 1,000.

FinalCollapse's picture

I will respectfully disagree with you. Let's assume that you properly corrected me. There are two parties communicating and both parties must ensure that the software that facilitates the communication between them is error free. 

The younger generation of developers came, replaced the experienced ones and threw out all quality standards. Results are as you see.

Did I make it clear? In software you don't want to have faulty bridge between two systems. It is responsibility of both parties to ensure that the software is error free, crash free, etc. 

Grab popcorn and coke. Enjoy the show as young and inexperienced are learning job on their own errors.

NotApplicable's picture

I understand and agree with what you're saying, but that doesn't cover the full scenario. The tester software is NOT part of the system. It is an add-on that replicates customer behavior. NYSE, for instance, would have absolutely no idea it was running.

jonjon831983's picture

Hmmm lets see if got this right.

 

More like say you (Client) connect to a FTP site (Server).  Whatever software you use does not matter as long as your authentication etc was successful.  As client on the FTP whatever you do within your access rights.  So, say you are allowed to upload/delete stuff, if you delete stuff accidentally - will the Server be responsible for your accidental delete?

FinalCollapse's picture

The answer is 'Absolutely Yes'.

The Server is responsible for its own contents and performance. If the FTP client rampages through out the Server deleting important production stuff - then the Server's Sys Admin should be hang even before the client's one.

This is where the younger and less experienced completely fail to understand the IT, and for me this is like talking to the wall.

Server should grant proper rights and make sure that the restrictions in place will protect the system stability. Anything that can cause Server's instability should be disallowed or properly tested by the Server (on top of the Client testing).

Not only the Knight blew up but the entire stock market was unstable for a while. We didn't have complete crash but the stock market was unstable and many investors lost money because of it.

It is such a simple and basic concept, I have trouble arguing about it. Really.

Silver Bully's picture

'The answer is 'Absolutely Yes'.'

 

Not quite. This is NOT like a sys admin allowing someone access to an FTP login.

This is more akin to something like a game client logged into an MMO server, controlled by a botting program. Think World of Warcraft gold farmers. An automated bot program logs in to the client and makes the character do things to earn money. The server is unable to automagically detect the bot because all it can see is the client. The only way to detect a bot exists is through the actions of the client. Once this occurs, THEN the server can cut off access to the damaging client, but not until that client does something wrong. That is how the system is designed.

Now take it one step further. Imagine a game where NO humans control the clients. Instead a horde (ahem) of HFT's run them. The server can only track what the clients do. So when one of the bots goes nuts and decides to buy high and sell low, disrupting the in-game economy, who's 'guilty'? The server? Heck no. The guilty party is the human being who ran the messed up bot, no one else. Would Activision-Blizzard offer everyone gold refunds from all the damage a lone bot created? No! They'd ban the offending client account and be done with it. The winners win, the losers lose.

Knight has done the same thing. They released their bot program into the game without making sure it was working properly. They paid the price, the market was temporarily disrupted, and the damage is done. A 1500 person company went bankrupt in 45 minutes. This is part of the risk anyone takes being involved in the current complex system. if you don't know what you are doing (which would be just about everyone with how complex this has become) then don't risk your (and someone else's) money.

 

Here's another gaming cliche,

TL;DR: don't play a game you do not understand. If you do, and you get burned, it is no one's fault but YOUR OWN.

icanhasbailout's picture

Only in the sense that the NYSE is knowingly facilitating fraud/illegal trading activities by letting these algos run in the first place.

Eternal Complainer's picture

"We can fully appreciate the nightmare their team must have experienced that morning."

Good on them! :)