This page has been archived and commenting is disabled.

Explaining The Knightmare

Tyler Durden's picture


On the day Knight blew up, and its stock tumbled initially to the $7 range, when the market speculated the loss may be "only" as large as $150-$250MM, we calculated courtesy of a Nanex analysis which suggested the modus operandi of the "berserk" algo, that the finaly loss would be far greater. This was confirmed a day later when it was made public that the final loss KCG experienced in just 45 minutes of trading was at least $440 million, and will be far greater when the losses associated with all the external trading reroutes are calculated. Nonetheless, with the SEC still completely mum on the whole issue (for one simple reason: it has no idea what happened, and is quiet not out of malice, but sheer incompetence), there is still an open question of just what happened. Here, once again from Nanex, is the complete post-mortem of a firm that was almost fully mortem, explaining everything that happened.

From Nanex:

The Knightmare Explained

We believe Knight accidentally released the test software they used to verify that their market making software functioned properly, into NYSE's live system.

In the safety of Knight's test laboratory, this test software (we'll call it, the Tester) sends patterns of buy and sell orders to its new Retail Liquidity Provider (RLP) Market Making software, and the resulting mock executions are recorded. This is how they could ensure their new market making software worked properly before deploying to the NYSE live system.

When the time comes to deploy the new market making software, which is likely handled by a different group, the Tester is accidentally included in the release package and started on NYSE's live system. On the morning of August 1st, the Tester is ready to do its job: test market making software. Except this time it's no longer in the lab, it's running on NYSE's live system. And it's about to test any market making software running, not just Knights. With real orders and real dollars. And it won't tell anyone about it, because that's not its function.

For stocks where Knight is the only one running market making software as a RLP, and the Tester is the only algo trading that's crossing the bid/ask spread, then we'll see consistent buy and sell patterns of trade executions, all marked regular, and all from the NYSE, and all occurring at prices just above the bid or just below the ask. Examples include EXC and NOK and you can see these patterns in charts here. The Tester is functioning just as it did in the lab, and Knight's market making software is intercepting these orders and executing them. Knight won't lose any money on these trades, but they will be generating a lot of wash sales.

For stocks where Knight is not the only market maker, or when there are other algos actively trading (and crossing the bid/ask spread), then some, or all of the orders sent by the Tester will be executed by someone other than Knight, and Knight will now have a position in the stock. Meaning it could be making or losing money. The patterns generated for these stocks will depend greatly on the activity of the other players.

Because the Tester indiscriminately buys at the ask and sells at the bid, and because the bid/ask spreads are very wide during the open, we now understand why many stocks moved violently at that time. The Tester was simply hitting the bid or offer, and the side it hit first, determined whether the stock opened sharply up or down.

Since the Tester doesn't think it's dealing with real dollars, it doesn't have to keep track of its net position. It's job is to send buy and sell orders in test pattern waves. This explains why Knight didn't know right away that it was losing a lot of money. They didn't even know the Tester was running. When they realized they had a problem, the first likely suspect would be the new market making software. We think the two periods of time when there was a sudden drop in trading (9:48 and 9:52) are when they restarted the system. Once it came back, the Tester, being part of the package, fired up too and proceeded to continue testing. Finally, just moments before an economic news release at 10am, someone found and killed the Tester.

We can fully appreciate the nightmare their team must have experienced that morning.


- advertisements -

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
Mon, 08/06/2012 - 13:45 | 2682173 Id fight Gandhi
Id fight Gandhi's picture

Skynet has become self-aware

Mon, 08/06/2012 - 13:58 | 2682206 sunaJ
sunaJ's picture

You have a firm believer here that one does not have to look for malicious intent as an explanation of events when sheer incompetence will suffice.

Mon, 08/06/2012 - 14:13 | 2682300 NotApplicable
NotApplicable's picture

Given the last two days EVERYTHING in my office that uses java database connectors is going haywire... yeah, I can see this.

Thankfully I don't have a single app using them (yay!)

*meanwhile I'm testing my code where? the production server, baby!*

Mon, 08/06/2012 - 14:43 | 2682410 LongSoupLine
LongSoupLine's picture

"We meet again...Mr. Anderson."

Mon, 08/06/2012 - 15:19 | 2682514 NotApplicable
NotApplicable's picture

How can it really be a true test otherwise?

Wed, 08/08/2012 - 21:41 | 2689672 Dr. Kenneth Noi...
Dr. Kenneth Noisewater's picture

There's a certain class of bug in model or order flow code that is only revealed when actual money is on the line.

Mon, 08/06/2012 - 15:50 | 2682606 calltoaccount
calltoaccount's picture

Except where the SEC is concerned, incompetence is the plausible "cover story" for massive long term gross criminal connivance and enablement.

Various sections of the US Securities & Exchange Acts expressly oblige the SEC to refrain from and prohibit any acts that harm the investing public — yet they have regularly, consistently, almost addictively ignored the statutes’ mandate– to instead permit and encourage explicitly proscribed conduct that unfairly and illegally steals from the public to enrich hedge funds, investment banks, broker-wheeler-dealers and other market insiders– all to the extreme prejudice of the investing public and the global credibility of US capital markets.

The crisis now decapitating the world’s economies didn’t have to happen– would not in fact have happened if those charged with protecting the financial system and the investing public— namely Congress in general and the Securities and Exchange Commission in particular— had simply done their jobs, upheld their oaths, and seen to the enforcement of laws already on the books, instead of consistently looking the other way; or worse still, directly aiding their campaign contributors, benefactors, patrons, employers and future employers by contemptible acts of public disdain such as repealing Glass Steagel, allowing Reg Sho’s grandfathering of billions in counterfeit shares, scuttling the systemic market protections provided by the uptick rule, and encouraging countless Trillions in credit default swaps.

Mon, 08/06/2012 - 22:47 | 2683608 Kiwi Pete
Kiwi Pete's picture

How is this not theft? I think it's called skimming or front-running here and is totally illegal. Straight to jail for anyone who does it. What has America become? At least you can land a buggy on Mars. Well done Nasa!

Tue, 08/07/2012 - 06:09 | 2684033 andrewp111
andrewp111's picture

The SEC allowed Madoff to operate for 30 years. They looked the other way at Stanford. They are as incompetent as it gets.

Mon, 08/06/2012 - 14:38 | 2682394 Zero_Sum
Zero_Sum's picture

This wasn't SkyNet. It wasn't self-aware. Knight accidentally released financial ebola into the NYSE well water.

Mon, 08/06/2012 - 14:43 | 2682413 MsCreant
MsCreant's picture

ebola that only made them ill...

Mon, 08/06/2012 - 18:37 | 2683047 Thomas
Thomas's picture

That's not true. There were all sorts of winners and losers. Some HFTs were certainly winners. Others, by definition, lost. This system is simply all fucked up. I have one thing to say to these algo traders: Just keep crashin' and burnin' boys and girls. You will eventually all commit mass suicide. 

"Look what they've done to my song, Ma."


Mon, 08/06/2012 - 15:24 | 2682535 MassDecep
MassDecep's picture

This video is a must see about Knight and them crossing TPTB.

Thu, 08/09/2012 - 09:28 | 2690682 DanP1966
DanP1966's picture

This still makes no sense.

A) Why would a test system have any connection to anything other than another test system & QA?


B) Why would you configure your test system to do what this article is claiming? It would have no test value even for negative testing.


C) It still does not explain why a major upgrade would be done mid-week.


D) It still does not explain how a test system, even if redirected to point to the exchanges production environement should be able to execute trades.


E) Assuming that the test system did have the flaw described (nonsensicle on its own) it still does not  explain HOW a test system gets pointed to the exchange production environement or how the exchange could allow it to happen. There HAS to be some form of certification on the exchange side when new code is linked to it.


F) How would this have gotten past even a basic QA?


This still smells to me like somebody did something intentionally.


Is it plausible that it was all an accident that resulted from a cascade of best practice failures? is plausible but so unlikely as to be nearly rediculous.


BUT....then.....given the number of rediculous actions that violate best practices that we have seen from Wall Street I suppose I would not be that shocked to know that their IT operations are slipshod.

Mon, 08/06/2012 - 13:47 | 2682176 Bazinga
Bazinga's picture

I can fully appreciate the nightmare their team experienced but I DON'T CARE. Companies that F-up need to go out of business if the F-up is big enough to bury them. If you can't stand the heat, get the hell out of Skynet's kitchen bitchez.

Mon, 08/06/2012 - 13:52 | 2682201 Karl von Bahnhof
Karl von Bahnhof's picture

Damn! And I hoped it was china!

Mon, 08/06/2012 - 14:13 | 2682299 MachoMan
MachoMan's picture

I haven't looked at anything more than headlines, but if I was the head of a now failed entity, I'd be apologizing for my complete and total idiocy...  it just seems like there is more of an apology for risk taking behavior biting you rather than your stewardship failing...  it's basically a deflection of the real problem...  come out, admit failure, and never manage money again...  spend the rest of your life on the beach with the spoils you made from a non productive activity.  Kudos.

Mon, 08/06/2012 - 15:29 | 2682552 You Didn't Buil...
You Didn't Build That's picture

Luckily, Knight didn't build that.

Mon, 08/06/2012 - 13:47 | 2682181 RingToneDeaf
RingToneDeaf's picture

It is all on the level

Mon, 08/06/2012 - 14:57 | 2682435 slewie the pi-rat
slewie the pi-rat's picture

kinda like a digital chernobyl?

btw i finally understood something from nanex, BiCheZ!   YaY! 

Mon, 08/06/2012 - 15:08 | 2682483 slewie the pi-rat
slewie the pi-rat's picture


new media:  new non-atomic record destruction of digits belonging to self & others

can we get a nurse in here with a shot of thorazine, please?  stat!

"intelligent design" has met the twilightZone where a lifetime of robo_T trades were condensed into a corporate nightmare on elmo street

video at 5!

Mon, 08/06/2012 - 13:47 | 2682182 bigwavedave
bigwavedave's picture

Goldman bought the entire book at a discount. Squiddly

Mon, 08/06/2012 - 13:47 | 2682183 Jupiter
Jupiter's picture


Could have happened to any firm, not evidence of any exploitative HFT, etc.


Mon, 08/06/2012 - 13:58 | 2682223 larz
larz's picture

except that the tester is supposed to be us traders out here and our 401k's etc

Mon, 08/06/2012 - 14:19 | 2682321 MachoMan
MachoMan's picture

I suppose it's inevitable...  however, just because an activity was bound to happen, doesn't mean it was bound to happen here and now.  It certainly is a plausible explanation, but it still leaves a lot to the imagination...  such as the degree of competency of the HFT implementers, their ties with competitors, how deep their pockets are, etc. 

The other thing is, was this their first rodeo?  Did they finally get a PC about the time that the content on the internet started to suck?  If they're just venturing into the world of HFT, then they're fucking dinosaurs (only a matter of time before the cigarettes kill them all, right kids?). 

Mon, 08/06/2012 - 15:02 | 2682468 Praetorian Guard
Praetorian Guard's picture

However, I wonder to what extent someone actually profited from this "mistake"? maybe it was an inside job? If another firm, ie GS knew this was going to be launched at a certain time, they could manipulate the market to force this algo to work in their favor and make tons of $$$$$$. Hmmm....

Mon, 08/06/2012 - 13:48 | 2682186 Concentrated po...
Concentrated power has always been the enemy of liberty.'s picture

Testy bitches!

Mon, 08/06/2012 - 13:48 | 2682187 Dr. Engali
Dr. Engali's picture

It's my belief that there are no accidents. I am willing to bet there is more to this story.

Mon, 08/06/2012 - 13:55 | 2682213 CvlDobd
CvlDobd's picture


I think the best we can hope for is the Knight CEO selling the story to Hollywood for millions and digging for tidbits of truth in John Malkovich's character.

Mon, 08/06/2012 - 13:58 | 2682237 magpie
magpie's picture

...and i guess India struck back at Reuters.

Mon, 08/06/2012 - 13:57 | 2682227 financial apoca...
financial apocalyptic contagion's picture

yea the explanation that "the tester" was randomly included in the market making software is hard to believe.

It's more likely Ethan Hunt/ James Bond were behind this

Mon, 08/06/2012 - 14:17 | 2682315 NotApplicable
NotApplicable's picture

Well, it wouldn't have been "random," but rather, sloppy code updates where someone either just copied the whole directory structure (which assumes it was segregated), or forgot to set the test flag to false somewhere.

Mon, 08/06/2012 - 15:17 | 2682509 CheapBastard
CheapBastard's picture

Despite losing $440 million, the silver lining is they probably saved $2,456 on hiring inexperienced DBAs and software engineers.

Tue, 08/07/2012 - 07:56 | 2684090 StandardDeviant
StandardDeviant's picture

Being a software developer myself, I don't find it at all hard to believe.

You can see on the Nanex charts (Chart 6 at that there were two points where volume dropped suddenly, then picked up again, suggesting that the Knight staff thought their new MM software was at fault and bounced it, unaware that the tester was also part of the software release and was being restarted right along with it.

(Just before 10:00, someone seems to have figured out what was going on, and did a quick "kill -9".  Imagine the look on his face...)

Mon, 08/06/2012 - 14:32 | 2682370 mac768
mac768's picture

common sense is not common


Mon, 08/06/2012 - 14:45 | 2682417 LongSoupLine
LongSoupLine's picture

nor is it a requirement of employment within the financial sector...whether it be operations or enforcement.

Mon, 08/06/2012 - 13:49 | 2682189 Meesohaawnee
Meesohaawnee's picture

im not joking when i joke. bullish!! seems to me the more fraud the worse earnings data are the bigger the vapor melt up to cover up how bad things are for obama. its payback for wall street for obama not locking any up and letting ben keep the bonuses flying.

Mon, 08/06/2012 - 13:51 | 2682194 CharliePrince
CharliePrince's picture

see ya knight

Mon, 08/06/2012 - 13:52 | 2682200 css1971
css1971's picture

If so, this'd be an example of cutting great big pages never mind corners in their testing environment.

Mon, 08/06/2012 - 13:52 | 2682202 FinalCollapse
FinalCollapse's picture

If Knight released faulty software into NYSE then both parties are equally guilty. NYSE failed to perform the due diligence.

Mon, 08/06/2012 - 14:05 | 2682271 KidHorn
KidHorn's picture

You don't understand. Night doesn't release software into NYSE. They have software that communicates with the NYSE. The NYSE had nothing to do with this.

It would be analogous to blaming nasdaq if you fat fingered a MSFT trade and bought 10,000 shares instead of 1,000.

Mon, 08/06/2012 - 14:14 | 2682297 FinalCollapse
FinalCollapse's picture

I will respectfully disagree with you. Let's assume that you properly corrected me. There are two parties communicating and both parties must ensure that the software that facilitates the communication between them is error free. 

The younger generation of developers came, replaced the experienced ones and threw out all quality standards. Results are as you see.

Did I make it clear? In software you don't want to have faulty bridge between two systems. It is responsibility of both parties to ensure that the software is error free, crash free, etc. 

Grab popcorn and coke. Enjoy the show as young and inexperienced are learning job on their own errors.

Mon, 08/06/2012 - 14:28 | 2682358 NotApplicable
NotApplicable's picture

I understand and agree with what you're saying, but that doesn't cover the full scenario. The tester software is NOT part of the system. It is an add-on that replicates customer behavior. NYSE, for instance, would have absolutely no idea it was running.

Mon, 08/06/2012 - 14:45 | 2682422 jonjon831983
jonjon831983's picture

Hmmm lets see if got this right.


More like say you (Client) connect to a FTP site (Server).  Whatever software you use does not matter as long as your authentication etc was successful.  As client on the FTP whatever you do within your access rights.  So, say you are allowed to upload/delete stuff, if you delete stuff accidentally - will the Server be responsible for your accidental delete?

Mon, 08/06/2012 - 16:13 | 2682653 FinalCollapse
FinalCollapse's picture

The answer is 'Absolutely Yes'.

The Server is responsible for its own contents and performance. If the FTP client rampages through out the Server deleting important production stuff - then the Server's Sys Admin should be hang even before the client's one.

This is where the younger and less experienced completely fail to understand the IT, and for me this is like talking to the wall.

Server should grant proper rights and make sure that the restrictions in place will protect the system stability. Anything that can cause Server's instability should be disallowed or properly tested by the Server (on top of the Client testing).

Not only the Knight blew up but the entire stock market was unstable for a while. We didn't have complete crash but the stock market was unstable and many investors lost money because of it.

It is such a simple and basic concept, I have trouble arguing about it. Really.

Mon, 08/06/2012 - 19:17 | 2683119 Silver Bully
Silver Bully's picture

'The answer is 'Absolutely Yes'.'


Not quite. This is NOT like a sys admin allowing someone access to an FTP login.

This is more akin to something like a game client logged into an MMO server, controlled by a botting program. Think World of Warcraft gold farmers. An automated bot program logs in to the client and makes the character do things to earn money. The server is unable to automagically detect the bot because all it can see is the client. The only way to detect a bot exists is through the actions of the client. Once this occurs, THEN the server can cut off access to the damaging client, but not until that client does something wrong. That is how the system is designed.

Now take it one step further. Imagine a game where NO humans control the clients. Instead a horde (ahem) of HFT's run them. The server can only track what the clients do. So when one of the bots goes nuts and decides to buy high and sell low, disrupting the in-game economy, who's 'guilty'? The server? Heck no. The guilty party is the human being who ran the messed up bot, no one else. Would Activision-Blizzard offer everyone gold refunds from all the damage a lone bot created? No! They'd ban the offending client account and be done with it. The winners win, the losers lose.

Knight has done the same thing. They released their bot program into the game without making sure it was working properly. They paid the price, the market was temporarily disrupted, and the damage is done. A 1500 person company went bankrupt in 45 minutes. This is part of the risk anyone takes being involved in the current complex system. if you don't know what you are doing (which would be just about everyone with how complex this has become) then don't risk your (and someone else's) money.


Here's another gaming cliche,

TL;DR: don't play a game you do not understand. If you do, and you get burned, it is no one's fault but YOUR OWN.

Mon, 08/06/2012 - 21:19 | 2683404 icanhasbailout
icanhasbailout's picture

Only in the sense that the NYSE is knowingly facilitating fraud/illegal trading activities by letting these algos run in the first place.

Mon, 08/06/2012 - 13:53 | 2682204 Eternal Complainer
Eternal Complainer's picture

"We can fully appreciate the nightmare their team must have experienced that morning."

Good on them! :)

Mon, 08/06/2012 - 13:54 | 2682209 ItsDanger
ItsDanger's picture

Did anyone ever think to pull the plug?

Mon, 08/06/2012 - 14:31 | 2682368 jayman21
jayman21's picture

If the testing program was on a different machine, the plug theory would not work.  If the testing program was on a cron job on a different machine, that would explain a lot.  Cron was working that day.

Mon, 08/06/2012 - 19:58 | 2683188 Pay Day Today
Pay Day Today's picture

Pulling the plug...didn't they try and fail that in T3?

Tue, 08/07/2012 - 08:00 | 2684095 StandardDeviant
StandardDeviant's picture

It looks like they tried, twice, before finding the right plug to pull: see Chart 6 at

Mon, 08/06/2012 - 13:57 | 2682221 DeFeralCat
DeFeralCat's picture

The testing team would be segregated from the implementation team. The implementation team would have no idea what they are implementing as it is a process. The server was probably co-located so that would require an upload to a server not on-site. The person who loaded up the software was probably late getting out to Jersey City as he had no reason to fear a bad morning and he was there late the night before. The password on the account was probably on its 30th day. No one could log into it so it was just throwing orders everywhere. They called the developpers who said it worked fine for them. They called QA who showed their checklist. The called the implementation team who said we just do what we are told. I don't smell conspiracy as much as a real stupid series of events.

Mon, 08/06/2012 - 14:25 | 2682347 tarsubil
tarsubil's picture

All modern systems seem to be dedicated to CYA. Pass the buck in a circular logic and no one is to blame. Oh well, not my fault!

Mon, 08/06/2012 - 16:11 | 2682690 Al Huxley
Al Huxley's picture

Failure in the build/release process, that honestly could have been guarded against by the team the built the test software if they'd considered embedding something that made it verify it was running in the lab before it executed.  But it's amazing how many companies are really weak in their development/build/test/release processes, even companies that live or die on their software. Too bad, but really completely plausible and not particularly surprising.

Tue, 08/07/2012 - 01:23 | 2683898 StychoKiller
StychoKiller's picture

No problemo!  We'll just inspect more "quality" into the system! :>D

Mon, 08/06/2012 - 13:58 | 2682231 reader2010
reader2010's picture

"The Force is strong with this one."

- Darth Vader

Mon, 08/06/2012 - 13:59 | 2682234 timbo_em
timbo_em's picture

"Accidentally released"...this can never happen to the nuclear arsenal of the US, Russia et al, right?!?

btw: Someone should make a movie about this incident. Anyone got JC Chandors number?

Mon, 08/06/2012 - 14:00 | 2682244 I Am Not a Copp...
I Am Not a Copper Top's picture

All of them can go to hell.  Every fucking one of them.

Mon, 08/06/2012 - 14:02 | 2682250 KidHorn
KidHorn's picture

The software wasn't faulty. The problem is they released an extra piece of software (The testing software) when they shouldn't have. It was a configuration management issue. Not a software issue.

Mon, 08/06/2012 - 14:03 | 2682259 Joshua_D
Joshua_D's picture

"We regret to inform you that we lost your money due to a computer error caused by a human error. Our bad."

- Knight Managment

Mon, 08/06/2012 - 14:06 | 2682275 Karl von Bahnhof
Karl von Bahnhof's picture

Naah, whats a 500 mellons nowadays...

Mon, 08/06/2012 - 14:06 | 2682276 insanelysane
insanelysane's picture

Also, their initial focus would have been on the actual application which was probably working fine.  The computer crew was probably doing all of their checks on that system and everything was honky dory.

Mon, 08/06/2012 - 14:57 | 2682454 NotApplicable
NotApplicable's picture

Seen that a million times too. Cause and effect can not only be distant, but have any number of intemediate steps that otherwise appear benign (especially if unchanged).

Mon, 08/06/2012 - 14:08 | 2682284 CrazyCooter
CrazyCooter's picture

This was an EXCELLENT post. Answered a lot of lingering questions for me. Simple, basic, and plausible explanation for what went wrong and why.



Mon, 08/06/2012 - 14:09 | 2682286 buzzsaw99
buzzsaw99's picture

so both HFTs and regulators spend much of their time playing with themselves, as we all suspected. interesting.

Mon, 08/06/2012 - 14:13 | 2682296 ebworthen
ebworthen's picture

"Testing" software that can buy and sell and does not know it's net position.


Mon, 08/06/2012 - 15:17 | 2682511 slewie the pi-rat
slewie the pi-rat's picture


...sounds like most ATM + check-writer's checkbooks, eb

but, they never run outa cash b/c the never use any! 

Mon, 08/06/2012 - 14:13 | 2682301 spentCartridge
spentCartridge's picture

Somebody forgot to isolate the vmware box.

Mon, 08/06/2012 - 14:15 | 2682304 bnbdnb
bnbdnb's picture

Oh my god, that's the wrong connection string!

Mon, 08/06/2012 - 14:15 | 2682307 adr
adr's picture

So the tester is supposed to act as a market player to test the market making software. The tester buys every ask and sells every bid.

Seriously, what the fuck is that testing? Who buys every ask and sells every bid?

I also must take issue with the existence of "market making" software. Doesn't that imply that the market is in fact made by software, ie completely fabricated by computers.

Am I paying $3.90 for gas because some "market making" software decided that was the price? Is Apple above its pre-earnings price because "market making" software determined that Apple should be $700 by september?

Pull the fucking plug on all of it, now!

Mon, 08/06/2012 - 14:21 | 2682328 I Am Not a Copp...
I Am Not a Copper Top's picture

Sky Net is self aware.  Too late to pull the plug...

Mon, 08/06/2012 - 18:11 | 2683004 Debt-Is-Not-Money
Debt-Is-Not-Money's picture

"It is the will of Landrew"

Mon, 08/06/2012 - 14:20 | 2682320 Cognitive Dissonance
Cognitive Dissonance's picture

"Finally, just moments before an economic news release at 10am, someone found and killed the Tester. We can fully appreciate the nightmare their team must have experienced that morning."

30 minutes over Tokyo Wall Street.

Mon, 08/06/2012 - 14:21 | 2682327 twh99
twh99's picture

I would hate to be the programmer that accidentally let that algo out into production.

Mon, 08/06/2012 - 15:17 | 2682499 Vagabond
Vagabond's picture

I would love to be that programmer that "accidentally" released the software.

Mon, 08/06/2012 - 18:16 | 2683017 Debt-Is-Not-Money
Debt-Is-Not-Money's picture

Looks to me like Knight had a visit from a mole working for a competitor (pulling down two paychecks)?

If so, could he be a double agent?

How the hell does anyone know who they are really hiring?

I saw this stuff happening way back in the mid-'70's, damn near put the company I was working for out of business!

Mon, 08/06/2012 - 14:22 | 2682331 bankruptcylawyer
bankruptcylawyer's picture

the fact that a company like this can obtain money overnight  , after such a huge accident is highly bullish for printing. money will be printed up the wazoo.

Mon, 08/06/2012 - 14:23 | 2682336 DeFeralCat
DeFeralCat's picture

The testing software would have to mimic a live market so it would have to create orders on both sides of the book. The testing software was probably already installed for all production releases as you would want to have a mechanism to do a production test. It seems more plausible to me that somehow the actual testing script got loaded into production. Somebody either ran the wrong script or it somehow got attached to the production device and executed. It is possible somebody forgot to change the startup script and it loaded the testing script and fired it into production before anyone could stop it. However, the testing script should not have been available to the production device.

Tue, 08/07/2012 - 03:44 | 2683972 superflyguy
superflyguy's picture

The way it usually works in smaller companies is that they're too cheap to buy a separate QA/UAT environment so they just have DEV and PROD and release directly from DEV to PROD.

Developer probably forgot to change some script parameters or bundled up the testing piece of code along side with the prod code and viola.

I believe most of the large trading institution potentially face similar fate as they replace their knowledgable staff and original code creators with the cheaper alternatives who are struggling to figure out the environment and are not skilled enough to do things right.

But maybe it's done on purpose and is exactly what they're expecting to happen so they can blame the whole global financial meltdown on a "computer glitch" rather than the broken phylosophy.


Mon, 08/06/2012 - 14:23 | 2682339 Disenchanted
Disenchanted's picture



Or someone put "the Tester" into the live system 'accidentally' on purpose...

Mon, 08/06/2012 - 14:33 | 2682376 NotApplicable
NotApplicable's picture

Those tricks are the bestest.

Plausible Deniability, FTW!

I wonder, does GS have connection to the software?

Mon, 08/06/2012 - 15:04 | 2682457 Disenchanted
Disenchanted's picture



"I wonder, does GS have connection to the software?"


August 2009:


At 9:20 p.m. on July 3, Mr. McSwain arrested Mr. Aleynikov, 39, at Newark Liberty Airport, accusing him of stealing software code from Goldman Sachs, his old employer. At a bail hearing three days later, a federal prosecutor asked that Mr. Aleynikov be held without bond because the code could be used to “unfairly manipulate” stock prices...


This spring, Mr. Aleynikov quit Goldman to join Teza Technologies, a new trading firm, tripling his salary to about $1.2 million, according to the complaint. He left Goldman on June 5. In the days before he left, he transferred code to a server in Germany that offers free data hosting...


Why he downloaded the open source software from Goldman, rather than getting it elsewhere, and how he could at the same time have inadvertently downloaded some of the firm’s most confidential software, is not yet clear...

Six days after Mr. Aleynikov’s arrest, Citadel, the hedge fund, sued Mr. Aleynikov’s new employer, Teza Technologies, which was founded in March by three former Citadel employees. While Teza is not yet conducting any trading, Citadel claimed the former employees had violated a noncompete agreement with Citadel and might even be trying to steal Citadel’s code, causing “irreparable harm...”


In its response, Teza said that it had never stolen or tried to steal Citadel’s software, did not ask Mr. Aleynikov to take code from Goldman, and had never seen the code he took. A lawyer for Teza did not return calls for comment.

excerpts from:


Did the 'stolen' Goldman code have a built in backdoor that Goldman could access after the theft? How many different places did that stolen Goldman code turn up, like maybe at Knight? Could the 'stolen' Goldman code have been a sort of 'trojan horse' into other's trading systems all along?


I don't know...just theorizing. :)


Is GS benefitting(profiting) from Knight's problems? Seems like that might be a yes:

Mon, 08/06/2012 - 14:25 | 2682349 working class dog
working class dog's picture



If enough investors say screw you NANEX and the horse you rode in on I am buying directly from a company and not the parasytic sucking market you provide, the Exchanges will dump the HFT pigs and return to a reality of buying and sellling on fundamentals for more than 5 milliseconds. 

Mon, 08/06/2012 - 14:36 | 2682383 NotApplicable
NotApplicable's picture

The acronym you seek is called DRIP. Though these days, I'm not sure if that route's any safer.


[oh, and you meant NYSE. Nanex are the good guys.]

Mon, 08/06/2012 - 15:06 | 2682470 jonjon831983
jonjon831983's picture



What you need, are stock certificates registered under your name - not your brokerage.  This is the old school way of stock exchanges, pre-electronic... and apparently still done in 3rd world stock exchanges.  If you don't hold it in your hands, you don't own it kinda thing.  Supposedly more safe as the brokerage would not be able to rehypothecate for short lending or what not.

From what I have seen, what you refer to is not necessarily a Dividend Reinvestment Plan (DRIP) - DRIP is another registration process on top the Direct Share Purchase Plan (DSPP) that some companies offer via a Stock Transfer Company.  This means you can send them a cheque and recieve shares directly from the company instead of the secondary market.  Basically, you now have registration of shares directly under your name.


See ComputerShare link for list of companies offering DSPP:


It should be possible to get stock certificates for Any company, not included in the list above - the process is to tell your brokerage to issue and mail out stock certificates to you under your name.  But... there is "sometimes" a large fee for share registration for the administration of it... and well they'd want to keep your business with them.


There actually are clubs out there that deal with stock certificates (ie. DRIP clubs) they basically trade stock certificates of companies (with DRIPs) with other members.

Mon, 08/06/2012 - 15:10 | 2682490 NotApplicable
NotApplicable's picture

I was thinking I might be misstating that, but was too lazy for teh googlez.

Mon, 08/06/2012 - 14:36 | 2682384 ebworthen
ebworthen's picture

You will have to go in the opposite direction as the robots.

Rip Van Winkle market timing.

Mon, 08/06/2012 - 15:06 | 2682479 AustriAnnie
AustriAnnie's picture

Yes.  That would likely happen, because companies want investors to focus on fundamentals and get a realistic picture of their company's value.....

The reality of buying and selling on fundamentals is just that: reality.  Reality doesn't pay.  Smoke and mirrors do.

Mon, 08/06/2012 - 15:33 | 2682558 bankruptcylawyer
bankruptcylawyer's picture

yea, they have those. they are called private companies not listed on the stock market.

Mon, 08/06/2012 - 14:26 | 2682351 rosiescenario
rosiescenario's picture

When Good Algos Go Bad....or, The Rogue Algo....IMHO, Goldman had a Fifth Column in Knight that reversed the + and - signs in the algo and GS took the opposite side of the trades....need to follow the money here and find out who made it on this fiasco.


Or, perhaps this is Son of Stuxnet.....GS took the original Stuxnet and with a few tweaks, inserted it into Knight's servers...

Tue, 08/07/2012 - 01:31 | 2683904 StychoKiller
Mon, 08/06/2012 - 14:33 | 2682374 q99x2
q99x2's picture

Labeling could have prevented it: Warning do not use coke and whores before installing this software.

Mon, 08/06/2012 - 14:34 | 2682378 NotApplicable
NotApplicable's picture

How else can you settle your nerves?

Mon, 08/06/2012 - 14:37 | 2682390 luna_man
luna_man's picture



Way to go, MY MAIN MAN!...(ALL KNOWING)..for me, anyway!


Now, do you think "knight", deserves a "do over"?...Before you answer, have not each of us "mortals" made mistakes?...


"do over"...H**L NO!

Mon, 08/06/2012 - 14:46 | 2682420 Karl von Bahnhof
Karl von Bahnhof's picture

But,but outsourcing improves bottom line,no?

Mon, 08/06/2012 - 15:01 | 2682433 FinalCollapse
FinalCollapse's picture

This is Peggy - may I help you?

In terms of Computer Science education, Russia, China and Poland are tops - as per results of the ICPC competition ( Computer Science World Championships). The last time any North American team won was in 1999  by the University of Waterloo (Canada).

India has won zero. Nothing. Nada.

Even the best education will not replace experienced System Engineering - and Canada and USA are still the best in this category. If you start giving this function to younger foreigners (here or else) then they will learn on their own errors, and these will be very expensive lesson. 

Mon, 08/06/2012 - 15:01 | 2682465 NotApplicable
NotApplicable's picture

Thing is, you can't trust those nerds who enter all of those competitions. They end up creating nuclear weapons, and all sorts of Big Brother technologies.

Mon, 08/06/2012 - 14:41 | 2682403 ThisIsBob
ThisIsBob's picture

Do I understand that for whatever reason they are loosing 10 million dollars a minute and it takes 45 minutes to find out and stop it?

Mon, 08/06/2012 - 14:42 | 2682406 exiledbear
exiledbear's picture

What do you call something that goes way beyond fat fingers? What do we call this? A really expensive brain fart?

Mon, 08/06/2012 - 16:41 | 2682784 fnordfnordfnord
fnordfnordfnord's picture


Mon, 08/06/2012 - 14:43 | 2682412 firstdivision
firstdivision's picture

It would have been even more fun for us to see it react to FOMC release.  That would have been an epic blowup for sure.

Mon, 08/06/2012 - 14:45 | 2682418 silverserfer
silverserfer's picture

aaaaaand its gone! currency wars bitchez! digital financial weaponry Fire!

Mon, 08/06/2012 - 14:47 | 2682424 slickrock
slickrock's picture

This means that the tester software were tied into a real account with real money that could place real trades.  Really?  All this time they were testing it they were using a real account?  I don't buy it.

Mon, 08/06/2012 - 15:06 | 2682481 NotApplicable
NotApplicable's picture

You're forgetting THEY are a market-maker. In that case, "real" means that it passes their sniff test. They would just assign an unused acct number (likely predesignated). It wouldn't have to be connected to anything on their back end (or the user front end), but rather, just not set off any errors.

So, there wouldn't be any real money involved at the time. That would only come later when GS approaches with their settlement bill in hand.


Mon, 08/06/2012 - 16:04 | 2682581 BlackholeDivestment
BlackholeDivestment's picture

...I don't buy it either Slick. There are two factors at play, one is the prophetic strong delusion, but, the primary one ties the Corzine Vapor to Benny and the 911 Jets Weill E Coyote that drove the Legislative White Van Mystery Van full of BTFD Cup of Fornication political payola to the Borderless Bailout Bonus Bitchez of the Great Wal Mart of China on Capitol Hill etc... Further analysis is needed to quantify this case, but, at this point we really do not need too. We are getting marked as PIIGS in a global prison pen of contempt for an offensive purpose that points directly to the usual suspects that established the U.N. and the drive for peace among all nations and Israel under a single global monitary system of control. This case is just more evidence of the Supreme Citizen United Peta Flop power that has overcome the voice of the human soul (in agreement with Christ/freedom and independence etc...) and the image of man's moral agreement with the sustaining moral principle return required of labor and for foundation for the market to survive the Suicide Vampire Squid's robot beast the global Elite socialist bastards want everyone to worship like Lloyd's god(s).

We have lost control, do to the same (lie) old (eat this fruit it will not kill you) offer(s) of temptation, and there is nothing we can do to overcome the system at hand short of all out war against the evil bastards behind the SkyNet world of wonders. We have reached that same old fiscal market call for war already. 

This case defines the (overcoming) virus of contempt and corruption.

Mon, 08/06/2012 - 14:58 | 2682460 FRBNYrCROOKS
FRBNYrCROOKS's picture

Singing ashes , ashes all fall down!

Mon, 08/06/2012 - 21:18 | 2682687 michael_engineer
michael_engineer's picture

Regarding HFT :

As a software engineer with market insights, I suspect that the quick algo swings up and down (and sometimes as shown at ZH before ( as growing in amplitude) are designed to increase nervous tensions that result in people or other algo's to adjust their limit orders or stop loss orders on both stocks and options. The algo may probe then stand back for 5 to 30 minutes to watch for changes then probe again. After determining weakness or panic then the algo can swing in for the profits such as in a short covering period.. An algo like this could easily be used in situations where the bid ask spread is large so no trades would even likely result but the information gained by watching the changes to the bids, asks, and limits in the next few minutes could be analyzed for exploitation.

Another manipulation could be to alter the true bid/ask prices towards the direction you want a stock to move, even though no trades may have occurred. The wildly swinging prices on sweeping orders might allow a trading house to claim any value in the swept through price range as the bid and ask values even though there was never any intent for the HF trade order to execute. An algo could even report one bid/ask spread to someone positioned long and a different spread to someone positioned short, and both of those bid ask ranges may have been true in the previous second. Does anyone know if the bid/ask spreads for all customer inquiries (or data pumps or data feeds) has ever been examined to look for a customer by customer bias where short and long positions are given slightly different sets of data?

That's how I would design the logic.

Statistics could be kept and a trading algo could become very confident in the outcomes. If an algo has been run hundreds of thousands (or even millions) of times in the past and fine tuned from the knowledge gained, then it could easily recognize places for it to attempt to step in with a trading strategy that has proven itself as successful in similar scenarios in the past. Part of the quants jobs are to quantify statistically the success of algorithms tried out on the markets

Mon, 08/06/2012 - 16:28 | 2682748 pragmatic hobo
pragmatic hobo's picture

it's inconceivable how something like this can happen ...

Mon, 08/06/2012 - 17:09 | 2682879 Paul Bogdanich
Paul Bogdanich's picture

On Monday morning at 1:24 EST rover Curiosity hit the Martian atmosphere doing 16,000 mph.  The heat shield slowed it down to Mach 2, a supersonic parachute deployed, the heat shield was jettisoned.  Activer radar scanned the surface looking for the landing area.  The parachute was cut away while the vehicle went under thrust and did an evasive manuve to avoid colliding with the parachute.  The vehicle comes in verticle on 4 motors till about 70 feet off the surface where it hovers and the cranes the rover to the surface so as not to cover it with dust kicked up by the motors.  When it senses that the rover is about 50% supported by the ground it cuts the bridal and flys away and crashes,  The problem here is that from the top of the atmosphere to the crash of the hover vehicle takes about 7 minutes.  It takes radio signals 14 minutes to reach Earth so the machine had to act totally independently.  Every possible error condition had to be though of in advance and programmed into the computer or it would have ended up as space junk.  And it worked.  Meanwhile back on Earth the genisus at Knight Capital bankrupted the firm with their "culture of risk" by rolling out a contaminated algo.  The financial elites are real, real far from the best and the brightest.  They just get paid hundreds of times more for no discernable reason and are imune from prosecution because of their collusion in political corruption.  What a contrast and what a system we have.  The way it's built the financial system will go tits up for some equally stupid reason given the gamblers that are working it.  It is not a question of if but when.       

Mon, 08/06/2012 - 17:43 | 2682950 Westcoastliberal
Westcoastliberal's picture

At least this time they didn't confuse centimeters with inches.

Mon, 08/06/2012 - 17:42 | 2682945 Westcoastliberal
Westcoastliberal's picture

I used to work for a software company that was just like this...beta testing done live. It makes for interesting customer service.

Mon, 08/06/2012 - 17:53 | 2682967 SaveTheGreenback
SaveTheGreenback's picture

"Knight’s investment came one day after the death of John Phelan, the former chairman and president of the New York Stock Exchange, who warned of an equity market “meltdown” spurred by trading technology one year before the crash of October 1987."


Mon, 08/06/2012 - 19:30 | 2683152 Centurion9.41
Centurion9.41's picture


In military aviation when a "mishap" is investigated to determine why it happened there are direct causes [e.g. the pilot f'd up, or the ___ broke] and indirect [e.g. poor leadership led to spoor training that resulted in the wrong ____, which led to _____].

But then there are these other two things called complacency and over confidence; i.e. folks get comfortable with the danger and push the envelope, fly to the limits more aggressively.

I don't know what happened.  But I wouldnt be surprised if after a decade of computers being integral to market making operations, especially at Knight where even back in 2000 their IT folks could take control of trading machines remotely via wireless while standing behind traders holding a laptop smaller than an iPad, that complacency played a role.

I could see a bunch of Wall Street MBA's running firms, who have nominal computer/coda experience, but who have been around computers being integral to trading operations for so long that they begin to push the envelope.  Or maybe, are just not as afraid of the software development process and risks, and begin to push the "geeks" more and more.  Pushing to cut testing time, etc.

Next thing you know, big ball of fire and dead people.


The fact is there is ZERO reason why this situation should have occurred.  You can be sure if the lives of the kids of the exec and IT folks were at stake and determined by the capital lost by testing this would have never happened.


Really, the scenario is not much different than all the MBA's walking around before 2007 thinking CDS'/MBS' and all the quant models were valid.   Somewhat understandable after 20 years of mathematician's & physicist's who didn't have a vested interest, because they werent working on Wall Street, warning them the models were flawed - but were proven "wrong" year after year, many WS/finance types drank the kool-aid.  [no offense to Kool-Aid].   The MBA's should have known better, but they got complacent and found it far easier to shovel shit they didnt understand and take the paycheck than really do proper risk analysis - and oh yea, tell their clients.  

Yea there were some really smart folks up at the top, who knew the BS and should be in jail - a kind handling IMO.  But the majority of MBA's are not as nearly as smart as they think they are.  Business should be a MINOR degree and most of the stuff taught in MBA programs should have been covered in under-grad.  The whole system is just a club scheme that colleges play to make money.


Complacency all around....

Next stop





Mon, 08/06/2012 - 21:08 | 2683365 icanhasbailout
icanhasbailout's picture

If true that this was a test algo gone live, it may well be the single most amusing event since I started watching the markets back in 2005.

Mon, 08/20/2012 - 22:03 | 2722397 serema
serema's picture

For cheap evening dresses those who attend formal get together, you can decide on lengthy white celebration dresses for juniors. In case you consider element in informal party,prom dresses short white get cheap flower girl dresses together dresses for juniors are suitable for you personally. Short type celebration dresses can present off your

Do NOT follow this link or you will be banned from the site!