You're now on the archive server. Commenting has been disabled.
Credit Suisse HFT Algo Gone Wild Slapped With Whopping $150,000 Fee
The recent focus on the dangers of HFT algos gone wild was validated earlier today when the NYSE slapped Credit Suisse with a massive $150,000 fee for "failing to adequately supervise development, deployment and operation of proprietary algorithm, including failure to implement procedures to monitor certain modifications made to algorithm." The action involved Credit Suisse algorithm known as SmartWB, implemented by the Swiss firm in 2007, whose function is to "examine the closing imbalances of various exchanges and to attempt to trade profitably based on the algorithm’s assessment of the imbalances and other market data." Yet on November 14, 2007, something went wrong... In fact quite a few somethings...
On November 14, 2007, the SmartWB examined over 800 securities and identified 129 securities that it wanted to trade based upon the algorithm’s parameters. At 3:40:01 p.m., the SmartWB began trading and immediately began receiving executions. After approximately 45 seconds of trading, the Firm trader/programmer changed the trading parameters by entering “7” into the “basis points” data field. The trader/programmer then double-clicked the up arrow in an attempt to generate cancel/replace requests for all unexecuted orders with new limit prices that were 14 basis points higher than the original orders.
However, instead of replacing all unfilled orders with new orders with limit prices that were fourteen basis points higher, each click resulted in the generation of a separate set of cancel/replace requests. Below is an example of the double-clicking issue:
- Original limit order in XYZ with a limit price of P is submitted (Order A)
- Click #1 results in a replacement order in XYZ with a limit price of P + 7 basis points (Order A2)
- Click #2 results in another replacement order in XYZ with a limit price of P + 14 basis points (Order A3)
The change to the SmartWB’s order parameters resulted in cancel/replace messages for 38 of the 129 total securities being targeted by the SmartWB on November 14th. The majority of these cancel/replace requests were processed without incident and were ultimately filled if they were marketable or canceled if they were not. However, for the last seven of the 38 securities included in the cancel/replace requests, the cancel/replace request resulted in “reject-unmatched cancel” messages from SuperDOT, signifying that SuperDOT could not locate the orders that the SmartWB was attempting to cancel and replace. Significantly, the seven securities were all at the end of the “list” of the 38 securities for which the double-clicking resulted in cancel/replace requests.
When the Firm trader/programmer double-clicked to change the order parameters, the first click generated new orders for the 38 securities almost instantaneously, and the second click did the same. Because the second click occurred a fraction of a second after the first click, the algorithm did not have sufficient time to send out all of the cancel/replace requests for all of the 38 securities resulting from the first click. Below is an example of how the double-clicking issue resulted in certain messages being skipped and therefore not being sent to SuperDOT:
- Click #1 generates cancel/replace requests for securities one through 38 of the list of 38 total securities. SmartWB is only able to send requests for securities one through 31 before Click #2 occurs.
- Click #2 generates a second set of cancel/replace requests. The requests for securities 32 through 38 resulting from Click #1 remain unsent.
- Both sets of cancel/replace requests for securities one through 31 are processed properly and without incident.
- For the remaining seven securities (securities 32 through 38), SmartWB only sends the most recent requests and disregards the earlier requests resulting from Click #1.
Therefore, for every one of the final seven of the 38 securities impacted by the change to the algorithm’s order parameters, the following occurred:
- Original limit order in XYZ with a limit price of P is submitted (Order A)
- Click #1 results in a new limit order in XYZ with a limit price of P+7 (Order A2), which is intended to replace Order A
- Click #2 results in a new limit order with a limit price of P+14 in XYZ (Order A3) and attempts to replace Order A2 with Order A3 before SmartWB is able to send the request to replace Order A with Order A2.
- SuperDOT rejects the request to replace Order A2 with Order A3 because SuperDOT never received Order A2 and therefore cannot locate it.
Because SuperDOT was unable to locate the subject of these cancel/replace requests, SuperDOT responded by issuing “reject-unmatched cancel” messages. However, the SmartWB was not programmed to properly respond to the “reject-unmatched cancel” message and therefore continued to repeatedly re-send the cancel/replace requests.
By the end of trading on November 14th, the SmartWB had sent approximately 600,000 cancel/replace messages. SuperDOT sent approximately 405,000 “reject-unmatched cancel” messages between 3:40 p.m. and 4:05 p.m. in response to cancel/replace requests generated by the SmartWB. The seven affected securities traded at five different trading posts on the NYSE Floor. Message traffic in all of the securities traded at these five Floor posts either slowed considerably or stopped completely.
In addition, although the messages stuck in queue for four of the five posts impacted by the increased message traffic were eventually able to be processed, for the remaining trading post, approximately 131,000 messages frozen in queue could not be processed and ultimately had to be deleted.
The SmartWB was not designed to generate an internal alert or error message in the event of rejected requests or unusually high message traffic. As a result, the Firm did not become aware of the rejected cancel/replace requests until the following day, when NYSE Regulation contacted the Firm.
For those of you with ADHD, the sequence of events, simplistically, was that in the mad trading rush prior to close (an event we have all noticed numerous times in Zero Hedge), the CS algo began trading per default parameters, which were subsequently changed by a trader, who however inadvertently generated a separate set of cancel and replace request, which in turn caused the NYSE to respond with 405,000 "reject-unmatched cancel" messages. This burst of activity clogged messaging traffic in five trading posts to stop completely, preventing trading to be completed by the normal closing time of 4 p.m., and in fact taking as long as 4.27 p.m. Amusingly, CS never knew anything wrong had happened and was not notified it snarled trading in numerous stocks until the next day.
While not quite the terminal "rise of the machines" event that many have expecting out of the SkyNet formerly known as the equity capital market, it does indicate how a simple human error can have dramatic downstream effects. Of course, the broader idea of which it is perfectly acceptable to take advantage of HFT pricing arbitrage at market close, when the only consideration is whether a stock falls into a given computer trading pattern, seems to have evaded the NYSE, which instead slapped CS with a token fine. Surely, Credit Suisse will never ever do this ever again: the idea of having to part with $150,000, or a one second's profit for the very same algo group, must cause nightmares for everyone in the Swiss firm.
Of course, the incident incited polarly opposing reactions by participants in the very same industry.
“It highlights the risks in electronic trading where development is many times done directly in a production environment - the traders are developers, and thus normal development protocols and thorough testing may not be effectively done,’’ said John Jacobs, director of operations at Lime Brokerage.
Tabb Group analyst Adam Sussman notes the “hype” around the potential for algorithms to wreak havoc, and that Credit Suisse’s problem was actually prompted by human interference.
“It is quite revealing that … the root cause of the November 2007 incident was the introduction of manual functionality that allowed the user to override certain parameters,” Sussman says.
Well, it is a good thing that at least the "experts" agree. And while this is one incident that has been brought to our attention, one wonders how many thousand other comparable fiascoes have been omitted from the public radar in recent market trading. Of course, when the "big unhideable one" comes, then it will be too late to ask Adam Sussman what his opinion on the matter, under new and improved circumstances, would have been.
»
- 4302 reads
- Printer-friendly version
- Send to friend
The "big unhideable incident" has already come. None of the machines predicted today's default/deflation scenarios.....in rre or cre.
All your exchange are belong to us.
Haha. They are having the same problems you have with online games. I remember back in dialup modem days people would have like 600ms to work with. They'd do things like sell one item to a vendor and then buy back 10 the server would get confused and only charge for one and dupe 9 items within the 600ms out of "time" frame. People would spend days trying to figure out exactly how to manipulate the lag or dumb time out zone.
That fee is terrible. Poor poor people having to pay $150,000 and none of it going to the counterparty their actions affected. This is almost as win as putting huge taxes on cigarettes and then paying hundreds of thousands of dollars to punk 18 year kids to do truth now commercials. With none of it going to the people the cigarrettes affected and only resulting in a bunch of kids owning houses by the time they were 20.
Justice isn't blind. It just keeps what it takes.
Just goes to show that automation combined with bad QA combined with fat-finger traders who don't know what is going on behind the scenes results in things FUBAR.
And if $150K is but a fraction of the profits realized from using something broken, they'll pay that and then some and leave it broken, because making the system run correctly can cost more than the fine.
cougar
Googles e-mail was hacked in China. Just wait until someone Hacks into Goldmans Computer. What if they lost 100 Billion in a day thru a Hacker? Thoes Computers would be shut down pronto.
Just a matter of time that a Hacker takes advantage of the HFT Computer for their own gain. Example short airline stocks then Hack the computer and tell it to sell all airline stocks no matter what price, that would make your Rich beyond imagination. Would make a great Movie.
Looks like it will not be Wall Street that will make you rich just Computer Programing.
Yeah, we'll just take that out of the petty cash drawer.
$150,000? What do they want to do, put CS out of business? This fine creates systemic risk in the financial system! We need Geithner here pronto to provide cash to Credit Suisse. $86B should do it. And crush the bondholders while you're at it
Somehow 150K doesn't feel "massive."
What would Buffett say?
Are the stock exchanges just for gamblers and suckers with the real trading occuring in dark pools etc?
Glad I have a real job.
Jimbob.
Who is programming these computers? 600,000 orders in 25 minutes is a problem? 6.7 orders per second? Pfft. "Algo gone wild" should be nothing less than a million orders per second. A single Core i7 is O(10^10) instructions per second, and it's perfectly feasible to do quite involved computations of size O(10^9) in 5 seconds. Maybe I should get a job programming these things. What does it pay?
I guess the assistant janitor won't get a bonus this year at Credit Suisse
now let me turn on my "optimistic" mode and propose a theory!
the timing of this fine is significant, because the case is synonymous with sponsored/naked access. for a typical trader, the message path is trader-broker-exchange-(and then back). so if some random-joe black box does this sort of a thing, the broker's system will prevent messages from getting out into the wild. but, if the trader is a sponsored access customer, then they could indeed cause the same flooding into an exchange or an ecn.
of course, in this case CS sounds to have been trading on its own behalf so there would not have been another layer (broker) between their systems and the exchange.
long theory short, i suppose you could hope the timing of this fine was synced with the SEC vote to ban sponsored access, since their rationale is to minimize the possibilities of exactly this type of a situation.
ZH is drawing the wrong conclusion. First, submitting 38 cancel/replaces isn't HFT. It is a trading application that is operated by a human trader. ZH is getting on its anti-HFT pulpit for a different issue.
The fact is that NYSE is the boss. I suspect that a NYSE defect left those 19 cancel/replaces in limbo. This triggered the defect in the CS software. NYSE software that accepts orders from broker software is deficient, if it allows a single broker to overload NYSE. NYSE should throttle the input from each broker. Of course, NYSE doesn't take responsibility. NYSE blames CS and fines them an amount that sounds substantial. NYSE is able to pass the buck, because they are NYSE.