This page has been archived and commenting is disabled.
HFT War Stories: The Algo That Couldn't Count
This is the first in MKTSTK’s HFT War Stories series. Submitted anonymously by a high frequency trader. Names of people, places, and products have been omitted:
Algo trading has a way of making your hair turn white. Sometimes it’s the wild market action. The response then is to cut risk and keep on trading. The big losses come from avoidable human errors. Those are the losses you might not come back from.
The trading world is increasingly automated. Every year there are fewer human eyes watching the market at any given time. Traders employ algorithms to cut latency and market impact. Dealers find automation indispensable in quoting thousands of different exchange traded derivatives.
If your automation is coded correctly most of the time it saves your ass. Risk management algorithms run on top of trading strategies and monitor underlying markets for signs of trouble — backing your quotes off if necessary. I have been saved by automation on numerous occasions, side-stepping big market moves and allowing me to keep trading when others ran to the sidelines.
When your automation goes haywire, however, that’s when you experience fear and anger in equal, blood-curdling, measures. Usually it’s not an obvious case of turning the strategy on and watching it go crazy, a la Knight Capital. The worst errors are subtle, they lurk in the corners of your strategy’s logic. They could be caused by code you did not write, living in lower level code that interacts directly with the market.
I discovered this risk first hand one day right before the markets closed. I was working with a new tool our developers had created to cut latency. Certain parts of the strategy which had existed within my code should have now been handled by lower level code written by developers. The changes had passed QA testing and were mission ready as far as I knew.
I had been testing this tool out on an obscure exchange listed derivative. Purchasing the contract bought a combination of separate contracts. The tool handled things fine for a week an a half. Then one day I went to turn it off and the strategy went insane, purchasing nearly two billion dollars worth of illiquid derivatives in literally the blink of an eye.
The risk limits I had coded at my level of code were worthless; the strategy launched so many orders at the market in such rapid succession that its PnL and position were not delivered to my strategy logic fast enough to catch. I had hard-coded a maximum of $80 million in risk before the strategy was supposed to kill itself.
HFT has a tendency to put you in these types of situations more often than not. You feel that visceral shot of adrenaline course through your veins. You perceive time, light, and sound as though in a tunnel. Panic must be suppressed, if only long enough to get hedged.
After fear comes anger. WHO THE FUCK DO I KILL? This emotion must be buried. If it isn’t I will lose my job and lifestyle — I might get combination sued by the government whilst given up as bait by my firm to regulators. All for code I did not fucking write, mind you.
Shit, now it was time to get the fuck out of my unwanted $2 billion position. Selling out directly was not possible: the market is too illiquid to reabsorb my gigantic inventory at this time of day. It’s a miracle I didn’t get worse prices. The only answer was to hedge it quickly into a more liquid market that was still open. No time to negotiate on price, I took a similarly large offsetting position in a product that I hoped would hedge me long enough to liquidate.
Now here is where shit got really fun. The product I hedged into was correlated to my unwanted position, but it certainly wasn’t a perfect hedge. This was around one of the debt ceiling farces, so I was shitting myself watching the news for someone from the government to inadvertently fuck me in the ass by causing my hedge to perform badly.
Now that the market was closed I was able to track down the developer who had coded the tool that went haywire. After talking it out and then checking his code, the problem was caused by a change he made yesterday. Because the contract was a combination of other contracts, it didn’t handle getting flat correctly. It kept buying contracts to get flat, not recognizing each fill using the correct multiplier. Thus it kept shoveling bigger orders into the market thinking it was closing out a position. In fact it was in an infinite loop, acquiring contracts until a risk check had a chance to kill it.
In the end, the code was fixed by correcting a minus sign. Me and my junior traders worked out of the position slowly over the course of the next day and only ended up losing around $40,000. HFT being HFT, we didn’t even lose money that day. Even so, it could have been a complete disaster. Every time I turn off a strategy I still get that feeling I am about to get fucked.
»
- 15386 reads
- Printer-friendly version
- Send to friend
- advertisements -
Cool story bro
Live by the sword,
die by the sword.
Even when they mess up, they make money. Why would the average joe even mess around with the stock market anymore? It's not even about investing. This is all a big scam.
"To err is human, but to really screw things up, you need a computer." Now when theres a ZH post about a market breaking, I'll know what to blame.
Did I miss the part about the massive pile of coke?
The Wolf of Wall Street would blush..
Maybe he can blow it in some hookers butt and call it stress relieving.
Wow if only we can get this software for the federal reserve maybe it could wipe out all debts in couple microseconds.
Why would the FED need the software. They have the company who lost their algo program to a intern. That company said, "In the wrong hands our software could manipulate, (rig), the markets".
That company would be Goldman Shafts. Godman Shafts accepts phone calls from the FED all the time. Afterall, they do run the FED. Especially after they got bank status from the Treasury under Neel Kashkari and Hank Paulson. All X Goldman Shafts past employees. The Squid is in every country and political regime everywhere.
At my HFT house all algos are fed cocaine 3 times a day. You just put it in the cd drive and spit on it. Usually around the market open and any data releases.
Every time I turn off a strategy I still get that feeling I am about to get fucked.
That's exactly what I was thinking after that georgeous blond ordered her third glass of wine.
Badda Bing!
The "georgeous" blonde??? Was S(He) a tranny? ARF!
Live by the keyboard, die by the keyboard.
How did the situation ever get this out of control
Dana Perino defends Obama's supposed legal right to declare amnesty for millions of illegal aliens:
http://tinyurl.com/mdkemhe
What did you expect? Its a brainwashing machine.
This is what happens when you let a Kenyan conman into the whitehouse.
LOVE getting fucked by Z Algo
This guy should just watch the gold traders. They dump tons of contracts into an illiquid market all the time.
I'm surprised we don't see this every day...but did you all catch this part: "After talking it out and then checking his code, the problem was caused by a change he made yesterday. "
Now, did QA testing fail to happen or did he just sign off based on assumptions? Where the hell is their risk dept??
EXACTLY.
There is no testing. There is no QA.
Just a bunch of adrenaline driven rocket scientistss. Wait rocket scientists would be fired for what these nothings did and do every day.
When rocket scientists screw the pooch, there is usually a report...
"Every time I turn off a strategy I still get that feeling I am about to get fucked."
If only..... Hey, a person can dream can't they?
Let's compare that to the number of ordinary joes getting fucked when he turns ON the strategy.
I think "Anonymous" should obtain the names of all HFT traders and publish them
"anonymous" is having a duel with Klan to busy
-----------> Forward into the fucking void
"I placed a huge bet on double 6's at 30-1 not noticing the dice had been swapped out by the croupier."
So nice to know that our lives and those of future generations are being gambled away.
"Liquidity" is what killed those folks that went down with the Titanic.
Almost to $18 TRILLION: http://www.usdebtclock.org/
But it passed 'QA'!!!!
I oversee software development outside of the financial sector. Whenever someone says they passed through QA without finding any errors I immediately call for a higher level review of the test. Not finding problems in code is a problem!
Any form of testing only catches a percentage of the bugs. As a result, QA can't make crap code good, and it can make the devs lazy. Not saying you shouldn't have it, but if you need QA to make the software usable then it'll fail.
Something is fishy, and missing from the story... like the part where the author ignored that email in his inbox about the change that was about to get rolled and would impact his algo... probably because he was too busy chasing tail, getting trashed or surfing ZH to do his own damn job correctly.
We don't want to give away the ending to the second Wolf of Wall Street movie do we?
Watch out for the day when one of these algo developer bankers is in severe depression, writes code to take the system out, and then takes a dive off the top of the building.
I hope they spray paint the camera lens before they push him
I suggest you watch the anime "Patlabor". Crazed programmer + giant construction robots + jumping from building. Fun fun fun for Tokyo!
'the day' I think it's happening all the time.
- we are approaching the singularity and it's gonna be tight!
Given the incidence of mood disorders and personality disorders in the population, and cocaine n' hookers, plus alcohol, one would think that has already happened and the banks covered it up with suicided banksters, threats and intimidation, extortion, et cetera. Suicidal banksters seem to be the norm in regressive market conditions. No growth means what it says and that's why corporations never give employees advanced notice of being fired. We are in a produce or get fired market that will only get worse
as time goes by. No correction is forthcoming in the current economic system until it is completely destroyed and rebuilt on a more suitable base.
When the substrate that holds up markets is no longer a suitable substrate
one must provide a new substrate entirely. Guaranteed that current market participants know that the market cannot continue on for much longer before one or another of the major faults in the system decline to a point of no return, and ultimately, chaos ensues. In brief, the system in and of itself is that suicidal code writer when it blows a HFT gasket. One domino on top of the other starts the cascade.
'so I was shitting myself watching the news for someone from the government to inadvertently fuck me in the ass by causing my hedge to perform badly."
Is it Danny Dyer?
Tyler - Why don't you post some stories where the HFT scum actually does end up getting fucked? Everyone likes a happy ending.
Blow and hookers for all!
Nothing a dwarf throwing competition can't fix
He only lost $40,000 on a bunch of illiquid trades amounting to $2bn?
Hahaha. The bullshit is thick with this one...
I think the algo missed a couple of zeros.
The bizarre part of that story is that he went though checks and reviews initially, and then the developer went in and changed another part of the code. Evidently there were no checks of that. Safety checks are meaningless if people skip them.
that is the point...it is so easy and they just steal more tomorrow. And worst case...the FED or Treasury bail you out if you lose! Social losses and private profit!
This is why market crash all time or that fat finger episodes
That's very true, about fuck-ups and near fuck-ups that no one ever knows about. I saw one case where some poor semi trader schmuk sold a hedge using an option instead of buying the hedge on a transition portfolio for a new major client, and they then made a sizeable loss on unwinding the fuck up.
great story!
As a coder; you made me laugh :)
No matter how intelligent the machine..
Speaking as a Wall Street trader who did well by actually using my brain on a few exchanges over many years, I hope the author of this article chokes one day on his own vomit at his desk because they are going to kill the market and usher in much, much more government into our lives. The death of capitalism started when "price discovery" became an equation in some H1B Visa guy's head.
I hope it happens fast, but the problem isn't the AssHat H1B.
The problem is the executive environment that pay themselves billions for all the "Innovative Financial Instruments" they take credit for and then skimp on some entry level flunky from a foriegn country in order to save a few pennies.
If there is aANYPLACE IN THE WORLD where skimping on ENGINEERING should not be done, it is WS.
WS SCUM
He feels like he is going to get fucked because HFT is fucking everyone
else and what goes around comes around eventually when you play with
HFT. Buffet characterized them as weapons of mass destruction for a logical reason based on simple mathematics of the law of large numbers
and probability theory. Sooner or later they fail over time in the long run of events. Ergo, they are indeed weapons of mass destruction by definition and Buffet was correct to operationally define them as such. This dude is merely conveying how anxious these deals make him feel overall, but he
plays the game because he likes the FED monetary heroin business and it pays him his living being a Wall Street thief.
Buffett is a thief, a liar, a scammer and seemingly also a hypocrite himself though.
tineye that image.
disturbing
http://www.zastavki.com/eng/3D-graphics/wallpaper-25563.htm
Who is the sucker in this take? Oh, yeah, the Muppets, right? Let skynet think for you or perhaps Hal 9000.
"If it isn’t I will lose my job and lifestyle — I might get combination sued by the government whilst given up as bait by my firm to regulators. All for code I did not fucking write, mind you."
Die in a fire bitch.
What is the strategy....stealing from pension funds, bankrupting states, destroying retail stock holders, making food too costly to eat, gaming people out houses, etc. please explain why HFT exists as anything but a game to steal from lots of people...FOR NO PURPOSE! Ever hear of killing the GOLD GOOSE? These FUCKHEADS are destroying the market. It is machine versus machine....FOR NO PURPOSE. The exchanges are not an ends...they are supposed to be a means. The FUCKHEAD that wrote this pieces is worthless! Nothing good comes of what he does...and too many don't realize it. Else we wouldn't be draining our childrens future(and piling on debt)...to fund this FUCKHEADS LIFESTYLE are narcissistic trading!
"Then one day I went to turn it off and the strategy went insane, purchasing nearly two billion dollars worth of illiquid derivatives in literally the blink of an eye."
Maybe you should as your task master IF you can turn him off first...
Advanced AI singularities are a Bitch!
What about when everything runs perfect, you know you've got the best algos in the marketplace and then you find out that you're being front run at the source by insiders that rank higher in the hierarchy foodchain so they get the drop on the info before you?
In the shit fraud that is this market you eat shit and smile as you lose everything
Still can't recall that m'fers name. Great interview though. Lots of details
"I might get combination sued by the government whilst given up as bait by my firm to regulators."
I notice the words 'criminal proceedings against me' did not even enter the thought process!!
That tunnel vision is an adrenalin dump in to your bloodstream.
BTW. Where's your automated test suite which checks what your software actually does? Where's your dev environment? Where's your test environment?
I'm betting you're an adrenalin junky and you'll push and push for that dump until you blow the whole company away. By not following any sane engineering practices are you ignorant or just ensuring you can get your adrenalin fix?
$4B in erroneous positions held open overnight; you'd think a firm with that sort of capital could afford to set up proper test & evaluation environments. 0.05% would pay for over a dozen developers and engineers...
should title the piece 'The Algo That Couldn't Lose '..... typical.
So this is far worse than a software consultant building and running a General Ledger system in production without any testing. No consultant in their right mind would dream about doing that, but that would only really affect the material representation of one company.
Yet WS is perfectly fine doing that on a far bigger scale and leaving a wrecked financial world in their wake.
FUCK THEM ALL
'
'
'
Can you imagine if we just outlawed all the bullshit this guy was doing, and made you hold your position to the next morning?
The markets wouldn't be able to stand the price discovery.
•?•
V-V
"If it isn’t I will lose my job and lifestyle"
MKTSTK, you don't have a lifestyle to lose. You don't even know what one is.
i hate the hft's, but i hate even worse is the control people have over the markets.
when they halt markets, until the hft's can get flush, is over the top.
they'll sit there all day and let hft's fleece the 90%, but they take to big of bite, the markets are halted.
probably the same hft team buying back the naked shorts, with naked longs while the market is down.
if their going to allow hft's to trade on their markets, knowing who they are, and knowing what can happen, and not letting other investors take advantage of these fat fingers, i would call it a rigged market.
IMHO all traders are blood sucking parasites that are indicative of how low humanity has devolved. It must be terribly depressing for a trader to face the end of their life, only to realize that they are a worthless sack of meat that has devoted their life to only make money (to buy stuff) and contributed nothing positive to the world. "Stuff" doesn't equal fulfillment. "He who has the most toys, wins" is bullshit.
I suppose that there might be exceptions. There might be some that contribute a majority of their winnings to some charity or some cause that lifts the human condition, rather than just using it to buy stuff; but I find that pretty unlikely.
"Don't hesitate to satisfy your needs; indeed, expand your needs and demand more." This is the worldly doctrine of today. And they believe that this is freedom. The result for the rich is isolation and suicide, for the poor, envy and murder.” -- Fyodor Dostoyevsky, The Brothers Karamazov
I know there are a lot of traders on ZH so I'm ready for the down-votes. Just don't be a chickenshit. I'm open minded, so tell me why I'm wrong.
I don't buy into that deathbed introspection.
Most people I know who were Assholes in life are Assholes when they die.
They may be worried about meeting their maker but I don't think that will prompt deep introspection, just more self centeredness.
I think I'm going to call BULLSHIT on this guy.
Reads too much like a very bad script for a redo of (God help us NO) - Wolf Of Wall Street. The guy sounds like he's auditioning for the lead role.
Nothing to glorify here.
Code is never written like that.
I don't care who you are or what you do for a living... you don't take code live without testing and putting it on a test system somewhere before it goes live... unless you are Knight or really don't give a fuck about your job.
Believe it or not The Knight Fuck did have repercussions for the HFT industry.
I'm certain of it.
Adding a random delay to all transactions would make this kind of problem go away and level the playing field for everyone. HFT is about high tech cutting in line. Does anyone like people who cut in line?
Algorithms be damned, ban it all and bring back the ticker tape machine....