The $12 Trillion Fat Finger: How A "Glitch" Nearly Crashed The Global Financial System - A True Story

Tyler Durden's picture

For all the talk of how the financial world nearly ended in the aftermath of first the Lehman bankruptcy, then the money market freeze, and culminating with the AIG bailout, and how bubble after Fed bubble has made the entire financial system as brittle as the weakest counterparty in the collateral chain of some $100 trillion in shadow liabilities, the truth is that despite all the "macroprudential" planning and preparations, all the world's credit, housing, stock, and illiquidity bubbles may be nothing when compared to the oldest "glitch" in the book: a simple cascading error which ends up taking down the entire system.

Like what happened in the great quant blow up August 2007.

For those who may not recall the specific details of how the "quant crash" nearly wiped out all algo and quant trading hedge funds and strats in a matter of hours if not minutes, leading to tens of billions in capital losses, here is a reminder, and a warning that the official goalseeked crisis narrative "after" the fact is merely there to hide the embarrassment of just how close to total collapse the global financial system is at any given moment.

The following is a true story (courtesy of b3ta) from the archives, going all the way back to 2007:

I.T. is a minefield for expensive mistakes

There's so many different ways to screw up. The best you can hope for in a support role is to be invisible. If anyone notices your support team at all, you can rest assured it's because someone has made a mistake. I've worked for three major investment banks, but at the first place I witnessed one of the most impressive mistakes I'm ever likely to see in my career. I was part of the sales and trading production support team, but thankfully it wasn't me who made this grave error of judgement...

(I'll delve into obnoxious levels of detail here to add color and context if you're interested. If not, just skip to the next chunk, you impatient git)

This bank had pioneered a process called straight-through processing (STP) which removes the normal manual processes of placement, checking, settling and clearing of trades. Trades done in the global marketplace typically have a 5-day clearing period to allow for all the paperwork and book-keeping to be done. This elaborate system allowed same-day settlement, something never previously possible. The bank had achieved this over a period of six years by developing a computer system with a degree of complexity that rivalled SkyNet. By 2006 it also probably had enough processing power to become self-aware, and the storage requirements were absolutely colossal. It consisted of hundreds of bleeding edge compute-farm blade servers, several £multi-million top-end database servers and the project had over 300 staff just to keep it running. To put that into perspective, the storage for this one system (one of about 500 major trading systems at the bank) represented over 80% of the total storage used within the company. The equivalent of 100 DVD's worth of raw data entered the databases each day as it handled over a million inter-bank trades, each ranging in value from a few hundred thousand dollars to multi-billion dollar equity deals. This thing was BIG.

You'd think such a critically important and expensive system would run on the finest, fault-tolerant hardware and software. Unfortunately, it had grown somewhat organically over the years, with bits being added here, there and everywhere. There were parts of this system that no-one understood any more, as the original, lazy developers had moved company, emigrated or *died* without documenting their work. I doubt they ever predicted the monster it would eventually become.

A colleague of mine one day decided to perform a change during the day without authorisation, which was foolish, but not uncommon. It was a trivial change to add yet more storage and he'd done it many times before so he was confident about it. The guy was only trying to be helpful to the besieged developers, who were constantly under pressure to keep the wretched thing moving as it got more bloated each day, like an electronic ‘Mr Creosote’.

As my friend applied his change that morning, he triggered a bug in a notoriously crap script responsible for bringing new data disks online. The script had been coded in-house as this saved the bank about £300 per year on licensing fees for the official ‘storage agents’ provided by the vendor. Money that, in hindsight, would perhaps have been better spent instead of pocketed. The homebrew code took one look at the new configuration and immediately spazzed out. This monged scrap of pisspoor geek-scribble had decided the best course of action was to bring down the production end of the system and bring online the disaster recovery (DR) end, which is normal behaviour when it detects a catastrophic 'failure'. It’s designed to bring up the working side of the setup as quickly as possible. Sadly, what with this system being fully-replicated at both sites (to [cough] ensure seamless recovery), the exact same bug was almost instantly triggered on the DR end, so in under a minute, the hateful script had taken offline the entire system in much the same manner as chucking a spanner into a running engine might stop a car. The databases, as always, were flushing their precious data onto many different disks as this happened, so massive, irreversible data corruption occurred. That was it, the biggest computer system in the bank, maybe even the world, was down.

And it wasn't coming back up again quickly.

(OK, detail over. Calm down)

At the time this failure occurred there was more than $12 TRILLION of trades at various stages of the settlement process in the system. This represented around 20% of ALL trades on the global stock market, as other banks had started to plug into this behemoth and use its capabilities themselves. If those trades were not settled within the agreed timeframe, the bank would be liable for penalties on each and every one, the resulting fines would eclipse the market capital of the company, and so it would go out of business. Just like that.

My team dropped everything it was doing and spent 4 solid, brutal hours recovering each component of the system in a desperate effort to coax the stubborn silicon back online. After a short time, the head of the European Central Bank (ECB) was on a crisis call with our company CEO, demanding status updates as to why so many trades were failing that day. Allegedly (as we were later told), the volume of financial goodies contained within this beast was so great that failure to clear the trades would have had a significant negative effect on the value of the Euro currency. This one fuckup almost started a global economic crisis on a scale similar to the recent (and ongoing) sub-prime credit crash. With two hours to spare before the ECB would be forced to go public by adjusting the Euro exchange rate to compensate, the system was up and running, but barely. We each manned a critical sub-component and diverted all resources into the clearing engines. The developers set the system to prioritise trades on value. Everything else on those servers was switched off to ensure every available CPU cycle and disk operation could be utilised. It saturated those machines with processing while we watched in silence, unable to influence the outcome at all.

Incredibly, the largest proportion of the high-value transactions had cleared by the close of business deadline, and disaster was averted by the most "wafer-thin" margin. Despite this, the outstanding lower-value trades still cost the bank more than $100m in fines. Amazingly, to this day only a handful of people actually understand the true source of those penalties on the end-of-year shareholder report. Reputation is king in the world of banking and all concerned --including me-- were instructed quite explicitly to keep schtum. Naturally, I *can’t* identify the bank in question, but if you’re still curious, gaz me and I’ll point you in the right direction…

Epilogue… The bank stumped up for proper scripts pretty quickly but the poor sap who started this ball of shit rolling was fired in a pompous ceremony of blame the next day, which was rather unfair as it was dodgy coding which had really caused the problem. The company rationale was that every blaze needs a spark to start it, and he was going to be the one they would scapegoat. That was one of the major reasons I chose to leave the company (but not before giving the global head of technology a dressing down at our Christmas party… that’s another QOTW altogether). Even today my errant mate is one of the only people who properly understands most of that preposterous computer system, so he had his job back within six months -- but at a higher rate than before :-)

Conclusion: most banks are insane and they never do anything to fix problems until *after* it costs them uber-money. Did I hear you mention length? 100 million dollar bills in fines laid end-to-end is about 9,500 miles long according to Google calculator.

* * *

And here is Zero Hedge's conclusion: the next time you think all those paper reps and warranties to claims on billions if not trillions of assets, are safe and sound in some massively redundant hard disk array, think again.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
One And Only's picture

This was the most exciting moment in my financial career.

http://www.bloomberg.com/apps/news?pid=newsarchive&sid=a5O2y1go1GRU

I saw that and thought....here we gooooo. And there we went.

Deathrips's picture

Blame the hacker/computer for the actions of the ZioFED?

They would never do that.

 

RIPS

TeamDepends's picture

If you quant put your hands on it....

realmoney2015's picture

Exactly... If you cant hold it, you dont own it. Buy gold and Silver!!

Trade in your electronic digits for real money. While most markets are in a bubble, silver has been manipulated downward for years. Great time to add to your stack. These candles with silver coin prizes make great gifts and help people start their own stack: https://www.etsy.com/shop/ScentSavers?ref=hdr_shop_menu

Ask yourself, how is wealth protected if all it takes is a power outage or a computer glitch or a hacker to completely wipe it out? If you cannot answer that question honestly, you may someday be wishing you that you spent those digits while they still 'existed' and were 'worth something'

0b1knob's picture

Phat Phinger Phuckups Phracture Phragile Phinancile Cystems.

cookie nookie's picture

Only an insider would put his money in the stock market.  It's too risky for the average guy.  Inside information is needed to win in a rigged game.  Better hiding your cash in the mattress.

VinceFostersGhost's picture

 

 

You can't fat finger physical......get thee some.

FinalEvent's picture

There is only one solution to this digital pile of shit, and that's called bitcoin.

Much faster than 1 to 5 days and imho never intended  to be coffee money but the solution for this story. (and the source of rainbows and unicorns of course, but that's another story)

One of these is not like the others..'s picture

 

I am poor as a church mouse, but I do own five generators, three invertors and a whole host of spare tools, radio transceivers and other real world useful stuff, in several caches and FUCK have I got skills...

Once being able to "repair anything" (and having the tools required to do so, or being able to improvise) has more value than having a continous work history or the ability to write a good CV, (I.E. when the shit gets real) conditions will favour the likes of me..

 

 

 

mt paul's picture

if you don't code it

you don't own it..

exi1ed0ne's picture

Ha.  Keep offshoring IT jobs to kraplocastan and fucking over your competent IT staff with furloughs, pay freezes, and no training you cheap cunts.  I've seen this time and again over 25 years in IT and it fucks over everyone who tries to cheap out, but at least there are handy scapegoats in IT putting 18 hour days to fix it.  Fucking assholes eat their seed corn all the time just to eek out quarterly numbers.

We are so fucked as a species.

vicorjh's picture

I was thinking the same thing.

"as the original, lazy developers had ..."

I claim this is not the matter of finding someone or something to claim as the fall guy.

But rather, this is precisely how these a-holes think. They actually believe that without their genius marketing and sales skills, the world would literally stop turning. Without their mad-hot abilities; the IT folk, the engineers, and the scientists would not be able to produce or create anything of value. Because, you know, the bankers, the marketing department, the sales department, along the executives create the value proposition.


been fired
been laid-off
received no cost of living increase
received no bonus, promotions, or typical corporate incentives
watched as their jobs were off-shored (while training their replacements)
forced to follow the managements ill-conceived and under-funding project plans (or else be fired)
been the brunt of management failure and out-of-control egos

CrazyCooter's picture

And since no one (i.e. engineers, scientists, etc) quits the good jobs (that are the opposite of what you describe), there is a strong bias towards any job opening being exactly as you describe.

Regards,

Cooter

One of these is not like the others..'s picture

Oddly enough the greater the pay, it seems the easier the work is and greater respect is paid to you.

Low paying IT work is just slavery and abuse all rolled into one shitty dealio. 

detached.amusement's picture

+++ you got that right (glad I got in on that game early)

Latitude25's picture

Yea I got laid off designing control systems for high voltage substations.  These same geniuses brought in a bunch of H1Bs who thought they knew it all.  In that business you need years of experience to design reliably.  Soon we will have infrastructure running like it does in India.  Intermittent periods when the lights come on.

CrazyCooter's picture

Having worked on large teams of H1B visas (as a coworker) doing major/complex projects, let me share a beautifully simple insight with y'all.

American's (i.e. the North American US variety - sorry Brazil) have peculiar workplace habits. One of those is calling out, discussing, and resolving problems. Sometimes this is just a conversation, sometimes it is a meeting, but the jist is that it is about solving problems, the first step of which is identifying/accepting problems.

From my experience, this is NOT part of the work place culture of folks from MENA, Asian/Pacific Rim, etc. We did not have many Europeans, so have to exclude them from my anecdote.

Further, many cultures have very weird behaviors, such as choosing team leads based on religion, dialect, and/or caste from their respective regions, regardless of qualifications or experience. This quickly turns into a "gang" that roves around clubbing other "gangs" that show up, except they use keyboards.

All the while, they will insist, with the house burning down around their ears, that everything is great, under budget, on schedule, etc etc etc ... because culturally they do NOT admit failure/mistakes. If you ever visited a large corporate web site and wondered how it could be such a dysfunctional pile of shit ... well now you have a good conspiracy theory!

But, most software systems don't kill people, they just piss them off, and the shit is becoming a commodity anyway.

<looks around for a new k-wave to hitch his trailer to>

Regards,

Cooter

i_call_you_my_base's picture

Been my experience too. For example, a lead will come to you the day something is supposed to go live and tell you it's not done rather than come weeks ahead. Even though they had to have known. I've worked with Europeans too (Germans, Frence, Italians) and they have been quite good and are not like that at all.

Uber Vandal's picture

LOL, and all of the people who put their 401(k), 403(b) assets into safe "cash" allocations, or "dry powder" may be in for an abrupt rude awakening when those money market funds "break the buck" again.

Then again, those warnings about such risks tend to be buried far in the prospectus.

Here is generic boilerplate from one such fund:

An investment in the Fund is not a deposit of a bank and is not insured or guaranteed

by the Federal Deposit Insurance Corporation or any other government agency.

Although the Fund seeks to preserve the value of your investment at $1 per share, it

is possible to lose money by investing in the Fund

 

neidermeyer's picture

What a bunch os ASSHATS! ,, critical applications run on MAINFRAMES... ALWAYS!!! Online games can run on the servers...

 

Oh, and if your work isn't documented and approved , you're fired.

Lumberjack's picture

Insitutionial memory and archives count. 

cowdiddly's picture

Don't like a rigged casino? Well, alternatively you can always buy a countries/companies bonds, but the only thing they ever promise to do is pay you in more fiat dollars, which may or may not be worth more than you originally gave them in buying power. And, even then the promise of return of original investment is iffy.

Rock, Paper, scissors.-- Ill take rocks for 1000 Alex, I know all about those damn scissors

Scooby Dooby Doo's picture

http://ronpaulmessage10.com

Collapse information from Dr. Ron Paul

SubjectivObject's picture

Cash in hand is king.

VinceFostersGhost's picture

 

 

That's why I've got a basement full of whiskey....can't eat it....but you can drink the hell out of it.

Pliskin's picture

Mmm, I get where you're coming from, I really do, but try going in to my favourite Go-Go bar in Itaewon and offering the smokin' hot girls there some fine grass fed mongolian beef, a bottle of Glenfiddich or a mag full of 45's!!

monad's picture

This is all build up to transferring the losses from the 1% who hold most of this overvalued paper to the 99% who had no part in this. This will end bloody.

Scooby Dooby Doo's picture

Get the pleebs to put their money in banks and bitcoins(the circus).

Suppress value of real money long enough for elite to gobble it all up.

Kill pleebs.

monad's picture

Debt is a wealth transfer weapon.

Latitude25's picture

Let pleebs kill pleebs.  Much easier.

pigs-n-space's picture

Ask Mary, Let them eat cake. ........

Mini-Me's picture

A well-placed EMP and we'll be back to barter in a big hurry.

Peter Pan's picture

All you need is a malevolent version of Edward Snowden who not only wants to expose the rot but also make it crash by accessing the system and making it crash in an unalterable and unstoppable way. Sooner or later this is going to happen and then we will be boiling our leather belts and shoes for something to chew on.

trader1's picture


God, it's good to see you. We were going to change the world, Marty, remember?
Did you ever get around to actually doing it? No, I guess not.
- Well, I think I can. - Really?
Yes.
What's wrong with this country, Marty?
Money. You taught me that.
Evil defense contractors had it, noble causes did not.
Politicians are bought and sold like so much chattel. Our problems multiply.
Pollution, crime, drugs, poverty, disease, hunger, despair.
We throw gobs of money at them.
The problems always get worse. Why is that?
Money's most powerful ability is to allow bad people to do bad things...
...at the expense of those who don't have it.
I agree.
- Who did you say you were working for? - That's just my day job.
While in prison, I learned everything in this world, including money...
...operates not on reality... - But the perception of reality.
Posit: People think a bank might be financially shaky.
- Consequence: - People start to withdraw their money.
- Result: Soon, it is financially shaky. - Conclusion: You can make banks fail.
I've already done that. Maybe you've read about a few?
Think bigger.
- Stock market? - Yes.
- Currency market? Commodities market? - Yes.
Small countries?
I might even be able to crash the whole damn system.
Destroy all records of ownership. Think of it, Marty.
No more rich people, no more poor people, everybody's the same.
Isn't that what we said we always wanted?
Cos, you haven't gone crazy on me, have you?
Who else is going to change the world, Marty?
Greenpeace?


https://www.youtube.com/watch?v=coDtzN6bXAM

trader1's picture

how do you get 25 upvotes for the same "theme" for which i received 3 downvotes.

humans act irrationally more than not and/or you have a lot of "friends";-)

 

Peter Pan's picture

My apologies. Double post.

Peter Pan's picture

I really can't help you there Trader1 but if you really had the same theme then what was the point of repeating it? Perhaps being a bit more succinct may also help.

As for " friends" I am not sure what you are insinuating but I can assure you I have on occasion been hammered for some of my comments.

By the way you may have been punished for leaving out the next line in the script,......

"you are crazy." 

VinceFostersGhost's picture

 

 

for which i received 3 downvotes.

 

Welcome to Fightclub bitch.

Bluntly Put's picture

As my friend applied his change that morning, he triggered a bug in a notoriously crap script responsible for bringing new data disks online. The script had been coded in-house as this saved the bank about £300 per year on licensing fees for the official ‘storage agents’ provided by the vendor.

What about MERS? As far as I remember no one has gone back and reconstructed proper title chains at the county recording offices. Seems to me that computerized trading has forfeited hundreds of years of legal property claims, and rights, in order to facilitate high speed trading.

.

Latitude25's picture

Yeah I know a guy in FL who is STILL living in "his " house.  He challenged the bank in court to produce the mortgage documents to reposess the house and won.  The bank walked away and he just stayed in the house that now belongs to no one on paper.

KnuckleDragger-X's picture

This is the kind of stuff IT nerds who work on big corporate systems talk about over beers and hoping it never happens to them. I worked for IBM when mainframes truly ruled and these kind of emergencies made us a huge buttload of money......

Nick Jihad's picture

Yeah, I recall a very similar situation, where more storage needed to be allocated from a storage device, something that had been done before on hundreds of occassions, but on this occassion, the storage device spazzed out, and took down itself, hence the database servers, and thus the entire website. The vendor was never able to identify the fault that caused this random failure, but the sysadmins were admonished to get clearance from management before doing this in the future.

Dorelei's picture

The financial infrastructure is by far the most complex IT system ever created !

Hundreds of banks with millions of traders and robots inputting billions of parameters at all time. When the proverbial black swan hit the CPU fan it will generate so much chaotic volatility that it's probable some systems will go down and a deadly paralysis will spread across the world markets. It's likely our collapse will be brought by the mother of all bugs ! And the reset button is nonexistent ...
Amish Hacker's picture

Ah, humans. This is why every pencil comes with an eraser.

Macon Richardson's picture

What a brilliant little comment. Thank you.

opport.knocks's picture

Except those stubby ones on the golf course. 

Cadavre's picture

@Amish Hacker .. "Ah, humans. This is why every pencil comes with an eraser."

The weak link is not the capicitity to erase, it's knowing when to erase. 

When I moved beyond bean counting solutions, and into financial services, as as a contracting engineer, my first gig was with the intitutional division of an investment bank servicing FASB and GASB clients. The VP of that division was a a salt of the earth broker. Honest and straight shooter. So were most of his crew. My task was to build a portfolio interface and compute the actuals (principal return, income, cash, yield, and market to book values). I was told I got the gig because I was naive, meaning not corrupted by the bias of tenure. I remember my "audition". A room full of 30 MBA's. That meeting taught me how to recognize an MBA: When one asks 30 MBA's about an accrual method, and one gets 30 different answers, one is with  30 MBAs. Fact was, most of those guys didn't know the difference between a reporting accrual and a settlement accrual ... but I passed the audition, got a big box full of books, a check, and flew home to spend 6 months reading.

One has to have calculated the interest income for a monthly float, 64 day delay,6 month pay, end of month interest income accrual for an amortized bond, like an ARM, to understand, and then to have delivered the solution that only tolerated a misprint of one penny. Delivered about 3 versions, each one with improved reporting, like charts and executive summaries, roll offs, etc etc.

Some bonds had exotic traunches peeled off, like IOETTES, IO's. The typical purchase ranged from a million to 10 million for each portfolio,. Some portfolios had over 150 cusips and more than a billion bucks invested. The division used the output as a "value added" service to their clients. 

Most of the brokers (series 7 certs), in the division like I said, were salt of the earth, and believe it or not, "honest" brokers.

But there was this one broker who seemed to be joy riding. He was maintaining a porn site he operated, while at work, and missed a lot of work. He lived by the grace of the division head who charitably "partnered  with him, providing the f-off made a few phone calls.

Then, one day, one of the client's confirmed a sizable purchase, and it was Mr Lackadaisical's job, as a term of the charity partnership with the honcho, to verify the confirmation's info on a hand written order from the back office staff, for factor, interest payout, dates, etc, and then submit the confirm for settlement. He got 50K commission share, from his partnership with the boss for doing that, for this one bond.

The next day he comes to work wearing a 1000$  camel hair coat. A week later he divorces his wife and moves in with a dancer from one of the many gentleman's clubs clustered around the financial district.

THEN:AT THE END OF THE MONTH, it is learned that the purchase factor was a month old, and the client was asking for for it's overpayment back. His boss covered the loss cause Mr Lackadaisical had blown his share of the margin stand on a coat, a divorce and a new car for his trophy girl friend.

About 8 month's later, the bank shut it's doors because some broker allowed a client's rep make a purchase decision related to a fat PO (principal only) position for a university named after a big city in Illinois. The PO was in inventory a a result of a haircut scam being ran by a few traders and sold with a misrepresented prepayment speed.

Moral: Sometimes a stripper and a camel hair coat will screw up the "short between the headsets" idiot holding the pencil.