This page has been archived and commenting is disabled.
The $12 Trillion Fat Finger: How A "Glitch" Nearly Crashed The Global Financial System - A True Story
For all the talk of how the financial world nearly ended in the aftermath of first the Lehman bankruptcy, then the money market freeze, and culminating with the AIG bailout, and how bubble after Fed bubble has made the entire financial system as brittle as the weakest counterparty in the collateral chain of some $100 trillion in shadow liabilities, the truth is that despite all the "macroprudential" planning and preparations, all the world's credit, housing, stock, and illiquidity bubbles may be nothing when compared to the oldest "glitch" in the book: a simple cascading error which ends up taking down the entire system.
Like what happened in the great quant blow up August 2007.
For those who may not recall the specific details of how the "quant crash" nearly wiped out all algo and quant trading hedge funds and strats in a matter of hours if not minutes, leading to tens of billions in capital losses, here is a reminder, and a warning that the official goalseeked crisis narrative "after" the fact is merely there to hide the embarrassment of just how close to total collapse the global financial system is at any given moment.
The following is a true story (courtesy of b3ta) from the archives, going all the way back to 2007:
I.T. is a minefield for expensive mistakes
There's so many different ways to screw up. The best you can hope for in a support role is to be invisible. If anyone notices your support team at all, you can rest assured it's because someone has made a mistake. I've worked for three major investment banks, but at the first place I witnessed one of the most impressive mistakes I'm ever likely to see in my career. I was part of the sales and trading production support team, but thankfully it wasn't me who made this grave error of judgement...
(I'll delve into obnoxious levels of detail here to add color and context if you're interested. If not, just skip to the next chunk, you impatient git)
This bank had pioneered a process called straight-through processing (STP) which removes the normal manual processes of placement, checking, settling and clearing of trades. Trades done in the global marketplace typically have a 5-day clearing period to allow for all the paperwork and book-keeping to be done. This elaborate system allowed same-day settlement, something never previously possible. The bank had achieved this over a period of six years by developing a computer system with a degree of complexity that rivalled SkyNet. By 2006 it also probably had enough processing power to become self-aware, and the storage requirements were absolutely colossal. It consisted of hundreds of bleeding edge compute-farm blade servers, several £multi-million top-end database servers and the project had over 300 staff just to keep it running. To put that into perspective, the storage for this one system (one of about 500 major trading systems at the bank) represented over 80% of the total storage used within the company. The equivalent of 100 DVD's worth of raw data entered the databases each day as it handled over a million inter-bank trades, each ranging in value from a few hundred thousand dollars to multi-billion dollar equity deals. This thing was BIG.
You'd think such a critically important and expensive system would run on the finest, fault-tolerant hardware and software. Unfortunately, it had grown somewhat organically over the years, with bits being added here, there and everywhere. There were parts of this system that no-one understood any more, as the original, lazy developers had moved company, emigrated or *died* without documenting their work. I doubt they ever predicted the monster it would eventually become.
A colleague of mine one day decided to perform a change during the day without authorisation, which was foolish, but not uncommon. It was a trivial change to add yet more storage and he'd done it many times before so he was confident about it. The guy was only trying to be helpful to the besieged developers, who were constantly under pressure to keep the wretched thing moving as it got more bloated each day, like an electronic ‘Mr Creosote’.
As my friend applied his change that morning, he triggered a bug in a notoriously crap script responsible for bringing new data disks online. The script had been coded in-house as this saved the bank about £300 per year on licensing fees for the official ‘storage agents’ provided by the vendor. Money that, in hindsight, would perhaps have been better spent instead of pocketed. The homebrew code took one look at the new configuration and immediately spazzed out. This monged scrap of pisspoor geek-scribble had decided the best course of action was to bring down the production end of the system and bring online the disaster recovery (DR) end, which is normal behaviour when it detects a catastrophic 'failure'. It’s designed to bring up the working side of the setup as quickly as possible. Sadly, what with this system being fully-replicated at both sites (to [cough] ensure seamless recovery), the exact same bug was almost instantly triggered on the DR end, so in under a minute, the hateful script had taken offline the entire system in much the same manner as chucking a spanner into a running engine might stop a car. The databases, as always, were flushing their precious data onto many different disks as this happened, so massive, irreversible data corruption occurred. That was it, the biggest computer system in the bank, maybe even the world, was down.
And it wasn't coming back up again quickly.
(OK, detail over. Calm down)
At the time this failure occurred there was more than $12 TRILLION of trades at various stages of the settlement process in the system. This represented around 20% of ALL trades on the global stock market, as other banks had started to plug into this behemoth and use its capabilities themselves. If those trades were not settled within the agreed timeframe, the bank would be liable for penalties on each and every one, the resulting fines would eclipse the market capital of the company, and so it would go out of business. Just like that.
My team dropped everything it was doing and spent 4 solid, brutal hours recovering each component of the system in a desperate effort to coax the stubborn silicon back online. After a short time, the head of the European Central Bank (ECB) was on a crisis call with our company CEO, demanding status updates as to why so many trades were failing that day. Allegedly (as we were later told), the volume of financial goodies contained within this beast was so great that failure to clear the trades would have had a significant negative effect on the value of the Euro currency. This one fuckup almost started a global economic crisis on a scale similar to the recent (and ongoing) sub-prime credit crash. With two hours to spare before the ECB would be forced to go public by adjusting the Euro exchange rate to compensate, the system was up and running, but barely. We each manned a critical sub-component and diverted all resources into the clearing engines. The developers set the system to prioritise trades on value. Everything else on those servers was switched off to ensure every available CPU cycle and disk operation could be utilised. It saturated those machines with processing while we watched in silence, unable to influence the outcome at all.
Incredibly, the largest proportion of the high-value transactions had cleared by the close of business deadline, and disaster was averted by the most "wafer-thin" margin. Despite this, the outstanding lower-value trades still cost the bank more than $100m in fines. Amazingly, to this day only a handful of people actually understand the true source of those penalties on the end-of-year shareholder report. Reputation is king in the world of banking and all concerned --including me-- were instructed quite explicitly to keep schtum. Naturally, I *can’t* identify the bank in question, but if you’re still curious, gaz me and I’ll point you in the right direction…
Epilogue… The bank stumped up for proper scripts pretty quickly but the poor sap who started this ball of shit rolling was fired in a pompous ceremony of blame the next day, which was rather unfair as it was dodgy coding which had really caused the problem. The company rationale was that every blaze needs a spark to start it, and he was going to be the one they would scapegoat. That was one of the major reasons I chose to leave the company (but not before giving the global head of technology a dressing down at our Christmas party… that’s another QOTW altogether). Even today my errant mate is one of the only people who properly understands most of that preposterous computer system, so he had his job back within six months -- but at a higher rate than before :-)
Conclusion: most banks are insane and they never do anything to fix problems until *after* it costs them uber-money. Did I hear you mention length? 100 million dollar bills in fines laid end-to-end is about 9,500 miles long according to Google calculator.
* * *
And here is Zero Hedge's conclusion: the next time you think all those paper reps and warranties to claims on billions if not trillions of assets, are safe and sound in some massively redundant hard disk array, think again.
»
- 110273 reads
- Printer-friendly version
- Send to friend
- advertisements -
This was the most exciting moment in my financial career.
http://www.bloomberg.com/apps/news?pid=newsarchive&sid=a5O2y1go1GRU
I saw that and thought....here we gooooo. And there we went.
Blame the hacker/computer for the actions of the ZioFED?
They would never do that.
RIPS
If you quant put your hands on it....
Exactly... If you cant hold it, you dont own it. Buy gold and Silver!!
Trade in your electronic digits for real money. While most markets are in a bubble, silver has been manipulated downward for years. Great time to add to your stack. These candles with silver coin prizes make great gifts and help people start their own stack: https://www.etsy.com/shop/ScentSavers?ref=hdr_shop_menu
Ask yourself, how is wealth protected if all it takes is a power outage or a computer glitch or a hacker to completely wipe it out? If you cannot answer that question honestly, you may someday be wishing you that you spent those digits while they still 'existed' and were 'worth something'
Phat Phinger Phuckups Phracture Phragile Phinancile Cystems.
Only an insider would put his money in the stock market. It's too risky for the average guy. Inside information is needed to win in a rigged game. Better hiding your cash in the mattress.
You can't fat finger physical......get thee some.
There is only one solution to this digital pile of shit, and that's called bitcoin.
Much faster than 1 to 5 days and imho never intended to be coffee money but the solution for this story. (and the source of rainbows and unicorns of course, but that's another story)
I am poor as a church mouse, but I do own five generators, three invertors and a whole host of spare tools, radio transceivers and other real world useful stuff, in several caches and FUCK have I got skills...
Once being able to "repair anything" (and having the tools required to do so, or being able to improvise) has more value than having a continous work history or the ability to write a good CV, (I.E. when the shit gets real) conditions will favour the likes of me..
if you don't code it
you don't own it..
Ha. Keep offshoring IT jobs to kraplocastan and fucking over your competent IT staff with furloughs, pay freezes, and no training you cheap cunts. I've seen this time and again over 25 years in IT and it fucks over everyone who tries to cheap out, but at least there are handy scapegoats in IT putting 18 hour days to fix it. Fucking assholes eat their seed corn all the time just to eek out quarterly numbers.
We are so fucked as a species.
I was thinking the same thing.
"as the original, lazy developers had ..."
I claim this is not the matter of finding someone or something to claim as the fall guy.
But rather, this is precisely how these a-holes think. They actually believe that without their genius marketing and sales skills, the world would literally stop turning. Without their mad-hot abilities; the IT folk, the engineers, and the scientists would not be able to produce or create anything of value. Because, you know, the bankers, the marketing department, the sales department, along the executives create the value proposition.
been fired
been laid-off
received no cost of living increase
received no bonus, promotions, or typical corporate incentives
watched as their jobs were off-shored (while training their replacements)
forced to follow the managements ill-conceived and under-funding project plans (or else be fired)
been the brunt of management failure and out-of-control egos
And since no one (i.e. engineers, scientists, etc) quits the good jobs (that are the opposite of what you describe), there is a strong bias towards any job opening being exactly as you describe.
Regards,
Cooter
Oddly enough the greater the pay, it seems the easier the work is and greater respect is paid to you.
Low paying IT work is just slavery and abuse all rolled into one shitty dealio.
+++ you got that right (glad I got in on that game early)
Yea I got laid off designing control systems for high voltage substations. These same geniuses brought in a bunch of H1Bs who thought they knew it all. In that business you need years of experience to design reliably. Soon we will have infrastructure running like it does in India. Intermittent periods when the lights come on.
Having worked on large teams of H1B visas (as a coworker) doing major/complex projects, let me share a beautifully simple insight with y'all.
American's (i.e. the North American US variety - sorry Brazil) have peculiar workplace habits. One of those is calling out, discussing, and resolving problems. Sometimes this is just a conversation, sometimes it is a meeting, but the jist is that it is about solving problems, the first step of which is identifying/accepting problems.
From my experience, this is NOT part of the work place culture of folks from MENA, Asian/Pacific Rim, etc. We did not have many Europeans, so have to exclude them from my anecdote.
Further, many cultures have very weird behaviors, such as choosing team leads based on religion, dialect, and/or caste from their respective regions, regardless of qualifications or experience. This quickly turns into a "gang" that roves around clubbing other "gangs" that show up, except they use keyboards.
All the while, they will insist, with the house burning down around their ears, that everything is great, under budget, on schedule, etc etc etc ... because culturally they do NOT admit failure/mistakes. If you ever visited a large corporate web site and wondered how it could be such a dysfunctional pile of shit ... well now you have a good conspiracy theory!
But, most software systems don't kill people, they just piss them off, and the shit is becoming a commodity anyway.
<looks around for a new k-wave to hitch his trailer to>
Regards,
Cooter
Been my experience too. For example, a lead will come to you the day something is supposed to go live and tell you it's not done rather than come weeks ahead. Even though they had to have known. I've worked with Europeans too (Germans, Frence, Italians) and they have been quite good and are not like that at all.
LOL, and all of the people who put their 401(k), 403(b) assets into safe "cash" allocations, or "dry powder" may be in for an abrupt rude awakening when those money market funds "break the buck" again.
Then again, those warnings about such risks tend to be buried far in the prospectus.
Here is generic boilerplate from one such fund:
And this too. http://www.zerohedge.com/news/2014-07-23/gates-are-closing-sec-votes-thr...
What a bunch os ASSHATS! ,, critical applications run on MAINFRAMES... ALWAYS!!! Online games can run on the servers...
Oh, and if your work isn't documented and approved , you're fired.
Insitutionial memory and archives count.
Don't like a rigged casino? Well, alternatively you can always buy a countries/companies bonds, but the only thing they ever promise to do is pay you in more fiat dollars, which may or may not be worth more than you originally gave them in buying power. And, even then the promise of return of original investment is iffy.
Rock, Paper, scissors.-- Ill take rocks for 1000 Alex, I know all about those damn scissors
http://ronpaulmessage10.com
Collapse information from Dr. Ron Paul
Cash in hand is king.
Try to Drink eat or shoot cash
That's why I've got a basement full of whiskey....can't eat it....but you can drink the hell out of it.
Mmm, I get where you're coming from, I really do, but try going in to my favourite Go-Go bar in Itaewon and offering the smokin' hot girls there some fine grass fed mongolian beef, a bottle of Glenfiddich or a mag full of 45's!!
This is all build up to transferring the losses from the 1% who hold most of this overvalued paper to the 99% who had no part in this. This will end bloody.
Get the pleebs to put their money in banks and bitcoins(the circus).
Suppress value of real money long enough for elite to gobble it all up.
Kill pleebs.
Debt is a wealth transfer weapon.
Well done.
Let pleebs kill pleebs. Much easier.
Ask Mary, Let them eat cake. ........
A well-placed EMP and we'll be back to barter in a big hurry.
All you need is a malevolent version of Edward Snowden who not only wants to expose the rot but also make it crash by accessing the system and making it crash in an unalterable and unstoppable way. Sooner or later this is going to happen and then we will be boiling our leather belts and shoes for something to chew on.
how do you get 25 upvotes for the same "theme" for which i received 3 downvotes.
humans act irrationally more than not and/or you have a lot of "friends";-)
My apologies. Double post.
I really can't help you there Trader1 but if you really had the same theme then what was the point of repeating it? Perhaps being a bit more succinct may also help.
As for " friends" I am not sure what you are insinuating but I can assure you I have on occasion been hammered for some of my comments.
By the way you may have been punished for leaving out the next line in the script,......
"you are crazy."
for which i received 3 downvotes.
Welcome to Fightclub bitch.
As my friend applied his change that morning, he triggered a bug in a notoriously crap script responsible for bringing new data disks online. The script had been coded in-house as this saved the bank about £300 per year on licensing fees for the official ‘storage agents’ provided by the vendor.
What about MERS? As far as I remember no one has gone back and reconstructed proper title chains at the county recording offices. Seems to me that computerized trading has forfeited hundreds of years of legal property claims, and rights, in order to facilitate high speed trading.
.
Yeah I know a guy in FL who is STILL living in "his " house. He challenged the bank in court to produce the mortgage documents to reposess the house and won. The bank walked away and he just stayed in the house that now belongs to no one on paper.
This is the kind of stuff IT nerds who work on big corporate systems talk about over beers and hoping it never happens to them. I worked for IBM when mainframes truly ruled and these kind of emergencies made us a huge buttload of money......
Yeah, I recall a very similar situation, where more storage needed to be allocated from a storage device, something that had been done before on hundreds of occassions, but on this occassion, the storage device spazzed out, and took down itself, hence the database servers, and thus the entire website. The vendor was never able to identify the fault that caused this random failure, but the sysadmins were admonished to get clearance from management before doing this in the future.
The financial infrastructure is by far the most complex IT system ever created !
Hundreds of banks with millions of traders and robots inputting billions of parameters at all time. When the proverbial black swan hit the CPU fan it will generate so much chaotic volatility that it's probable some systems will go down and a deadly paralysis will spread across the world markets. It's likely our collapse will be brought by the mother of all bugs ! And the reset button is nonexistent ...Ah, humans. This is why every pencil comes with an eraser.
What a brilliant little comment. Thank you.
Except those stubby ones on the golf course.
@Amish Hacker .. "Ah, humans. This is why every pencil comes with an eraser."
The weak link is not the capicitity to erase, it's knowing when to erase.
When I moved beyond bean counting solutions, and into financial services, as as a contracting engineer, my first gig was with the intitutional division of an investment bank servicing FASB and GASB clients. The VP of that division was a a salt of the earth broker. Honest and straight shooter. So were most of his crew. My task was to build a portfolio interface and compute the actuals (principal return, income, cash, yield, and market to book values). I was told I got the gig because I was naive, meaning not corrupted by the bias of tenure. I remember my "audition". A room full of 30 MBA's. That meeting taught me how to recognize an MBA: When one asks 30 MBA's about an accrual method, and one gets 30 different answers, one is with 30 MBAs. Fact was, most of those guys didn't know the difference between a reporting accrual and a settlement accrual ... but I passed the audition, got a big box full of books, a check, and flew home to spend 6 months reading.
One has to have calculated the interest income for a monthly float, 64 day delay,6 month pay, end of month interest income accrual for an amortized bond, like an ARM, to understand, and then to have delivered the solution that only tolerated a misprint of one penny. Delivered about 3 versions, each one with improved reporting, like charts and executive summaries, roll offs, etc etc.
Some bonds had exotic traunches peeled off, like IOETTES, IO's. The typical purchase ranged from a million to 10 million for each portfolio,. Some portfolios had over 150 cusips and more than a billion bucks invested. The division used the output as a "value added" service to their clients.
Most of the brokers (series 7 certs), in the division like I said, were salt of the earth, and believe it or not, "honest" brokers.
But there was this one broker who seemed to be joy riding. He was maintaining a porn site he operated, while at work, and missed a lot of work. He lived by the grace of the division head who charitably "partnered with him, providing the f-off made a few phone calls.
Then, one day, one of the client's confirmed a sizable purchase, and it was Mr Lackadaisical's job, as a term of the charity partnership with the honcho, to verify the confirmation's info on a hand written order from the back office staff, for factor, interest payout, dates, etc, and then submit the confirm for settlement. He got 50K commission share, from his partnership with the boss for doing that, for this one bond.
The next day he comes to work wearing a 1000$ camel hair coat. A week later he divorces his wife and moves in with a dancer from one of the many gentleman's clubs clustered around the financial district.
THEN:AT THE END OF THE MONTH, it is learned that the purchase factor was a month old, and the client was asking for for it's overpayment back. His boss covered the loss cause Mr Lackadaisical had blown his share of the margin stand on a coat, a divorce and a new car for his trophy girl friend.
About 8 month's later, the bank shut it's doors because some broker allowed a client's rep make a purchase decision related to a fat PO (principal only) position for a university named after a big city in Illinois. The PO was in inventory a a result of a haircut scam being ran by a few traders and sold with a misrepresented prepayment speed.
Moral: Sometimes a stripper and a camel hair coat will screw up the "short between the headsets" idiot holding the pencil.
Thanks, though I only understood a third of it.
The room full of MBA s was crystal clear after all these years, and still gives me the willies.
This must of been a long time ago.
The hand written confirm gives it away.
A fat finger may cause a few days of downtime, but it's unlikely to bring the whole system to an end or cause irreparable damage. These systems are designed to survive all kinds of disaster scenarios including some amount of human error.
Hardware and software is designed with intentional flaws (holes) to allow for intelligence spying. Those same holes can be used to exploit the system for personal gain or systemic chaos.
typedef enum {
PRIMARY_REQUEST_HANDLER,
BACKDOOR_REQUEST_HANDLER,
NUM_REQUEST_HANDLERS
} ANY_SOFTWARE_REQUEST_HANDLER_EVER_MADE;
p.s. hardware backdoors... we question the authenticity (goodness/honesty) of electronics manufacturers. That makes youa kook, doesn't it. "d00d, they're spying on us. The camera's watching man." Response: "Ok, Dr. Buzzkill."
The Titanic was also designed that way.
I think they way it would go would be more along the lines of the Challenger Shuttle...
@Zoomorp ... "A fat finger may cause a few days of downtime, but it's unlikely to bring the whole system to an end or cause irreparable damage."
How true those words be.
To bring down a system requires time to setup for the outcome. First, the RICO actors have to sync with the FED and DoT. Then they have load up on put options, knowing that their chumps at the SEC will loose or flame the paper trail. Then they made need the services of a CIA operative/contractor, hopefully one whose wardrobe includes a shemag or keffyeh or gutrah and at least one laundered thobe (to make it look authentic). A bonus benefit might be some real-estate actor, who recently purchased some assets that can be used as the target, but under some kind of IBC violation, so the ask will be low, and insured for the precise gambit the plan will use.
Then plan managers just have to sit back and guide the players to their targets.
Those mini crashes could be low end short players painting the board, or some dufus that farked up an order entry, but the big ones, the really really big ones ... well those guys take a time and planning. All the actors have to be on the same page.
You should re-read the article. This system was designed to survide disasters, but it was that disaster-recover code that caused the disaster in the first place. Unsurprisingly, DR is the least-tested portion of most IT systems.
Fuck 'em.
Some people worry about financial armageddon, and many more people each day are praying for it.
Please give me a date for Armageddon. I have a $22k master card I want to buy PMs with
When a company goes belly up what can i get for a put option on it's stock.
The boyz reminiscing about their deeds/misdeeds while feeding the fiery furnace of misallocation, a so-called "honest day's work".
FUCK. OFF.
PS message about the precariousness of the IT systems holding our "financial system" together noted.
"'wafer-thin" margin"
The term is "razor thin", and the inspiration for my handle. We are all hanging on by a mere micron of shit...for now.
He's talking about weekly church rations for the hungry.
He;s British, I assume. They don't speak proper English good like us'uns do.
"Wafer thin" is another reference to Mr. Creosote from Monty Python's "Meaning of Life".
A single mint, despite being "wafer-thin" was too much for the gluttonous, already overstuffed Mr. Creosote, causing his entire torso to explode. The metaphor doesn't fit exactly, but it does fit loosely.
Rest easy, when it pops Jesus will be there.
I've worked with banksters on and off for over twenty years, and a shiftier, more corner-cutting bunch of sleazeballs there is not on this Earth, always looking to just violate protocol and break the system one more time. Now that there's nobody to stop them but a bunch of little brown foreigners who don't even speak the language, they get away with it more often than not, too. Until they don't. Oops.
Are you talking about Preet Bharara lol?
Don't fuck with them coolies. They are very bright, very polite, very tricky and very crooked. The Jews are afraid of them, one day they will take over Wall Street.
What an interesting story..but never fear..if a crash actually took place...well I guess we would have to blame the Chinese or North Korean hackers.
MF Global was a testing bed. ^^^ you forgot to blame the Russians. It's never our fault.
/sarc
Banks are always 10 years behind in implementing technology because they are cheap motherfuckers.
not cheap, but nobody ever got a billion dollar bonus for a bulletproof IT system.
But take a trillion dollar bet? Now you are a banker.
Which is why my DBA team was replaced with an offshore Indian team.
For a while.
most bank auto-teller machines are still on XP, the banks pay MS a shit-load of money (they charge to you) to continue supporting the XP teller machines.
Security? Fuck that, it's not our money.
Just as Capitalism is a bastardization of the concept of Free Markets, "Finance" is a corruption of the concept of money.
Very good.
I feel like such a putz.
I once worked on a billing system, and well, long story short, the system lost a quarter million of billables.
I always thought that was a cool story.
Fear Porn.
Giving suggestions, and executiongs commands?
Long Super Volcanos
Sounds like Flash Crash Epitaph. Google "Flash Crash Ephitaph Kelley".
Here is why gold prices are so low:
http://michaelekelley.com/2015/07/20/dear-fed-plz-raise-gold-price/
http://www.zerohedge.com/news/2015-07-09/are-big-banks-using-derivatives-suppress-bullion-prices/
Here are some more signs of a coming recession.
http://michaelekelley.com/2015/05/29/mergers-and-acquisitions-set-record...
http://michaelekelley.com/2015/02/20/fed-warns-of-two-bubbles/
http://michaelekelley.com/2015/02/24/would-you-pay-39-more-than-asked/
http://www.zerohedge.com/news/2015-07-27/when-will-we-ever-learn/
Here is how to prepare.
http://michaelekelley.com/2014/10/16/8-things-to-do-when-recession-happens/
Here is how to get your mind off this stuff.
http://michaelekelley.com/category/humor/
Good luck!
Why even bother bringing this to our attention from 8 years ago!...
How many times has DOW, NASDAQ and S&P just thrown the power button "off" then "on" again at varying degrees of length during trading hours in those 8 years???...
Too many times to even fucking count! The frog has been in the water and the pot it's bathing in has been at a full rolling boil for 15 years now!...
Didn't need Michael Lewis on 60 minutes to tell us that's been the case either!!!!
In the next crisis or black swan someone’s untested little used crap code may destroy your assets. Code is written for everyday use. High transaction volumes or large complex transactions may break these systems in crisis conditions meant to run in an everyday controlled environment.
A good analog I once heard is a nuclear sub. They run fine under normal conditions. But when the order is called for nuclear war in these subs they may fail. They fail because too many of these war systems have never been really tested or used. The same is true with a bank's computers. Rarely are critical systems thoroughly stress tested today because of costs.
You write a program with 100 user settable binary switches. There are probably 9.332622e+157 possible setting combinations over the range of the 100 switches, they system users typically use 50 of them. Five years later somebody trys the 51st and they system goes down. Whose gonna sit there and test the full range? Add to that the fact that most primary systems grow organically, like cancer. I would never want to be an IT adminstrator. The only good hope is that the IRS and NSA will get fatally glitched one day.
They're getting us ready for when the actual crash happens they can blame it on a glitch.
Lol, I once met a psychologist that thought it was appropriate to colllect rent at " MidNight" (sp)
One for Ned!
The chaotic system is entering turbulence. Time to head for the hills.
As a retired Database Administration (DBA) manager in IT finance I have been in his shoes many times before I retired. Banks are cheap and expect HUGE system processing for little investment. So primary systems often get built piecemeal with devastating results. I do not miss the adrenaline filled recoveries at 3:00 A.M. after my wake up call and then later calls with the top IT management reminding me the database downtime is costing $10K a minute.
Then the were the lessons learned sessions where I had to explain in polite terms to a programming manager that his “imbecile” programmer had brought down the system because of poor testing methods while I listened to taunts that my databases (that were not changed in years) had really been the cause of the downtime. So people were very surprised when I retired early to taunts of “...why retire you have such a soft easy job…” I don’t miss it at all...
The beauty of Zero Hedge, is that it's readers are already " well versed".
We don't need countless ideas, and mindless diatrabe.
I feel you. "It's ALWAYS the Database."
I've slept on many a computer room floor.
I've since transferred to an internal consulting role.
I sleep better, now.
"I've slept on many a computer room floor."
Used to work the hardware side. The coldness really hits at about 5 AM. Body temp down, tired, constant noise, drinking coffee where coffee isn't allowed. Ugh, good times!
Lot's of ways to screw it all up, that is for sure. Humans put way to much stock in technology. I designed microchips and it was always amazing to see bugs show up in parts that had been in production for years.
--> Barclays
--> Deutsche Bank
First law of System Admin:
NEVER. TOUCH. PRODUCTION. WITHOUT. AUTHORIZATION.
Agree with First law of System Admin:
I once sub'd in for the network admin at a trading firm. Around 11am on my first day, all the trader's desktops stopped communicatiing out of the network. Back-end trading systems still worked. (Fortunately, I was still in training and hadn't been given any access yet, so I had plausible deniability.) No one could figure out the problem, and oddly enough, the problem cleared itself up after 2 hours.
It turned out, a technician had connected and started configuring a new blade chassis during trading hours. During the config, the networking blade on the chassis started advertizing its MAC address as the default gateway (arp proxy), effectively cutting off everything on that segment (included the traders' desks) from the rest of the network. The network only reconnected after the tech shut off the blade chassis on a lark. (He was still in denial about causing the outage, but I had the ARP records to prove it.)
Leason learned - do not touch production except during authorized maintenance windows, even to add unconfigured hardware. The unexpected can happen.
this article is a work of fiction
Great story!
Current electronic infrastructure and protocol implementations are a joke.
The Intercept has a good article on the "dirty tricks" and exploits GCHQ is using to manipulate the real world to match their fantasies; screwing with financial systems is right up their alley.
https://firstlook.org/theintercept/2015/08/07/psychologists-work-gchq-de...
As a system administrator, I can't tell you how many times my staff and I have done year long studies to determine a proper conversion to a newer system, and then presented a recommendation for a new system based on those studies. And then the corporate CEO's don't pay any attention to the recommendation and instead decide to buy some horseshit system that's cheaper. But it's always a horseshit system. We tell them it won't work, it will never work, there is no way to make it work, and it will cause nothing but heartache and continual trouble.
But they don't care, they go ahead and buy that anyway, and then we have to try to make it work, but it never does... And then they usually fire a bunch of us because they say we are incompetent. And why are we incompetent? Because we can't make the horseshit computer system they decided to buy function properly.
To summarize: if you're in the IT field and you are really smart, the best thing to do is to only be in the business of getting paid to do year long studies to recommend systems to CEO's and then leave there, go to another company the next year and do the same thing. Do that for about 20 years and you'll have made yourself a nice little retirement and you'll never be blamed for anything.
So...took a three pound magnet and had it pulverized into powder.
Guess where it went?
Got gold?
Wow, bank IT people are apparently overpaid incompetent hacks, but I suppose that helps them blend.
Bernie got away with less then a Commodore 64 for years.
If you are corrupt, you just lie.
If you are a bank, you just lie.
All large organizations have legacy programs written by programmers who are no longer there, undocumented, written in source codes that no longer exist and that current staff cannot even read, much less follow the logic of the original programmer. Add to this the widespread fakery practiced by personnel who pad their resumés. Most systems are implemented on a rush basis, so adeqate testing is not done. Unusual data input can trigger bugs never tested for. This conglamoration of weaknesses make for fragility that can shatter a system at any moment, even if there is no obvious hacking. Add to this the volumes of input implemented by high frequency traders and failure changes from a possibility to a probability.
I don't make fat-"fingered" mistakes.