This page has been archived and commenting is disabled.
Shining Some Light On CME's Huge "Dummy Order" Error
Submitted by a reader
The top U.S. futures regulator said on Thursday it is looking at test orders that inadvertently went live on CME Group Inc's (CME.O) electronic Globex trading platform on Monday, adding "there may be lessons" from the mistake.
http://www.nasdaq.com/aspx/stock-market-news-story.aspx?storyid=201009161105dowjonesdjonline000427&title=cme-issues-apology-letter-to-clients-affected-by-botched-order-tests
Monday's mistake, which CME has attributed to human error, sent 30,000 test orders onto Globex, CME's electronic platform. The orders were placed there for 6 minutes, and wound up leaving some traders with unexpected positions. In addition, the exchange mistakenly sent out tens of thousands of messages, which typically include system updates or changes to particular markets or contracts.
Thursday's letter said that while it's not possible to cancel the trades, CME would "work with all affected customers to neutralize the financial effect of these transactions, including making customers whole for net losses they incurred in exiting positions."
Problems with this story...
1) Why would anything send out 30,000 test orders on the system simultaneously? Why would anyone couple that with an additional tens of thousands of messages? More than 30,000 orders are sent simultaneously every time the market goes through an inventory number on Wednesdays at 10:30. Just looks at that data.
2) The Oct11/Dec11 spread in both RB (gasoline) and HO (heating oil) contracted by at least $400 per contract in this 6 minute window. CL (crude oil) had significant moves by more in the longer dated part of the curve. NG (nat gas) was also effected.
3) Clearinghouses contacted customers late Monday night and asked them to verify trades that they had done against their statement. CME couldn't tell the clearinghouses what trades were put in error. CME sent out additional lists of customers that may have been affected at approx. 2am.
4) Trades went into customer accounts as positions, not necessarily as trades. Then position based algorithms went and hedged out exposure in an illiquid market. How do you determine what the net effect of the trade loss is in this case?
5) If these were test orders errantly put into the marketplace/Globex, why would they show up as positions in your trading statement but not trades on your TT machine?
6) One clearing firm alone reported over several thousand NN (nat gas swaps contracts) that hit their books near the bust range of an error trade. That would put it approx $800 per normal size contract. NN contracts for Globex on Bloomberg have this at about 300 in volume. So, if this was a real contract and it represents a real position, where is the print?
7) Investment banks and hedge funds lit up phones all over the place to figure out what was going on. Its not like no one noticed.
This thing has got to be over $30mil error. And just points out to the fact that they don't have proper risk controls. Additionally, they aren't really telling the whole story. Which is scary, because it just sounds like they are covering something up.
»
- 7739 reads
- Printer-friendly version
- Send to friend
- advertisements -
For the life of me, I do not understand the term "test orders". It is or it is not. Why is anyone allowed to put out feelers of any kind? Help!!!
Here, hear or whatever that phrase is....
Been around the futures markets since the CBT T-Bond conrtact (Way Back Machine POV) and I too would be most pleased to find out just WTF a "Test Order" is and to see the printed reference and guidelines to such in the Procedures and Delivery Manuals, including Viability and Resolution/Restitution of Trades.
And who claims they don't know, for Christ's Sake. The Clearing Corp has to match up each and every last buyer and seller. Or caught wanking again, little fellow?
Bullshit I say, Bullshit.
To paraphrase that which was so beautifully spoken the other day, by whom I remember not; "Easy there, Smedley, that sponge has corners, you know."
For every significant live computer system there is a non-live copy which is used to test software changes before the changes are implemented in the live system.
In the CME's test exchange system there are pretend "test" orders which cross and the resulting executions are all used to verify changes.
Routing so many test orders to live is a monumental cock-up which should cost the IT manager his job. If there is a conspiracy as ZH alludes - it is smoke and mirrors inthe coverup hoping that not many people would notice! The root cause will most likely be a config error in routing servers which exist between computer systems. The test orders and market alerts could be from a random generator program which was connected to the wrong router.
The interesting part is saying that they can't cancel the executions. There is no 2nd counterparty to DK (don't know) the original trades (so these MUST be cancelled). BUT, other live trades were done by the victims unwinding fake positions - which can't be unwound. The unwinding gave rise to risk.
However, this is part of the holistic market risk of being a trader! On a rare occasion positions can be wrong and this is like slippage - a small cost of being in this business.
The market spikes of ramaging algos which ZH document should never be DK'd either. The trades should stand costing the algo owners dearly when their software fails. This dictat should be across the board for all software trading errors.
Your explanation makes the most sense. IMO
Right On!
Whatever the cause, the counter-parties to the trades made full faith, best efforts to be there in the markets, exposed and just because some other entity makes an error such as these, the erring party is the one who should be held responsible, the trades left to stand.
Breaking trades such as these brings into question the very integrity of the marketplace; after all, many would love to cross the line between an error test order and just plain don't want to honor the transaction. From order devolves chaos...
Sorta like the drubbing that the senior bond holders in the GM deal took. One big MF test order dissolving the sanctity of the law of contracts. This is another example of the very filament of the financial system failing. Welcome to Zimbabwean markets.
And we were told the oligarchs were in Russia.
The root cause will most likely be a config error in routing servers which exist between computer systems. The test orders and market alerts could be from a random generator program which was connected to the wrong router.
I agree, this has all the appearances of an IT error, and routing (among other things) would explain it. Except, why is it possible for the test network to be so easily connected to the live one?
In the old days, you could tell your network was under attack when you saw your IT guy run down the hall into the wiring closet where he'd start frantically unplugging T-1s. In high-speed trading space, a whole lot of orders could be "placed" if someone saved the wrong config on a router before it would be noticed and corrected, or "unplugged."
100% correct. It might have been a router misconfig, or database, I suspect the latter; testing and QA was directed into the live database, and the records once created were stuck together with all the relational glue.
The real problem though is not the cock-up, those happen. It's not having a way to instantly (via scripts) back these things out without resort to manual SQL. That will be the discussion Monday morning; How to we prevent these things happening, and how can we clean it up next time it does.
But yeah, the IT director gets his head handed to him regardless. And/or they contract with another outsourced QA services provider (snicker)
+1 +1 and +1
I like your IT assessment.
And I love your don't-cancel-trades proposal! Let everyone bear the risks as well as the rewards for their chosen strategy.
I agree. While this is a serious cockup, it is not a complex one. Brokers and exchanges testing connectivity have screw-ups every now and then. Other times, there is a database setting screwup and a market-making algo goes haywire. Someone loses their job. There is a hunting expedition to find the root of the problem, which is then duly found and fixed, and Compliance, Risk Management, and Management all feel like they have done the right thing. Until the next time it happens.
The easiest way to combat test environment errors is to only allow access to exchange connectivity testing environments in off-hours (which is kind of an issue with Globex) or to a different address.
The easiest way to combat coding or database error is to run everything in a test environment.
It is really tough for users to protect themselves against exchange screwups (and yet another reason why moving everything to an exchange is not necessarily desirable.
I have been considering this "Exchange Balls-Up of the Century" and there is evidence that the risk of it has been baked in the cake for years.
Consider point 4 in the article. "4) Trades went into customer accounts as positions, not necessarily as trades." This implies that the test orders had real counterparty codes on them and passed internal checks. (The trades must exist but got held up somehow - probably due to invalid content).
Every live (and test) order will have a counterparty code of the originator of the order. It might be "PP" (Pimpco) or "VS" (Vampire Squid) for example.When the test orders were routed to the live system with valid counterparty codes - they passed the internal validation checks against the database before reaching the orderbook. This implies it has long been standard practise to have live account data in the test system.
Test counterparties should be dummy ones - like "T1" or "T2". These codes would not exist in the live system and orders for these will be rejected internally without causing harm or disruption to the orderbook.
The above control should have been right up there in the top 10 controls to prevent this type of disaster. There has been a systemic failure of institutional professional standards as well as that of every audit which allowed this state of affairs to continue.
A simple printout of counterparty codes and names from the test system to ensure that live ones are not present, and a second printout from the live system to ensure test counterparties are not present - would have revealed the risk of this failing years before.
That was exactly my thought. Testing with real counterparty codes??? Absurd. It seems virtually impossible that anyone is doing that.
I'm inclined to suspect a hacker, crazy algos, industrial espionage/sabatoge, or a disgruntled employee (in an inadequately controlled environment) instead.
I think they overlooked the risk.
Keeping the test system free of live codes is too much hard work and discipline. Much easier just to replicate live database tables into test periodically. The focus would be on keeping the systems similar to make it easier to verify software changes.
Gawd, it's hard to conceive of such sloth and irresponsibility. Even in Corporate America.
Just when I think I've seen it all, even my most cynical imagination fails me.
Beats me! But your conclusion does make for the simplest explanation.
thanks for the input---- reader
Looks like the FED fake price server is having a few hiccups. Prices need to go up every day at a steady 3% YOY growth rate with no deflationary action. To ensure this - it is best to switch off the real quote servers and just use more malleable price servers.
"In addition, the exchange mistakenly sent out tens of thousands of messages..."
Why is this significant, and exactly how many messages does this mean?
These are the same guys that own the Comex ain't it ?
Don't know about the Comex, but I'm fully aware of the Crimex.
Here's a good one, and the graph is phenomenal:
Selling Gold That Grows on Trees
http://news.goldseek.com/RickAckerman/1285308000.php
120 day months. It's the key to staying young.
You are one smart, incredible, guy. It's amazing to see you work.
More CME bull$h!t. It never ends...
Oh fuck! That's right.
(whimpering softly, imagining bad bad things)
Yes and Threeggg, too. I missed that...
Just a part of the design; multiple ways to manipulate. Keep it simple, stupid.
And the "regulators" say "there may be lessons." (per the Reuters article)
Yeah, like the integrity of US financial markets has reached the credibility and functionality of Zimbabwe.
WTF in Christ's name are any people domiciled offshore doing business in this country for anymore? I cannot for the life of me, if I were a senior portfolio manager at some major offshore entity, imagine trying to explain to my boss or an investment committee why I'm transacting in US Mafia markets if another viable alternative exists elsewhere. One of the allures of US markets historically was that they were well regulated and bounded by rules of law. All shot to fuckola in the last several years.
Does not bode well for the value of the dollar and inflows of foreign capital. T'is really sad. Seems more of thoughts/commentary on the Decline of Nation than traditional mundane Investment Strategy.
knukles.......i used to visit a grave site in a far away place where a young man was buried from a war long from my past.
it was a quiet place and beautiful and unknown except to only a few.........when i stopped to look at the grave stone i used to think about his life and the innocnce of it before i ever came to know the privildge of his resting place.......
i was always struck by the fortune i was given in my life by his responsibility and the responsibility of others, although we never knew one another, but in ways also came close.
the fight this time is not so obvious.
Eloquent, touching and beautiful. Thank you.
+100
By Eric Bogle
Well, how do you do Private William McBride?Do you mind if I sit here down by your graveside?
And rest for awhile neath the warm summer sun
I've been walking all day, and I'm nearly done
And I see by your gravestone, you're only nineteen
When you joined the great fallen in nineteen sixteen
Well I hope you died quickly, I hope you died clean
Or poor Willy Mcbride, was it slow and obscene?
Did they beat the drums slowly?
Did they play the pipes lowly?
Did they bugles carry you over as they lowered you down?
And did the band play 'The Last Post' in chorus?
Did the pipes play 'The Flowers Of The Forest'?
And did you leave a wife or a sweetheart behind?
In some loyal heart is your memory enshrined?
And though you died back in nineteen-sixteen
In that faithful heart are you always nineteen?
Or are you a stranger without a name?
Forever enshrined behind some glass pane
In an old photograph, torn and tattered, and stained.
And faded to yellow in a brown leather frame.
Did they beat the drums slowly?
Did they play the pipes lowly?
Did they bugles carry you over as they lowered you down?
And did the band play 'The Last Post' in chorus?
Did the pipes play 'The Flowers Of The Forest'?
Well the sun's shining down on these green fields of France
The warm wind blows gently, and the red poppies dance
The trenches have vanished long under the plow
There's no gas, no barb wire, there's no guns firing now
But here in this graveyard that's still no-man's land
The countless white crosses stand mute in the sand
To man's blind indifference to his fellow man
The whole generation was butchered and damned
Did they beat the drums slowly?
Did they play the pipes lowly?
Did they bugles carry you over as they lowered you down?
And did the band play 'The Last Post' in chorus?
Did the pipes play 'The Flowers Of The Forest'?
And I can't help but wonder young Willy McBride
Do those that lie here know why that they died?
And did they really believe you when you told them the cause
Did they really believe that this war would end wars?
Well the suffering, and the sorrow, the glory of pain
The killing and dying they were all done in vain
For young Willy McBride it's all happened again,
And again, and again, and again, and again...
Did they beat the drums slowly?
Did they play the pipes lowly?
Did they bugles carry you over as they lowered you down?
And did the band play 'The Last Post' in chorus?
Did the pipes play 'The Flowers Of The Forest'?
I've been several times to the National Cemetery in Little Rock. It is known locally as the Confederate Cemetery. There are small white grave markers as far as the eye can see over the knolls. From the Civil War to those who died last week. It is a somber place. I've put a link to the Google view for those who'd like to walk with me. The actual site will make the hardiest warrior weep.
http://tinyurl.com/26ljqad
Just stroll down the E. 26th Street side as far as you'd like to wander.
Then:
Zoom in on the Google Earth Map:
http://tinyurl.com/29etveu
Good thing that one of the "unintended" consequences of regulation is to protect incumbents, even if they can't manage technology, change, or operations.
Has the CFTF approved another exchange to compete against COMEX?
I am an ex IT engineer who quit the BS system. Last place was typical corporate structure. Managers were all yes men that would never rock the boat. 75% of staff was H-1B because indentured servants are the most exploitable. Stupid policies from top to bottom. Quality and Precision, what the fuck does that have to do with anything.
Were these "test orders" also test securities? If not why not?
If test securities were used - what system was dumb enough to trade off/against them?
Were algos actually exchanging real money for TESTA?
What am I missing here? There's not a lot of detail in the linked story.
It was test (well, 'test') orders with real securities. Weird, yes. The systems saw it as real orders so any type of algo that was required to hedge against X position did so by picking it up Y, hence leaving participants with open positions which weren't actually required since the hedge was false to begin with.
Yikes. Sounds like they need to figure out how to do what they need to do with test securities like they do with equity trading systems.
This is a result of TBTF Bail Outs. I aint buy'n nuttin. They are cutting off the water in 48 hours. I dont care. Let the chaos begin. The end is near.
Let's not forget this, either, y'all:
http://www.zerohedge.com/article/was-april-28-mini-flash-crash-grand-rehearsal-real-deal-week-later-if-so-whom
Strange things are afoot at the Circle K.
They're up to something. Be careful.
dup
In the evening ceremony chanel bags,chanel handbags sale as the first high-level chanel designer handbags custom Chinese star chanel bags prices uk XuQing alone in Paris – 2010 Shanghai chanel bags online uk,chanel bags uk online shopping early series dress coach outlet as ceremony. coach outlet store is Karl Lagrange coach outlet online the anfield fantasy coach outlet 2010 is 30-40 in Shanghai outlet 2010 coach handbags,coach handbags oulet China’s amorous feelings chanel 2.55 handbags,chanel handbags black different dress.
Maybe it was Son of Stuxnet.