This page has been archived and commenting is disabled.

NYSE Issues Official Explanation Why Its Was Offline For Nearly 4 Hours, Blames Software Upgrade

Tyler Durden's picture




 

Moments ago the NYSE released its explanation why its was offline for nearly 3.5 hours. Here is the official narrative:

On Tuesday evening, the NYSE began the rollout of a software release in preparation for the July 11 industry test of the upcoming SIP timestamp requirement. As is standard NYSE practice, the initial release was deployed on one trading unit. As customers began connecting after 7am on Wednesday morning, there were communication issues between customer gateways and the trading unit with the new release. It was determined that the NYSE and NYSE MKT customer gateways were not loaded with the proper configuration compatible with the new release.

 

Prior to the market open, gateways were updated with the correct version of software and stocks opened at 9:30am. However, the update to the gateways caused additional communication issues between the gateways and trading units, which began to manifest themselves mid-morning. At 11:09am, NYSE issued a Market Status message that a technical issue was being investigated. At 11:32am, because NYSE and NYSE MKT were actively trading but customers were still reporting unusual system behavior, the decision was made to suspend trading on NYSE and NYSE MKT. NYSE ARCA, Arca Options and NYSE AMEX Options were not impacted by this event and continued to trade normally.

 

NYSE and NYSE MKT began the process of canceling all open orders, working with customers to reconcile orders and trades, restarting all customer gateways and failing over to back-up trading units located in our Mahwah, NJ datacenter so trading could be resumed in a normal state. In consultation with regulators and industry, we determined that we would implement a complete restart and that NYSE MKT primary listings would resume trading at 3:05pm and NYSE primary listings and NYSE MKT Tape C symbols would resume trading at 3:10pm. Trading resumed as scheduled and the closing auctions accepted orders and executed normally. All NYSE and NYSE-MKT listed securities traded for the entire day either on NYSE and NYSE MKT or other market centers.

Odd: no mention of what CEO Farley told Bloomberg TV as to why there was no rollover to the NYSE backup system:  "we chose the least disruptive option for customers.  So if we had moved to our Disaster Recovery Center, which was an option, customers would have had to do a good deal of work to be able connect to that new Disaster Recovery Center."

So, as we noted yesterday, the NYSE has a disaster recovery center which they choose not to use because it is an inconvenience to clients who would rather be unable to trade for over 3 hours.

Gotcha.

As for software upgrades and, of course, "glitches", always being the cause of historic market events: funny how these always take place on major market down days.

Then again, as we also said yesterday, in the grand scheme of things the NYSE hasn't mattered in a while:

In fact, the only "program" that does matter in this centrally-planned market is the CME's CBIP or Central Bank Incentive Program: lose that and things may not be as generally "calm" as they  were yesterday,

 

- advertisements -

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
Thu, 07/09/2015 - 12:41 | 6290554 WTFUD
WTFUD's picture

More Skullfuckery.

Thu, 07/09/2015 - 12:45 | 6290576 BullyBearish
BullyBearish's picture

...and, you can keep your own doctor

Thu, 07/09/2015 - 12:46 | 6290588 Four chan
Four chan's picture

capitalism takes a break, maybe not such a bad thing?

Thu, 07/09/2015 - 12:48 | 6290603 Pladizow
Pladizow's picture

I smell the strong stench of BULLSHIT!

Thu, 07/09/2015 - 12:51 | 6290619 pods
pods's picture

Fitting that all this was caused by UpgrayeDD.

pods

Thu, 07/09/2015 - 12:57 | 6290636 nope-1004
nope-1004's picture

Baffle'em with bullshit.

Of course, the timing is purely coincidental.  "Software upgrades" could mean anything, as in going offline for plunge protection.  There's your software upgrade in the midst of zero evidence to substantiate the "official" release.

 

Thu, 07/09/2015 - 12:57 | 6290656 SickDollar
SickDollar's picture

The tribe is so good at their lies

Thu, 07/09/2015 - 13:00 | 6290674 Fun Facts
Fun Facts's picture

They never tell us the truth about anything, and they certainly would never admit to being hacked by the Chinese because that portends weakness.

It's all about keeping up appearances to the donkeys they rule over.

Thu, 07/09/2015 - 13:12 | 6290722 Tom Servo
Tom Servo's picture

2 "Ds" for a double dose of this pimpin...

 

 

Thu, 07/09/2015 - 13:32 | 6290825 Beam Me Up Scotty
Beam Me Up Scotty's picture

"capitalism takes a break, maybe not such a bad thing"

Capitalism has been on break for far longer than one 3 hour period.  In fact, its been on break for so long, its dead.

Thu, 07/09/2015 - 14:01 | 6290982 thebigunit
thebigunit's picture

Really?

Capitalism has been on break for far longer than one 3 hour period.  In fact, its been on break for so long, its dead.

Did I miss it, or are corporations now passing out equity shares "fairly" so that no one makes a profit by buying or selling?

Thu, 07/09/2015 - 14:34 | 6291229 mvsjcl
mvsjcl's picture

After 15+ years in the corporate world (Fortune 500 company) as a systems analyst/architect, this "explanation" is pure, 100% hogwash.

Thu, 07/09/2015 - 16:37 | 6291901 Liposuction
Liposuction's picture

Is he like an art dealer or somethin'?

Tue, 07/14/2015 - 12:16 | 6311163 ZeroPoint
ZeroPoint's picture

He's Obama's 'manager'...

Thu, 07/09/2015 - 13:00 | 6290673 RiverRoad
RiverRoad's picture

+1  Make that the United Airlines, New York Stock Exchange, Wall Street Journal, Zerohedge  and  who else? simultaneous GLITCH!!

Thu, 07/09/2015 - 13:44 | 6290875 weburke
Thu, 07/09/2015 - 15:39 | 6290904 Paveway IV
Paveway IV's picture

"...Make that the United Airlines, New York Stock Exchange, Wall Street Journal, Zerohedge and who else?..

A German SAM site?

Hackers take over Patriot missile battery in southern Turkey

PressTV - July 8, 2015

Unknown hackers have reportedly taken over the German-owned Patriot air and missile defense systems stationed in southern Turkey on the border with crisis-hit Syria, and given it unspecified commands.

 

According to a report published by German-language Behörden Spiegel magazine on Tuesday, the US-built systems, consisting of six launchers and two radars, were briefly compromised and carried out orders.

 

The report, however, did not provide any information about when the orders were carried out and what they were.

...

“It is certainly the case that foreign governments, intelligence agencies do try to hack into these systems, and it may well be that the software built into the missile has been compromised in some way by some foreign government,” Schifreen said.

So, we can look for the passenger airliner shootdown over Southern Turkey, which will give the Turkish troops massed on the Syrian border causus belli to invade Syria. Mostly to save their JAN and ISIS mercenaries that are getting their ass kicked and about to lose Aleppo. Time to throw those used-up stooges under the bus - sorry ISIS.

I suppose they could shoot down one of the occasional passenger airliners that transit Syria itself. Israel could always load up one of their fake airliners with dead Gazan children (there's plenty) and claim it was Israeli citizens.

Kobane makes sense now - the agreement with the evil U.S. must have been that the Rojava Kurds will be immune from Turkish attack in any Syrian territory they control. The Turks would just as soon kill all Kurds (and will if they ever have the chance) but for now, they have to tread lightly because Uncle Sugar would get mad.

Israeli's JAN lackeys in Northern Golan are also getting their asses handed to them, so I would expect either an Israeli attack, or more likely a Jordanian ground attack backed by Israeli missiles. Either way, the Israelis want another 20-mile buffer beyond the UNDOF buffer land protecting their other stolen land in Syria and Lebanon. Israel is a giant real estate aquisition scheme.

This will not be billed as an attack on Syria, but an attack on ISIS. I guess for the fake airliner shoot-down with a 'hacked' German patriot in the current scenario. The exact method is irrelevant - it may be something completely different. It WILL happen in the next month or two and a false-flag will start it. They got nothing else.

Thu, 07/09/2015 - 16:55 | 6292009 Paveway IV
Paveway IV's picture

Has Germany's Patriot missile system been hacked?

DW 8.7.2015

Summary: German Defense Ministry strenuously denies and suggests DW not write about it, but original source, Behoerden-Spiegel, insists as reported.

Thu, 07/09/2015 - 20:05 | 6292753 Crash Overide
Crash Overide's picture

Total bullshit, used to work in IT/market/finance, these "upgrades" are well planned and network setting and configurations for new software are tested and information distributed to all involved.

At least China told it how it was:  We will kick your ass and throw you in jail if you sell and further crash the game.

Others decided to blame a software update rather than see the cough... cough... market crash.

 

There are no "markets" left, just interventions.

Thu, 07/09/2015 - 12:48 | 6290607 New England Patriot
New England Patriot's picture

Public beta?

Thu, 07/09/2015 - 12:49 | 6290608 Dr. Engali
Dr. Engali's picture

There's some capitalism going on some place? Do tell?

Thu, 07/09/2015 - 12:51 | 6290614 BandGap
BandGap's picture

....and unemployment is 5.3%

Thu, 07/09/2015 - 12:43 | 6290560 nuubee
nuubee's picture

Software upgrade? You mean someone hit "Yes" to their McAffee notice?

Thu, 07/09/2015 - 12:51 | 6290623 madcows
madcows's picture

Or, one of their 500 daily Java notices.  Fucking Java.  How about an update a month.  Instead of a dozen per day.

Thu, 07/09/2015 - 13:03 | 6290689 nosam
nosam's picture

It took them 3.5.hours to roll back a limited upgrade??

Thu, 07/09/2015 - 13:12 | 6290728 Colonel Klink
Colonel Klink's picture

Riiiiiiiiiiiiiiiigggggggggggggggghhhhhhhhhhhhhhhhttttttttt!!!!!!!!!!!!!!!

Thu, 07/09/2015 - 12:43 | 6290562 Soul Glow
Soul Glow's picture

Damn Windows always crashes at the worst times lol

Thu, 07/09/2015 - 12:43 | 6290563 Bazinga
Bazinga's picture

"the NYSE has a disaster recovery center which they choose not to use because it is an inconvenience to clients who would rather be unable to trade for over 3 hours."

Thanks for the laugh NYSE!

Thu, 07/09/2015 - 12:43 | 6290564 i_call_you_my_base
i_call_you_my_base's picture

Not a very good disaster recovery plan if all of your customers have to reconfigure their systems to accomodate.

Thu, 07/09/2015 - 12:44 | 6290567 Bay of Pigs
Bay of Pigs's picture

Yeah, and that fuckan dog keeps eating my homework too.

Thu, 07/09/2015 - 12:44 | 6290570 gcjohns1971
gcjohns1971's picture

H1B imports don't know what the University can't teach.

The university doesn't employ the people who invented this stuff, but only those who observed them.

The people who invented this stuff are the ones you fired to hire Mr H1B.

Mr Invented-That-Shit is going to ask a premium to come out of retirement and fix what would never have broken were he still at his desk.

Mr H1b won't be so cost competetive then.

Thu, 07/09/2015 - 12:47 | 6290592 Big Corked Boots
Big Corked Boots's picture

Generally, the H1B doesn't know what the University does teach, either.

Orientation for new Chindian hires: This is a bathroom. You sit on this white thing called a toilet. This lever flushes it.

Thu, 07/09/2015 - 12:52 | 6290627 Dr. Engali
Dr. Engali's picture

And what do I do while sitting on this toilet?

Thu, 07/09/2015 - 13:07 | 6290706 Paveway IV
Paveway IV's picture

Tech support.

Thu, 07/09/2015 - 13:27 | 6290798 large_wooden_badger
large_wooden_badger's picture

Don't be stupid, you bill your client. YOU ALWAYS BILL THE CLIENT

Thu, 07/09/2015 - 13:01 | 6290675 nosam
nosam's picture

How did you figure out it wa the fault of an H1b, you racist?

Thu, 07/09/2015 - 12:44 | 6290571 carbonmutant
carbonmutant's picture

The first rule of combating a cyber-attack is to deny it is a cyberattack.

Thu, 07/09/2015 - 12:44 | 6290572 NoDebt
NoDebt's picture

<Cough> <Cough> <Bullshit> <Cough>

 

Thu, 07/09/2015 - 13:14 | 6290740 Colonel Klink
Colonel Klink's picture

The bullshit part deserve caps, bold, underline, and numerous exclamation points because of the true bullshittery of it.

Thu, 07/09/2015 - 12:44 | 6290573 buzzsaw99
buzzsaw99's picture

the squid had a losing trade so they shut it down

Thu, 07/09/2015 - 13:26 | 6290786 large_wooden_badger
large_wooden_badger's picture

Best, simplest, most succinct explanation so far. It's true, the house always wins.

It's like they went full reverse "Duke Brothers". TURN THOSE MACHINES OFF!

Thu, 07/09/2015 - 12:45 | 6290574 rosiescenario
rosiescenario's picture

Chinese officials are considering this same "software upgrade" as a quick means to halt their collapsing market....upgrades would occur on any day that the market was dropping.....

Thu, 07/09/2015 - 12:45 | 6290578 KnuckleDragger-X
KnuckleDragger-X's picture

the HFTs likely ate it alive......

Thu, 07/09/2015 - 12:45 | 6290583 Kaiser Sousa
Kaiser Sousa's picture

la, la, la, la,.....

BULLSHIT!

Thu, 07/09/2015 - 12:45 | 6290584 Lady Jessica
Lady Jessica's picture

"Oh, and one more thing, margin calls on Chinese investors meant they tried to liquidate their BABA, and we just couldn't have that."

Thu, 07/09/2015 - 12:46 | 6290587 Youri Carma
Youri Carma's picture

Why did WWIII break out? Software upgrade which launched all the missiles. LOlz

Thu, 07/09/2015 - 12:49 | 6290591 NubianSundance
NubianSundance's picture

United Airlines and WSJ must also have been doing major updates yesterday that went snafu all at the same time. Not buying that.

Thu, 07/09/2015 - 12:51 | 6290624 Amish Hacker
Amish Hacker's picture

And don't forget the ZH updates, similarly scheduled.

Thu, 07/09/2015 - 12:55 | 6290644 NubianSundance
NubianSundance's picture

Some AM outlets are saying that Anonymous posted a tweet the previous evening predicting a bad day for Wall Street.

Thu, 07/09/2015 - 12:59 | 6290663 L Bean
L Bean's picture

That's akin to predicting the sky will be blue on that day too.

Thu, 07/09/2015 - 13:02 | 6290679 NubianSundance
NubianSundance's picture

:)

Thu, 07/09/2015 - 13:26 | 6290790 detached.amusement
detached.amusement's picture

because it was obvious the tribe was going to lose money - if you've got some actual data in your hands

Thu, 07/09/2015 - 12:56 | 6290650 L Bean
L Bean's picture

Even Megyn Kunty of Fox News said she had that "spinning rainbow ball thingy" on her iphone all day.

Thu, 07/09/2015 - 12:47 | 6290594 Dr. Engali
Dr. Engali's picture

Yeah, and nineteen cave dwelling hijackers are responsible for knocking down the twin towers too. "Official" narratives crack me up.

Thu, 07/09/2015 - 12:55 | 6290643 Mike Honcho
Mike Honcho's picture

I've seen it a million times.  A group of people vastly out number attackers, but they have 1 inch blades.  Curl up in fetal position everytime.  I would rather go out in flames than remove a bandaid.

Thu, 07/09/2015 - 14:37 | 6291247 Ignorance is bliss
Ignorance is bliss's picture

And building 7 and blowing a hole in the pentagon.

Thu, 07/09/2015 - 12:48 | 6290600 Amish Hacker
Amish Hacker's picture

Today's Thursday. If they can just keep spinning the laughable official explanantion until the markets close tomorrow, the whole thing will be down the rabbit hole and forgotten by Monday.

Thu, 07/09/2015 - 12:48 | 6290606 ted41776
ted41776's picture

yeah, ok... i totally buy that

Thu, 07/09/2015 - 12:49 | 6290609 Seasmoke
Seasmoke's picture

I would be even more worried after hearing that.

Thu, 07/09/2015 - 12:49 | 6290610 Roving reporter
Roving reporter's picture

We locked some folks out of their portfolios ...

Thu, 07/09/2015 - 12:50 | 6290612 Mike Honcho
Mike Honcho's picture

Anyone think to do this over the weekend, no?

Thu, 07/09/2015 - 12:51 | 6290617 clade7
clade7's picture

 

We would easily believe the switchman was getting a hummer by a bucket boy from the late night Nicarauguan cleaning crew....but no, you had to go all poindexter and blame it on sunspots..

Thu, 07/09/2015 - 12:51 | 6290622 atlasRocked
atlasRocked's picture

Ok show us the deployment email orders that went out to coordinate the deployment.  

Thu, 07/09/2015 - 12:54 | 6290638 Mike Masr
Mike Masr's picture

Good point atlasRocked!

Thu, 07/09/2015 - 12:51 | 6290626 replaceme
replaceme's picture

Maybe the HFT crowd doesn't have their ultrafast connections to the backup facility, and they wouldn't allow a failover. That I could believe over changing to the other site being too difficult for the other traders.

Thu, 07/09/2015 - 12:52 | 6290628 p00k1e
p00k1e's picture

What was the name of the software company? 

Thu, 07/09/2015 - 13:18 | 6290763 CPL
CPL's picture

ACME Consulting, they sell Widget software services according to the fictional situation in the textbook.

Thu, 07/09/2015 - 19:53 | 6292712 Skiprrrdog
Skiprrrdog's picture

Pretty sure it was Hu Flung Pu Software, LTD.

Thu, 07/09/2015 - 12:53 | 6290633 Mike Masr
Mike Masr's picture

How does that explain UAL and ZH also down at the same time?  

Thu, 07/09/2015 - 12:54 | 6290642 ZeroPoint
ZeroPoint's picture

The White House assured us they are confident the incidences are completely unrelated.

Thu, 07/09/2015 - 13:51 | 6290906 large_wooden_badger
large_wooden_badger's picture

So they really are all related...

Thu, 07/09/2015 - 12:59 | 6290664 p00k1e
p00k1e's picture

The Internet is more controlled than we imagined.  It’s one big VM in Hillary’s shed. 

Thu, 07/09/2015 - 13:22 | 6290779 large_wooden_badger
large_wooden_badger's picture

We must always "endeavor to persevere".

Thu, 07/09/2015 - 12:54 | 6290637 ZeroPoint
ZeroPoint's picture

Funny how those software upgrades always happen when foreign exchanges are in a full panic......

Thu, 07/09/2015 - 12:58 | 6290651 cowdiddly
cowdiddly's picture

If you ever get the impression that your money will get put on lockdown in the next collapse of even a mere 2%? Don't worry it will be.

 

Thu, 07/09/2015 - 13:00 | 6290669 replaceme
replaceme's picture

...and this outage has nothing to do with the B3 being deployed to Turkey. I will not take any questions on the B3, sorry for even bringing that up.

Thu, 07/09/2015 - 13:00 | 6290672 The Delicate Genius
The Delicate Genius's picture

the chinese threat map.... arch neocon and hubby of Victoria Nuland Fred "Israel First" Kagan... and Norse Corporation {creator of the China-did-it threat map}

http://rightweb.irc-online.org/profile/Kagan_Frederick

[is there anyone with a more punchable face???]

http://www.aei.org/events/honeypots-and-sticky-fingers-the-electronic-tr...

Iran is the big hacking threat.

Pay no attention to the massive Israeli espionage in the US .gov and Silicon Valley - home to about 20k Israelis... clearly Israel would never, ever hack American systems...

Why, we never hear about Israeli hacking, so it must not happen, right??

http://www.norse-corp.com/about-us/who-we-are/index.html
http://sluent25.net/sam-glines/

Think Sam might be a Zionist?

http://kuer.org/post/experts-divided-over-irans-cyberactivity-start-nucl...

All the world's a stage...

Thu, 07/09/2015 - 13:01 | 6290676 Mike Masr
Mike Masr's picture

The MSM media spent more time yesterday telling us it wasnt a cyber attack than on any of the other news of the day. 

It's great to be lied to over and over again!

Thu, 07/09/2015 - 13:02 | 6290680 p00k1e
p00k1e's picture

Only invest (relinquish control of) money you can afford to do without.    

Once the market finally seizes, we’ll all be paid.  It just may take 100 years before we get a check. 

Thu, 07/09/2015 - 13:02 | 6290682 LightWalker
LightWalker's picture

  Wouldn't it been smarter to wait until after close on friday then update and test their shit? Fools....

Thu, 07/09/2015 - 13:06 | 6290701 I Write Code
I Write Code's picture

Nah, do all development and testing right on the production systems, it's called "DevOps" and it's all the rage.

Thu, 07/09/2015 - 13:48 | 6290896 mbutler101
mbutler101's picture

LMAO...too funny. Sometimes that is often the harsh reality in small environments that have no sand boxes, no budget, and no sense. I work in IT and the running joke on whatever project is being worked on is "this will be totally seamless"

Thu, 07/09/2015 - 14:22 | 6291155 I Write Code
I Write Code's picture

Also a lot of bleeding edge Internet startups are all about DevOps.  I mean why not, when all you're doing is showing kitten videos and matching ads to eyeballs, it's not like anybody is auditing you.

You just might find that a lot of the biggies like Yahoo and Google are into this too, when you have 100,000 production systems and it takes a month to update them all, wtf if you kill fifty or a hundred?

Thu, 07/09/2015 - 13:03 | 6290687 cookies anyone
cookies anyone's picture

but... but... chinese hackers, russian hackers, north korea...

Thu, 07/09/2015 - 13:20 | 6290771 large_wooden_badger
large_wooden_badger's picture

Keep going... remember where it went with Sony? Inside job to promote a bomb of a movie.

Thu, 07/09/2015 - 14:14 | 6290911 L Bean
L Bean's picture

And it was offered free on Netflix/Amazon Prime about a week later anyway.

Thu, 07/09/2015 - 13:05 | 6290698 I Write Code
I Write Code's picture

Oh, well, if it followed standard practice, then, um, yeah ok fine.

Thu, 07/09/2015 - 13:08 | 6290704 localizer
localizer's picture

I wonder if software "upgrades" are normally done during business hours? Sounds like BS explanation.

Thu, 07/09/2015 - 13:12 | 6290729 Laughinggrizzley
Laughinggrizzley's picture

As an IT Project Manager, which deals specifically with Data Centers, the NYSE explanation is complete BS.

Thu, 07/09/2015 - 13:30 | 6290817 detached.amusement
detached.amusement's picture

+++ any computer tech worth a penny more than an H1B salary knows that!

Thu, 07/09/2015 - 15:41 | 6291633 Clowns on Acid
Clowns on Acid's picture

The H1B staff @ NYSE is most probably the cause behind the breach.  

Thu, 07/09/2015 - 14:16 | 6291103 localizer
localizer's picture

Doesn't take an IT expert to figure out this is complete bull. Seriously, couldn't they come up with something better?! "Software upgrade" LMAO, what are they taking people for - complete idiots?

Thu, 07/09/2015 - 13:12 | 6290727 katchum
katchum's picture

Audit the business impact assessment, disaster recovery plan and crisis management plan please.

Thu, 07/09/2015 - 14:22 | 6291156 moonman
moonman's picture

Audit?

Ha ha ha ha ha ha ha ha ha

 

 

Thu, 07/09/2015 - 13:13 | 6290737 f16hoser
f16hoser's picture

Someone or some gubernment, wanted to off-load a bunch of JPM stock. Guess they were Cock-Blocked...

Thu, 07/09/2015 - 15:01 | 6291397 lakecity55
lakecity55's picture

*ring*

Hello, Bath House here.
Barry, it's Jamie! You gotta shut down the market! We gotta move some stuff before the open!
No problem, I'll let Joe push the button. He likes the strobe light that goes with it.
Whew! Thanks, bud!
Anytime.

Thu, 07/09/2015 - 13:14 | 6290743 tnquake
tnquake's picture

NYSE was installing the new "Hyper Print" function for the connection to the FED. 

Instead of CTL-ALT-PRINT they hit CTL-ALT-DELETE.

Servers rebooted and all is well! Move along...

Thu, 07/09/2015 - 13:15 | 6290749 jay28elle
jay28elle's picture

Is the NYSE run by the same idiots that run OPM or IRS?

Thu, 07/09/2015 - 13:17 | 6290758 Joebloinvestor
Joebloinvestor's picture

IMO it was to disconnect any China HFT collusion.

Thu, 07/09/2015 - 13:18 | 6290762 large_wooden_badger
large_wooden_badger's picture

This is all so... plausible?

Thu, 07/09/2015 - 13:19 | 6290764 kchrisc
kchrisc's picture

They lie about everything. Why would they lie about this?

Liberty is a demand. Tyranny is submission.

Thu, 07/09/2015 - 13:25 | 6290785 Caveman93
Caveman93's picture

Software upgrades are smoketested and ususally performed over the weekends. Bullshit.

Thu, 07/09/2015 - 13:32 | 6290826 detached.amusement
detached.amusement's picture

and usually thoroughly tested in a Dev platform first

I mean unless one doesnt have the money to have a dev area, but we all know damned well they have that money x a million

Thu, 07/09/2015 - 13:30 | 6290814 messy
messy's picture

I worked in a Wall St Bank IT before.  Minor upgrades are allowed only at night.  Major upgades are allowed only on weekends.  

Thu, 07/09/2015 - 13:42 | 6290864 large_wooden_badger
large_wooden_badger's picture

Any current operation that is in business to actually make a profit does this, never ever upgrade during operating hours. Plan a shutdown if you have to, but you never stop production. Same for filling soda bottles as it is for a trading desk.

Queue the hapless H1b that's going to get blamed for this: Hello, I am Ashkar from Mumbai.

Thu, 07/09/2015 - 14:51 | 6291328 Palladin
Palladin's picture

Yep, it is true.

On the west coast at 6:30am everything had better be up and running, no excuses tolerated. No upgrades were ever done before or during market hours. Ever.

Any upgrade was always tested at night, and run on a separate small development network. Never the active trading floor.

The trading floor was broken up into 4 segments and any upgrade even after it passed the isolated server, was only rolled out to one segment for a week or so. And the least used segment, Never OTC, Muni or Corporates.

Look at detail the CME went into over a Leap Second. One Second, and it wasn't even during regular trading hours:

http://www.cmegroup.com/tools-information/lookups/advisories/electronic-...

So the NYSE would have you believe that they had a "glitch" as the morons on CNBC called it. It wasn't a "glitch", it was a massive system wide failure.

Welcome to the Matrix.

 

Thu, 07/09/2015 - 14:58 | 6291379 lakecity55
lakecity55's picture

Errr, it was a massive intervention on the NYSE to prevent a melt-down, IMO.

Thu, 07/09/2015 - 13:52 | 6290916 large_wooden_badger
large_wooden_badger's picture

NYSE techs better check their A-Plus certification!

Thu, 07/09/2015 - 14:00 | 6290919 Able Ape
Able Ape's picture

Software engineers forgot to say, "Ba da boom, Ba da bing", before activating the new software....This is a crucial MAJOR step that they forgot...

Thu, 07/09/2015 - 14:14 | 6290995 The Delicate Genius
The Delicate Genius's picture

The NYSE must have a parallel system, a back up...

this tale just doesn't make sense, and too many things happened at the same time.

https://investmentwatchblog.com/ceo-nyse-shutdown-probably-a-cyberattack...

Who knows, but where we are is anything like this blamed on Iran, China or Russia is just as, perhaps more likely, to be an "inside job."

In this case - maybe too many sell orders coming from Asia, eh???

Thu, 07/09/2015 - 14:07 | 6291035 Happy Camper
Happy Camper's picture

Next stop, the Fed with a side trip to at least one mobile network...

Damn if feels good to be a gansta..

Thu, 07/09/2015 - 14:56 | 6291354 lakecity55
lakecity55's picture

*Ring*

Hello, Janet here.
Bath House here, Ms Yellen. That 'software glitch' is great!
Thank you, sir/madam. We can use it for quite a few scenarios.
Well, keep up the good work. I need time to smuggle moar ISIL cells into the country.
Sir, I can't delay the Reset forever...
Just a few moar months, Janet! Soon we hope to activate the ISIL cells and finish training our Black Panther cadres.
Tel Aviv will be pleased, mr pResident!

Thu, 07/09/2015 - 15:08 | 6291443 asfffasfff
asfffasfff's picture

LIE

Thu, 07/09/2015 - 15:40 | 6291626 cinderalle
cinderalle's picture

Ok we buy all these BS, so most of the western nation were upgrading their softwares, in Paris the Metro was down, in London the tube was down, and even in New York was the underground down, are kidding me!

Thu, 07/09/2015 - 16:25 | 6291850 diana_in_spain
diana_in_spain's picture

This is my theory of what happened.

China was right, the USA was making the Chinese market go down by massive selling (US was trying to disrupt the BRICS meeting yesterday)  .  So the chinese hackers decided to get back at the US by hacking United airways, WSJ and New York stock exchange. 

The US promised to stop making china go down in return for the chinese stopping the hacking of the US sites.

 

 

Thu, 07/09/2015 - 17:26 | 6292161 gcjohns1971
gcjohns1971's picture

A Software Rollout?

During business hours?

Bullshit.  Not done.

If it were done, these people would be far too stupid to trust with your money.

Thu, 07/09/2015 - 18:28 | 6292386 Sokhmate
Sokhmate's picture

It could not have been CIM, nor creampie, release. Those releases are rarely problematic.

It must be a type of release I'm not aware of.

Thu, 07/09/2015 - 22:14 | 6293225 VW Nerd
VW Nerd's picture

Ha, ha, ha!  Do I have "stupid" witten on my forehead???

Do NOT follow this link or you will be banned from the site!