Moments ago the NYSE released its explanation [4] why its was offline for nearly 3.5 hours. Here is the official narrative:
On Tuesday evening, the NYSE began the rollout of a software release in preparation for the July 11 industry test of the upcoming SIP timestamp requirement. As is standard NYSE practice, the initial release was deployed on one trading unit. As customers began connecting after 7am on Wednesday morning, there were communication issues between customer gateways and the trading unit with the new release. It was determined that the NYSE and NYSE MKT customer gateways were not loaded with the proper configuration compatible with the new release.
Prior to the market open, gateways were updated with the correct version of software and stocks opened at 9:30am. However, the update to the gateways caused additional communication issues between the gateways and trading units, which began to manifest themselves mid-morning. At 11:09am, NYSE issued a Market Status message that a technical issue was being investigated. At 11:32am, because NYSE and NYSE MKT were actively trading but customers were still reporting unusual system behavior, the decision was made to suspend trading on NYSE and NYSE MKT. NYSE ARCA, Arca Options and NYSE AMEX Options were not impacted by this event and continued to trade normally.
NYSE and NYSE MKT began the process of canceling all open orders, working with customers to reconcile orders and trades, restarting all customer gateways and failing over to back-up trading units located in our Mahwah, NJ datacenter so trading could be resumed in a normal state. In consultation with regulators and industry, we determined that we would implement a complete restart and that NYSE MKT primary listings would resume trading at 3:05pm and NYSE primary listings and NYSE MKT Tape C symbols would resume trading at 3:10pm. Trading resumed as scheduled and the closing auctions accepted orders and executed normally. All NYSE and NYSE-MKT listed securities traded for the entire day either on NYSE and NYSE MKT or other market centers.
Odd: no mention of what CEO Farley told Bloomberg [5]TV as to why there was no rollover to the NYSE backup system: "we chose the least disruptive option for customers. So if we had moved to our Disaster Recovery Center, which was an option, customers would have had to do a good deal of work to be able connect to that new Disaster Recovery Center."
So, as we noted yesterday, the NYSE has a disaster recovery center which they choose not to use because it is an inconvenience to clients who would rather be unable to trade for over 3 hours.
Gotcha.
As for software upgrades and, of course, "glitches", always being the cause of historic market events: funny how these always take place on major market down days.
Then again, as we also said yesterday, in the grand scheme of things the NYSE hasn't mattered in a while [6]:
[7]
In fact, the only "program" that does matter in this centrally-planned market is the CME's CBIP or Central Bank Incentive Program [8]: lose that and things may not be as generally "calm" as they were yesterday,
