NASDARK Admits "Software Flaws"; Blames NYSE

Unsurprisingly NASDARK, while admitting to "latent software flaws" in the SIP, choose to place the blame for causing the 'glitch' at the foot of the NYSE Arca...

• *NASDAQ HALT REPORT SAYS DATA FLOOD EXPOSED SOFTWARE FLAW
• *NASDAQ SAYS CONFLUENCE OF EVENTS 'VASTLY EXCEEDED' SIP CAPACITY
• *NASDAQ SAYS SOME ISSUES 'CLEARLY WITHIN' COMPANY'S CONTROL
• *NASDAQ:'UNPRECEDENTED VOLUME' OF MSG TRAFFIC TO SIP CAUSED HALT
• *NASDAQ: SIP RECEIVED OVER 20 SEQUENCES FROM NYSE ARCA
• *NASDAQ: SIP GOT INACCURATE SYMBOL QUOTE STREAM FROM NYSE ARCA

Provides Details on Preliminary Findings of Internal Review

 

Company Intends to Recommend Steps to Strengthen the Industry Quote System; Identifies Key Areas 0f Focus for SIP Improvements

 

NASDAQ OMX today provided the following update on the preliminary findings of its internal review of the events leading to the August 22 temporary trading halt in trading of NASDAQ-listed securities.

 

A preliminary internal review has identified a combined series of technology events that caused the initial market problems and extended the halt period. A number of these issues were clearly within the control of NASDAQ OMX. As the Securities Information Processor (SIP) for NASDAQ stocks, we are responsible for them, regret them, and intend to take all steps necessary to address them to enhance stability and functionality of the markets. Other issues contributing to the halt are more endemic to technology issues across today’s complex markets and will require a broader industry-wide effort to resolve.

 

NASDAQ OMX is currently identifying potential design changes to further strengthen the SIP’s resiliency, including architectural improvements, information security, disaster recovery plans and capacity parameters. NASDAQ OMX will present our initial recommendations related to these changes to the SIP governing committee within 30 days.

 

NASDAQ OMX is also undertaking a comprehensive review of the policies and procedures for communicating with customers, market participants and the broader public during market-wide events. We are committed to reviewing and improving our communications practices with all stakeholders.

 

Summary of Preliminary Findings

 

NASDAQ OMX determined to halt trading on August 22 after the SIP could not process quotes thereby impacting the fair and orderly functioning of the public market. As outlined in more detail below, the catalyst for the SIP failure was a confluence of unprecedented events that overwhelmed the processing capacity of the SIP.

 

As with any technology system, the SIP has finite capacity. The SIP’s governing committee plans for capacity based on the needs of its member exchanges, adding an appropriate cushion of extra capacity to handle heavier than expected volume. In January 2013, a regularly scheduled systems capacity test showed the SIP system was capable of handling approximately 500,000 messages per second across 50 of the SIP system’s ports, for an average peak of approximately 10,000 messages per-port, per second. The tests are conducted in a controlled environment - performance in real-life scenarios will typically be lower.

 

On the morning of August 22, however, a sequence of events combined to create an unprecedented volume of message traffic into the SIP; well beyond the system’s tested capacity of 10,000 messages per-port, per second.

 

On August 22, the SIP received more than 20 connect and disconnect sequences from NYSE Arca, each of which consumed significant resources. Available capacity was further eroded as the SIP received a stream of quotes for inaccurate symbols from NYSE Arca, and generated quote rejects. Both of these actions served to degrade the system below the tested capacity of 10,000 messages per per-port, per second. During this period, NYSE Arca sent multiple bursts with each connect and disconnect, topping more than 26,000 quote updates per-port, per second as it attempted to reconnect. By comparison, a typical August day for NYSE Arca would peak at less than 1000 messages per-port, per second. The events of August 22 were 26 times greater than the average per-port, per-second activity. The confluence of these events vastly exceeded the SIP’s planned capacity, which caused its failure and then revealed a latent flaw in the SIP’s software code.

 

This latent flaw prevented the system’s built-in redundancy capabilities from failing over cleanly, and delayed the return of system messages. The combination of large system inputs and delayed outputs ultimately degraded the ability of the SIP system to process quotes to an extent that a shutdown of the system was in the broader public interest, to prevent information asymmetry and ensure fair conditions for all market participants.

 

Although the problem was quickly identified and data feeds were operational within 30 minutes of the halt, additional time was required for NASDAQ to consult UTP SIP committee members and market participants, and to test and evaluate scenarios to re-open the market to ensure that trading could be resumed in a fair and orderly manner. As soon as that process was completed, connectivity to the SIP was made available to UTP participants. The markets reopened without incident.

 

NASDAQ OMX is deeply disappointed in the events of August 22 and our performance is unacceptable to our members, issuers and the investing public. While getting to 100% performance in all of our activities, including our technology is difficult - it is our objective.

 

NASDAQ OMX is committed to enhancing our systems and is actively engaging with customers, partners and stakeholders to strengthen the infrastructure that supports our markets.

 

We will continue to provide updates on our findings and progress.