This page has been archived and commenting is disabled.

Air Conditioner Failure At CenturyLink Data Center Takes ITG Dark Pool Offline

Tyler Durden's picture




 

Earlier today, we first reported that the massive CenturyLink NJ2 data center in Weehawken was in shut down mode as a result of a critical failure in its HVAC system. This promptly led BATS to issue a warning that "the cage will not have cooling over the next couple of hours at a minimum, so BATS would like to advise Members utilizing the BATS NJ2 PoP of potential impact that could result from overheating of equipment. BATS has shutdown all non-critical infrastructure at the NJ2 PoP at this time and will continue to monitor the situation."

We commented that "while BATS may be safely offline and trading out of a redundant location, one wonders just how many other "wealth effect" mission critical HFTs clients of CenturyLink are about to go offline, and whether the entire market is about to go down with them?"

Moments ago we found at least one, appropriately enough the company whose dark pool was recently busted in the biggest criminal scam involving a private trading company's dark pool: ITG, which announced minuets ago that it would shut down its dark pool POSIT for the rest of the day.

  • NEW JERSEY DATA CENTER WHERE ITG OPERATES HAS HVAC ISSUE
  • ITG SHUTDOWN INCLUDES POSIT DARK POOL
  • ITG SAYS IT WON'T BE TRADING FOR REMAINDER OF DAY

Whether or not this added to the market tumble that resulted from the JPM note, first and so far only posted here, is unclear but the fact that many are wondering if the market may be crashing as a result of a blown air conditioner should tell you all you need to know about the current state of the so-called market.

 

- advertisements -

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
Thu, 08/27/2015 - 15:40 | 6478643 ParkAveFlasher
ParkAveFlasher's picture

I believe they call this "fragility".

Thu, 08/27/2015 - 15:43 | 6478665 remain calm
remain calm's picture

I say Bullshit. And that's my boy Putin letting you know he is not dead

Thu, 08/27/2015 - 15:45 | 6478684 ZeroPower
ZeroPower's picture

I tweeted this out around noon today. ITG lost all street connectivity. Lost of substitute brokers had to take up the slack of buy side firms seeking alt routing.

Thu, 08/27/2015 - 15:50 | 6478693 remain calm
remain calm's picture

Its the Russian. Their hackers are just as good as ours

Thu, 08/27/2015 - 15:56 | 6478732 kliguy38
kliguy38's picture

I just can't wait to borrow some paper to buy Stawks.......who wouldn't wanna be in this "market"?

Thu, 08/27/2015 - 15:57 | 6478736 NoDebt
NoDebt's picture

Remember a few months ago when the NYSE went dark and ZH posted up pics of guys in work trucks at some data center in NJ where all that stuff ran?

Same place?

Thu, 08/27/2015 - 17:45 | 6479129 natty light
natty light's picture

That was Mahwah.  

Thu, 08/27/2015 - 15:57 | 6478734 Eirik Magnus Larssen
Eirik Magnus Larssen's picture

A 24-hour shutdown? Something a bit off about this story. As mentioned, HFT firms tend to be well-equipped to deal with events like these.

Thu, 08/27/2015 - 17:44 | 6479125 Itgoestoeleven
Itgoestoeleven's picture

The critical systems at data centers are  bomb proof. They also have backup sites. I have direct knowledge of the redundency associated with these installations. Anything short of building destruction, they are up 99.99% of the time. I call bullshit!

Thu, 08/27/2015 - 19:49 | 6479423 Ralph Spoilsport
Ralph Spoilsport's picture

Pre-Lehman, I had a contract working at a large data center run by Computer Science Corporation. They took the operation over from a large corporation that had started outsourcing everything. Before CSC, the data center only supported engineering and research along with some government work related to company business. The contract that CSC negotiated allowed them to start stuffing the place with servers and mainframes from anybody, which made sense since this data center had 4 floors and 2 basement levels and had been connected to ARPANET before anybody knew what that was. Needless to say they had a lot of connectivity and it was so important that the Feds came and locked the place down on 9/11 and escorted non-US citizens off the property.

CSC is a lousy low-bid company that pinches pennies. How cheap were they? They shut off the auto-flushing urinals on weekends and holidays that you had to use if you worked on those shifts.The data center went offline for 6 hours because they cut so many corners. They stuffed that place with so many servers, mainframes and tape drives that it was difficult to work on them or do backups it got so cramped. They added the bare minimum AC capacity so that if it failed for any reason, and it did, the temperature rose very quickly. Some systems did an auto-shutdown they got so hot on occasion.

Another area they neglected was spending enough money on the wiring for the incoming power feed or the battery backup wiring. One night we lost the power feed because of a substation explosion a few miles away. They were supposed to have redundant feeds but guess what hahaha. The battery backup came on but there were no lights except for emergency lights and all our terminals and PCs were dead. It started to get pretty warm for January so the AC must have been having problems too. This lasted for about an hour when the power was restored. Two hours later the power failed again because the incoming power feed cables caught on fire. Same deal, no lights or PCs. 5 minutes later, everything went dead and I mean mainframes, servers, phones, etc.

Why?The wiring coming from the battery backup caught on fire. To make CSC look even more like total assholes, none of the failovers to backup locations worked. A lot of data was lost and CSC paid hefty fines by the minute as systems all over the world went completely dead. The wiring smell got so bad the building had to be evacuated. It took weeks trying to recover from this epic example of incompetency and corporate greed. I think CSC still runs the place because they always under-bid everybody.

Thu, 08/27/2015 - 15:46 | 6478686 Zirpedge
Zirpedge's picture

RUT ROH HFT Stuxnet? 

Thu, 08/27/2015 - 15:48 | 6478696 remain calm
remain calm's picture

Bingo baby

Thu, 08/27/2015 - 16:04 | 6478779 J Jason Djfmam
J Jason Djfmam's picture

We could see what a stock market looks like when nobody shows up.

Thu, 08/27/2015 - 15:50 | 6478683 ThroxxOfVron
ThroxxOfVron's picture

One less dark pool for retail muppets to be drowned in...

Thu, 08/27/2015 - 15:58 | 6478742 Eireann go Brach
Eireann go Brach's picture

So it just takes the air conditioner to take them offline? I am shocked that someone has not taken out more than the air conditioning on this building!

Thu, 08/27/2015 - 15:40 | 6478644 Haus-Targaryen
Haus-Targaryen's picture

If only that documentary Blackout America would happen to the various HFT centers -- that would be great.  

(If you want to scare the shit out of yourself -- watch that on youtube here;

https://www.youtube.com/watch?v=FYoXxVnTePA

Thu, 08/27/2015 - 15:52 | 6478651 remain calm
remain calm's picture

Who is fucking in the electrical closte? is that you Krugman humping your cat?

Thu, 08/27/2015 - 15:41 | 6478654 I woke up
I woke up's picture

At&t (aka The NSA) and Verizon will be looking to grab this business

Thu, 08/27/2015 - 15:44 | 6478675 KnuckleDragger-X
KnuckleDragger-X's picture

Probably not. That would require customer service, which I guarantee does not exist at AT&T......

Thu, 08/27/2015 - 16:14 | 6478844 gmrpeabody
gmrpeabody's picture

...or Verizon.

Thu, 08/27/2015 - 15:42 | 6478657 Callz d Ballz
Callz d Ballz's picture

Good, now take one of those new flame throwers to it.

Thu, 08/27/2015 - 15:42 | 6478658 KnuckleDragger-X
KnuckleDragger-X's picture

The server people were honest with their customers. Too bad the same thing can't be said about their customers themselves....

Thu, 08/27/2015 - 15:42 | 6478659 agent default
agent default's picture

So this is why silver is up today.  Back on the treadmill tomorrow I guess.

Thu, 08/27/2015 - 15:43 | 6478666 Jason T
Jason T's picture

i can hear some old man screaming .. "turn those machines back on"

Thu, 08/27/2015 - 15:44 | 6478667 Bill of Rights
Bill of Rights's picture

Just as long as the " Crush Gold and Crush Silver and Crush miners " servers are all up and running and staying cool all is well.

Thu, 08/27/2015 - 15:44 | 6478671 Rainman
Rainman's picture

Cooling system = liquidity

Thu, 08/27/2015 - 15:44 | 6478672 KCMLO
KCMLO's picture

Market rally currently restored but that dip tho...

Navel gazing:  NJ2 better beef up their security if just taking them offline could cause a major market move.  That's a very ripe soft target for would be "terrorists"

Thu, 08/27/2015 - 17:52 | 6479140 Buster Cherry
Buster Cherry's picture

All it takes is a ream of copier paper inserted into a cooling tower basin....

Thu, 08/27/2015 - 19:56 | 6479459 Ralph Spoilsport
Ralph Spoilsport's picture

Hehehe I'll have to remember that one.

Thu, 08/27/2015 - 15:51 | 6478707 youngman
youngman's picture

Yes we had a little dip there and the CNBC talking heads almost had a heart attack.....but now the machines must be back on...as here we go again..up up and away.....except for gold and sliver..

Thu, 08/27/2015 - 15:51 | 6478708 khnum
khnum's picture

sort of reminds me of the time the inflation rate went up the primary culprit-bananas wiped out in a storm,the bullshit this week is biblical

Thu, 08/27/2015 - 15:59 | 6478743 GMadScientist
GMadScientist's picture

Ooohh....they let out the magic smoke!

Thu, 08/27/2015 - 15:59 | 6478750 ToSoft4Truth
ToSoft4Truth's picture

Everyone will get their money back, they must wait… just a little longer. 

Thu, 08/27/2015 - 16:02 | 6478768 WTFRLY
WTFRLY's picture

yassss

Thu, 08/27/2015 - 16:05 | 6478782 Duc888
Duc888's picture

Yea. RRAYYYT.

Thu, 08/27/2015 - 16:14 | 6478838 KCMLO
KCMLO's picture

Center is now completely offline.  Looks like they limped in to close and just shut down.

Thu, 08/27/2015 - 16:20 | 6478883 XitSam
XitSam's picture

A plot straight out of TV's Mr. Robot.

Thu, 08/27/2015 - 16:23 | 6478899 armageddon addahere
armageddon addahere's picture

After the last week's gyrations I'm not surprised the computers were smoking hot.

Thu, 08/27/2015 - 16:28 | 6478912 Chuck Knoblauch
Chuck Knoblauch's picture

The old air conditioner not working trick.

Thought they would have backups?

LOL

The dog ate my repairman.

Thu, 08/27/2015 - 16:54 | 6479002 kchrisc
kchrisc's picture

Hey wait!

All of those computers and air conditioners aren't "green." What about the planet, and "climate change," and what not?!

Somebody's been lying...

Zion is a scheme, not an ethnicity..

Thu, 08/27/2015 - 16:32 | 6478940 mrdenis
mrdenis's picture

Call in Hillary's tech support team .....

Thu, 08/27/2015 - 16:44 | 6478975 Oliver Jones
Oliver Jones's picture

I am sorry, but Mr. Takagi won't be trading with us for the rest of the day...

Thu, 08/27/2015 - 16:50 | 6478996 kchrisc
kchrisc's picture

See, the so-called Information Age is fragile.

Tyranny can criminally store all the data in Utah and Texas on me they like. Hanging on to it without water and electricity is going to be the real trick. As will be utilizing it without connections to it.

Zion is a scheme, not an ethnicity..

 

Tyranny is fragile.

Thu, 08/27/2015 - 17:04 | 6479029 Bunga Bunga
Bunga Bunga's picture

Some shit hit the fan.

Thu, 08/27/2015 - 17:47 | 6479133 Atomizer
Atomizer's picture

They should of called Friedrich Air Conditioner.

http://www.consumeraffairs.com/homeowners/friedrich-air-conditioner.html

:0

Thu, 08/27/2015 - 17:50 | 6479137 Buster Cherry
Buster Cherry's picture

Wait till the bitcoiners face this shit.

Fri, 08/28/2015 - 00:53 | 6480285 onmail
onmail's picture

The Game Theory says : Its better not to participate

stay aloof , stay safe

pullout

leave

sell

Buy gold & platinum 

Do NOT follow this link or you will be banned from the site!