This page has been archived and commenting is disabled.
HFTs Are Overheating: CenturyLink Reports "Catastrophic" HVAC Failure At NJ2 Data Center, Starting "Safe Shut Down"
If today's ridiculous move is the kind that no retail investors would chase, it is precisely what HFTs around the globe love: nothing but momentum, momentum, momentum. In fact, HFTs are trading so much, they are literally overheating!
According to a notice mailed out moments ago by CenturyLink, its NJ2 data center has just experienced a "critical" HVAC data failure.
Incident Notification
All times listed are in Central time zone.
Time and Date of Event: 10:08, 08/27/2015
Location: ZZNJ2
Event Description: The south side of the NJ2 data center is having a critical HVAC event. Clients are being requested to begin a safe shut down of devices immediately.
Current Status: Steps are being taken by the data center to safely shut down client devices by request due to temperature concerns on the South side of the building.
Next Status Update: 30 mins
* * *
Here are some specifics on the NJ2 data center:
Real Estate Summary
Located near Newark International Airport; 15 minutes from Manhattan, NY
Four story building
Total building interior (sf) = 223,022
Raised floor (inches) = 12
Electrical Summary
PSE&G provides power feeds
Power density minimum (W/sf) = 150
Generator configuration = N + 1
Total Power Capacity = 16 MW
Minimum two fuel replenishing companies
Mechanical Summary
Cooling system configuration = N + 1
CenturyLink manages temperature and humidity to strict ASHRAE standard
Fire Detection and Suppression Summary
VESDA provides early warning detection
FM200
How do we know HFTs are involved? Moments ago, BATS issued an advisory:
BATS Weehawken Network Point-of-Presence (NJ2 PoP) Advisory
August 27, 2015 12:26:21
Please be advised that the CenturyLink Weehawken, NJ data center (NJ2) maintaining a BATS PoP has reported a critical HVAC issue. Per CenturyLink, the cage will not have cooling over the next couple of hours at a minimum, so BATS would like to advise Members utilizing the BATS NJ2 PoP of potential impact that could result from overheating of equipment. BATS has shutdown all non-critical infrastructure at the NJ2 PoP at this time and will continue to monitor the situation. There is currently no customer impact but BATS will advise with material updates to this situation in NJ2 and if customer impact is expected.
IMPORTANT: There is no customer impact at this time in NY5 or NY4 (Secaucus) and all BATS exchange platforms are operating normally.
And while BATS may be safely offline and trading out of a redundant location, one wonders just how many other "wealth effect" mission critical HFTs clients of CenturyLink are about to go offline, and whether the entire market is about to go down with them?
This is a developing story: more as we see it.
»
- 40294 reads
- Printer-friendly version
- Send to friend
- advertisements -
Just a glitch, bitchez!
ANNNND...Market volume drops to zero. DOW re-priced to $1.
I entered a limit order @ $0.5
Ooops, looks like someone has actually implemented the infamous HCF (halt and catch fire) instruction in some hardware.
Way back when employees were encouraged to think, a simple placement of standing floor fans would have prevented this. The new techs aren't allowed to think for themselves...
Can someone technical explain to me that if you have potentially nine-figure-dollar-sums coursing through your rackspace you would not figure out a way to rig/invent/engineer some sort of liquid cooling system that can leach off the natural, largely-gravity-driven, always-on pressure of the municipial water systems of the NYC metro area? Or, why not a freon based cooling system, a refrigerated rackspace? It seems if you want big bucks you would insure that physical, technical side against this kind of goofy stuff. Overheating? Really? Is this a Lenovo laptop left on the patio?
It's a matter of too many people getting in the cart instead of pulling it. Remember that picture of the Greece office of Finance or whatever it was where they just had piles of paperwork and all sorts of junk strewn about like the Tazmanian devil paid them a visit?
It's easy to be competent when you have enough cooks in the kitchen. What we are seeing now on many levels is what happens when you try to get an average Joe to perform the work of two average Joes. If he could do that, he wouldn't be average.
+1
The heat has to be extracted and dumped outside. Setting an oscillating fan in a corner just wont do it.
Wow -- HowdyDoody must be as old as me. The "halt and catch fire" comes from an **ancient** list of "undocumented but universal" assembly instructions that was going around in the '70s.
It included his HCF - halt and catch fire
along with other gems such as....
SDJ - send data to japan
EPT - eject printer paper twice
WBT - water binary tree
BST - backspace and stretch tape
CPAR - crumple paper and rip
ESD - emit smoke and die
IBP - insert bug and proceed
EOP - execute operator
and many others i've long forgotten. good memory, HowdyDoody!
" oscillating fan in a corner just wont do it"
Correct..., but it buys the time needed to call in the HVAC team to examine the problem before it all breaks. It could be as simple as airflow being blocked because of that new rack that was installed last month, when the outside temps hadn't yet reached 100 degrees.
Wait, is that a picture of Lois Lerner's email server?!
Maybe someone didn't understand the leave no trace after reading coding requirement.
Hillary wiped it with her nitrocellulose cloth.
Rule one is don't cook your servers. If they set it up correctly, when the temp went up the system automatically started shedding load and shutting down. If they didn't do it right, somebody is getting a panic order or a buttload of new servers.....
Most HFT firms have redundant infrastructure in place as a contingency against precisely this sort of event.
Are you talking about..., (wait for it...,)
FANS?
I've seen redundant systems that weren't.....
... tested
Calling Mr. Robot.
Exactly what I thought of when I saw this article... How many technicians are checking all the thermostats now?
that's what you get for putting your high dollar critical shit in a Tier 3 datacenter...
Burn, baby. Burn.
Can somebody please convert MW into Trades/Second?
0.0000000000mW= 0 trades
Burn baby burn.
Depends on what they are using for processors which are likely fast Xeon quad core that are very fast, use a lot of power, then it all comes down to software which for HFT's work on microsecond slices, so the answer is a large shitload.....
1 MWH =2.246942291e+28 electron volts
Asuuming a million, billion electrons at 110 volts per trade that works out to 2e+11 trades per hour or 27.8 million trades a second
Too bad they did not burn down.
That would have leveled the playing field
MeltUP MeltDOWN !
I trade on Amada Drury tweets. She said the correction is over.
simple as that
In three years, Cyberdyne will become the largest supplier of military computer systems. All stealth bombers are upgraded with Cyberdyne computers, becoming fully unmanned. Afterwards, they fly with a perfect operational record. The Skynet Funding Bill is passed. The system goes online August 4th, 1997. Human decisions are removed from strategic defense. Skynet begins to learn at a geometric rate. It becomes self-aware at 2:14 a.m. Eastern time, August 29th. In a panic, they try to pull the plug.
That could actually fit with 2017.
cyberdyne = google "deep learning" and big data---only a matter of time---
Ahh...those Cyberdyne computers are in Weehauken. Poor computers. Even robots and computer think being in NJ is shit. I would get burned up too if I was stuck in Weehauken.
This would help explain why they are learing at a geometric rate. The ones in the Hamptons are spoild brats having binge parties, dumbing themselves down.
LOL! The ones in Weekhauken, NJ are learning at a geometric rate trying to figure out how they can get the F*** out of NJ.
I think the place burned down because of shitty electrical union woring and Chris Christie showed up with two pizza and his body heat made everything melt down.
If that's an N+1 data center, how did the fucking backup CRACs not take over? And 223,000 sq ft and only 12" of raised floor? WTF. Why bother at all. What a shitty data center.
Update (per the picture):
That's not an overheating, that is what we call a fucking fire. That rack doesn't have anything in it that should have gotten very hot even if locked in a sealed closet. Judging by the door in the right side of the picture, this doesn't look like a colocation facility at all. It looks like a closet in a circa 1950s school. I think this picture was just a stock image. Come on tylers...
Hillary's home-brew.
The first pic definitley looks like it came from a book about how NOT to wire your network racks. Although it is all low voltage and won't catch fire anyways.
Flash Melt Up? Any play on CNBC or are they all just masturbating to all the green?
the new "capitol crime" -- killing a cooling tower pump--- death penalty -----
clear shot from the "freeway" eight lanes of turn off--through the landfill ---Port Elisabeth NJ Prime realestate--- The badlands ---
Robots must be up again. Crude wanting to go up.
This reminds of one of those Rocky movies where he is fighting Mr. T. Just holding tough taking the full fury of his punches until he burns the guy out
NJ union electrician wiring work. i.e. total shit.
Proprietary Systems: The Endangered Species Act
Funny, when I was there, UC San Diego was the top premedical school in the United States, and the professors were afraid of me back then too, because I caught them giving the frat boys answers to the tests, and I worked with the rabbits from China, who worked 7/24 so their families back home would not be negatively ‘affected’ by authorities, several of whom jumped to their deaths from the library rooftop, the same one the frats threw pumpkins off of as an experiment.
Healthcare operates under the illusion that it has an information advantage. What it has is an arbitrary process, with layer upon layer of technology, which doesn’t work, gobbling up the real economy like Pac-Man, and replacing it with consumption, through arbitrary, self-biased, selective breeding. Naturally, like Pop-Goes-the-Weasel, each round takes out more participants, replacing them with temporary, traveling make-workers, incapable of bearing their own children, expert lactation specialists that have never breastfed their own babies.
Among the many things I learned firing people for US Navy is that if you cut off the head at the right time and the right way, several things happen. The arbitrary peer pressure group controlling the organization collapses, those capable of actually working begin to do so, and out of the ashes something far better is built. Modern healthcare is simply a temporary token, holding a place, waiting to be dissolved, so industrious young people have a recycle pile.
Unlike Bush and Clinton, Trump is at least amusing, when you have nothing better to do than watch the latest version of General Hospital. Contrary to popular assumption, that an irrational market of the majority can outlast all intelligent individual investors, History repeating itself with a new dress, economics is not a top down operation.
All a majority can do is consume, with increasing efficiency, to feed capital, the past. Without intelligent labor, there would be no capital, and at the end of the iteration, where you are, all capital is a sunk cost, waiting to be recycled. The problemsolution of globalization is that it is a closed system, which can only produce cancer, like a virus, because life will not tolerate a closed system.
The State in no way has an information advantage, despite the mountain of debt it has created for the purpose. What it has is arbitrary laws, a waste of time, as the means of maintaining the status quo, until it collapses, which it always does. Denying a baby nutrition, oxygen and hydration, and more importantly its parents, resulting in autism, is not healthcare; it’s an insurance business. Hospitals are not run like concentration camps by accident.
Did you know that Germany installed Lenin, and Stalin was the derivative?
It's a circle of stupidity, a closed system with a positive feedback loop.
Idiocracy on steroids....
Your punctuation, grammar and spelling are all good (so you are not likely a complete loon) but exactly what are you trying to tell us? All I'm getting is the message is that you are a brilliant, intellegent, important insider with a wide variety of skills who has observed and influenced the course of modern history.
What am I missing?
I would up arrow this ten times if I could.
nicely stated
+1
One always wonders (there's a little 'tard in all of us) if this is just cover or whether the A/C really is the problem.
The Fed can spend all it wants to 'fix' things but in the end every dollar they cut loose to do the job has the potential to be the one that starts the avalanche. Dollars can be like snow flakes that way.
Who would be surprised at all if this isn't just a conjured up convenient 'mechanical false flag' so that in weeks ahead they have some sort of excuse... any kind of excuse... for the next 'Lehman event' that is just around the corner.
That speculation aside, in my opinion ZH loses a ton of credibility every time they take a story like this and morph it into something it's not... and then paste a spectacular image at the top of the article that has nothing to do with the story. This is not a case of HFT machines melting down.
The algo robots go insane.
Buy Stawks!
The timing of this couldn't be more "moronic"...
I hope they store a buch of chemicals illegaly in the warehouse too...it would be nice to see a China firework and get rid of these computers
"Rod of God" ...
(that's my porn star name)
Where's the Kaboom?
There was supposed to be an Earth-shattering Kaboom!
--Marvin the Martian
http://tinypic.com/r/2efsw06/8
Hey, maybe this explains the fire at the Fed building yesterday. They must have HFT there as well..
Need more cooling fans
"Expect Unforeseen Consequences."
"Expect The Unexpected."
Ya HFT's use DSL internet. Totally.
CRAP! HFT MELTDOWN!
DOUBLE CRAP!
https://www.youtube.com/watch?v=vYhtqys3t_4
HVAC systems are mechanical systems that fail sometimes for a large variety of reasons. Well designed and maintened data centers have redundant cooling so that one chiller can be taken off line for repairs while one or more other chillers take the load. The tendency is to become complacent when one chiller is down and take too long to repair it. Then another chiller goes down and it's an emergency. In the winter, sometimes you can open the doors and let cold outside temps bail you out. In the summer, not so much...
What data center chief engineer would do chiller stop inspections in August?
N+1 means need + 1 total backup.
In August, if I had a chiller bust a tube or whatever, Id be on the phone securing a truck mounted chiller plant, if only to have the documentation that I did all I could to keep cooling online....to save my job.
"Well designed and maintened data centers have redundant cooling so that one chiller can be taken off line for repairs while one or more other chillers take the load."
Well,,, There's your problem!
Well designed and especially maintained,,, went out with the 20th century. And that goes double for telecommunications industries, except where glorious government is concerned.
The cheaper,,, the better. More money for them thar bonuses.... -I have some experience in this area-
Well at least they are coming up with alternatives excuses as opposed to blowing up China's industrial infrastructure!
CenturyLink? You've got to be fucking kidding me.
There's your problem.
I realize that I, the retail homeowner consumer, am the last of CenturyLink's concerns. But they started a new e-mail filtering process about May 2014. Immediately I noticed very long delays in receiving messages from certain corporate domains. I called, texted, online-chatted, and e-mailed CenturyLink's customer service and support. I had my corporate clients put both my CenturyLink and Gmail addresses in the "To" line of messages. I got the Gmail messages instantly, and the CenturyLink messages were usually delayed between 2 minutes and 20 hours; but not always. Whether or not the messages contained attachments, or were to a larger group, didn't seem to matter. It really seemed random.
I'd seen something like this earlier with another ISP, and then they freely admitted it was a glitch in graylisting, and if I didn't mind being exposed to more SPAM, they'd exempt me from it. So I suggested this to CenturyLink, and they claimed they weren't doing anything new and there wasn't a problem at all.
One of CenturyLink's competitors is installing fiber to the home in my neighborhood, unlike CenturyLink, and as soon as service is available, CenturyLink will no longer sell me anything. And then I'll have gotten rid of both them and Comcast; things I feel good about.
If BATS is relying on CenturyLink, no wonder they go tits-up all the time.
Doesn't Century link go back to either Arkansas or Louisiana? Maybe connected to Stephens in AR aka friend of Bill Clinton and lots of dead business associates?
o/`The roof, the roof, the roof is on fire
We don't need no water ... o/`
Looks like it's Blackened Swan again. Sweet, savory and spicy will cost you extra though.
False Flag for the up coming crash.....lol
the point
The other day I was talking so fast that my phone exploded.......................................no not really.
HFT (High Frequency Trading) must be banned from the investment and trading exchanges. It is a creature of greed and has indirectly corrupted equity price discovery. You can read about its history here (see: http://www.ftpress.com/articles/article.aspx?p=1924987&seqNum=2 ).
The markets are now focused on short-term trading which redirects capital away from investing in new and successful businesses.
This hurts growth opportunities for the country.
Just WTF do you mean my bitcoins melted?....
You will need bigger hardware in order to hunt for the ellusive liquidity particle. -CERN
gimme 2 cabals roasted