NSA's Utah Spy Supercenter Crippled By Power Surges

Long before Edward Snowden's whistleblowing revelations hit the world and the Obama administration's approval ratings like a ton of bricks, we ran a story in March 2012 which exposed the NSA's unprecedented domestic espionage project, codenamed Stellar Wind, and specifically the $1.4+ billion data center spy facility located in Bluffdale, Utah, which spans more than one million square feet, uses 65 megawatts of energy (enough to power a city of more than  20,000), and can store exabytes or even zettabytes of data (a zettabyte is 100 million times larger than all the printed material in the Library of Congress), consisting of every single electronic communication in the world, whether captured with a warrant or not. Yet despite all signs to the contrary, Uber-general Keith Alexander and his spy army are only human, and as the WSJ reports, the NSA's Bluffdale data center - whose interior may not be modeled for the bridge of the Starship Enterprise - has been hobbled by chronic electrical surges as a result of at least 10 electrical meltdowns in the past 13 months.

The facility above is where everyone's back up phone records and emails are stored.

Such meltdowns have prevented the NSA from using computers at its new Utah data-storage center which then supposedly means that not every single US conversation using electronic media or airwaves in the past year has been saved for posterity and the amusement of the NSA's superspooks.

This being the NSA, of course, not even a blown fuse is quite the same as it would be in the normal world: "One project official described the electrical troubles—so-called arc fault failures—as "a flash of lightning inside a 2-foot box." These failures create fiery explosions, melt metal and cause circuits to fail, the official said. The causes remain under investigation, and there is disagreement whether proposed fixes will work, according to officials and project documents. One Utah project official said the NSA planned this week to turn on some of its computers there."

More from the WSJ on this latest example of what even the most organized and efficient of government agencies ends up with when left to its non-private sector resources:

Without a reliable electrical system to run computers and keep them cool, the NSA's global surveillance data systems can't function. The NSA chose Bluffdale, Utah, to house the data center largely because of the abundance of cheap electricity. It continuously uses 65 megawatts, which could power a small city of at least 20,000, at a cost of more than $1 million a month, according to project officials and documents.

Utah is the largest of several new NSA data centers, including a nearly $900 million facility at its Fort Meade, Md., headquarters and a smaller one in San Antonio. The first of four data facilities at the Utah center was originally scheduled to open in October 2012, according to project documents. The data-center delays show that the NSA's ability to use its powerful capabilities is undercut by logistical headaches. Documents and interviews paint a picture of a project that cut corners to speed building.

Backup generators have failed numerous tests, according to project documents, and officials disagree about whether the cause is understood. There are also disagreements among government officials and contractors over the adequacy of the electrical control systems, a project official said, and the cooling systems also remain untested.

The Army Corps of Engineers is overseeing the data center's construction. Chief of Construction Operations, Norbert Suter said, "the cause of the electrical issues was identified by the team, and is currently being corrected by the contractor." He said the Corps would ensure the center is "completely reliable" before handing it over to the NSA.

But another government assessment concluded the contractor's proposed solutions fall short and the causes of eight of the failures haven't been conclusively determined. "We did not find any indication that the proposed equipment modification measures will be effective in preventing future incidents," said a report last week by special investigators from the Army Corps of Engineers known as a Tiger Team.

The architectural firm KlingStubbins designed the electrical system. The firm is a subcontractor to a joint venture of three companies: Balfour Beatty Construction, DPR Construction and Big-D Construction Corp. A KlingStubbins official referred questions to the Army Corps of Engineers.

Tsk tsk: this is what happens when you use taxpayer dollars to pay the lowest bidder - you can't even build an efficient totalitarian superstate!

The first arc fault failure at the Utah plant was on Aug. 9, 2012, according to project documents. Since then, the center has had nine more failures, most recently on Sept. 25. Each incident caused as much as $100,000 in damage, according to a project official. It took six months for investigators to determine the causes of two of the failures. In the months that followed, the contractors employed more than 30 independent experts that conducted 160 tests over 50,000 man-hours, according to project documents.

This summer, the Army Corps of Engineers dispatched its Tiger Team, officials said. In an initial report, the team said the cause of the failures remained unknown in all but two instances. The team said the government has incomplete information about the design of the electrical system that could pose new problems if settings need to change on circuit breakers. The report concluded that efforts to "fast track" the Utah project bypassed regular quality controls in design and construction.

Contractors have started installing devices that insulate the power system from a failure and would reduce damage to the electrical machinery. But the fix wouldn't prevent the failures, according to project documents and current and former officials.

Contractor representatives wrote last month to NSA officials to acknowledge the failures and describe their plan to ensure there is reliable electricity for computers. The representatives said they didn't know the true source of the failures but proposed remedies they believed would work. With those measures and others in place, they said, they had "high confidence that the electrical systems will perform as required by the contract."

A couple of weeks later, on Sept. 23, the contractors reported they had uncovered the "root cause" of the electrical failures, citing a "consensus" among 30 investigators, which didn't include government officials. Their proposed solution was the same device they had already begun installing.

Wait, we know: it's the Syrians.

So for those who have no choice but to live in a totalitarian banana republic, may we suggest at least laughing about it. We present the Domestic Surveillance Directorate.

Comments

[A Lunatic]

Love the header.......

Domestic Surveillance Directorate; Defending our Nation (by) Securing the Citizens.

[rlouis]

A. Disinformation

B Chinese/Islamist 'stuxnet'

C. US embassy - Iraq

[Zer0head]

Why Privacy Matters Even if You Have 'Nothing to Hide' (long essay)

excerpt

Privacy is rarely lost in one fell swoop. It is usually eroded over time, little bits dissolving almost imperceptibly until we finally begin to notice how much is gone. When the government starts monitoring the phone numbers people call, many may shrug their shoulders and say, "Ah, it's just numbers, that's all." Then the government might start monitoring some phone calls. "It's just a few phone calls, nothing more." The government might install more video cameras in public places. "So what? Some more cameras watching in a few more places. No big deal." The increase in cameras might lead to a more elaborate network of video surveillance. Satellite surveillance might be added to help track people's movements. The government might start analyzing people's bank rec­ords. "It's just my deposits and some of the bills I pay—no problem." The government may then start combing through credit-card records, then expand to Internet-service providers' records, health records, employment records, and more. Each step may seem incremental, but after a while, the government will be watching and knowing everything about us.

"My life's an open book," people might say. "I've got nothing to hide." But now the government has large dossiers of everyone's activities, interests, reading habits, finances, and health. What if the government leaks the information to the public? What if the government mistakenly determines that based on your pattern of activities, you're likely to engage in a criminal act? What if it denies you the right to fly? What if the government thinks your financial transactions look odd—even if you've done nothing wrong—and freezes your accounts? What if the government doesn't protect your information with adequate security, and an identity thief obtains it and uses it to defraud you? Even if you have nothing to hide, the government can cause you a lot of harm.

"But the government doesn't want to hurt me," some might argue. In many cases, that's true, but the government can also harm people inadvertently, due to errors or carelessness.

When the nothing-to-hide argument is unpacked, and its underlying assumptions examined and challenged, we can see how it shifts the debate to its terms, then draws power from its unfair advantage. The nothing-to-hide argument speaks to some problems but not to others. It represents a singular and narrow way of conceiving of privacy, and it wins by excluding consideration of the other problems often raised with government security measures. When engaged directly, the nothing-to-hide argument can ensnare, for it forces the debate to focus on its narrow understanding of privacy. But when confronted with the plurality of privacy problems implicated by government data collection and use beyond surveillance and disclosure, the nothing-to-hide argument, in the end, has nothing to say.

http://chronicle.com/article/Why-Privacy-Matters-Even-if/127461/

[shinobi-7]

The risk goes beyond mere privacy and concerns absolute control. People do not understand that when you unleash the power of statistics on meta-data, you get an unbelievably rich harvest. Soon, our I-phones will predict what we want and need better than we know ourselves based on historic paterns. With a larger computer and a bigger sample of people and data, you can do better. You can modelise and predict. You can control behaviors, groups or individuals, look for "deviation"... But worst of all, you learn from your mistakes until the system becomes fail-proof!

[IndyPat]

Aw, fuck it.
Just go out, dig a hole, cut out a six foot section of fiber, go home and drink a few cold ones.
That's what Walt and Jessie would do.

Or...there's always "Magnets, bitches!"

[world_debt_slave]

anyone check for rats?

[Pullmyfinger]

They did. It turns out they were in charge of the project.

[Hulk]

Never chew through high voltage wire with gold teeth !!!

[quasimodo]

60 fucking standby gensets and they can't figure out a way to feather or prevent surges. I have better systems at my workplace and never a single arcflash. Granted it's not this place, but hells bells.

[shovelhead]

Oooohhhh! The Tiger Team.

Kevlar pocket protectors with desert camo clipboards.

Relax everyone...Your info is safe.

Govt. project.

[shovelhead]

I'll bet Alexander's Star Trek office door makes that cool 'swipt' sound though.

Priorities.

[Downtoolong]

Tiger Team was the name of my football team............

When I was six years old.

[shinobi-7]

Glossing over the failures of the system is no help, the writing is on the wall. If we do not control technology, it will control us sooner than later. If you have enough information, anybody can be intimidated, harassed or arrested. Compiling all records forever is an unbelievably powerful weapon against democracy. It should be stopped immediately as the benefits are minuscules compared to the risks. We are ignoring the warnings at our peril.

[shovelhead]

I have registered my disapproval.

They wept.

[shinobi-7]

True, you can't fight the machine. Ethics would help but paradoxically we started having “ethic” classes in the City, just as the last shreds were jettisoned...

[SubjectivObject]

It's all disinfo.  All systems are up an on schedule. 

Show me a crater, and maybe I'll hope things are changed.

[Chaos_Theory]

Matrix needs those human batteries. 

Other movie analogy...can't wait 'till Skynet becomes self-aware!

[redwater]

A hive of scum and villainy.

[Vooter]

Does that sign in front of the building actually say "If you have nothing to hide, you have nothing to fear," or was that Photoshopped in?

[Pullmyfinger]

I'm thinking that every other government in the world is/will be backing any and every effort to shut this project down. This is not something intended to just form a totalitarian police state in this country alone, and everyone in power anywhere knows that now. The societal dynamics of this are fascinating --even if it is something akin to watching an asteroid heading in one's direction.

[optimator]

THQL breakers?  NO, they should have been Magna Blast breakers.

[tony wilson]

hope and belief will solve the problems.

if not 

change can be activated like a bandar bush mossad chechen terror cell.

on and off change

yes sir.

like the dreams of his kenyan cia mom and dad by chanting we can prevail.

yes we can 

yes we can.

because

obama

[Downtoolong]

It’s a new age problem for surveillance computers everywhere, caused by a septerraxial event. That’s when the following seven messages enter the system simultaneously and the computer can’t decide the sequential priority to label the senders as terrorists:

Fuck you NSA

Fuck you Google

Fuck you Facebook

Fuck you MSN

Fuck you Amazon

Fuck you Wall Street

Fuck you Bernanke.

[DadzMad]

As an electrical engineer specializing in mission critical power systems, I can assure you there are a few sweat-stained shirts, dry mouths, and puckered assholes at a few consulting companies.  A project of this scale has so many fingers on it they'll never really find the problem.  It will be a bunch of passing the buck on who fucked up the selective coordination.  High end selective coordination typical of that done in hospitals and data centers is used to compartmentalize problems, but it has it's own drawbacks, one of them being arc fault incidents with a higher energy incident thant usual. 

Serves 'em right.  Any engineer who worked on this project is nothing more than a traitor in my eyes.

[Pullmyfinger]

Let's me see if I understand you correctly: Here we have a project whose sole purpose is to support and implement a totalitarian police state on a global scale ---and you are calling anyone who may have fucked up its implementation to date A TRAITOR ?!? WTF?

I have nothing but contempt for born and bred slaves like you.

[DadzMad]

I think you are misunderstanding me.  I'm calling anyone who chose to work on it's implementation a traitor, not the fuck-up.

[Pullmyfinger]

My profound apologies. I had in fact already realized my error and was attempting to edit said post, but encountered a glitch in resending. The scale and scope of the project has apparently made me a bit skittish and trigger happy. But again, my sincere apologies. I promise it won't happen again... : )

[DadzMad]

Don't sweat it.  It's nice to have dialog with pissed off people in lieu of the zombies I'm surrounded by every day.  I'm just glad to see others give a fuck.  Beisdes, this is fight club.

[Pullmyfinger]

ha! I'd forgotten that point. It wasn't a bad rant, as rants go, though certainly directed to the wrong person. Maybe I should look through the immediate list and seek out another victim and do a copy/paste. hmmmm, I do see one or two possible candidates as we speak....

..or maybe I'll just mosey on ; )

[SubjectivObject]

Well, in that case, pull his finger!

[replace his last sentece with sarc/ and we're good]

[Ident 7777 economy]

DadzMad squawks: 

" As an electrical engineer ...  one of them being arc fault incidents with a higher energy incident thant usual. "

So, dumbass, was it DC or AC switchgear involved in the arc flashes?

You can't say, can you?

Idiot ...

[Reader1]

Yeah, I'm sure it's all an accident and not some contractor "accidentally" mis-wiring the fuses...

Watch, they're going to be having lots of QC issues and "accidents" for months to come.  Once the place is up and running, watch folks try to take out the local infrastructure, too.  I bet the water lines get tampered with and their transformers end get taken out.  With all the attention the place gets, I bet there will be constant attempts at sabotage, not to mention the foreigner interest in stopping them.

[New World Chaos]

Sniping at transformers to drain the cooling oil.

Arrows / model rockets trailing thin copper wire.

Took me two seconds to think of these.  I'm sure every redneck has thought of them also.  Imagine what they could come up with if they were sufficiently angry and had a lot of time on their hands.

TPTB can't guard everything at once.  Mayhem is coming.

[Ident 7777 economy]

It ain't 'QC' that is the isssue or 'transformers' even.

So far, you guys score a big ZERO on getting to the issue  ....

[epwpixieq-1]

Electricity. What is it?

No one knows, the only thing we know with certainty is that it behaves as an incompressible fluid, as Tesla mentioned 121 years ago in his honorable lecture in front of the annual IEEE meeting at Columbia University.

We detect certain (related) physical phenomena, and we try to conceptualize and put in a frame, for our easier understanding.

There is a fundamental engineering design with all these massive facilities, they all need power in massive quantities and they need a very good grounding system with a fundamental assumption that the GROUND IS THE NEGATIVE sink for the electricity (whatever it is). Now what happens if this is not true, apropo: there are ground electrical high voltage surge waves. All this infrastructure can/will be nicely fried, for there is no protection for high voltages COMING FROM THE GOURD, it is supposed to be the sinking place not the source of such waves.

And this is not theoretical, it has been made to work on massive scale 114 years ago, by the Genius who lit the world (and "invented the 20th century"), and highly likely will happen again. Especially, with the realization that such massive tool can be used as a weapon without harming the humans.

[Ident 7777 economy]

Your assessment looks to be well off-base. You ever 'breaker-ed' a DC system? Of over 24 V? Capable of impulse currents in excess of, say, 5,000 Amps?

[Caveman93]

Too big for their own britches I guess? Does a starship need that much power from the warp drive core?

[seabiscuit]

It occurs to me that the sign is absolute truth. "DEFENDING OUR NATION. SECURING THE CITIZENS"

Defending whose nation? And 'securing' the citizens for whom?  Secure, as in the useage of a verb - to get hold or possession of.

[SMC]

Aww... Perhaps some electrons have standards and will not drift like the others. ROFL!

[joego1]

Good! Fuck those traitors! 

[DosZap]

I NUB the caption of their SIGNAGE, almost like a Germany of olden days,eh?.

[Widowmaker]

FUCK THE NSA!

The only power surge is that of the people and unprecedented message that has been sent to spooks and faggots monetizing your Constitutional rights.

Look up the Congressional vote for Patriot Act renewal in 2011 with inclusion of section 2 that says "gut your privacy, eat national security shit."

VOTE EVERY ENCUMGUZZZLER OUT THAT VOTED IN FAVOR OF IT!

FUCK THE NSA!

[W74]

50,000 man hours /160 tests = 312.5 hours per test.

50,000 man hours / 30 independent contractors = 9 months worth of work per man.

Good thing somebody's on top of that shit.

[dexter_morgan]

Tyler, you confused the crap out of a lot of ZH'rs by putting that parody in there.....many apparently did not catch that.

LOL

[Ident 7777 economy]

What do you want to bet these 'morans' didn't account for long DC line runs and high in-rush currents of various power supplies and converters fed off those DC buses when 'switched' (breaker-ed) on?

It would  NOT be the first time 'physics' of transmission lines (see: the telegraphers equation) and stepped impulses affected the unwary and ill-informed (DON'T ask me how I know this.)

[DadzMad]

What you say above makes absolutely no sense, and we are all a bit dumber for having read it.  The distribution is AC all the way to the device level.  The only DC in the power system would be the AC-DC-AC conversion within the UPS inverter and the fucking batteries to start the generators.  With the exception of the supercomputer(s), they are likely using the same basic equipment you and I could get, just a LOT more of it, hence the need for 1M ft² of floor space.  Switching mode power supplies don't really present in-rush problems.  They can cause triplen harmonic problems, but these are easily accounted for in the design, properly sized grounded circuit conductors being the easiest way to handle them.

You'll notice I seldom post when the thread topic is about economic technicals.  There's a reason for that.  I might say something stupid in front of someone who actually knows what they are talking about.

[IndyPat]

Fucking stellar. Since I'm clearly in the presence of a pro, it only seems appropriate to ask. How many Gigawatts, volts, amps and whatever else went into the bean burning I just watched?

[Ident 7777 economy]

Your brown nose is not near far enough up that other poster's ass; PUSH HARDER. 

[Ident 7777 economy]

DadzMad: " What you say above makes absolutely no sense  ... "

Of course not. Because you're just a stupid f*cking moron who has NEVER worked with the 'physics' of power distribution at the hands-on energineering and DESIGN phase as an OEM.

WHY do you think they are having arc-flash events taking out gear?

I guarantee ya IT AIN'T THE 60 Hz MAINS THAT ARE ARC FLASHING!!!!!

[SmittyinLA]

LOL, all that "smart grid" technology is open to hackers, and the "solar grid" with wildly variable voltages doesn't help. 

Amerika is investing in a more expensive less reliable electrical grid.

[SmittyinLA]

be funny if it was the iranians stuxnexting us back 

[August]

Too bad the Utah data center hasn't been "hobbled" by a thermobaric surge.

One lives in hope.....

[ThisIsBob]

Jeezus, its just electricity.  Fairly well understood how it works. 

(I wonder if they've tried turning everything off, waiting 3 minutes,  and turning back on again.)

[Ident 7777 economy]

No, it isn't Bob, not when you're breakering DC ... 

Buncha- GD loons and STOOPIDS on the board today.

We need a submariner who can INFORM you idiots the hazards of DC switching ...

AGAIN, DO YOU MORONS THINK IT IS AC SWITCH-GEAR THAT IS ARC FLASHING!!!!???

[Ident 7777 economy]

You mean to tell me, NONE of you A Holes reading ZH have ever welded with DC ... LET ALONE understand the physics behind the need for a big-assed inductive 'ballast' inductor (coil) in line?

[Ident 7777 economy]

Those who know everything can skip this post; those who want to learn read on.

From: http://www.battcon.com/PapersFinal2013/1-Bill%20Cantor%20Steve%20McCluer%20-%20DC%20Arc%20Flash.pdf

Excerpt:

DC Arc Flash Hazard. The arc flash hazard around batteries is potentially the most severe, yet it is the least understood. A 2012 Battcon paper[8] summarized the potential arc flash hazard with respect to batteries and dc systems. The problem with dc arc flash, as was outlined in the 2012 paper, is that there is very little dc arc flash test data available to be able to create accurate models. The models that are available are very conservative and do not take into account the dynamic energy of an electro?chemical device. 

- - - - - - - - - - - - 

And:

http://www.battcon.com/PapersFinal2012/Bill%20Cantor%20-%20DC%20ARC%20Flash.pdf

Abstract    Arc flash is fairly well understood in ac systems and NFPA 70E  provides significant guidance for Personal Protective  Equipment (PPE) to prevent arc flash injuries from occurring in ac systems. In the 2012 version of NFPA 70E, new guidance was added that addresses arc and shock hazards for dc systems including storage batteries.

[Ident 7777 economy]

From:

http://www.hss-ltd.com/assets/files/DC%20for%20Newnes%20Whipp%20%20Bourne%20contribution%202.pdf

Principles of DC Circuit Interruption 

The most obvious difficulty in interrupting DC current is that there are no natural current zeros as occur in AC supplies, DC current has to be forced down to zero by the circuit breaker. To force the current to zero the breaker must generate an arc voltage across the break contacts which is greater than the system voltage. As the characteristics of the loads and also fault currents is highly inductive the circuit breaker interrupting device must be capable of dissipating all of the energy in the circuit until arc extinction. A momentary natural extinction of the arc due to current zero conditions is a luxury not afforded in these circuit breakers. 

Typically the circuits that are being switched have a few salient features, these include a very fast potential rate of rise, a high rated current, and inductive loads. Until the early 1970’s the main method of arc extinction was to stretch and cool it in an arcing chamber under the influence of the magnetic field produced by a series connected “magnetic blow-out coil”. Figures have been quoted in the past of a volt drop in DC arcs of about 1 Volt/millimetre. This figure obviously is dependent on arcing conditions, but it can be readily seen that for system voltages of above a few hundreds of volts the arc needs to be fairly long. This is not only relatively difficult to achieve, but also promotes the chance of the arc re-striking to other components within the switchgear. 

[StychoKiller]

In a typical PC, there are only four voltage levels present:  +12V, +5, +3.3V, and -12V,

with 3.3V powering the majority of the computing circuitry.  If thousands of Blades (whatever) are all tied into one central power distribution point, they deserve all the fried parts they generate!

[StychoKiller]

Hmm, codename "Bumblehive" -- well part of it is accurate right now! :>D

[Burticus]

This news seems too good to be true...so is probably the usual disinformation propaganda, like everything else coming from the not-shutdown gubbermint.