NSA's Utah Spy Supercenter Crippled By Power Surges

Tyler Durden's picture

Long before Edward Snowden's whistleblowing revelations hit the world and the Obama administration's approval ratings like a ton of bricks, we ran a story in March 2012 which exposed the NSA's unprecedented domestic espionage project, codenamed Stellar Wind, and specifically the $1.4+ billion data center spy facility located in Bluffdale, Utah, which spans more than one million square feet, uses 65 megawatts of energy (enough to power a city of more than  20,000), and can store exabytes or even zettabytes of data (a zettabyte is 100 million times larger than all the printed material in the Library of Congress), consisting of every single electronic communication in the world, whether captured with a warrant or not. Yet despite all signs to the contrary, Uber-general Keith Alexander and his spy army are only human, and as the WSJ reports, the NSA's Bluffdale data center - whose interior may not be modeled for the bridge of the Starship Enterprise - has been hobbled by chronic electrical surges as a result of at least 10 electrical meltdowns in the past 13 months.

The facility above is where everyone's back up phone records and emails are stored.

Such meltdowns have prevented the NSA from using computers at its new Utah data-storage center which then supposedly means that not every single US conversation using electronic media or airwaves in the past year has been saved for posterity and the amusement of the NSA's superspooks.

This being the NSA, of course, not even a blown fuse is quite the same as it would be in the normal world: "One project official described the electrical troubles—so-called arc fault failures—as "a flash of lightning inside a 2-foot box." These failures create fiery explosions, melt metal and cause circuits to fail, the official said. The causes remain under investigation, and there is disagreement whether proposed fixes will work, according to officials and project documents. One Utah project official said the NSA planned this week to turn on some of its computers there."

More from the WSJ on this latest example of what even the most organized and efficient of government agencies ends up with when left to its non-private sector resources:

Without a reliable electrical system to run computers and keep them cool, the NSA's global surveillance data systems can't function. The NSA chose Bluffdale, Utah, to house the data center largely because of the abundance of cheap electricity. It continuously uses 65 megawatts, which could power a small city of at least 20,000, at a cost of more than $1 million a month, according to project officials and documents.

 

Utah is the largest of several new NSA data centers, including a nearly $900 million facility at its Fort Meade, Md., headquarters and a smaller one in San Antonio. The first of four data facilities at the Utah center was originally scheduled to open in October 2012, according to project documents. The data-center delays show that the NSA's ability to use its powerful capabilities is undercut by logistical headaches. Documents and interviews paint a picture of a project that cut corners to speed building.

 

Backup generators have failed numerous tests, according to project documents, and officials disagree about whether the cause is understood. There are also disagreements among government officials and contractors over the adequacy of the electrical control systems, a project official said, and the cooling systems also remain untested.

 

The Army Corps of Engineers is overseeing the data center's construction. Chief of Construction Operations, Norbert Suter said, "the cause of the electrical issues was identified by the team, and is currently being corrected by the contractor." He said the Corps would ensure the center is "completely reliable" before handing it over to the NSA.

 

But another government assessment concluded the contractor's proposed solutions fall short and the causes of eight of the failures haven't been conclusively determined. "We did not find any indication that the proposed equipment modification measures will be effective in preventing future incidents," said a report last week by special investigators from the Army Corps of Engineers known as a Tiger Team.

 

The architectural firm KlingStubbins designed the electrical system. The firm is a subcontractor to a joint venture of three companies: Balfour Beatty Construction, DPR Construction and Big-D Construction Corp. A KlingStubbins official referred questions to the Army Corps of Engineers.

Tsk tsk: this is what happens when you use taxpayer dollars to pay the lowest bidder - you can't even build an efficient totalitarian superstate!

The first arc fault failure at the Utah plant was on Aug. 9, 2012, according to project documents. Since then, the center has had nine more failures, most recently on Sept. 25. Each incident caused as much as $100,000 in damage, according to a project official. It took six months for investigators to determine the causes of two of the failures. In the months that followed, the contractors employed more than 30 independent experts that conducted 160 tests over 50,000 man-hours, according to project documents.

 

This summer, the Army Corps of Engineers dispatched its Tiger Team, officials said. In an initial report, the team said the cause of the failures remained unknown in all but two instances. The team said the government has incomplete information about the design of the electrical system that could pose new problems if settings need to change on circuit breakers. The report concluded that efforts to "fast track" the Utah project bypassed regular quality controls in design and construction.

 

Contractors have started installing devices that insulate the power system from a failure and would reduce damage to the electrical machinery. But the fix wouldn't prevent the failures, according to project documents and current and former officials.

 

Contractor representatives wrote last month to NSA officials to acknowledge the failures and describe their plan to ensure there is reliable electricity for computers. The representatives said they didn't know the true source of the failures but proposed remedies they believed would work. With those measures and others in place, they said, they had "high confidence that the electrical systems will perform as required by the contract."

 

A couple of weeks later, on Sept. 23, the contractors reported they had uncovered the "root cause" of the electrical failures, citing a "consensus" among 30 investigators, which didn't include government officials. Their proposed solution was the same device they had already begun installing.

Wait, we know: it's the Syrians.

So for those who have no choice but to live in a totalitarian banana republic, may we suggest at least laughing about it. We present the Domestic Surveillance Directorate.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
Debtonation's picture

I thought the government was shutdown?

CH1's picture

Hope the place burns to the ground.

GetZeeGold's picture

 

 

The solar panel farm was a terrible idea.....should have stuck a coal plant next to that place.

Who knew Utah had so many cloulds?

Gringo Viejo's picture

"Ruprecht, do you want the genital cuff?"

NoDebt's picture

"The facility above is where everyone's back up phone records and emails are stored."

Super duper.  I can quit wasting money on Carbonite every month.  I love the government.  Always thinking ahead.

Joe Davola's picture

Gotta be the largest repository of pron also, which begs the question - can the administrators of this be arrested for possession of kiddie?

Overfed's picture

You know full well that those rules also don't apply to them.

Boris Alatovkrap's picture

Boris notice Bluffdale Edifice of Liberty is surround by barb wire and guard tower. How nostalgic.

Stackers's picture

Hopefully the"unknwon" faults were intentionally installed by an outraged design engineer or Army Corp construction crew ........

Zymurguy's picture

Stuxnet strikes us back?

old naughty's picture

Stellar Wind had nothin to do with it...

Bluff.

CPL's picture

Build ANY building on the current bidding system that ALL agencies use and guess what's going to happen?  It's going to be built with the same care as the US Health Care portal because money doesn't buy quality if the money offered isn't worth anything.  Stuxnet is a minor annoyance in comparison to 'sort of' met building and electrical standards because the builder squeezing their profit margin by ordering seconds.

When the seconds are pirate parts for things like joists, panels, fuses, millions of miles of copper, even concrete (there's been shortages, so not all that was poured is concrete).  The place is a fire trap.  Good looking fire trap.  So I'll say it again to the NSA, get the fuck out of that building, it's not safe.  Period.  Full Stop.  Leave.  Do not trust the building inspection.  Work from home.  Be anywhere BUT that building.

Can always build another one if it's really that important, it's just stuff.  The occupants inside, not so much.

 

Oh regional Indian's picture

The only way to build this right would be to A naval ship's military specs...

 

Oracle 911's picture

Or somebody use psychic powers anyway, this is what I referring them:

Nelson Ha! Ha! 10 hour edition.

Freddie's picture

A weird web site was saying OabamCare ACA web site was sabotaged.   Probably they spent $150 million and used bad programmers.

NSA Utah reminds me of the govt MIC location that gets it in T3 or T4.  When the drones get linked to SkyNet and kill all the govt workers like Sequestration should do.

IndyPat's picture

Obamacare FUXNET virus.
Lame Cherry rocks.
I actually have you to thank for pointing me in her direction. I dig her style. Hell, I donated. Worth every penny.

SilverSavant's picture

Stop worrying about this place right now.   If a Corp Tiger Team is involved, it will be fucked up for 10 years at the minimum.

CPL's picture

There's no place like 127.0.0.1

August's picture

I lke the arrangement of the buildings in an arc, visually suggesting that one day a vast circle will have been built.

Too bad Albert Speer wasn't available to enhance the esthetics a bit.

Zer0head's picture

Does it actually say

"If you have nothing to hide, you have nothing to fear" ?

or is that a Banzai embed

 

If true, I should think that every voyeur in America will be using that as a defence

 

why wouldn't they just have the motto "the end of privacy/ the end of freedom"

Winston Smith 2009's picture

"Does it actually say "If you have nothing to hide, you have nothing to fear" ? or is that a Banzai embed"

That hi-res photo and web page snapshot is from a joke site (.info domain) with a good bit of info on that facility:

http://nsa.gov1.info/utah-data-center/index.html

CoolBeans's picture

Your comment reminded me of the YouTube video of the middle eastern gentleman (believe he was in NYC) who took a video of himself calling the NSA for help in retreiving an email -- pretty funny...

Ying-Yang's picture

You live by Stuxnet you die by Stuxnet.

TPTB have not learned about opening Pandora's box

Pandora's box is an artifact in Greek mythology, taken from the myth of Pandora's creation in Hesiod's Works and Days. The "box" was actually a large jar given to Pandora which contained all the evils of the world.

Manthong's picture

"Securing the Citizens"

Think about the real meaning of that.

BadDog's picture

Kind of like the old twilight zone How To Serve Man.  Run! It's a cookbook.

insanelysane's picture

The solar panels cause power surges on those partly cloudy days.  Cloud, sun, cloud, sun, cloud, sun, 

Debt-Penitent's picture

The overall design specifications of the power supply grid are highly compartmentised and secret to "protect the integrety" of the secure power supply.  THIS is why no one subcontractor can find the arc-fault issues.

I too hope it burns to the ground.

 

Imagine the havoc a Citizen Drone could induce.  Hmm.

Freddie's picture

I hope the place has an earthquake and it gets sucked into the earth.   Evil govt is in a location with the best clean coal in the world but Clinton made in a national park for a kick back from Indonesia for less competition.

There is enough coal there to power America for the next 300 years.

salvadordaly's picture

yet another gubment project that they can not get right. Bet they will do a better job with healthcare though!

Dr. Richard Head's picture

While the idea of a totalitarian state is concerning, this example demonstrates the utter lack of competence of a so-called uber and all-knowing government beaucracy and is a bit comforting.

viahj's picture

they don't have to be competent to be totalitarian, just forceful

Kirk2NCC1701's picture

The word 'Schadenfreude' comes to mind.  :-)

You don't get the Captain's Chair that easily.  ;-)

DoChenRollingBearing's picture

Aye, aye, Cap'n!  Getting those Captain's bars ain't no piece o' cake!  Schadenfreude is the perfect word that makes me smile when I see the NSA all FUBAR...

 

:-)

Ignatius's picture

"Hope the place burns to the ground." -- CH1

Maybe one day soon we can all celebrate as tanks roll up and take it out (in the name of 'we the people').

ATM's picture

A better irony would be to take it out via drone missiles.

WillyGroper's picture

Drones would be nice. I'd kinda like some of that gigunda CAT mining equipment.

Like monster trucks on steroids crushing it.

New World Chaos's picture

The puppetmasters surely put a nuke in the basement in case of rebellion.  This data center would be quite useful in hunting them down.  Satan forbid the little people ever getting their hands on it!

In the meantime, we will have to rely on occaisonal spiders for sabotage.

petolo's picture

Need some Joozish lightning ,eh?

firstdivision's picture

The real center is located at Groom Lake with the backup center at Dugway Proving Grounds

SgtShaftoe's picture

Nevada doesn't have to give them grid power or water to mini-Mordor and that place needs lots of both.  The people can force the local government to cut it off. 

 

On a different note, I'm suprised that Banzai hasn't done a rework of the NSA logo with the all seeing eye of Sauron yet.

sushi's picture

How do they know that a foreign state has not reverse engineeried the NSA Stuxnet virus and has now deloyed it against its originators.

Electrical stuff blowing up in unexplained ways. Sounds pretty Stuxnettey to me.

I guess now NSA will need billions more in appropriations to defend against a technology they unleashed on thw world.

john39's picture

>>shutdown

just the show government... the real machine behind the scenes is grinding away.

IndyPat's picture

Yet too busy to "approve" my tax stamps on my fucking GemTechs!

Cursive's picture

 

....partners who had the vision and fortitude....

 

Fortitude?  Say what?  Sweet Jesus, yes!  Fry the fucker!

 

ETA:  CPL or any other IT hardware guys here, is static electricity in the desert a big problem for Steller Wind?

replaceme's picture

If there is a static electricity problem in general, I would assume the server areas are completely climate controlled, including humidity... I know our server areas, much less of a big deal, have big units that will call facilities (the people) if the humidity and temp get outside of a preset range.  Wonder what they pay -anyone know the going rate for a soul and a background in data warehousing?