The Warehoused GPU Problem is Worse Than You Think.

^{Hyperscalers are apparently trying to out-do the warehouse scene at the end of Indiana Jones and the Raiders of the Lost Ark.}

Ed Zitron (@edzitron on TwitX, website is Where’s Your Ed At) has been telling the world that the big hyperscalers have hundreds of billions of dollars of GPUs (contained in, presumably, pre-assembled cabinets of GPU servers because that’s how they’re sold), either sitting around in warehouses or installed in data centers but still awaiting power:

This is not idle or baseless speculation.

Zitron bases this accusation on things like research into active data center power capacity (some of which has been confirmed by Bloomberg), and on the total amounts listed as CIP (Construction In Progress) on hyperscaler and neocloud balance sheets. There really isn’t anything else expensive enough to create dollar amounts in the hundreds of billions there besides GPU servers. The whole article is worth reading if you want to see the receipts. 

This has since been confirmed for Amazon by Business Insider.

Of the two storage options (warehoused vs sitting in data centers without power), I lean towards the latter for two reasons:

1. You can temporarily fire them up and perform quality control checks to ensure that your many-millions-of-dollars GPU server cabinets don’t have any manufacturing defects and weren’t damaged in shipping.

2. Performing this quality control check by sharing a production load (even briefly) technically means the server has been brought online, which allows hyperscalers, neoclouds, and chip vendors to make some extremely sketchy claims about “gigawatts brought online” without technically lying. See my article Gigawatts of Compute is Ridiculous and Meaningless for more detail on this and other nonsense being peddled.

So the image is probably more like below instead of a warehouse, but this image doesn’t work without context and who can resist leading with an Indiana Jones reference?

^{“Now you know!” “And knowing is half the battle!” (iykyk)}

No, They Can’t Fight the (Lack of) Power.

Either way, it’s still a crazy problem: hyperscalers and neoclouds have purchased far more compute than they can power, and the industries that supply power have been severely constrained for a long time. Things were tight before the COVID lockdowns, then the lockdowns threw buckets of sand into the gears, and that’s where the industry was when the AI construction boom hit. Some things, like the blades for turbine generators, can’t be easily scaled up. The process for making them is eerily similar to fabricating the precise crystalline geometries in the ultra-pure silicon ingots used for semiconductor manufacturing. And the turbine blades incorporate crazy-rare stuff like rhenium to keep them from melting. Only 50 tons of rhenium is produced per year, globally, it’s all spoken for, and without it your gas turbine generator becomes substantially less-efficient. It’s not like you can just mine more, because you can’t mine rhenium. It’s too rare. You can get a very tiny bit of rhenium (a few hundred parts per million) as a byproduct of refining molybdenum. And you can’t just mine molybdenum. It’s too rare. You get a very small amount of molybdenum as a byproduct when refining certain copper ores (not all mines have traces of molybdenum). So out of one metric ton of molybdenum-rich copper ore you can expect between _¹/_²⁰ and _¹/^₅ of a gram of rhenium. The point is this: you can’t just get more rhenium right away even by throwing ridiculous mountains of money at the problem. And there is no substitute for rehenium. So that’s why you can’t say “Hey, we suddenly need much faster production of top-notch, high-efficiency combined cycle gas turbine generators! We’ll pay top dollar!” The supply chain responds with “LOL. LMAO.”

Yes, there are other ways to generate electricity, but at gigawatt scale combined-cycle gas turbine generators are the cost-efficiency kings. Solar and wind are intermittant and require huge amounts of land and huger amounts of batteries at that scale. Nuclear is expensive and takes forever to build (at least for now). Fuel cells are an interesting (more expensive) alternative, but production capacity and supply chains can’t produce the scale necessary right now and that will take years (and risk) to ramp up. There are plenty of other bottlenecks: switchgear production, copper for wiring, etc. And all of this is competing with the normal uses that have been going on forever, like factories and municipal power production to support expanding populations. Remember, all of this stuff was highly constrained when the AI boom it. Now it’s a nightmare mess for everyone. And everyone wants more power now.

So Eventually They Get Power. Then What?

Eventually (if a giant ticking debt bomb doesn’t detonate) they get their power. Great. They can finally plug in their GPUs!

But which GPUs do you plug in? The hyperscalers and neoclouds won’t get all the new capacity of their dreams all at once. They have a pile of GPUs (inventory) in warehouses or datacenters, and are probably many, many years away from having enough power for all of them. Zitron got close in his article, but didn’t follow it to the conclusion:

Then the Bad News Hits.

With warehouses, there are different ways to manage the flow of inventory. The most common is FIFO (First-In, First-Out). You send out the oldest products first, which is good for accounting reasons (not going into it) and minimizes storage age-related degredation or damage. Generally-speaking, this is how things are done, but there are exceptions.

Exceptions like these GPU servers. The most sensible approch for them is LIFO (Last-In, First-Out). This is because your newest GPUs can do the most compute, and can do that compute with less electrical cost-per-unit. LIFO maximizes AI factory production capacity and minimizes OpEx. Going the FIFO route would be shooting yourself in the foot both in terms of product quality advantage and cost efficiency. A very bad strategy.

But this brings us to the Dark Side of the LIFO, and why it’s seldom used: if you can’t ship everything and clear out your warehouse, then your oldest inventory keeps getting older without ever getting shipped out, forever. With newer generations of GPUs arriving continuously, the value of deploying that older inventory drops every single year and never stops.

With GPU production and sales continuing to outpace power production, it’s looking quite likely that a lot of these warehoused GPUs will never be deployed, ever.

The oldest GPUs will be always economically-unviable compared to newer ones. They will be unable to run the newer frontier models that require more memory. They will become e-waste before they process a single token. They will be among the biggest and dumbest write-off losses in human history. The only conceivable way out is that demand for GPU compute shoots to infinity, outpacing GPU production capacity andthere is also sufficient excess supply of the highly-specialized AI data center space, power, and cooling capacity to accommodate them. If (when?) the AI bubble pops, there will be much better GPU servers available at a fire-sale prices.

But why? Why?

This is the $375 billion question (according to Zitron’s numbers). Companies like Microsoft, Google, Meta, and Oracle have a decade or more of experience building data centers. It’s tough to believe that they have been completely blindsided by the delays. Zitron’s speculation is that maybe Nvidia is threatening to prioritize customers who buy up present capacity when the latest chips become available, and the hyperscalers are afraid of being frozen out down the road. That being said, AMD’s offerings are competitive at a silicon level, and Nvidia’s CUDA software moat is not what it once was for the big players. Broadcom and Amazon have thrown their hats in the ring. On the other hand, Nvidia has made some very aggressive purchase commitments with TSMC, Samsung, SK Hynix, and Micron, reserving huge swaths of their production capacity with leaked details indicating that some is pre-paid. AMD and the rest may not be able to physically build enough GPUs if push comes to shove. And all of this assumes that demand for GPU compute goes to infinity and beyond.

The fact is, we just don’t know [I know]. People will say that “Oh, these experts are super smart and wouldn’t mess this up,” but that’s what people say during every bubble. And when the bubble pops we see the actual data and ask “How could these experts have possibly been so insanely stupid?” In hindsight, the prevailing wisdom turned out to be pure madness.

We’ll have to wait to find out. [I figured it out. It will be covered in my next posted article.]  Meanwhile, I’m wondering if an unused Nvidia NV72 compute cabinet would make an interesting conversation piece in my home.

^{Nice, but needs a lizard or two climbing up it. Maybe put it next to a window.}

This article was originally published on Oct 9 on my Substack.