The Pulse: a new trend of CPU shortages

Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. In every issue, I cover Big Tech and startups through the lens of senior engineers and engineering leaders. Today, we cover one out of four topics from a past issue of The Pulse. Full subscribers received the article below fourteen days ago. If you’ve been forwarded this email, you can subscribe here.

I was at dinner with a bunch of CTOs and Head of Infrastructure folks recently, and from the conversation it was clear that many companies are struggling to source CPUs in the current climate, and are coming to terms with the end of juicy discounts from cloud providers for machines in the new era of surging demand fueled by AI.

The ‘memory crisis’ afflicting sectors like video gaming is well established and has been extensively covered in terms of shortages of GPUs, but now it seems like things are just as hard for businesses in need of CPUs from cloud providers.

In a sign of how things are changing, the disappearance of CPU spot pricing was mentioned at the table. Customers used to be able to pay up to 90% less than the standard price for CPUs, as cloud providers slashed CPU prices for machines that were lying dormant and unused. But that’s no longer the case. It seems that spot pricing has vanished because there’s no longer any lack of demand for CPUs – quite the opposite.

I was surprised, but a lot of people chimed in; apparently, it’s now nearly impossible to get CPUs on spot instances without long-running connections with cloud providers. Also, reserving specific CPUs now needs to be done months in advance, and cloud providers will even turn down certain reservations because they don’t have enough CPUs or the right type of CPUs.

Even big players struggle to reserve CPUs

I have asked turbopuffer CEO Simon Eskildsen about their experience of CPU availability in the cloud, since turbopuffer, as a product, runs on CPUs, not GPUs. They operate in AWS, GCP, and Azure, so I asked how easy it is to get CPUs these days. Simon’s response:

“Getting CPUs is not easy anymore. As Reinforcement Learning (RL) is becoming a large amount of the workloads: RL needs a lot of CPUs. So the labs are sucking up a lot of CPUs. During RL, they need to teach the models how to do things, like searching, and then they need the model to run software, which then takes CPUs to run.

Then, outside of RL, agents need to do all kinds of very general purpose things on a CPU. So as the demand curve is shifting to general purpose agents, CPU demand is also going up.

Even the big companies are fighting each other for the right to get the CPU allocations. I would assume that it gets a lot worse before it gets better on the CPU side.”

I was able to confirm what Simon said about larger companies struggling; a VP of Engineering at a large inference provider told me they are at the limit on how much GPU and CPU capacity they can buy from their cloud providers. They have cash to spend and want to rent more capacity, and are willing to accept the longest leases. Despite that, cloud providers tell them no more is available!

AI hogging CPUs

Katelyn Lesse, Head of Platform Engineering for Claude Platform, has written about the reasons for the massive CPU demand increase:

“In the past few years, AI-fueled demand has skyrocketed, and these few companies suddenly needed multiple years and tens of billions of dollars to actually add enough capacity. We ended up with 3 separate bottlenecks in factory capacity that AI is exacerbating. At TSMC, GPUs are competing with CPUs (and with Apple, Qualcomm, and Broadcom) for production lines. And at SK Hynix, Samsung, and Micron, HBM [High Bandwidth Memory] is competing with regular DRAM for wafers.

What we’ve ended up with is CPUs getting squeezed from both sides. AMD doesn’t own fabs [semiconductor fabrication plants], so its CPUs need to come out of TSMC’s constrained allocation. Intel does own fabs, but it’s been working through yield problems and is now pulling some of its capacity from PC chips in order to make more server chips. And CPUs need DRAM which has gotten more expensive because memory production has shifted toward HBM. Analysts are expecting CPU supply to add more comfortable headroom before memory does, but their expectation is that it’s still going to be multiple quarters away.”

AI-fueled demand does increase CPU load, as shown in this graph from Uber, displaying the growth in agent requests over the past six months:

Increasingly, “agent requests” not only generate code which is inference-heavy – and therefore needs GPUs – but they also run tools that compile the code, run tests, run linters, and all of this is CPU-heavy. At companies like Uber, Ramp, and others, AI agents no longer run on the dev’s local machine, but on a dedicated instance in the cloud. So, the companies reserve more CPUs on their respective cloud providers for agentic workloads. We recently covered how Ramp built and runs its cloud agent, Inspect.

Basically, the problem is:

• AI applications use more and more CPUs, thanks to agents running a lot more software. AI data centers used to have a ratio of 1 CPU to 8 GPUs. Now the ratio is more 1:4, and it could shrink to 1:1.

• Companies that can manufacture more CPUs are busy on other hardware. TSMC is busy producing GPUs, which might be more profitable than CPUs. Meanwhile, CPUs also need DRAM, but DRAM manufacturers (SK Hynix, Samsung, and Micron) are instead producing high-bandwidth memory (HBM) because it’s more profitable. This is why memory prices are spiking; even Big Tech is unable to buy RAM, as previously covered.

To secure CPUs, it’s necessary to do capacity planning up to 12 months in advance. Katelyn says that server orders are being fulfilled in ~six months, instead of 1-2 weeks’ time as previously, and that prices are up by between 10-20%. So, it’s probably time for capacity planning. Katelyn:

“Most of us have never capacity-planned CPUs. We planned databases, we maybe planned accelerators if we needed them, and we autoscaled on-demand into CPU capacity as much as our budgets allowed us to. But general purpose compute is now something many teams will need to commit to ahead of time, which means you should probably start to forecast and plan around it. If you’re operating at scale, there are some things to spend your energy on.”

Using existing CPUs more efficiently is something to do, as of now. The CPU capacity shortage won’t go away, and any new CPU allocations requested could take months to turn up. So, what can we do if new capacity lags? One option is utilizing current resources more efficiently!

This is a great time to review and to establish now which services are CPU-intensive, and whether or not they need to be. Also check on services which are utilizing little CPU: can they run on fewer nodes, so that some CPU capacity can be allocated to services that need it more?

The best time to secure more CPU capacity is most certainly right now. I’m hearing rumors that certain cloud regions no longer accept new tenants because all CPU capacity is leased, or negotiations elsewhere are difficult. I’m also hearing that customers are already paying today to reserve capacity that will only come online in data centers from December. This seems predatory by providers, but demand is so high that this is how they likely prioritize new capacity allocation – while earning much higher profits than usual.

If your company has dynamic workloads, and you’ve used spot instances in the past, now could be a good time to allocate fixed capacity – even if it’s more expensive. If you expect meaningful growth, doing so now might mean having options at some cloud providers or in some regions.

It seems like this issue has spread everywhere as a corollary of widespread AI adoption. There’s a GPU shortage, memory shortage, and now a growing CPU shortage as well. Back at the end of last year, there was even an hard drive shortage. The only compute primitive not in short supply seems to be networking!

Read the full issue of The Pulse this is from, or check out this week’s The Pulse. This week’s issue covers:

1. Writing code by hand: is it over? In his Rails World keynote, David Heinemeier Hansson (DHH) declared the end for writing code by hand for professional work – at 37Signals at least. Is this change no