Power Capping: The single most common tool data centers use to control the energy cost of AI does almost nothing for the workload that matters most. The mechanism runs exactly as designed, but during the phase that dominates production inference it never engages, so a fleet capped in the name of efficiency saves nothing where it counts. That gap costs money and power every day it goes unnoticed.
Running a large language model in production happens in two very different phases. First, the model reads the prompt, a phase known as prefill, then it generates the answer one token at a time, a phase known as decode. In real services (chat assistants, coding copilots, document tools) decode dominates: it is where the serving cluster spends most of its time and most of its electricity. Any serious conversation about inference energy is really a conversation about decode.
The Illusion of Power Capping
The standard way to control electricity in a GPU cluster is power capping. You set a ceiling on how many watts each GPU may draw, and the hardware throttles itself to stay under it. This is the knob every facility manager reaches for, and it works well for the workloads it was designed around, namely training runs and dense number-crunching that push a chip to its rated limit.
Our measurements show it does almost nothing for decode. On an NVIDIA H200 rated at 700 watts, decode draws only 137 to 290 watts. The cap sits far above where the chip is actually operating, so it never engages. You can set the ceiling anywhere from 280 to 700 watts, and throughput barely moves (under 3 percent), while energy consumption does not change at all. An operator who power-caps an inference fleet to reduce energy consumption achieves no savings across the workload that fleet was designed to serve.The knob is on, the dashboard looks responsible, and the effect is zero. We call this the illusion of power capping.
The Real Bottleneck Is Moving Data, Not Doing Math
The cause is the memory wall that shapes everything else in modern large language models. To generate a single token, the model does very little arithmetic but must haul its entire set of weights out of high-bandwidth memory. The compute cores sit more than 88 percent idle, waiting on that traffic. Because the chip is starved for data rather than for compute, running the cores faster, or capping their power, changes almost nothing: the memory bus sets the pace.
That insight points straight at the correct lever. Instead of capping total power that the chip never reaches, we lock the compute clock to a lower speed. Since the cores are idle most of the time anyway, slowing them down costs almost no throughput while cutting the power they waste. In our tests this recovers up to 32 percent of decode energy at under 1 percent throughput loss, consistently across every architecture we examined. It is a single command applied once when a server starts. At fleet scale the arithmetic is stark: 50 watts saved per GPU across 10,000 GPUs is half a megawatt of continuous power reduction that power capping simply cannot deliver.
Where the Model Architecture Comes In
The choice of model architecture compounds the effect, and this is where our long-context work meets the energy question. Standard transformers keep a cache of everything they have read so far and must stream all of it back on every step, so their energy per token climbs steeply as documents grow longer. Newer architectures avoid that growth.
The architecture Mamba2 is the clearest example. It carries a small fixed-size internal state instead of an ever-growing cache, so its cost per token stays flat no matter how long the input is. A widespread assumption is that these newer designs pay for their efficient decode with a slower prompt-reading phase. Our corrected measurements dispel that myth: at long context, the newer architectures are actually 25 to 43 percent cheaper when reading the prompt too, because they sidestep the escalating cost that standard attention pays as context grows.
The two savings stack rather than cancel. For a realistic request with a 16K-token prompt and a 1K-token answer at production scale, Mamba2 uses 267 joules where a standard transformer uses 495, a 46 percent reduction, and it is cheaper from the very first token of output with no penalty to repay. This is the crossover we keep observing: dense transformers are fine for short and medium inputs, but past roughly 8K tokens the newer architectures pull decisively ahead on energy. For the long-document and agentic workloads that increasingly define real usage, that gap is the line between a service that is affordable and one that is not.
Getting Both Levers Right
Two levers, one wrong and one right. The industry-default control, power capping, is structurally blind to the phase that dominates AI inference. A simpler and rarely used control, locking the compute clock, delivers the real savings the standard knob only appears to. Layer the right model architecture on top, and the long-context workloads that used to look like an energy nightmare become genuinely affordable. Getting both right, the hardware control and the architecture choice, is what makes sustainable and cost-effective inference possible at scale.
The full study behind these findings has been accepted for presentation at PPAM 2026.
Author: Bole Ma from NRH@FAU
