Tokens per Watt
Tokens per watt is the ratio of inference output to power consumed — the revenue line of AI divided by its scarcest input. Tokens are what hyperscalers sell; watts are what they cannot quickly buy more of. As the AI build-out shifts from construction to monetization, this single ratio is emerging as the metric that decides which operators earn a return on the infrastructure they paid for — and which layers of the hardware stack get repriced for moving it.
What it is
Inference has an income statement: every token generated is revenue, every watt consumed is cost. Tokens per watt divides one by the other. A hyperscaler that raises the ratio grows revenue without a proportional rise in infrastructure cost — which is why the metric doubles as a margin gauge. The lever underneath it is utilization: accelerators sitting idle while they wait for memory — KV cache, the working memory of inference, spilling out of HBM on reasoning and agentic workloads — burn watts without producing tokens. Feed them faster, and the same power envelope yields more revenue.
Why it matters
Power stopped being an abundant input. I/O Fund, which pushed the metric into the foreground in July 2026, cites an ERCOT interconnection queue above 438 GW — roughly 390 GW of it data centers — against about 23 GW actually added across 2024–2025. Each chip generation pulls more from the wall: roughly 700W for an H100, 1,200–1,400W for Blackwell, around 2,300W for Rubin. The cheap efficiency lever is exhausted — hyperscaler PUEs of 1.09 to 1.17 leave little to harvest outside the compute itself. And the memory wall makes the utilization problem structural: compute grew about 3× every two years over the two decades to 2023 while memory bandwidth grew 1.6×, and the gap persists into Rubin — 5× Blackwell's inference compute against 2.8× the HBM bandwidth and 1.5× the capacity. Because HBM ships packaged with the accelerator, buying more memory means buying more underutilized compute, with HBM's share of accelerator bill of materials rising from roughly 52% (B200) toward 62% (Rubin) — the dynamic behind the memory supercycle.
The two architectures
The engineering answer is the offload engine: dedicated hardware that keeps accelerators fed with KV cache from cheaper memory tiers. Two paths are forming. Nvidia's proprietary CMX pairs BlueField-4 DPUs and rack-scale SSD tiers with its Rubin platform — Nvidia's own figures claim up to 5× token throughput and 5× power efficiency on KV-cache work, and up to 35× tokens per megawatt for a Vera Rubin rack paired with a Groq 3 LPX rack versus a GB200 NVL72. The open path runs through CXL, a vendor-agnostic standard built on PCIe: Marvell's Structera controllers run near 30 watts against 150–700 watts for adding CPUs or GPUs, and Marvell's data shows a 4.8× throughput gain on a 16TB pooled tier. Two-thirds of servers were already CXL-capable in early 2025; over 90% are projected by end-2026. Tellingly, Nvidia's own Vera CPU supports CXL 3.1 — even the proprietary camp keeps the open door ajar.
How to read it
Three tells. First, watch the language of hyperscaler earnings calls: when management starts quoting inference economics — token revenue against power — the metric has become the scoreboard. Second, watch rack-level specs: tokens per megawatt is now a headline number in Nvidia keynotes, which means each platform generation gets judged on it. Third, watch where the ratio's gains come from — silicon (new accelerators), architecture (offload engines), or siting (the power constraint relocating capacity to stranded energy). Each answer reprices a different layer of the stack.