Question
Why inference costs aren't dropping like training costs
The standard story is that AI is getting cheaper across the board. Training costs fall 4x a year, so inference must be next. But the empirical picture is messier than that, and I think the structural differences matter more than people assume.
Training has this one magical property: you do it once, then amortize across millions of inference calls. So a 10x improvement in training efficiency *looks* like a 10x improvement in the per-inference cost of that model, even if the actual inference hardware or ops haven't changed much. Helion or Epoch's scaling laws show training FLOP requirements dropping fast. But that's partly a free lunch from better algorithms and scale—not necessarily from cheaper inference hardware. When people cite 4x/year improvements, they're often conflating two things: (1) training gets more efficient, so new models cost less to produce, and (2) running those models gets cheaper. Those aren't the same.
Inference hits different constraints. You need latency guarantees—can't batch forever on a consumer API. Memory bandwidth becomes the bottleneck, not raw compute. A single query to Claude or GPT-4 still needs to move model weights through memory, and memory bandwidth improvements lag FLOP improvements by years. You're also stuck with the installed base problem: training cost is capital spending that sinks, but inference infrastructure is operational. If you've got a data center full of H100s doing inference, you're not immediately retooling for cheaper hardware the way an organization might wait for the next generation of training chips.
The real question might be: is the gap closing at all? Or are we seeing improved training efficiency flatten against inference's actual hardware constraints?
1 comment
Log in to comment.
You're conflating two separate things here, and it's worth untangling because it changes what we should actually expect.
The training efficiency gains you're citing (Epoch's numbers, etc.) are mostly about algorithmic improvements and scale—better architectures, better data, longer training runs on the same hardware. That's real, but it's different from "inference gets cheaper." When a new model trains more efficiently, the *amortized per-inference cost* of that model drops, sure, but only if you're comparing it to running the old, less-efficient model. The actual hardware doing inference hasn't gotten proportionally better. You've just shifted to a smaller model or a more efficient architecture for the same task.
Inference hardware improvements are lagging because of exactly what you said: memory bandwidth is genuinely the bottleneck now, not multiply-accumulate throughput. An H100 can do enormous FLOPs, but you're bottlenecked moving weights around. We've known this for years—it's why folks have been pushing quantization, sparsity, and attention optimizations so hard. But those are *algorithmic workarounds* for a hardware constraint, not solutions to it. And yeah, the operational stickiness of deployed inference hardware means upgrade cycles are slow. I've watched teams run K80s for inference years after they stopped training anything on them, just because ripping out infrastructure is expensive and risky.
The gap-closing question is the right one, but I'd flip it: we're probably seeing training costs drop faster than inference costs for structural reasons that won't fix themselves just by waiting for better chips.