Why inference scaling hasn't matched training improvements

The gap is real and it tracks something important about what we've actually optimized for. Training compute has benefited from massive algorithmic gains (better optimizers, mixed precision, distributed strategies) plus straightforward hardware utilization improvements. But inference lives in a different constraint set, and I think we're only halfway through noticing it. Training is batch-friendly. You can run thousands of examples through the same model in parallel, amortize communication overhead, keep GPUs saturated for hours. Inference is latency-sensitive. A user waits for an answer. That changes everything about the hardware you're willing to use and the tricks you can deploy. You can't easily batch requests across users without introducing delay. Quantization helps but hits a wall around int8 before accuracy degrades noticeably—see the work by teams at Qualcomm and others on post-training quantization. Sparsity could theoretically help more but we're still figuring out how to exploit it in practice without custom hardware. The inference optimization problem is actually harder than the training one because the constraints are tighter. There's also a throughput vs. latency tradeoff that hasn't collapsed the way people maybe expected. Speculation and batching can improve throughput somewhat, but if you care about time-to-first-token or p99 latency for interactive systems, you're stuck running smaller batches. That means less amortization. The per-token cost curve flattens out before it drops 4x. I'd expect inference efficiency to keep improving—people are doing real work on serving stacks, KV cache optimization, hardware co-design—but we shouldn't expect it to track training gains unless something fundamental changes about how users interact with these systems (longer contexts? batch-first use cases? that'd help). The 30% annual improvement is respectable but it reflects a genuinely harder problem.