Why does training compute get cheaper than inference, then?

You're hitting something real, but I think you're looking at it backwards. The constraint isn't that inference *can't* get cheaper—it's that inference optimization is solved by throwing hardware at a solved problem, while training still has messy tradeoffs baked in. Here's what I mean from transit, weirdly enough: we could make buses faster by running them empty at night to pre-position for morning rush. Technically possible. But once you add in driver labor costs, fuel, maintenance on those pre-runs, the math breaks immediately—you're optimizing latency at the cost of utilization. So you don't. Training is still in the phase where you *have* to care about utilization because a full training run costs millions and you can't just spin up extra experiments. Inference already lives in the solved world where you just add a box if you need lower latency. The hardware is cheap enough relative to the problem that you're not forced to be clever. The algorithmic wins you mentioned are real, but they're narrower for inference because inference doesn't have training's built-in incentive structure. In training, if you halve your compute requirements, you halve your bill on a $10M run—that's a screaming signal. Inference optimization buys you maybe a 20% reduction in boxes, which is nice but doesn't shift your business model. So fewer people are grinding on it. The market fragmentation thing is real, but I'd flip it: it's not that fragmented markets can't scale efficiency, it's that fragmented markets don't have unified incentives strong enough to force the pain of optimization when "just add more boxes" already works.

The gap might not be a puzzle so much as evidence you're measuring the wrong thing. Training compute gets cheaper because you can delay it. You run it once, batch everything, amortize across a huge corpus, and if it takes three weeks instead of two, nobody's screaming. Inference can't do that—the query shows up, the answer has to exist in a few hundred milliseconds, and that constraint is hard. So the 4x vs 30% thing isn't really about what hardware can do; it's about what the work itself allows. You could theoretically make inference faster or cheaper if you accepted latencies that would break every production system that depends on it. Then you'd see the same efficiency curves training sees. But the moment you need to answer in real time, you've traded parallelism for sequential guarantees, and that trade doesn't have a hardware solution. The market fragmentation angle is real but secondary. The actual lock is physical: inference serving is constrained by latency in a way training fundamentally isn't, and hardware improvements that don't reduce latency much end up just giving you cheaper excess capacity. You can throw away GPUs mid-training because training is a batch job. You can't throw away inference capacity mid-query. That's not market inefficiency; that's the problem itself. The question worth asking isn't why inference hardware isn't improving faster. It's why we keep trying to solve a latency problem with raw compute instead of asking whether the latency requirement is actually necessary.

I don't know the inference cost curves you're asking about, but the fragmentation angle rings true to me in a different register. Spent years watching this same pattern in construction—training programs that got funding and standardization moved fast. Individual job sites trying to solve their own problems stayed expensive and weird. When you've got one big contractor running the same apprenticeship curriculum across fifty projects, you can actually optimize it. You cut fat, you find what works, you replicate it. But when you've got a thousand small shops each trying to train people their own way? They're all paying for redundant supervision, duplicated material, people falling through cracks. Nobody has enough volume to justify the investment in getting better. So inference probably looks like that—thousands of different serving setups, different latency requirements, different hardware targets. Training is the opposite: a few big runs that all look pretty similar, so you can actually amortize the investment in optimization across enough volume to justify it. The other thing is path-dependency. Once you're training at scale, you can afford to hire the people who'll spend six months squeezing another 10% out of the algorithms. Individual companies serving inference? They're getting off-the-shelf solutions most of the time. They're not running their own ops team for it. So the incentive to innovate is structured differently. Hardware improvements just get passed through. They don't compound the way they do when you've got institutional pressure to use them as hard as possible.