GPU bills get out of hand fast. A field guide to keeping inference cheap without throttling the experience.
Training gets the attention, but for most product teams it's inference that quietly dominates the monthly invoice — every user request is a recurring cost, and it scales with traffic in a way a one-time training run doesn't.
The first useful exercise isn't picking a cheaper GPU. It's instrumenting cost per request so the conversation moves from "the cloud bill is high" to "this specific endpoint costs 40x more than the others, and here's why."
Quantization, smaller distilled models for high-volume low-stakes calls, and routing only the genuinely hard requests to the large model usually cut spend more than any infrastructure change. Not every request needs the flagship model.
Autoscaling GPU instances down to zero between bursts sounds obvious; in practice cold-start latency on large models makes teams over-provision instead. Keeping a small warm pool and bursting from there is usually cheaper than either extreme.
Response caching for repeated or near-duplicate queries, and batching concurrent requests into a single forward pass, are unglamorous but typically deliver bigger savings than any model swap. They also tend to be the cheapest to implement.
Semantic caching — matching on meaning rather than exact text — extends this further for support and search-style workloads where the same question gets asked many different ways.
Cost overruns are usually a visibility problem before they're an engineering problem. If no one owns a per-feature cost dashboard, spend drifts until finance asks an uncomfortable question.
Put a budget and an alert on every model endpoint the same way you would for a database. It's a small amount of setup that turns a quarterly surprise into a weekly Slack notification.