How to slash AI inference costs without upgrading your hardware

Inference is where AI spending really adds up. While training a frontier model happens once, inference happens repeatedly—and that cumulative bill can easily dwarf the initial training investment. Machine learning engineer Chip Huyen laid out the economics clearly: the ratio of training to inference costs typically falls somewhere between 1:10 and 1:100, with reasoning models pushing those ratios even wider.

The challenge is stark: if inference becomes prohibitively expensive, organizations never recover their investment in model development. That's why understanding how to optimize inference without buying new hardware matters so much.

Understanding what to measure#

Before optimizing, you need the right metrics. Huyen recommends tracking several key latencies:

• Time to first token (TTFT): How quickly users see their first response
• Time per output token (TPOT): The interval between successive tokens in the response
• End-to-end latency: Sum of TTFT plus (TPOT × number of tokens)

With reasoning models, the situation gets more complex. As Huyen noted, "The first generated token might not be the same as the first visible token." These models think internally before showing output, so you may also need to track time to first visible token separately.

Beyond raw throughput—the raw count of requests processed—consider measuring goodput: the percentage of requests that actually meet your latency targets. An application processing 10 requests per minute at full capacity might only satisfy quality targets on three of them.

Two paths to cheaper inference#

Huyen focuses on two optimization levers—the model itself and the service layer that manages requests—while setting aside hardware improvements as out of reach for most teams.

Model optimization#

Modifying model weights directly reduces what you need to store and compute:

Quantization shrinks the numerical precision of weights and activations—for instance, dropping from 32-bit (four bytes per parameter) to 8-bit (one byte). This cuts memory footprint and speeds up calculations: an 8-bit operation requires only eight bit-by-bit additions instead of 32. The quality loss is typically minimal and scales well across different architectures, which explains why Huyen observes, "I rarely see any companies running a model at full precision anymore."