Why Tokens Per Second Is the Wrong AI Speed Metric

Inference, serving and runtime-performance engineering decides whether an AI system finishes its work inside a deadline, at a cost the business can sustain, not how many tokens per second a demo can post. A stack can raise its throughput number while requests wait longer, hard cases fail more, or an agent repeats paid-for work. The only speed claim worth funding is one measured against the workload and hardware the business will actually run.

What it is

Inference is the raw evaluation of a model, one pass producing one token. Serving wraps that in the machinery a live product needs: batching, queues, memory allocation, admission control and the network path to the accelerator. Runtime performance adds time spent outside the model itself, including compilation and warmup, so a number counting only inference time says nothing about what a user experiences.

Prefill reads the entire input and builds the reusable attention state a model needs to continue, scaling with the square of sequence length. Decode extends that state one token at a time, limited by data movement not computation. Caching means two things: prefix caching reuses computation that is genuinely identical, same tokens, model revision, tokenizer and permissions, while semantic caching reuses an answer judged similar and needs its own freshness argument. A closed API rarely exposes the kernel, cache layout or batching state a claim depends on.

Why it matters now

Three lines of 2026 evidence show how a headline number can hide what matters. A January study, revised through March, recovered a Nemotron 3 Nano model's accuracy from 66.0 under plain quantization-aware training back to 72.7, against 73.0 at full precision, under four-bit quantization using Quantization-Aware Distillation, selected from the best of ten checkpoints on that same benchmark average, evidence of recoverable accuracy there, not proof the model matches the original everywhere. A July prototype called Lynx cut time-to-first-token from 4.4 to 1.6 seconds on a 32 billion parameter Qwen model reading a 16,000-token input, using four-bit progressive key-value transfer on paired eight-accelerator NPU servers, but the measurement starts at decoding and excludes prefill, so it evidences one transfer bottleneck, not the latency a full request would show. An August prototype called OasisKV raised measured decode throughput from 1,235 to 2,083 tokens per second with a 0.1-point accuracy loss overall, while a long-context subset lost close to three accuracy points, a decline a pooled average would have hidden.

The architecture

I call the operating discipline that keeps these numbers honest the goodput ladder: four rungs of serving investment, each earned only when a measured bottleneck below justifies its cost, with goodput, verified work completed inside its deadline, as the only number that decides promotion.

The first rung, the dense baseline, sets one supported model server, a bounded input and output length, tenant-separated cache keys, and a scheduler whose admission and eviction choices have passed a quality test, not only a speed test. The second rung, disaggregation, splits prefill and decode across separate workers so each phase gets hardware suited to its bottleneck, earning its place only when the baseline shows measured queueing or a phase imbalance large enough to justify the network cost.

The third rung, approximate caching, quantizes the key-value cache, evicts old state, or offloads it beyond accelerator memory with prefetching. Recomputing evicted state can recover it, discarding it cannot, so this rung is admitted only after a long-context check and a check against rare, evidence-bearing cases, since a pooled score can stay flat while the request that mattered loses the fact it needed. The fourth rung, speculative execution, has a small model propose tokens a larger one verifies, matching the larger model's own output, earning its place only when the expected extra tokens accepted per cycle outweigh its drafting and verification overhead. A rung that fails its check runs one rung lower until the evidence exists.

How to lead it

Ownership sits with whoever owns the serving platform and its production on-call, not whichever team signed the model contract, since every rung is a release decision carrying its own regression risk.

Fund three things: workload replay against timestamped real traffic before a configuration takes live requests, a pipeline reporting time-to-first-token, accepted-token rate, verified-completion latency and goodput by percentile, and a per-tenant budget reservation that queues a request it cannot honor. Stop funding a promotion resting on a pooled quality score, a vendor's own benchmark, or a bit-width ratio standing in for a measured result.

Two decisions belong to the executive alone: the quality margin a new rung is allowed before rollback, weighing cost savings against user harm, and when a routine change counts as a new configuration needing the full ladder again rather than a silent swap. My rule: no rung goes live on staged or synthetic traffic alone, and every promoted configuration carries a versioned record of what it was tested under.

What it is worth

The money shows up in three places: accelerator cost per unit of verified work, the overhead each higher rung adds, and the cost of a regression a customer discovers before the approving team does. Measure all three with the same paired workload, before and after. A worked calculation in the dossier, a teaching approximation rather than a measured benchmark, makes the tradeoff concrete: a draft-and-verify cycle accepting roughly four tokens out of five on a four-token draft yields about 3.36 tokens per cycle, or roughly 26 tokens per second once drafting and verification time are counted, slower than the plain 30 tokens per second baseline it was meant to beat.

The evidence here supports testing precision recovery, progressive cache transfer and sparse offload on the workloads where they were measured. It does not support a universally fastest server, a lossless replacement for every model, or a gain that transfers automatically across accelerator families. Evidence on CPU and edge deployment, and fairness across tenants under real bursts, remains thin enough to treat both as open questions.