Inference lab
A trained model is a file of numbers. Serving it means moving those numbers through a chip fast enough, for enough people at once, at a price that makes sense. Six exhibits take the bill apart: why the first token costs differently from the rest, the cache that grows with every token, batching a stream of requests, rounding weights to fewer bits, letting a small model guess ahead, and a budget that ties it all together.
Everything runs in this tab. The attention decoder, the network you quantise and the speculative decoder are real, tiny, and seeded, so every visitor sees the same numbers. The hardware and model tables are approximate published peak figures and Llama-style shapes (shape only, not a specific product); real kernels reach about 30 to 70 percent of peak, so treat every time here as a floor, not a benchmark. Prices are illustrative, not quotes.
Lab exploredYou have watched prefill and decode land on either side of the ridge, run a model out of memory and back, batched a request stream, rounded a network to a few bits and back again, proved that speculative decoding changes nothing but the speed, and priced a 70B-class service. The Model workshop shows what is inside the numbers being served.
Two costs for one answer
A request runs in two phases. Prefill reads the whole prompt in one pass: every weight is fetched from memory once and used for every prompt token, so there is plenty of arithmetic per byte. Decode then writes one token per step, and every step fetches every weight again to do the work of a single token. The first phase is usually limited by arithmetic, the second by memory bandwidth, and that is why the time to the first token and the time for each token after it behave so differently.
The roofline on the right says which limit applies. Across is arithmetic intensity (floating-point operations per byte moved), up is the speed the chip can reach at that intensity. Left of the ridge the chip waits for memory; right of it, for its arithmetic units.
The maths
tmem = (W + B·L·kv) / BW tcmp = B·(2P + 4·nlayers·dq·L) / F step = max(tmem, tcmp) + t0
A decode step must read every weight (W bytes) and the cached keys and values of every sequence (B sequences, L tokens each, kv bytes per token) through memory bandwidth BW. It does one multiply and one add per weight per sequence (2P operations, where P counts the weights used in matrix multiplies; the input embedding is a table lookup and is left out), plus attention: a query of width dq scored against L keys and used to mix L values, 4·dq·L operations per layer. The chip overlaps the two, so the slower one sets the step, plus a fixed t0 = 0.3 ms for launching kernels and sampling.
prefill FLOPs = B·(2P·n + 4·nlayers·dq·n2/2) bytes = W + B·n·(kv + a), a = 4·d·nlayers
Prefill does the same work for all n prompt tokens in one pass, but reads the weights only once. Attention is causal: token i looks at tokens 1 to i only, which is half of the n by n square, hence the /2. Besides the weights it writes each token's keys and values and moves each layer's activations (d numbers in and out at 2 bytes, a = 4·d bytes per layer).
I = FLOPs / bytes ridge = F / BW compute-bound when I ≥ ridge
The ridge is where both limits take the same time: tcmp ≥ tmem is the same inequality as I ≥ F/BW. For decode at batch B the intensity is about B times the number of operations per weight byte, so raising the batch walks the dot to the right until B·(2P + 4·nlayers·dq·L)/F ≥ (W + B·L·kv)/BW, which solves to B* = (W/BW) / ((2P + 4·nlayers·dq·L)/F − L·kv/BW). If the bracket is zero or negative, the cache alone keeps decode memory-bound at any batch.
In practiceServing systems report the two numbers separately: time to first token (TTFT) for how long the user waits before anything appears, and time per output token (TPOT) for how fast the text streams. Long prompts cost TTFT; long answers cost TPOT. Peak figures here are published peaks; real kernels reach about 30 to 70 percent of them.
Watch outA benchmark at batch 1 measures memory bandwidth, not compute. Comparing chips on TFLOP/s alone predicts prefill and says almost nothing about decode.
Memory that grows with every token
Attention needs the keys and values of every earlier token. Recomputing them at every step would make each step slower than the last, so servers keep them in a cache: two vectors per token, per layer, per key/value head. That cache lives in the same memory as the weights, and at long contexts or large batches it is bigger than the model.
Fill the card below until it runs out, then get it back two ways. Under it, a real (tiny) attention decoder proves the cache changes nothing but the cost: it generates the same tokens twice, with and without the cache, and counts every multiply-add.
- Cache per token
- Cache in use
- Weights + cache
- Most sequences at this context
- Longest context at this batch
The maths
kv bytes = 2 · nlayers · nkv heads · dhead · bytes · tokens · batch
Each layer keeps one key and one value vector (the 2) of width dhead for every key/value head, for every token of every sequence. Grouped-query attention lets g query heads share one key/value head, so nkv heads = nheads / g; multi-query attention shares one for all (nkv heads = 1). The weights do not change size in this sum (in a real model the key and value projections shrink too, by a little).
most sequences = ⌊(M − weights) / (kv per token · context)⌋ longest context = ⌊(M − weights) / (kv per token · batch)⌋
with the cache, step t: 4d2 + 2d(t+1) + dV without: 2d2 + 2d2(t+1) + 2d(t+1) + dV
Step t (counting from 0) sees t+1 positions. Both versions project the new query (d2), score it against t+1 keys and mix t+1 values (d each, 2d(t+1)), project the output (d2) and the vocabulary (dV). With the cache only the new token's key and value are projected (2d2); without it all t+1 are, 2d2(t+1). Summed over N steps: with the cache N(4d2 + dV) + dN(N+1), without it N(2d2 + dV) + (d2 + d)N(N+1). The projections go from quadratic in N to linear; the attention sums stay quadratic in both, but are d times smaller. The softmax and the scaling are not multiply-adds and are not counted.
In practiceServers allocate the cache in fixed-size pages (paged attention) so memory is not reserved for tokens a request never writes, and they evict or swap sequences when it runs out. Grouped-query attention is now standard in large open models for exactly the saving you just measured.
Watch outA context window is a promise the memory has to keep for every concurrent user. Capacity planning that counts only the weights runs out of memory at the first busy hour.
Many requests, one pass
Decode is memory-bound, so a step for 32 sequences costs little more than a step for one: batching is how a server earns its throughput. The question is how to batch a stream of requests that arrive at random and want answers of very different lengths. Static batching forms a batch and runs it until the longest request finishes, while the finished ones sit in their slots. Continuous batching lets finished requests leave and queued ones join at every step.
Below, requests arrive at random (a seeded Poisson stream) for 60 seconds or 400 requests, with prompt and answer lengths drawn from seeded log-normal laws. The simulation runs on virtual time using exhibit I's step model, then replays. In this model, prefill is not chunked: when requests join, the batch stalls while their prompts are read.
| Policy | Tokens/s | Mean latency | p95 | Wait | Slot use |
|---|
The maths
L = λW
Little's law: the average number of requests in the system equals the arrival rate times the average time each spends there. Draw the number in the system over time: every request adds a strip as long as its time in the system, so the area under the curve is the sum of the latencies. Divide by the length of the window and you have L on one side and λW on the other. It holds for any policy and any length law, as long as the system is stable (what arrives also leaves inside the window); the gap you see comes from requests still in the system when the window closes.
slot use = busy slot-steps / (Bmax × decode steps) static, full batches: (E[o] − 1) / (E[omax of B] − 1)
In a static batch every slot is held until the longest request finishes, so a slot does o − 1 useful decode steps out of max − 1. With long-tailed lengths the expected maximum of B draws is far above the mean, which is the waste you see hatched.
E[omax of B] = Σk ≥ 0 P(max > k) = Σk (1 − F(k)B)
The maximum of B independent lengths is at most k only if all B are, which has probability F(k)B, where F is the length law's distribution function; summing the chance that it exceeds each k gives its mean.
stable only while λ < μ queue grows at about (λ − μ) per second when λ > μ
μ is the rate at which a full batch finishes requests. Past it, nothing settles: the queue grows for as long as requests keep arriving, and every latency number depends on how long you watched.
In practiceContinuous (in-flight) batching is what modern servers do, usually with chunked prefill: long prompts are cut into pieces and mixed into decode steps so they do not stall everyone else. The model here stalls on purpose so you can see the bands.
Watch outAverage latency hides the tail, and throughput numbers taken past capacity describe a queue, not a service. Measure p95 at the load you expect, and keep headroom below μ.
Fewer bits per weight
Decode time is mostly the time to read the weights, so storing each weight in fewer bits makes every step faster and lets bigger models fit. Quantisation rounds each weight to one of a few evenly spaced levels and keeps a scale to turn the level back into a number. The damage depends on how many levels there are and on how the scale is chosen.
The network below is real: a two-layer network (2 inputs, 16 ReLU units, 2 outputs) trained in this tab on two interleaved spirals. Round its weights and watch the decision boundary, the accuracy and the error. The biases stay at full precision, as they usually do.
Training…
| Full | Quantised | |
|---|---|---|
| Accuracy | ||
| Loss | ||
| Weight bytes | ||
| SQNR measured | ||
| SQNR predicted |
The maths
scale = max|w| / (2b−1 − 1) q = clamp(round(w / scale), −(2b−1 − 1), 2b−1 − 1) w′ = q · scale
Symmetric quantisation maps the largest weight to the largest level, so nothing is clipped and zero stays exactly zero. With b bits there are 2b−1 − 1 levels on each side of zero.
e = w′ − w ∈ [−scale/2, scale/2] E[e2] = scale2 / 12
Rounding moves a weight at most half a step. If the weights land anywhere within a step with equal chance, the error is uniform on that interval, and a uniform law of width s has variance s2/12.
SQNR = 10 log10(σw2 / E[e2]) = 10 log10(12 (2b−1 − 1)2 σw2 / max2) ≈ 6.02b + 4.77 − 20 log10(max|w| / σw) dB
Substitute the scale into the noise: E[e2] = max2 / (12 (2b−1 − 1)2). With 2b−1 − 1 ≈ 2b−1, 20 log10 2b−1 = 6.02b − 6.02 and 10 log10 12 = 10.79, which sum to 6.02b + 4.77. σw is the root mean square of the weights. Every bit buys 6 dB, and every doubling of max|w| over σw costs 6 dB: one outlier sets the scale for everything that shares it. Per-channel and group scales give each small set of weights its own max, at 16 bits per scale. Dropping the −1 costs 20 log10(2b−1 / (2b−1 − 1)): 1.2 dB at 4 bits, 0.07 dB at 8, so the worked example shows both. SQNR is also an average over the signal: with the outlier, the two big weights carry most of the power, so the number can look healthy while almost every small weight rounds to zero. Watch the accuracy, not only the decibels.
bytes = ⌈b · nweights / 8⌉ + 2 · nscales
The outlier here is exact, not a guess: one hidden neuron gets its incoming weights and bias multiplied by 20 and its outgoing weights divided by 20. ReLU satisfies relu(20z) = 20 relu(z), so the full-precision network computes exactly the same function, yet its first weight matrix now holds two weights twenty times larger than the rest.
In practiceWeight-only int8 and int4 with group scales (groups of 64 or 128 weights) are the everyday formats for serving large models; methods such as GPTQ and AWQ choose the rounding more cleverly than nearest. Large language models do have outlier features, which is why per-tensor int8 of activations was hard and per-channel or mixed schemes won.
Watch outAccuracy on a benchmark can hold while rare behaviours break. Measure the quantised model on your own traffic, and never compare speed at different bit widths without comparing quality too.
Guess ahead, check in one pass
A decode step of a big model costs about the same for one token or several, because it is memory-bound. Speculative decoding exploits that: a small draft model proposes k tokens, and the big target model checks all k in one pass. Each draft token is accepted with probability min(1, p/q), where p is the target's probability and q the draft's; at the first rejection a replacement is drawn from what is left of p, and if all k pass the target adds one more token of its own.
The accept rule is exact: the text that comes out has precisely the target's distribution, whatever the draft does. Here the target and the draft are seeded first-order Markov models over 8 symbols, and the draft is a blend of the target and noise.
The maths
α = Σx min(p(x), q(x)) = 1 − TV(p, q)
A draft token x arrives with probability q(x) and survives with probability min(1, p(x)/q(x)), so it is accepted with probability Σ q(x) min(1, p(x)/q(x)) = Σ min(p(x), q(x)). Since Σ p = Σ q = 1, that is one minus the total variation distance between the two laws.
P(emit x) = min(p(x), q(x)) + (1 − α) · r(x), r(x) = max(0, p(x) − q(x)) / (1 − α) so P(emit x) = min(p, q) + max(0, p − q) = p(x)
The two-line proof: the accepted path contributes min(p, q), and the rejection path contributes (1 − α) times the residual. The residual's normaliser Σ max(0, p − q) equals 1 − α, so the two cancel, and min(p, q) + max(0, p − q) = p for every x. Every emitted token, accepted, replaced or bonus, is an exact draw from the target.
E[tokens per step] = 1 + α + α2 + … + αk = (1 − αk+1) / (1 − α) speed-up = E / (1 + c·k)
With the same α at every position, the i-th draft token counts only if all before it passed (αi), and the step always emits one more token (the replacement or the bonus): a geometric series. A step costs one target pass plus k draft steps of relative cost c. Adding a draft token helps while αk+1/(1 − α) is worth more than c times the current step, so the best k is found by trying each; the curve does exactly that. Here α depends on the context, so the page also computes the exact expectation over the draft's paths and the chain of step starts, and the series with the long-run α is the textbook estimate.
In practiceDrafts are small models of the same family, extra prediction heads on the target itself, or plain lookups of text already in the prompt. Acceptance rates of 0.6 to 0.8 are common on easy text, and the gain shrinks as the batch grows, because a big batch has already made decode less memory-bound.
Watch outWith sampling, a variant that accepts “close enough” tokens is no longer exact; it trades quality for speed without saying so. Check the rule, and check the output distribution, as the panel does.
What it costs to keep it running
Put it together. Pick a model, a weight type and cards, split the model across them (tensor parallel, which costs some efficiency in all-reduce traffic), and say how many people it must serve at what context and speed. The planner checks memory first, then works out the decode speed from the roofline and the price of every million tokens.
- Fits
- Most users at this context
- Per user
- All users together
- First token (full context)
- Cost per million tokens
Serve a 70B-class model to 32 users at an 8,192-token context with at least 25 tokens per second each, at the lowest cost per million tokens. Costs here use the listed prices.
The maths
fits when P·bytes + kv·context·users ≤ (1 − 0.1) · cards · M
The weights and every user's cache must sit in the cards' memory, with a tenth kept back for activations and the allocator. Tensor parallel splits both across the cards.
step = max((W + U·L·kv) / (N·e·BW), U·(2P + 4·nlayers·dq·L) / (N·e·F)) + t0 per user = 1 / step all users = U / step
Exhibit I's step, with the rates of N cards scaled by the tensor-parallel efficiency e (1, 0.9, 0.8, 0.7 for 1, 2, 4, 8 cards, illustrative), every user at the full context (the worst case). While decode is bandwidth-bound, per-user speed is about N·e·BW / (W + U·L·kv): adding users adds their caches to every step, so each user slows down while the total grows. That is the trade batching makes.
cost per million tokens = (price per hour · N / 3600) / (all users' tokens per second / 106)
In practiceReal plans add a margin for traffic peaks, a prefill budget (long prompts compete with decode for the same cards), redundancy, and the fact that kernels reach only part of the peak figures used here. Every number on this page is a floor on the real cost.
Watch outCost per token only means something together with the speed and context it was measured at. A quote at batch 256 and 1k context is not a price for 32 users at 8k.