Everything runs in this tab. The decoders and the network you quantise are real but tiny, and seeded, so everyone sees the same numbers. Hardware figures are approximate published peaks and prices are illustrative, so treat every time here as a floor.
Inference lab
A trained model is a file of numbers. Serving it means pushing those numbers through a chip fast enough, for enough people, at a price that works. We'll take the bill apart in six exhibits, from the first token to a full serving budget.
Lab exploredThat's the whole serving stack, from the first token to the bill. The Model workshop shows what's inside the numbers being served.
Two costs for one answer
A request runs in two phases. Prefill reads the whole prompt in one pass, so every weight it fetches gets used for every prompt token. Decode then writes one token per step and fetches every weight again for that single token. Prefill usually hits the arithmetic limit and decode hits memory bandwidth, which is why the first token behaves so differently from the rest.
The roofline on the right shows which limit you're hitting. Across is arithmetic intensity (operations per byte moved), and up is the speed the chip can reach there. Left of the ridge the chip waits on memory. Right of it, it waits on arithmetic.
The maths
tmem = (W + B·L·kv) / BW tcmp = B·(2P + 4·nlayers·dq·L) / F step = max(tmem, tcmp) + t0
A decode step reads every weight (W bytes) plus the cached keys and values of every sequence (B sequences, L tokens each, kv bytes per token), at bandwidth BW. It does one multiply and one add per weight per sequence (2P operations, P counting the weights in matrix multiplies). Attention adds 4·dq·L operations per layer for a query of width dq against L keys and values. F is the card's dense fp16 tensor-core rate, two to four times what its CUDA cores reach. The chip overlaps the two, so whichever is slower sets the step, plus a fixed t0 = 0.3 ms for launching kernels and sampling.
prefill FLOPs = B·(2P·n + 4·nlayers·dq·n2/2) bytes = W + B·n·(kv + a), a = 4·d·nlayers
Prefill does the same work for all n prompt tokens in one pass, but reads the weights only once. Attention is causal: token i looks at tokens 1 to i only, which is half of the n by n square, so that's the /2. Besides the weights it writes each token's keys and values and moves each layer's activations (d numbers in and out at 2 bytes, a = 4·d bytes per layer).
I = FLOPs / bytes ridge = F / BW compute-bound when I ≥ ridge
The ridge is where both limits take the same time: tcmp ≥ tmem is the same inequality as I ≥ F/BW. At batch B the decode intensity is about B times the operations per weight byte. So raising the batch walks the dot to the right until B·(2P + 4·nlayers·dq·L)/F ≥ (W + B·L·kv)/BW. That solves to B* = (W/BW) / ((2P + 4·nlayers·dq·L)/F − L·kv/BW). If the bracket is zero or negative, the cache alone keeps decode memory-bound at any batch.
In practiceServing systems report time to first token (TTFT) and time per output token (TPOT) separately. Long prompts cost TTFT and long answers cost TPOT.
Watch outA benchmark at batch 1 measures memory bandwidth and barely touches compute. Comparing chips on TFLOP/s alone predicts prefill and says almost nothing about decode.
Memory that grows with every token
Attention needs the keys and values of every earlier token. Recomputing them would make each step slower than the last, so servers cache two vectors per token, per layer, per key/value head. The cache shares memory with the weights, and at long contexts it outgrows the model.
Fill the card below until it runs out, then get the memory back two ways. Under it, a tiny attention decoder generates the same tokens with and without the cache and counts every multiply-add.
- Cache per token
- Cache in use
- Weights + cache
- Most sequences at this context
- Longest context at this batch
The maths
kv bytes = 2 · nlayers · nkv heads · dhead · bytes · tokens · batch
Each layer keeps one key and one value vector (the 2) of width dhead for every key/value head, for every token of every sequence. Grouped-query attention lets g query heads share one key/value head, so nkv heads = nheads / g. Multi-query attention shares one for all (nkv heads = 1). The weights don't change size in this sum (in a real model the key and value projections shrink a little too).
most sequences = ⌊(M − weights) / (kv per token · context)⌋ longest context = ⌊(M − weights) / (kv per token · batch)⌋
with the cache, step t: 4d2 + 2d(t+1) + dV without: 2d2 + 2d2(t+1) + 2d(t+1) + dV
Step t (counting from 0) sees t+1 positions. Both versions project the new query (d2), score it against t+1 keys and mix t+1 values (d each, 2d(t+1)), project the output (d2) and the vocabulary (dV). With the cache only the new token's key and value are projected (2d2). Without it all t+1 are, 2d2(t+1). Summed over N steps: with the cache N(4d2 + dV) + dN(N+1), without it N(2d2 + dV) + (d2 + d)N(N+1). The projections go from quadratic in N to linear. The attention sums stay quadratic in both, but are d times smaller. The softmax and the scaling aren't multiply-adds, so they aren't counted.
In practiceServers allocate the cache in fixed-size pages (paged attention) and evict or swap sequences when it runs out. Grouped-query attention is now standard in large open models for the saving you just measured.
Watch outA context window is a promise the memory has to keep for every concurrent user. Capacity planning that counts only the weights runs out of memory at the first busy hour.
Many requests, one pass
Decode is memory-bound, so a step for 32 sequences costs little more than a step for one. That's how a server earns its throughput. Static batching runs a batch until its longest request finishes, and the finished ones just sit in their slots. Continuous batching lets requests leave and join at every step.
Below, requests arrive at random (a seeded Poisson stream) for 60 seconds or 400 requests, with prompt and answer lengths drawn from seeded log-normal laws. The simulation runs on virtual time using exhibit I's step model, then replays. Prefill isn't chunked here: when requests join, the batch stalls while the server reads their prompts.
| Policy | Tokens/s | Mean latency | p95 | Wait | Slot use |
|---|
The maths
L = λW
Little's law: the average number of requests in the system equals the arrival rate times the average time each spends there. Every request adds a strip to the curve as long as its time in the system, so the area under it is the sum of the latencies. Divide by the window and you have L on one side and λW on the other. It holds for any policy while the system is stable, and the gap you see is requests still inside when the window closes.
slot use = busy slot-steps / (Bmax × decode steps) static, full batches: (E[o] − 1) / (E[omax of B] − 1)
In a static batch every slot is held until the longest request finishes, so a slot does o − 1 useful decode steps out of max − 1. With long-tailed lengths the expected maximum of B draws is far above the mean, which is the waste you see hatched.
E[omax of B] = Σk ≥ 0 P(max > k) = Σk (1 − F(k)B)
The maximum of B independent lengths is at most k only if all B are, with probability F(k)B for the length law's distribution function F. Summing the chance that it exceeds each k gives its mean.
stable only while λ < μ queue grows at about (λ − μ) per second when λ > μ
μ is the rate at which a full batch finishes requests. Past it, nothing settles: the queue grows for as long as requests keep arriving, and every latency number depends on how long you watched.
In practiceModern servers batch continuously, usually with chunked prefill, which cuts long prompts into pieces so they don't stall everyone else. The model here stalls on purpose so you can see the bands.
Watch outAverage latency hides the tail, and throughput measured past capacity describes a queue. Measure p95 at the load you expect, and keep headroom below μ.
Fewer bits per weight
Decode time is mostly the time to read the weights, so fewer bits per weight means faster steps and room for bigger models. Quantisation rounds each weight to one of a few evenly spaced levels and keeps a scale to turn the level back into a number. How much damage that does depends on how many levels there are and on how you choose the scale.
The network below is real: a two-layer network (2 inputs, 16 ReLU units, 2 outputs) trained in this tab on two interleaved spirals. Round its weights and watch the decision boundary, the accuracy and the error. The biases stay at full precision, as they usually do.
Training…
| Full | Quantised | |
|---|---|---|
| Accuracy | ||
| Loss | ||
| Weight bytes | ||
| SQNR measured | ||
| SQNR predicted |
The maths
scale = max|w| / (2b−1 − 1) q = clamp(round(w / scale), −(2b−1 − 1), 2b−1 − 1) w′ = q · scale
Symmetric quantisation maps the largest weight to the largest level, so nothing is clipped and zero stays exactly zero. With b bits there are 2b−1 − 1 levels on each side of zero.
e = w′ − w ∈ [−scale/2, scale/2] E[e2] = scale2 / 12
Rounding moves a weight at most half a step. If the weights land anywhere within a step with equal chance, the error is uniform on that interval, and a uniform law of width s has variance s2/12.
SQNR = 10 log10(σw2 / E[e2]) = 10 log10(12 (2b−1 − 1)2 σw2 / max2) ≈ 6.02b + 4.77 − 20 log10(max|w| / σw) dB
Substitute the scale into the noise: E[e2] = max2 / (12 (2b−1 − 1)2). With 2b−1 − 1 ≈ 2b−1, 20 log10 2b−1 = 6.02b − 6.02 and 10 log10 12 = 10.79, which sum to 6.02b + 4.77. σw is the root mean square of the weights.
Every extra bit buys 6 dB, and every doubling of max|w| over σw costs 6 dB, because one outlier sets the scale for everything that shares it. Per-channel and group scales give each small set of weights its own max, at 16 bits per scale.
Dropping the −1 costs 20 log10(2b−1 / (2b−1 − 1)): 1.2 dB at 4 bits and 0.07 dB at 8, which is why the worked example shows both. SQNR is an average too. With the outlier, the two big weights carry most of the power, so the figure can look healthy while almost every small weight rounds to zero. Watch the accuracy as well as the decibels.
bytes = ⌈b · nweights / 8⌉ + 2 · nscales
The outlier here is exact: one hidden neuron gets its incoming weights and bias multiplied by 20 and its outgoing weights divided by 20. Since relu(20z) = 20 relu(z), the full-precision network computes the same function, yet its first matrix now holds two weights twenty times larger than the rest.
In practiceWeight-only int8 and int4 with group scales of 64 or 128 weights are the everyday serving formats, and GPTQ and AWQ choose the rounding better than nearest. Large models do have outlier features, which is why per-channel and mixed schemes won.
Watch outAccuracy on a benchmark can hold while rare behaviours break. Measure the quantised model on your own traffic, and never compare speed at different bit widths without comparing quality too.
Guess ahead, check in one pass
A decode step of a big model costs about the same for one token or several, because it's memory-bound. Speculative decoding uses that: a small draft model proposes k tokens and the big target checks all k in one pass. Each draft token is accepted with probability min(1, p/q), where p is the target's probability and q is the draft's. At the first rejection the target draws a replacement from what's left of p, and if all k pass, it adds one more. an intern drafts and a senior reviews lol.
The accept rule is exact: the text that comes out has precisely the target's distribution, whatever the draft does. Here the target and the draft are seeded first-order Markov models over 8 symbols, and the draft is a blend of the target and noise.
The maths
α = Σx min(p(x), q(x)) = 1 − TV(p, q)
A draft token x arrives with probability q(x) and survives with probability min(1, p(x)/q(x)). So it's accepted with probability Σ q(x) min(1, p(x)/q(x)) = Σ min(p(x), q(x)). Since Σ p = Σ q = 1, that's one minus the total variation distance between the two laws.
P(emit x) = min(p(x), q(x)) + (1 − α) · r(x), r(x) = max(0, p(x) − q(x)) / (1 − α) so P(emit x) = min(p, q) + max(0, p − q) = p(x)
The two-line proof: the accepted path contributes min(p, q), and the rejection path contributes (1 − α) times the residual. The residual's normaliser Σ max(0, p − q) equals 1 − α, so the two cancel, and min(p, q) + max(0, p − q) = p for every x. Every emitted token, accepted, replaced or bonus, is an exact draw from the target.
E[tokens per step] = 1 + α + α2 + … + αk = (1 − αk+1) / (1 − α) speed-up = E / (1 + c·k)
With the same α at every position, the i-th draft token counts only if all before it passed (αi), and the step always emits one more token (the replacement or the bonus). That's a geometric series. A step costs one target pass plus k draft steps of relative cost c. Adding a draft token helps while αk+1/(1 − α) is worth more than c times the current step, so the curve tries each k. Here α depends on the context, so the page also computes the exact expectation over the draft's paths. The series with the long-run α is the textbook estimate.
In practiceDrafts are small models of the same family, extra heads on the target, or lookups of text already in the prompt. Acceptance of 0.6 to 0.8 is common on easy text, and the gain shrinks as the batch grows.
Watch outWith sampling, a variant that accepts “close enough” tokens is no longer exact and trades quality for speed without saying so. Check the rule and the output distribution, as the panel does.
What it costs to keep it running
Pick a model, a weight type and some cards, split the model across them, and say how many people it has to serve, at what context and speed. The planner checks memory first, then works out decode speed from the roofline and the price per million tokens.
- Fits
- Most users at this context
- Per user
- All users together
- First token (full context)
- Cost per million tokens
Serve a 70B-class model to 32 users at an 8,192-token context with at least 25 tokens per second each, at the lowest cost per million tokens. Costs here use the listed prices.
The maths
fits when P·bytes + kv·context·users ≤ (1 − 0.1) · cards · M
The weights and every user's cache must sit in the cards' memory, with a tenth kept back for activations and the allocator. Tensor parallel splits both across the cards.
step = max((W + U·L·kv) / (N·e·BW), U·(2P + 4·nlayers·dq·L) / (N·e·F)) + t0 per user = 1 / step all users = U / step
It's exhibit I's step with the rates of N cards scaled by the tensor-parallel efficiency e (1, 0.9, 0.8, 0.7 for 1, 2, 4, 8 cards, illustrative). Every user sits at the full context, the worst case. While decode is bandwidth-bound, per-user speed is about N·e·BW / (W + U·L·kv). Each added user adds a cache to every step, so each user slows while the total grows.
cost per million tokens = (price per hour · N / 3600) / (all users' tokens per second / 106)
In practiceReal plans add margin for traffic peaks, a prefill budget and redundancy, and real kernels reach only part of these peak figures. Every number here is a floor on the real cost.
Watch outCost per token only means something together with the speed and context it was measured at. A quote at batch 256 and 1k context isn't a price for 32 users at 8k.