AriakiI read one table and concluded "don't switch." I read a different cell of the same table and concluded "2.58x faster."
Both verdicts came from the same sweep, on the same machine, on the same day. Not a single number changed. I acted on the second verdict, switched configs, then opened the chat client I use every day and asked it a few questions. The UI said 38 tok/s. Hitting the same server directly with the same question gave 74.1.
The client was eating almost half the speed, and on prose my 2.58x had shrunk to about 10% to begin with. No wonder it felt the same.
If you just want the formula, jump to section 3. If you want to know how to take apart a speedup figure on a model card, see section 4.
1. One table, two verdicts
The hardware is a mining-edition datacenter card, software-unlocked to 64GB of memory, permanently hosting a 27B dense model. I wanted to compare two setups:
- Old: FP8 weights plus the model's built-in MTP head, drafting 5 tokens at a time
- New: INT8 weights plus a separately trained DFlash2 draft model, drafting 7 at a time
Speculative decoding means a cheap helper guesses the next few tokens and the big model verifies them in one pass. Correct guesses are free tokens; a wrong guess gets thrown away. Whether it pays off depends on how good the guesses are, how expensive one verification pass is, and one thing people keep forgetting: how many requests are running on the card at the same time.
I used the same streaming probe, on the same day, swapping the two configs in turn, and swept concurrency at 1, 8, 32 and 64. Aggregate throughput only (total output tokens across all requests divided by wall-clock time):
| Concurrency | Code, old | Code, new | Ratio | Prose, old | Prose, new | Ratio |
|---|---|---|---|---|---|---|
| 1 | 122.6 | 111.2 | 0.91× | 62.2 | 69.0 | 1.11× |
| 8 | 274.7 | 708.4 | 2.58× | 289.0 | 302.0 | 1.04× |
| 32 | 554.3 | 714.5 | 1.29× | 340.5 | 308.5 | 0.91× |
| 64 | 663.7 | 711.2 | 1.07× | 359.8 | 306.9 | 0.85× |
The first time through, I went straight to the bottom row. The reasoning sounded professional: the server was launched with --max-num-seqs 64, so judge it at 64. Code only gained 7%, prose lost 15%, and p95 time-to-first-token went from 12.0 s to 13.5 s. Verdict: don't switch.
Only later did I ask the question I should have asked first: how many requests does this server actually handle at once?
Awkward answer. It's just me. One chat client, plus the occasional background agent, four requests at most. The 64 was a ceiling written into the config out of caution. Nothing ever got near it.
Read the second row instead and code aggregate throughput is 2.58x, p95 time-to-first-token drops from 2472 ms to 333 ms, and prose goes from a 15% loss to a 4% gain. The verdict flips: switch.
Flags like --max-num-seqs are configuration ceilings, and they tell you nothing about the operating point. They record what whoever wrote the config was worried about. They don't tell you how many people are actually using it. And concurrency happens to be the axis speculative decoding is most sensitive to. At low concurrency the card is sitting around waiting on memory, so drafting 7 tokens is free money. At high concurrency the card is already saturated with real requests, and stuffing guesses in just steals compute from them.
These days I pin down the operating point in this order. Ask the people using it (or yourself). If you can't, sample num_requests_running from the server's /metrics over a while and look at the distribution rather than the peak. Failing that, look at the callers: a single-user chat client or agent sits around 1 to 4; only a backend sitting in a load-balancer pool serving a fleet of services might be in the dozens.
2. At low concurrency, don't read aggregate throughput
There was one more cell I nearly misread: concurrency 1, code, new config at 0.91x. Looks like a loss.
Same cell, different metric. Per-request speed (each request's output tokens divided by its own elapsed time, median across requests): old 137.4, new 194.5, 1.42x faster.
The reason the two disagree so much is mundane:
aggregate throughput = total output tokens across all requests ÷ total wall clock
per-stream speed = one request's output tokens ÷ that request's own elapsed time
At concurrency 1 the wall clock includes warmup and gaps between requests, and those take about the same time on both sides. The new config finishes each request faster, so the gaps make up a bigger share of its wall clock, and that drags its aggregate number down. When it's just you, per-stream speed is what you feel. The aggregate column in that cell is basically noise.
So "which cell" is really two questions: which concurrency level, and which column at that level. At high concurrency read aggregate, because you care how much the card produces in total. At low concurrency read per-stream, because you care how long your own request takes.
3. After switching, the client said 38
Once the verdict flipped I switched. Acceptance testing was three bare requests, tool calls worked, and the concurrency probe at 4 showed 490.2 aggregate, 147.5 per-stream, 229 ms p95 time-to-first-token. All green.
Then I asked a prose question in my own chat client. The UI showed 38 tok/s and the first token took 6.1 seconds.
My first instinct was to explain that "prose just doesn't gain much." True, but it didn't answer why the number was 38. I also made a dumber mistake: the client runs on another machine, so I went off and benchmarked the local inference servers on that machine, none of which had anything to do with it. Eventually I did the boring thing and read the client's config, followed its endpoint through to the backend, and confirmed it really was hitting this card.
What finally worked was the dumbest, most accurate method: open a browser and actually click through the client. Same model, same content, same day:
| How it was measured | Prose | Code |
|---|---|---|
| Direct to the inference server | 74.1 tok/s | 208.8 tok/s |
| Through the load-balancing gateway | 75.1 tok/s | — |
| Through the chat client | 38 tok/s, first token 6.1 s | 185 tok/s, first token 3.3 s |
| Client ÷ direct | 0.51× | 0.89× |
The gateway costs nothing. The client eats half the speed on prose and almost nothing on code.
The reason was sitting in the client's own stats. Every turn, it prepends three chunks: a 1509-token system prompt, 6603 tokens of tool and skill definitions, and 1347 tokens of my actual message plus history. I type one sentence and it sends 8112 tokens as the starting point.
That 8k prefix costs twice. Time-to-first-token goes up, because the model has to read all 8k first (this phase is called prefill, and speculative decoding does nothing for it: both configs scored 0.99x on prefill at 6.3k and 25k tokens). Decode slows down too, because every new token has to attend back over a longer context. Prose answers are long and pay that tax on every step. Code answers stop after a hundred-odd tokens, so the effect is much smaller.
The client computes its speed as decode tokens ÷ decode time. My turn was 377 tokens in 10.003 seconds, which is 37.7, shown as 38. The whole turn took 16.087 seconds, 6.084 of which was waiting for the first token.
With those numbers you can work out the speedup a user can actually feel:
whole-turn speedup = (time to first token + decode time × s) ÷ (time to first token + decode time)
Here s is the decode speedup measured at the engine, and both times are the post-switch measurements. The assumption is that time-to-first-token doesn't change, which holds here because prefill is identical on both sides.
Plug in the probe's prose number at concurrency 1, 1.11x:
(6.084 + 10.003 × 1.11) ÷ (6.084 + 10.003)
= 17.19 ÷ 16.09
≈ 1.07
The whole turn gets 7% faster. Nobody can feel 7%. Even if I force the code-question 1.42x onto this turn, it comes out at 1.26x. Plug in the prettiest number, 2.58x, and it only just reaches 1.98x. Those 6 seconds before the first token are a fixed cost, and they sit there no matter how fast the engine gets.
You can run this formula on your own numbers: pull time-to-first-token and decode time out of your client, then multiply decode time by whatever speedup your probe reported. The bigger the share of the turn spent waiting for the first token, the more an engine-side gain gets diluted.
Looking back, when I reported 2.58x I had picked the most favorable side of all three axes:
| Axis | What my number used | What I actually do |
|---|---|---|
| Content | Code questions | Mostly prose Q&A |
| Prompt length | A 33-token prompt | 8112 tokens injected every turn |
| Concurrency | 8 | 1 |
All three land on the side with the least gain, and multiplied together that's "feels the same." The switch did pay off on code: 185 in the client, versus a bit over 140 hitting the server directly before the switch. I just don't ask the client many code questions.
The next optimization fell out of this too, and it's on the client side: those 6603 tokens of tool definitions have nothing to do with the engine, and trimming them would move time-to-first-token more than another week of engine tuning.
4. Taking apart a number on a model card
The draft model in the new config comes in two versions, one at full precision and one with 4-bit weights (W4A16). The model card says the 4-bit version reads 2.7GB less per step with the same acceptance rate. Reading less means going faster, or so I assumed at the time.
I did a quick calculation at the time and thought it was rigorous:
upper bound on time saved = bytes no longer read ÷ memory bandwidth
2.7 GB ÷ 1133 GB/s ≈ 2.4 ms per step
The measured step time was 17.6 ms. Subtract 2.4 and I predicted 214 tok/s.
Measured: 185.5 with the full-precision draft, 193.3 with the 4-bit one. 4.2% faster. Each step saved about 0.6 ms, a quarter of my prediction.
Two mistakes, both worth remembering.
First, that division only covers one stage: reading weights. A forward pass of the draft model also includes its own attention, kernel launches, CUDA graph replay, and picking the top candidates. None of that gets faster when the weights get smaller. Dividing one stage's theoretical saving into the whole step's time assumes that stage is the whole step, which is only true when it actually is.
The second mistake matters more. The model card's number was measured on a 24GB consumer card. A 27B target plus a full-precision draft model doesn't fit in 24GB, so the full-precision draft has to be shuffled in and out, and the 4-bit version mostly saves that shuffling. It saves capacity. My card has 64GB and holds both comfortably. There's no shuffling to save, only the small read-bandwidth difference.
Acceptance length barely moved (3.273 vs 3.296 tokens accepted per round on average), which matches the model card. The card wasn't lying. Its benefit mechanism just doesn't exist on my hardware.
So when someone reports "switching quantization format made it X% faster," my first question is: does it save bandwidth or capacity? A capacity saving drops to zero on a machine that isn't short on capacity. For a bandwidth saving, compute the upper bound with the division above, then discount it by how much of the step that stage actually takes.
5. Where these numbers can't be trusted
I got the bandwidth constant wrong. The 1133 GB/s in section 4 came from a single run with a power cap and no warmup. A careful re-measurement later gave 1674. With 1674, the theoretical saving from reading 2.7GB less is 1.6 ms against 0.6 ms measured. The conclusion points the same way, but my 214 prediction was built on a wrong denominator.
The 1.07x whole-turn figure is an estimate. The 1.11x is the probe's prose aggregate ratio. I never ran a paired measurement of the old config inside the client (it didn't occur to me to test the client before switching). Applying the probe's ratio to a client turn assumes the decode speedup inside the client matches the probe's, and I haven't verified that.
I don't have per-stream numbers for prose. The 1.42x per-stream speed in section 2 was only measured on code. For prose at concurrency 1 I only have the 1.11x aggregate ratio, and the true per-stream number may be a bit higher.
Historical data didn't match. The old config's launch script had a note recording a concurrency ladder from three days earlier: 94, 248, 398, 468, 478 at concurrency 1/4/8/16/32. On the same server this time I measured 122.6, 274.7, 554.3 at 1/8/32. Higher at both ends, lower in the middle, so not even the same direction. Most likely a different prompt mix and output length. I didn't chase it down; the verdict uses only same-day, same-probe paired data, and the old ladder is marked as not comparable.
I nearly used an even worse ratio. An earlier draft of the comparison divided a new-config number measured at 65k context, single concurrency and non-streaming timing (172.3) by an old-config number from a server with a 64-request ceiling and streaming timing (140.9), and got 1.22x. Four variables differed at once, so that number was useless. The tables above hold all four fixed.
Quality came out even. 169 questions, same base weights, temperature 0: 8.74 overall versus 8.79, paired t of 0.21, 95% confidence interval of ±0.449, well inside noise. I used the 40 questions where both configs gave byte-identical answers to calibrate the judge's noise; the mean score difference there was -0.05 (it should be 0 by construction). All I can say is that I didn't see it get worse.
6. Before you read the next speedup number
- Ask which cell it came from: what concurrency, what content, what prompt length. A speedup that names none of the three should be assumed to come from the best-looking cell.
- Don't take the operating point from the server config.
--max-num-seqsis a ceiling. Ask the users, or sample the distribution ofnum_requests_running. - At low concurrency read per-stream speed; at high concurrency read aggregate throughput. Print both columns and don't drop either.
- Take acceptance testing all the way to the client you actually use. Pull time-to-first-token and decode time from the client's own stats, and find out how it computes its speed first.
- Run
(time to first token + decode time × s) ÷ (time to first token + decode time)to see how much of the engine-side gain survives a whole turn. - If the client and a direct request differ by more than 2x, check how many tokens the client prepends every turn before you blame the engine.
- When a model card or someone else says "saves X GB, so Y% faster," ask whether it saves bandwidth or capacity, then compute the ceiling as "bytes no longer read ÷ bandwidth" and multiply by that stage's share of the whole step.
The 2.58x I tuned for lost, inside my own client, to a 6603-token list of tool definitions.