AriakiBefore I ran a single request, I wrote myself a rule: this model is split across three GPUs in a pipeline, and if 8 concurrent requests don't give me at least 3x the aggregate throughput of one request, the setup isn't worth it and I go back to the drawing board.
First result: 60.3 tok/s for one request, 171.2 tok/s for eight. That's 2.84x. Short by 0.16.
By my own rule, that was a death sentence. But something bugged me. The whole point of feeding a pipeline concurrent requests is to fill its idle time, so why did 8 requests and 16 requests come out identical? I went back and measured differently. Same machine, same model, same 8 requests: 606 tok/s aggregate, 4.31x. Not one line of config changed in between.
Later I also pulled 72 and 359 out of that same machine. All four numbers are real measurements, all at 8-way concurrency. This post is about what each of them actually measured.
If you just want the formula, it's in section 2. If you want to see how much of a benchmark number can come from what the model is writing, section 4 has a result that surprised me too.
1. The 2.84x that almost killed the plan
The hardware: three mining-edition datacenter cards, software-unlocked to 64GB each, and each one hanging off a PCIe 2.0 x4 link, which is painfully narrow. On top runs DeepSeek-V4-Flash: 284B total parameters, 13B active per token, 43 layers split 15/16/12 across the three cards. That's pipeline parallelism: card 1 computes the first 15 layers and hands the result to card 2, which hands it to card 3.
With a single request, that pipeline is mostly empty. While card 1 works, cards 2 and 3 wait. That idle time is called a "bubble", and the whole reason to run concurrent requests is to stuff other requests into the bubbles so all three cards stay busy. So I expected scaling close to 3x at minimum.
My first measurement was casual. Each request carried a 352-token prompt, the model answered normally with max_tokens=256 and stopped whenever it was done. Fire 8 at once, divide total output tokens by wall time.
N=1 76 tokens out wall 1.26s aggregate 60.3 tok/s
N=8 486 tokens out wall 2.84s aggregate 171.2 tok/s
N=16 906 tokens out wall 5.27s aggregate 171.9 tok/s
From 8 to 16 it doesn't budge. 171 looked like a wall. My conclusion at the time: aggregate throughput tops out around 170.
The problem is that 486. Eight requests produced 486 tokens between them, about 61 each. The model said a short sentence and stopped. A big chunk of those 2.84 seconds had nothing to do with generating tokens.
2. One division: what's inside the wall clock
Aggregate throughput is a division. Total output tokens on top, wall time on the bottom. All the trouble is in the denominator. Break it apart:
wall ≈ total prompt tokens ÷ prefill speed + longest output ÷ per-request decode speed
aggregate = N × output tokens per request ÷ wall
Prefill is the phase where the model reads the whole prompt before emitting anything. The first term is reading time, the second is writing time. The decode throughput you think you're measuring lives only in the second term. The bigger the first term and the shorter the second, the less your number has to do with decode.
I'd measured this machine's prefill speed separately: one 200K-token request, 3,005 tok/s. Plug it in:
The 171 run (short prompts, natural stop): 8 × 352 = 2,816 prompt tokens, ÷ 3,005 = 0.94 s. Measured median time-to-first-token: 0.974 s. Matches. A third of the 2.84 s wall went to reading, and in the rest each request wrote an average of 61 tokens.
The 72 run (long prompts from the same sweep, 5,634 tokens each, also natural stop): 8 × 5,634 = 45,072 prompt tokens, ÷ 3,005 = 15.0 s. Wall was 17.77 s, so 84% of it was reading. That run's 72.03 tok/s is basically a prefill measurement. Decode barely shows up.
The 606 run: each prompt is one sentence, plus ignore_eos and min_tokens=512, which forces every request to keep going for exactly 512 tokens. Prefill is near zero, first token at 0.23 s. At 8-way, median per-request decode was 79.45 tok/s:
predicted wall = 0.23 + 512 ÷ 79.45 = 6.67 s
predicted aggregate = 8 × 512 ÷ 6.67 = 614 tok/s
measured = 4096 ÷ 6.76 = 606 tok/s
Off by 1.3%. Ninety-five percent of the wall was writing. That's the thing I actually wanted to measure.
Single-stream works the same way. Forced to 512 tokens, N=1 gives 140.6 tok/s, N=8 gives 606, 4.31x. It clears the bar. Same rule, different measurement, and the verdict flips from "not worth it" to "worth it".
Put plainly: the first version measured how quickly a pile of requests finished talking, and that depends mostly on how much they planned to say.
3. 171 → 359 → 606, in two cuts
But the full 3.54x from 171 to 606 isn't all about the stopping rule. When I went back over it, I realized two things had changed between the first and third runs: how requests stop, and what they're about. The first was a knowledge-base Q&A answer in Chinese. The third was "write some small Python functions".
Luckily I'd also run forced-512 Chinese prose that day (explain how Transformer self-attention works), which lands right in the middle:
natural stop · Chinese Q&A 8-way 171.2 median per-request decode 40.95
forced 512 · Chinese prose 8-way 359.3 median per-request decode 47.25
forced 512 · code 8-way 606.1 median per-request decode 79.45
Rows one and two differ by only 15% in per-request decode, so the content behaves about the same. Aggregate differs by 2.10x. That cut is mostly the stopping rule.
Rows two and three stop the same way. Aggregate differs by 1.69x. That cut is entirely content.
2.10 × 1.69 = 3.54, which is exactly 606 ÷ 171.
I'd assumed the stopping rule was the whole story. It's about half. The other half was hiding in something I hadn't even treated as a variable: what the model was writing.
4. Why code runs 1.7x faster than prose
Same model, same cards. Why would code be faster?
Because speculative decoding was on. A small draft model guesses the next few tokens, the big model checks them all in one pass, keeps the ones that are right, and redoes from the first miss. Every correct guess is free throughput.
Code is easy to guess. Indentation, brackets, def, return, type hints: very formulaic. Open-ended Chinese prose is hard to guess, because there are ten ways to say the same thing. I measured acceptance rates separately that same day: about 68.9% for code and JSON, about 30.5% for open-ended Chinese prose.
The cleanest evidence comes from turning speculative decoding off and measuring again:
single-stream decode, spec off spec on
code 54.04 143.01
Chinese prose 52.66 115.30
With speculation off, code and prose are nearly identical, 2.6% apart. With it on, they're 24% apart. Every bit of the content effect on speed comes in through the draft model's hit rate. The big model underneath works just as hard on code as on prose.
Concurrency widens the gap:
aggregate tok/s spec off spec on speculation multiplier
code N=1 53.0 140.6 2.65x
code N=8 199.8 606.1 3.03x
prose N=1 52.3 113.8 2.18x
prose N=8 228.3 359.3 1.57x
For code, the gain from speculation grows with concurrency. For prose it shrinks. The table also separates batching from speculation: with speculation off, code goes from 53.0 at 1 request to 199.8 at 8, so batching alone gives 3.77x; at a single request, speculation alone gives 2.65x. The two mostly act independently.
3.77x is more than the 3x I expected. My guess is that besides filling bubbles, eight requests sharing one read of the weights saves a lot too. I didn't verify that separately.
Back to the rule I started with. If I'd tested prose, with speculation on, forced to 512 tokens, 8-way would have been 359.3 ÷ 113.8 = 3.16x, a narrow pass. If I'd tested short naturally-stopping answers, 2.84x, a fail. Same machine, and the verdict depended on which prompt I happened to grab that day.
5. Questions to ask of any concurrency benchmark
When someone posts "8 concurrent requests, X tok/s aggregate", here's what I ask first:
Fixed-length output, or natural stop? A natural-stop number measures how long the answers to that particular prompt set are. Shorter answers mean time-to-first-token and the tail eat a bigger share, and aggregate drops. Flip it around: pick prompts with long answers and the number looks great. If this isn't stated, the number can't be compared to anything.
How long are the prompts? On my machine, one-sentence prompts with forced length give 606 at 8-way. 5,634-token prompts with natural stop give 72. That's 8.4x. Both prompt length and stopping rule changed there, but by the section 2 arithmetic, 84% of the 72 run's wall was reading, so prompt length is the bulk of it. Use a one-sentence-prompt throughput number to size an agent that stuffs 5K tokens of context into every call and you'll be off by an order of magnitude.
What's the content, and is speculation on? Any number with speculative decoding on has to come with a content type. Code 606, prose 359, same configuration, 1.69x apart. If only one number is reported, it's probably the flattering one.
How was "aggregate" computed? One method is to add up each request's self-reported decode speed. For my 16-way prose run, that gives 840.1 tok/s. Total tokens over wall time gives 401.3. That's 2.09x apart. The server only admits 8 at a time, so the second 8 sit in a queue and median time-to-first-token climbs to 4.72 s, but queue time doesn't count toward "each request's own decode speed". Everyone reports their own full-speed number and the sum is inflated. I covered this trap in another post; the short version is: trust only total output ÷ wall time.
6. Where my numbers are soft
- Almost every point is a single run. The only repeat was code at N=1: aggregate 140.55 and 133.27, about 5% apart, mostly because time-to-first-token was 0.062 s once and 0.316 s the other time; decode itself was 143.01 vs 145.24, only 1.6% apart. Single-stream aggregate numbers are inherently jumpy, so discount the scaling ratios a bit.
- The two-cut decomposition in section 3 has a gap: the natural-stop run used Chinese Q&A prompts and the forced-length run used Chinese prose, not the same prompts. I'm arguing the content is similar because per-request decode is close (40.95 vs 47.25), which is indirect evidence. The rigorous version runs the same prompts once with natural stop and once forced.
- The 3,005 tok/s prefill speed comes from a single 200K-token request. Using it to estimate reading time for 8 concurrent short prompts is a different shape of workload. The 171 run lines up with measured first-token latency; for the 72 run I have no independent first-token check.
- The 68.9% / 30.5% acceptance rates came from a different prompt set on the same day, not exactly the prompts in this post.
- The spec-off runs only cover code at 1/2/4/8 and prose at 1/8. The middle prose points are missing.
- The "shared weight reads" explanation in section 4 is a guess.
- One machine, one model, temperature=0. On cards with normal PCIe bandwidth the bubble fraction changes, so don't copy these scaling ratios.
7. Next time you measure concurrent throughput
- Run a fixed-length set first:
ignore_eos+min_tokens+ short prompts. That's your decode ceiling. Compare everything else against it. - Work out how much of the wall is reading:
N × prompt length ÷ prefill speed. If it's over 30%, you're no longer measuring just decode. - Then run your real workload: real prompt lengths, real stopping behavior. That's what you'll actually get, and its gap from step 1 is where your optimization budget should go.
- With speculative decoding on, test at least code and prose separately, then turn speculation off and test again, to check whether the whole gap comes from hit rate.
- Compute aggregate only as total output tokens ÷ wall time. Never sum self-reported per-request speeds.
- Any pass/fail threshold you set before testing should state which throughput definition it uses. My 3x rule didn't, and it nearly killed the whole setup on the least appropriate definition available.