AriakiThree GPUs were already serving a 284B-parameter model, and I wanted to know how much faster a fourth would make it.
Before touching anything I wrote down a prediction: by card count, +33% throughput. Then I compared against the four-card number an open-source project had published. 0.90x. Adding a card made it 10% slower.
I had an explanation ready on the spot. Pipeline parallelism gets one stage longer, so the serial path gets longer too. 33% more compute, 33% more road, net zero. Sounded great.
I made that mechanism up. Worse, on that very same machine, running the exact same config six times in a row gave throughput anywhere between 366 and 561. A ruler that can't measure itself consistently has no business being held up against someone else's four-card rig. That 0.90x told me nothing.
If you just want the formula, section 3 is one division that tells you how large a difference your ruler can see. If you want to watch the new ruler lie with its highest reading, that's section 6.
1. The machine and the wall
Three mining-edition datacenter GPUs, software-unlocked to 64GB each, running DeepSeek-V4-Flash on vLLM. The 43 layers are split 15/16/12 across the three cards. That's pipeline parallelism: card 1 computes the first 15 layers and hands the result to card 2, and so on.
I'd set a 200W power limit on these cards. Under load, all three spend 100% of their time pressed against it. In nvidia-smi, the throttle-reason bit for "software power cap" never turns off. SM clocks get held at 1418 MHz against a ceiling of 1695, so 19.5% of the clock is locked away.
Temperature is a non-issue: peaks of 66/55/63°C against an 85°C limit. The cards aren't hot. They're just starved of watts.
I wasn't going to touch the cap itself. I wanted to know whether any other knob inside the cap was worth turning: how many tokens speculative decoding guesses at once, how the layers are split. The expected effects of those changes were all in the 5% to 10% range.
2. Same config, six runs
Baseline first. Same config, change nothing, run it six times:
aggregate throughput: low 366, high 561
coefficient of variation (stdev / mean): 15%
and it's bimodal: one cluster decodes at 60.9 tok/s per stream, the other at 72.1, a 1.18x gap
The two clusters tell you more than the 15% does. Random noise smears out into a cloud. It doesn't pile up into two heaps.
The cause was the clock sinking over time. The three cards started a run at 1455/1410/1335 MHz and were down to 1335/1380/1245 after 37 seconds. Under a power cap, the hotter the card gets, the less frequency the same watts buy. Whatever thermal state the cards were in when a run started decided which cluster it landed in.
Put bluntly, half of what this ruler measured was my config. The other half was how long the machine had rested since the last run.
3. Before you run anything: how big a difference can this ruler see?
I never used to do this step. I'd just run things and look. It only takes one division:
smallest difference you can resolve with n runs per arm ≈ 2 × CV × √(2/n)
CV is the coefficient of variation from above. The √(2/n) comes from subtracting two means, since both arms carry noise. The leading 2 roughly corresponds to "95% sure this isn't noise."
Plug in my 15%:
1 run each: 2 × 15% × √2 ≈ 42%
5 runs each: 2 × 15% × √(2/5) ≈ 19%
With a single run each, two configs have to differ by 42% before I can call them different. Five runs per arm still only resolves 19%. The things I wanted to measure were 5% to 10%.
Solve it the other way to get how many runs you need:
runs per arm n ≈ 8 × (CV ÷ difference you want to detect)²
To detect 10%: 8 × (15/10)² = 18 runs per arm. To detect 5%: 8 × (15/5)² = 72 runs per arm.
That factor of 2 only guards against mistaking noise for signal. It doesn't account for missing a real difference. For proper sample-size planning these numbers need to go higher, so 72 is the optimistic estimate.
Now look again at that 0.90x from the opening. My machine ran at 200W. The four-card data ran at 180W, on a different machine, and it was a single number. Comparing one single measurement against another single measurement needs a 42% gap to mean anything by the math above. 0.90x is a 10% gap. Building a mechanism on it was writing a story about noise.
The irony is that the data to kill my mechanism was sitting in my own work log: at each pipeline boundary, decode moves 112 KiB per step, which is 1.15% of the time per token. One more pipeline stage adds one more hop of that tiny payload. It can't cancel 33%. I read that number when I wrote the log entry. I didn't read it when I drew the conclusion.
4. Fine, interleave: A/B/A/B
Running more trials assumes the trials are independent. Clock sag is monotonic. If I run A five times and then B five times, B inherits all of A's heat.
So I alternated the power cap between 200W and 180W, A/B/A/B, 64 concurrent requests per run, reading the setting back after every switch to confirm it actually took, and logging clocks throughout. This time the cap itself was the variable: if dropping 20W doesn't show up, comparing someone's 180W number against my 200W number means even less.
200W: 754.3 770.0 639.3 567.7 759.6 mean 698.2 CV 12.9%
180W: 630.1 572.6 710.8 595.3 746.0 mean 651.0 CV 11.5%
ratio 0.932 Welch t = 0.90, not significant
pairwise ratios: 0.835 0.744 1.112 1.049 0.982
Look at the pairwise row. Pool the first two pairs and 180W is 21% slower than 200W. Pool the last three and 180W is 4% faster.
If I'd stopped after the first two pairs, I'd have walked away with "dropping 20W costs 21%, same direction twice in a row." Self-consistent, replicated, and wrong.
Within each arm, the highest and lowest runs differ by 36% and 30%. Between arms the difference is 6.8%. Noise is about five times the signal. The clock telemetry shows the mechanism is real: average clocks of 1339 vs 1290 MHz, a 3.7% gap. It's just buried.
At that point the conclusion was clear. This machine's wall-clock noise floor sits around 35%, and three-to-four cards could give at most 33% in theory. On this machine, that comparison cannot be done with wall-clock time, no matter how many runs you throw at it. What I saved was several hours I would have wasted.
5. A ruler that doesn't care about heat
Wall-clock was out, but I still had other questions, like how many tokens speculative decoding should guess per step.
Speculative decoding means a small draft model guesses the next few tokens and the big model verifies them in one pass. Correct guesses are free tokens. One quantity here is the acceptance length: on average, how many tokens the big model keeps per verification. It depends only on how good the guesses are. Whether the GPU is running at 1245 or 1455 MHz doesn't enter into it at all.
vLLM periodically logs a line, SpecDecoding metrics: Mean acceptance length. I wrote a script that reads it before and after a workload and takes the delta, so earlier traffic doesn't contaminate the window.
Across windows on the same workload, acceptance length had a CV of 0.9% to 4%. Back into the section 3 formula:
CV 4%, 1 run each: 2 × 4% × √2 ≈ 11%
CV 0.9%, 1 run each: 2 × 0.9% × √2 ≈ 2.5%
CV 4%, to detect 5%: n ≈ 8 × (4/5)² ≈ 5 runs
To see a 5% effect, wall-clock wanted 72 runs per arm. Acceptance length needs five or fewer. In practice I had answers in twenty minutes.
This ruler has a hard limit, and it belongs up front: it only answers questions where the change affects how well the guesses land. The power cap changes clocks, not guesses, so acceptance length doesn't react to it at all. The question in section 4 is simply outside what this ruler can measure.
6. The highest reading was the worst config
Measuring "how many to guess per step" with the new ruler:
| Tokens guessed per step | Acceptance length | CV | Acceptance rate | Tokens returned per verified position |
|---|---|---|---|---|
| 5 (original) | 4.733 | 4.0% | 73.4% | 0.789 |
| 7 | 5.177 | 0.9% | 60.5% | 0.647 |
| 10 | 4.746 | 1.4% | 37.4% | 0.432 |
Look only at acceptance length and 7 wins with 5.18, and the gaps between neighbouring settings are significant (Welch t of −3.96 for 5 vs 7, +9.44 for 7 vs 10). Report that column alone and the recommendation is "switch to 7."
But the big model pays per position when it verifies. Guess k tokens and it processes k guesses plus 1 more position. So the comparison that matters is:
tokens returned per verified position = acceptance length ÷ (tokens guessed + 1)
5: 4.733 ÷ 6 = 0.789
7: 5.177 ÷ 8 = 0.647
10: 4.746 ÷ 11 = 0.432
5 beats 7 by 22% and beats 10 by 83%. Seven slots buy back only 0.44 more tokens and cost two more positions.
The acceptance-rate column says the same thing: 73% → 60% → 37%, falling monotonically. After the fifth token, the draft model is mostly guessing blind. 5 sits right at the edge of what it can do.
One more trap: this draft module won't let you go lower. I tried 3 and vLLM refused to start, complaining that the number of guessed tokens can't be smaller than the draft block size of 5, or the output will be wrong. So "guess fewer tokens on prose" isn't an option. You can only go up, and going up loses.
7. Suspiciously low variance is a signal too
Two more experiments ran on wall-clock. The noise was large, but they weren't worthless.
Layer split. My guess was that the third card, which drew the most power (219W), would go faster with fewer layers. To get all three splits to start at all, I shrank the context window to 128k and compared:
15/16/12: 487.7 ± 42.0
16/16/11: 488.3 ± 90.9
14/16/13: 470.2 ± 79.3
Welch |t| ≤ 0.39 for every pair
Indistinguishable. The more interesting part is what happens at the full context window: the other two splits don't start, failing with not enough KV cache. Besides its own layers, the third card also holds the roughly 12 GiB draft model plus the output layer, so adding layers there overflows it. Adding layers to the first card doesn't work either, because the KV cache pool is sized by whichever card is most crowded. The real value of the original 15/16/12 split is that it's the only one that can open the full window; speed is secondary. An experiment that refuses to start is itself a result.
The other knob was the logits computation chunk size (DSV4_LOGITS_ROW_CHUNK, an environment variable this model's implementation ships with):
128 (original): 500.3 ± 50.2 CV 10.0%
64: 453.2 ± 6.9 CV 1.5%
256: 501.3 ± 56.8 CV 11.3%
No t-test came out significant. But 64 is odd. All four runs landed between 446 and 462. The other two settings had CVs above 10% and occasionally spiked into the 570s, and 64 never did. Lowest mean, tightest spread: that looks like "consistently slow" more than bad luck. I lean toward calling it genuinely slightly worse, but it's a lean.
256 couldn't be told apart from 128 on speed, but it drew noticeably more power at idle (84.3W vs 55.2W on one card). On a machine already pinned against its power cap, burning extra watts while idle is a real cost.
8. Using this on other people's numbers
"I changed X and got 8% faster." Ask how many runs and what the CV was. If the author never measured how much the same config wobbles run to run, that 8% can't be read. Plug into 2 × CV × √2: once CV goes above 3%, a single-run 8% sits inside the noise. Laptops, mini PCs and mining cards that live against a power or thermal cap easily hit 10% CV.
"Acceptance length 5.2, beats the other one's 4.7." High acceptance length often comes from guessing more. Always ask how many tokens are guessed per step, then compute acceptance length ÷ (tokens guessed + 1) yourself. In my table, the highest reading was the worst deal.
"I added a card, and look, their four-card number is only 0.9x our three-card one." Same power cap on both sides? How many runs each? On my machine, just alternating 200W and 180W, the first two pairs "proved" a 21% slowdown. Cross-machine, cross-power-cap, one number each: treat it as rumor.
9. What this post didn't finish
- "Per verified position" is a proxy. I assumed verifying k+1 positions costs proportionally to the number of positions. Decode is often dominated by reading weights from memory, so verifying 8 positions may cost little more than verifying 6, with things only becoming close to linear as concurrency rises. I never measured end-to-end wall-clock for guessing 7, because wall-clock couldn't measure it reliably. So "5 beats 7 by 22%" holds under this proxy, and it also leaves out the draft model's own compute.
- The 2 in section 3 is an approximation. It ignores statistical power and assumes runs are independent. Clock sag makes neighbouring runs not independent, so the real noise floor is only worse than the calculation says.
- This post doesn't answer how much a fourth card helps. It only shows that wall-clock on this machine can't answer it. Answering it means either lifting the power cap or finding another quantity that ignores clock speed, and I've done neither.
- That 15% was measured on one config under one workload. A different workload shape may have a different CV and needs measuring again.
- "Consistently slow" for chunk size 64 is my interpretation. Its t value was −1.84, short of significance.
10. Next time you compare two configs, do this first
- Change nothing, run the same config five or more times, compute the CV. Check whether the numbers form one cloud or two clusters. Two clusters means go find what's grouping them: clocks, temperature, caches, how long the machine rested since the last run.
- Use 2 × CV × √(2/n) to compute the smallest difference this ruler can see, and compare it against the effect you expect. If the expected effect is smaller, don't start.
- To detect a given difference, estimate runs per arm with 8 × (CV ÷ difference)², multiply by the time per run, and see if you can afford it.
- If you can't, find a reading that the noise source doesn't touch. If clocks are dragging you around, look for an algorithm-level quantity, and first confirm the question you're asking can actually be answered by it.
- Don't read just one column on the new ruler either. Any metric you can raise by spending more, divide by what you spent before comparing.
- Always interleave A/B runs, read the setting back after every switch, and log clocks or temperature alongside.
- Don't draw conclusions from cross-machine, cross-power-cap, one-number-each comparisons. If you really can't resist, put your own machine's same-config run-to-run spread right next to it.