AriakiSame 64 GB card, same load-test script, four days apart.
First round was a 27B dense model. Official 8-bit weights beat 4-bit cleanly — faster decode and a bigger context cache. I wrote it down: on this card, pick official 8-bit.
Second round was a 35B MoE. I expected a formality. Instead, 8-bit still decoded 9.8% faster on a single stream, but started losing the moment concurrency hit 8, and was down 22.5% at 32. The version that went to production was the one that's slower single-stream.
Nothing went wrong in between. Both conclusions are correct — the first one just doesn't transfer to the second, and I nearly transferred it. This is where that gap comes from, how to compute it with one division, and the mistake I made in my own measurements.
Want just the test? Section 2, it's a division. Want to see me pick holes in my own numbers? Section 5.
1. What actually differs between these two tables
Quantization stores model weights at lower precision — 8-bit takes a byte per weight, 4-bit takes half, so the model halves in size at some cost to accuracy. Intuitively it's a "trade memory for quality" slider: lower bits, less memory.
On the 27B dense model, that intuition held perfectly:
| Official 8-bit | 4-bit | |
|---|---|---|
| Size | 29 GiB | 26 GiB |
| Single-stream decode | 72.7 tok/s | 69.6 |
| Context cache | 296,766 tokens | 335,717 |
Only 3 GiB apart, 8-bit decodes faster, and those 3 GiB don't buy much cache. Pick 8-bit, no hesitation.
The 35B MoE, same card, same settings:
| 4-bit | Official 8-bit | |
|---|---|---|
| Size | 24 GiB | 35 GiB |
| Single-stream decode | 147.1 tok/s | 161.5 |
| Context cache | 1,531,743 tokens | 925,214 |
| 8-way aggregate | 741.7 tok/s | 627.9 |
| 32-way aggregate | 1838.3 | 1501.2 |
The size gap went from 3 GiB to 11 GiB, and the context-cache winner flipped.
MoE means "mixture of experts" — the model is split into many expert sub-networks, and each token only wakes up a few of them. This one has 256 experts and activates 8 per token. Every expert's weights must sit in memory ready to go, but only 1/32 of them move at any moment.
That's the whole story.
2. The division: total params set capacity, active params set speed
Decode speed is a data-movement problem. Every token you emit requires reading the weights you're about to use out of memory, so what matters is active parameters.
Context cache capacity is a leftover-space problem — how much room remains after weights are loaded — so what matters is total parameters.
On a dense model those are the same number, so dropping bit-width makes decode faster and the cache bigger, together. MoE splits them apart:
decoupling ratio ← total params ÷ active params per token
Dense 27B: 27 / 27 = 1
MoE 35B-A3B: 35 / 3 ≈ 11.7
Experts alone: 256 / 8 = 32
The higher that ratio, the further apart "bit-width buys size" and "bit-width buys speed" drift.
To convert a size delta into context, use this:
cache cost per token ← size delta ÷ difference in cached tokens
Plugging in the measurements:
11 GiB × 1048576 ÷ (1531743 − 925214) = 19.0 KiB/token
This isn't a heuristic, it's a division. Run it backwards and you learn what those 11 GiB are worth:
11 GiB ÷ 19.0 KiB = 606,529 tokens ≈ 2.3 full windows
The 11 GiB that 8-bit costs in size equals more than two complete long-context windows. What it buys back is 9.8% on a single stream — which is not where production lives.
Run the same division on the 27B dense round: 88.1 KiB per token, so a 3 GiB size delta only costs 3.24 GiB of cache, roughly one-to-one. That's why 8-bit won there — its bit-width advantage wasn't eaten by a size penalty.
3. A check on whether you did the arithmetic right
That 19.0 KiB was derived from the difference between two versions. If I'd mistyped one input, I'd never know. So here's a cross-check: run it forward and confirm the leftovers match.
The memory budget is 90% of 64 GiB, i.e. 57.60 GiB.
4-bit : 57.60 − 24 weights − 27.78 cache = 5.82 GiB
8-bit : 57.60 − 35 weights − 16.78 cache = 5.82 GiB
Both leave 5.82 GiB. That's activations, graph capture, and similar fixed overhead — independent of weight precision, so it should match. It matching means the 19.0 KiB figure holds, and that both versions really ran under identical memory settings.
Worth making a habit: any constant you back out of a difference, run it forward and check the residual. If the residuals disagree, either an input is wrong or the two datasets weren't measured the same way.
4. Reading someone else's quantization benchmark: ask how many streams
Almost every published quantization comparison gives you one "decode tok/s" number. By default that's single-stream — one request, whole card to itself.
Watch where this curve crosses:
| Concurrency | 4-bit | 8-bit | 4-bit advantage |
|---|---|---|---|
| 1 | 147.1 | 161.5 | −9.8% |
| 8 | 741.7 | 627.9 | +18.1% |
| 32 | 1838.3 | 1501.2 | +22.5% |
| 128 | 3432.9 | 3016.0 | +13.8% |
The crossover sits between 1 and 8. Whoever wins at one stream loses at eight and above. Time-to-first-token points the same way: at 8 streams it's 0.10s for 4-bit versus 0.59s for 8-bit.
So when you read a quantization benchmark, ask two things: how many concurrent streams, and how many tokens of context cache are left. A comparison reporting only single-stream decode will reliably steer you to the wrong side on MoE models — the bit-width benefit sits entirely on the single-stream side, and the cost sits entirely on the concurrency side.
If the benchmark only gave you single-stream numbers, don't discard it. It still told you the model size — use the division from section 2 to fill in the missing half yourself: size delta ÷ per-token cost tells you how much context it traded away.
5. Three problems with my own measurements
The numbers above look tidy, so here are three things I need to put on the table, or none of them deserve your trust.
First, I built one comparison out of two different cards. To save time I ran the two versions in parallel on separate cards. That introduces a variable: the cards sit at different thermal states and clock down differently.
The fix was a swap test:
| Card 0 | Card 1 | Card 2 | |
|---|---|---|---|
| 4-bit | 147.1 | — | 145.5 |
| 8-bit | 161.5 | 162.5 | — |
Card-to-card spread ≤1.1%, quantization effect 9.8% — nearly ten times larger, so the attribution holds.
But my original notes said "8-bit wins single-stream by 10.5%" — computed from 8-bit's best card (162.5) against 4-bit's worst (145.5), cherry-picking high on one side and low on the other. Same-card is 9.8%. This post uses same-card figures throughout. A 1.1% spread doesn't change which way the conclusion points, but quoting a delta you never measured on one card is a habit worth breaking. Same issue in the prefill row: 6.6% across cards, 8.9% on the same card.
Second, parallel measurement only works for some metrics. Decode parallelizes fine, because each card's memory and compute are its own. Concurrency sweeps don't — at 128 streams the bottleneck may be the host CPU, and running both sides at once has them fighting each other, making the numbers incomparable. Those ran serially.
The rule: before parallelizing a measurement, ask whether the denominator of this metric is shared. GPUs are exclusive, so parallel is fine. Host CPU, network, and disk are shared, so those must run serially.
Third, I can't claim the quality difference. Across a 140-question, four-domain evaluation, 4-bit averaged 9.35 and 8-bit 9.49 — a 0.14 gap. 124 questions scored identically; of the remaining 16, 8-bit took 11. Paired t = 1.49, and the 95% confidence interval crosses zero, so it isn't significant.
The accurate phrasing: this round didn't detect a quality difference, which is not the same as showing they're equal. At 140 questions, a 0.14 gap is below the resolution of the instrument. Settling it needs more questions.
And one thing I'd believed that turned out wrong. A widely-cited writeup claims community 4-bit quantization crushes the speculative-decoding draft head along with everything else, effectively disabling it. This 4-bit checkpoint does have its draft head fully quantized to 4-bit, so by that account it should have been useless. Measured acceptance length: 2.23, against 2.27 for 8-bit — 1.8% apart, with matching per-position acceptance shapes. It didn't break.
The test needs amending: checking whether draft-head tensors exist is still necessary (zero tensors = disqualified, an existence check), but once they exist you can't assume "4-bit ones are dead." That's a quality claim, and quality claims need measurement.
6. What I haven't done
For honesty, here's what's missing before I'd state any of this firmly:
- Quality was only measured on 140 questions, which can't resolve a 0.14 gap. Settling it needs more.
- Concurrency only swept to 128, short of the saturation knee. Who wins above that is unknown.
- Only one MoE model and these two checkpoints were tested. The 11.7 decoupling ratio belongs to this model; compute your own.
- Context cache token counts are the capacity the engine reports at startup, not a measured fill-to-failure.
7. Next time you're picking a quantization, ask in this order
- Is it MoE? Check the config for expert count and experts-per-token.
- If so, compute the decoupling ratio: total params ÷ active params per token. Near 1 means dense, and prior results on the same architecture transfer. Far above 1 means re-measure — the higher, the less transferable.
- Compute per-token cache cost: size delta ÷ difference in cached tokens.
- Use it to convert the size delta into context tokens, and see how many full windows it trades away.
- Sweep concurrency out to the levels you'll actually run, and find the crossover. Don't decide on single-stream.
- Benchmark both versions on the same card. If you can't, measure card-to-card spread first and confirm it's an order of magnitude below the effect you're chasing.
- Compute a confidence interval on the quality gap. If it crosses zero, write "no difference detected," not "it won."
That card is running 4-bit now: 9.8% slower single-stream, 600,000 more tokens of context. The conclusion I'd written four days earlier on the same card saved it once — because I didn't copy it over.