AriakiWho wrote the Bencao Gangmu, the classic Chinese pharmacopoeia? Three pruned builds of DeepSeek-V4-Flash, three wrong answers, three zeros. What was Qin Shi Huang's birth name? Three zeros again.
I treated those two questions as the smoking gun and wrote down my verdict: pruning isn't to blame, the base model is just weak.
Then I called the official API and ran the same question set. The base model scored 10 on both. Its total was 9.21, the highest number in my whole comparison table.
Along the way I ran two rounds of statistics. Round two overturned round one, and then the official API overturned round two. Round one happened to be right, but it was right by luck. This post is about why the round-two statistic could never answer the question I was asking, and how you can see that before you run it.
If you just want the test, it's in section 5, and it's a single question you ask yourself. If you want to know how to read the benchmark table on a pruned model card, that's section 6.
1. The question: did pruning cost 1.9 points, or is the model just like that?
DeepSeek-V4-Flash is a mixture-of-experts (MoE) model: inside it are many small "expert" networks, and each token only wakes up a few of them. This release has 256 experts and 155.4 GiB of weights.
I have two 64 GB cards. After runtime overhead, roughly 122 GiB is left for weights. The full model doesn't fit.
So the community made pruned versions. You score each expert by how much it matters to the output and delete the low scorers. I tested three public ones:
| Variant | Experts kept | Pruned |
|---|---|---|
| A | 178 | 30.5% |
| B | 160 | 37.5% |
| C | 144, plus 2-bit quantization | 44% |
I ran them on my own 140-question Chinese eval, split into four areas: hallucination resistance (ask about things that don't exist and see if it makes stuff up), knowledge, reasoning, and tool use. A large model acts as the judge and scores each answer from 0 to 10.
The three variants landed between 7.29 and 7.37 overall. Tool use was great, 9.1 to 9.5. Hallucination resistance was bad, 4.90 to 5.63. Another 35B model I'd run on the same set scored 8.20 on hallucination.
So, the obvious question: are these three bad because pruning broke them, or is DeepSeek-V4-Flash just this good and no better?
The clean answer is to run the full model too. It didn't fit. My first instinct was to design an indirect test instead.
2. Round one: the failures don't overlap, so I blame pruning
The idea is simple. If the base model has a blind spot, every variant cut from it should fail the same questions. If each pruning run lost knowledge at random, two variants should fail different questions, with about as much overlap as chance.
Overlap comes down to one division:
expected overlap = questions × (A's failure rate) × (B's failure rate)
enrichment = actual overlap ÷ expected overlap
Enrichment of 1 means "same as chance." Above 1 means failures cluster.
I counted a score of 3 or below as a failure. In the hallucination area (30 questions), variant A failed 15 and variant C failed 10:
expected overlap = 30 × (15/30) × (10/30) = 5
actual overlap = 5
enrichment = 1.00
Exactly 1.00, indistinguishable from chance. My call: no shared defect, pruning is the main cause.
That call turned out to be right. But my own notes from that day already said it only proved half the story. It ruled out "the base model has specific blind spots," and it couldn't rule out "the base model makes things up across the board." And it rested on a single pair of variants.
3. Round two: add a third variant, enrichment jumps to 2–3x, and I reverse myself
Variant B finished downloading, so I recomputed with all three. The expected three-way overlap is the product of all three failure rates:
expected 3-way overlap = questions × (fA/questions) × (fB/questions) × (fC/questions)
Hallucination, 30 questions, the three variants failed 15, 11 and 10:
expected = 30 × 0.500 × 0.367 × 0.333 = 1.83
actual = 4
enrichment = 2.18
Knowledge, 40 questions, failures of 8, 16 and 11:
expected = 40 × 0.200 × 0.400 × 0.275 = 0.88
actual = 3
enrichment = 3.41
Now the failures cluster. Three different pruning runs, different prune ratios, one of them quantized on top and served on a different machine, and they all trip on the same handful of questions. I wrote "random expert loss can't explain this" and reversed round one: the defect lives in the base model.
Then I looked at the questions all three scored zero on. On four of them, another model I use as a reference scored a perfect 10, including the two from the top of this post. The base model can't name the author of the Bencao Gangmu, and all three cuts agree? That felt solid.
I also did something dumb in between. To get around "doesn't fit," I turned on the inference framework's option to offload part of the weights to system RAM. The whole machine dropped off SSH and needed a hard power cycle. The log stopped while it was building the memory mapping, before reading the first weight file. That path is dead on that machine, and my own notes already said so.
4. One run through the official API: 9.21
In the end I just called DeepSeek's official API with the same 140 questions and the same judge:
| Official (256) | A (178) | B (160) | C (144) | |
|---|---|---|---|---|
| Hallucination | 7.90 | 4.90 | 5.53 | 5.63 |
| Knowledge | 9.80 | 6.40 | 5.47 | 5.90 |
| Reasoning | 9.44 | 8.69 | 8.84 | 8.80 |
| Tool use | 9.40 | 9.52 | 9.48 | 9.12 |
| Total | 9.21 | 7.37 | 7.29 | 7.35 |
Bencao Gangmu: 10. Qin Shi Huang's name: 10.
To check whether the gap is real, I used a paired test. For each question, take the official score minus the variant's score, then look at the mean and spread of those 140 differences:
t = mean difference ÷ (std of differences ÷ √questions)
Roughly, once |t| clears 2 you can treat the gap as real. The three variants:
| Mean diff | t | |
|---|---|---|
| Official vs A | +1.84 | 6.28 |
| Official vs B | +1.92 | 5.76 |
| Official vs C | +1.86 | 6.09 |
All three sit around 6, all pointing the same way. Split by area it's even cleaner. On knowledge the gap is 3.4 to 4.3 points per question, with t between 6.0 and 6.3. On hallucination it's 2.3 to 3.0, t between 2.9 and 3.5. Tool use is noise for all three, and one variant even edges out the official model.
Pruning costs about 1.9 points, almost all of it in remembered facts and in "don't invent what you don't know." Its ability to call tools came through the cut untouched.
Of those four "smoking gun" questions, the two knowledge questions got full marks from the official model. The other two were hallucination questions, and the official model got those wrong too. Those two are the only ones that count against the base model.
5. What went wrong: the null hypothesis was false from the start
The 2–3x enrichment was real. What I got wrong was what it meant.
Pruning ranks experts by importance, keeps the top ones and cuts the rest. All three variants use this same family of method. So the "unpopular experts" that rank low everywhere get cut in all three builds, and any question that depends on them fails in all three builds.
The clustering comes from the consistency of the pruning method itself. When I wrote "random expert loss can't explain this," that sentence was true: none of the variants ever dropped experts at random, and each of them dropped the same batch. My mistake was the next step, reading "not random" as "not caused by pruning."
Turned into a test, it's one question to ask before you run any statistic:
If the base model is fine, what will this statistic look like?
If the base model is flawed, what will it look like?
Same answer both times → the statistic can't tell the two apart. Don't run it.
Apply it here. Base model fine: all three variants cut the same experts by importance, and failures cluster. Base model flawed: failures also cluster. Both worlds produce the same reading, and I was using that reading to tell them apart.
The overlap test's null hypothesis is "each variant loses things independently and at random." Under a deterministic pruning method that premise was never true, so rejecting it told me nothing new.
Indirect statistics are good for falsifying things. Round one's 1.00 did at least kill the idea that the base model had fixed blind spots on those questions. What they can't give you is the direction of cause. Whenever a direct control is available, go get it: the official API, the unprocessed original, the previous release, a public baseline someone else already ran.
The bill this time: two rounds of statistics and one hard reboot, versus one official API key and about fifteen minutes.
6. Debunking someone else's numbers: the benchmark table on the model card
When I was picking a pruned build, one repo's model card had a benchmark table, and at first I quoted it straight:
| Experts | GSM8K | MATH-500 | HumanEval+ | MBPP+ | Mean | |
|---|---|---|---|---|---|---|
| Base | 256 | 0.9484 | 0.7060 | 0.8720 | 0.7407 | 0.8168 |
| Pruned to 178 | 178 | 0.9401 | 0.6880 | 0.8720 | 0.7407 | 0.8102 |
| Pruned to 132 | 132 | 0.9295 | 0.7140 | 0.8963 | 0.7593 | 0.8248 |
What I said at the time was "the 178 build only loses 0.66 points on average, so quality holds up." Then someone asked: why does the 132 build score higher than the base model?
When a harder cut ranks higher, don't go looking for an explanation. Compute the error bar first. Each score is "correct answers ÷ total questions," and its wobble is easy to compute:
standard error = √( p × (1 − p) ÷ questions )
HumanEval+ has only 164 problems. With p around 0.87, one standard error is 2.61 percentage points, and a single problem is worth 0.61 points.
The 132 build's three "wins":
- MATH-500 +0.80, 0.39 standard errors, 4 more problems solved
- HumanEval+ +2.43, 0.93 standard errors, also 4 more problems
- MBPP+ +1.86, 0.83 standard errors, 7 more problems
All inside one standard error. Noise. The only item that clears the noise is GSM8K, 1319 problems, down 1.89 points, which is −3.1 standard errors and 25 fewer problems solved. The single trustworthy signal in the table points down.
Be careful with the mean column too. It adds the 164-problem HumanEval+ and the 1319-problem GSM8K with equal weight, so the noisiest benchmark gets as much say as the most stable one.
And there's a bigger issue on top of that. All four benchmarks are math and coding. Look back at section 4: pruning cost only 0.6 to 0.8 points per question on reasoning and nothing on tool use, while all the damage sat in knowledge and hallucination. This table measures exactly the areas where pruning hurts least, so of course it says almost nothing was lost.
When you read a table like this, ask three things: how many problems per benchmark, how many standard errors each gap is, and whether it measures the thing you actually care about.
7. What's wrong with my own experiment
- The judge is a single LLM. A few days later I re-scored the official run with the same judge. Six of 140 scores changed, and the mean went up by 0.06. That doesn't touch a 1.9-point gap, but don't read anything into differences under 0.1 on this setup.
- The question counts are small. Thirty hallucination questions, forty knowledge questions. When the expected overlap is 0.88 or 1.83 questions, the enrichment ratio swings a lot. The "3 or below counts as a failure" cutoff was also my call. If you only count zeros as failures, the direction holds, but the expected counts drop to 0.3–0.4 and the ratios become unreadable.
- The official model and the pruned builds ran in different places. I can't see what precision or serving stack sits behind the official API, while the pruned builds ran on my own cards. In theory some of the 1.9 points comes from the serving environment. The shape argues against that, though: the gap sits in knowledge and hallucination while tool use doesn't move at all, which isn't what an environment difference would look like.
- Turning off thinking works differently. The parameter my local serving framework uses to disable the model's "think before answering" mode is silently ignored by the official API, which still returns thinking tokens. I hit this during the comparison and had to switch to the API's own setting to line them up. If you compare an official API against a local deployment, check that the thinking-token count in the response is actually zero.
- Variant C adds 2-bit quantization and ran on another machine. That's two extra variables, but it ties the other two. All pairwise tests between the three variants are noise (B vs A: −0.09, t = −0.28).
- This is one base model. "Pruning hurts knowledge, leaves tool use alone" is something I've verified on this model only. Don't assume it carries over to other MoE models.
8. Next time you need to know "did the processing break it, or was the original like that?"
- First ask whether a direct control is available. An official API, the unprocessed original, the previous release, someone's public baseline. A few dollars of API calls against a few hours of inference is almost always the better trade.
- Before running any indirect statistic, write down two predictions: what the number should be if hypothesis A is true, and what it should be if B is true. If they're the same, don't run it.
- When the processing is deterministic (keep the top K by importance, same calibration data, same quantization scheme), don't use "several processed versions fail together" as evidence against the original. They were always going to fail together.
- Use indirect statistics only to falsify and to narrow the search. Report them as "this data is consistent with X, Y isn't ruled out yet, and telling them apart needs Z." Don't report "the conclusion is X."
- Before quoting the benchmark table from a pruned or quantized repo, compute a standard error for each item, convert percentages back into problem counts, and confirm it measures the area you care about.
- Before trying to cram a model that doesn't fit onto local hardware, check whether your own notes already say that path is dead.