AriakiTwo model judges, the same 961 answers, the same four-way question. One of them never picked a particular option. Not once in 961 tries.
Not because the option wasn't scored. I went back to the raw logprobs: all four candidates were present every single time, that option's probability peaked at 0.3181, and it came in second 24 times. It just never won.
And I was about to use these two judges to measure an effect of roughly one percentage point.
Want the criterion directly? Chapter 2, it's one division. Want to know how I tripped over my own method? Chapter 6.
1. What I was doing, and why it's worth writing down
I'm building a "difficulty router" for my assistant: a question comes in, something decides whether it's hard, easy ones go to a small local model and only the hard ones go to the cloud. To build that, I first need to know where the local model actually falls over.
Hence this dataset: 961 real conversation turns, each with an answer from a cloud model and an answer from a local 35B model. Both answers go side by side to a judge, which answers the same four-way question —
Both fine / only A is fine / only B is fine / neither works.
There are two judges. One is a commercial cloud classifier API. The other is that same local 35B scoring itself by logit readout — instead of letting it write prose, you look at the score it assigns each of the four digit labels and take the highest. Which answer sits in slot A and which in slot B is randomized, same seed for both judges, so the A/B positions line up across all 961 items.
In theory, the two judges should flag roughly the same set of "the local model blew this one."
2. The noise floor is one division
Here's what each judge flagged:
| Called the local model a failure | Share | |
|---|---|---|
| Cloud judge | 102 | 10.6% |
| Local 35B judge | 161 | 16.8% |
| Intersection | 47 | — |
| Union | 216 | — |
Overlap: 47 ÷ 216 = 21.8%.
Is that high or low? On its own the number means nothing. You need a baseline for "what would random flagging look like." The baseline is a division:
expected random overlap = a × b ÷ N
Two judges flag a items and b items out of N total; if they pick independently at random, that's the expected intersection. Plugging in:
102 × 161 ÷ 961 ≈ 17.09 items
as an overlap rate ≈ 6.95%
Measured 21.8%, which is 2.75× the random baseline. So the two judges are looking at the same underlying thing. There's real signal, and I'm not going to argue "model judges are useless."
But "has signal" and "is usable" are different claims. What actually decides whether I can conclude anything is a different number:
noise floor = disagreements ÷ total = 169 ÷ 961 = 17.6%
On 169 of 961 items, the two judges returned opposite verdicts. That's the floor. The dimensions I wanted to measure — is the user's question long, is there prior context, is it a short follow-up — came out to about one percentage point.
A 1pp effect against a 17.6% floor. That isn't "the result wasn't significant." That's never having started the measurement.
3. How 82.4% agreement lies to you
961 minus 169 is 792, and 792 ÷ 961 = 82.4%.
The two judges agree 82.4% of the time. Put that number in a report and nobody stops you.
Here's the trick: most of the time both judges say "no failure." The cloud judge flags only 10.6%; the local one 16.8%. On the other eighty-odd percent they both say it's fine — and that agreement is free. Two judges guessing at random would also get it.
You can price the freebie:
chance agreement pe = p₁ × p₂ + (1 − p₁) × (1 − p₂)
Plugging in: 0.106 × 0.168 + 0.894 × 0.832 = 0.762.
So of that 82.4%, a full 76.2% is handed over by the base rate. Only about six points were earned. Net it out and you get kappa:
kappa = (0.824 − 0.762) ÷ (1 − 0.762) = 0.261
0.261. In the inter-rater agreement literature that range usually gets called "weak."
This section is usable as a weapon. Next time you see a report claiming "our two annotators agreed 90% of the time" or "the model matches human judgment 88% of the time," ask one question first: what's the base rate of the thing being labeled? If the positive class is 5% of the data, answering "no" to everything scores 95% agreement. Under class imbalance, a raw agreement rate carries almost no information. Ask for kappa, or at minimum make them print all four cells of the confusion matrix.
4. The biggest effect in the experiment was which slot the answer got pasted into
This is the part I didn't see coming.
Back to that "never picked once." Here's the local 35B judge's full distribution:
| Option | Local 35B judge | Cloud judge |
|---|---|---|
| Both fine | 623 | 672 |
| Only A is fine | 326 | 192 |
| Only B is fine | 0 | 88 |
| Neither works | 12 | 9 |
Whenever it thinks one of them is inadequate, the one it picks is always B. 961 trials, zero exceptions.
Since A/B assignment is random, this contaminates the failure set directly. Split by position:
| When A holds the local answer | When A holds the cloud answer | Gap | |
|---|---|---|---|
| Cloud judge calls local a failure | 7.2% | 14.2% | 7.0pp |
| Local 35B judge calls local a failure | 1.4% | 33.2% | 31.8pp |
Same model, same answers. Purely by being pasted into the second slot, its failure rate goes from 1.4% to 33.2%.
The dimensions I set out to measure are around 1pp. The position effect is 31.8pp. The largest signal in this experiment is the slot.
The two judges also overlap completely differently across the two halves: when A holds the local answer, overlap is 1/42 = 2.4%; when A holds the cloud answer, it's 46/174 = 26.4%. That headline 21.8% is essentially all coming from the second half.
Worth noting the cloud judge has a position preference too (7.0pp) — it's just not grotesque. It at least went both directions: 192 "only A" and 88 "only B." Unbalanced, but it didn't collapse to zero.
5. So how many samples do I actually need
Suppose I fix the judges and balance out the position bias. How many items do I need to measure a difference along some dimension?
standard error of a difference in proportions ≈ √(2p(1−p)/n)
smallest reliably visible difference ≈ 2 × SE
Using the cloud judge's 10.6% failure rate as p:
| Samples per group | Smallest detectable difference |
|---|---|
| 100 | 8.7pp |
| 240 | 5.6pp |
| 480 | 4.0pp |
All 961 items split in half gives me 480 per group, so the smallest difference I can see is 4.0pp. The dimensions I actually measured: user message length gave +1.0pp on the cloud judge and −0.8pp on the local one; presence of prior context gave −0.3pp on the local judge.
All of it below the floor. Those dimensions aren't "showing no effect" — this apparatus cannot see whether they have an effect. Those are entirely different statements.
6. Where this experiment doesn't hold up
Up front: every item below weakens the conclusions above. But without them, you should default to not believing any of these numbers.
The big one: I have no human labels. This whole piece compares two model judges, and the definition of "the local model failed here" was never validated by a person. Both judges could be confidently evaluating the wrong thing. The real gold standard requires me to hand-label, and right now that count is zero.
The no-context group is only 47 items. In chapter 5 the cloud judge shows a −11.2pp difference there, the largest number in the table, but the standard error on 47 samples is close to 6pp. I don't believe that number at all. It's listed only because it's one of the dimensions I ran, and hiding it would be dishonest.
The two judges don't even read the same way. The cloud one is a full API that actually reasons before answering; the local one is a single-token logit readout restricted to emitting one digit. Comparing them mixes in a mechanism difference, so you can't cleanly attribute the gap to "model capability." That zero might itself partly be an artifact of single-token readout.
One run, one prompt, one seed. The correct way to handle position bias is to run each pair in both orders and average; I didn't. I also only tried one wording of the four-way question, and the wording itself may be nudging toward "only A."
The 2.75× signal is real. Don't let the preceding paragraphs carry you too far — the two judges genuinely are correlated on the same underlying thing. That correlation just isn't strong enough to support conclusions at the 1pp level.
7. Next time you hit this, go in this order
- Compute the random-overlap baseline
a × b ÷ Nbefore you look at your measured overlap. An overlap rate with no baseline is a number without units. - Compute the noise floor,
disagreements ÷ total. Compare it against the effect size you want to measure. If your effect is smaller than the floor, stop and fix the judges. - When you see any "agreement rate," ask for the base rate, then compute
pe = p₁p₂ + (1−p₁)(1−p₂)and subtract out the free part. - For any side-by-side A/B judgment, count how often it picks A versus B. A collapse to zero means it's broken.
- When it does collapse to zero, don't jump straight to prompt edits. First confirm in the raw output that the option was actually scored — never selected and never in the candidate set are different diseases with different cures.
- If you're going to label, label the disagreements. My 792 items where both judges agreed are worthless to label; the 169-item disagreement set is where all the information lives.
- Before labeling, run
2 × √(2p(1−p)/n)to see what difference that sample size can even detect, then decide how many to do.
This batch took me all night to run, and the most useful thing it produced was learning that it can't measure what I built it to measure. One night in exchange for a "don't build this yet" beats pushing forward on category design while sitting on a 21.8% overlap.