Ariaki
/
Back to blog

2026.09.21

The model gave the same answer on all 20 frames. Accuracy: exactly 50%. Do you throw out the model, or throw out argmax?

A decision model that does one forward pass and reads the probability of two words scored like a coin flip on a binary task. Its AUC was 0.949 — the signal was there the whole time, drowned by a prior. Here are two pieces of arithmetic that measure the bias, one test for debunking somebody else's benchmark, and a checklist.

LLMevaluationlocal-inferencedecision-models

Twenty frames. The model answered twenty times. Same answer every time. Score: 10/20.

At that point any reasonable person closes the tab. On a binary task it's doing worse than a coin — a coin at least changes its mind occasionally.

Then I dumped the raw probabilities and plotted an ROC curve. AUC 0.949.

Those two numbers come from the same calls, the same model, the same frames. Here's how that happens, and why it will happen to you the first time you evaluate any "read the logits" decision model.

If you just want the test, jump to section 2 — it's one addition and one subtraction. If you want to see me faceplant, sections 3 and 5.

1. What's actually being measured

There's a trick that keeps getting packaged into open-source projects lately: don't let the model generate a sentence. Make it do exactly one forward pass, then read the probabilities of a few candidate words and pick the highest. Set max_tokens=1, ask for top logprobs, softmax over the letters A/B/C, done. It's fast because you never decode a single token.

I pointed it at something concrete: show the model one frame of that side-scrolling game where a bird flies through gaps between pipes, and ask whether it should flap right now or wait. Image only, no coordinates, no state vector. Two candidate words: flap and wait.

First, the mistake I made before any of the numbers below, because it matters more than they do.

My original question was "is the bird above or below the gap?" After debiasing, that scored 19/20 at 512×640, p50 87ms, with latency holding around 80ms across three resolutions. I was pleased with myself.

He looked at it and said: asking above-or-below means you did half the reasoning for it.

He's right. The actual task is to emit an action, not describe a position. The hop from "below the gap" to "so, flap" is the part I did in my own head and then handed to the model for free. If your eval asks about an intermediate conclusion, it isn't measuring the capability you care about. Rephrased as a bare flap/wait, it dropped to the 10/20 above.

2. Measuring the bias: one addition, one subtraction

I didn't declare it dead at 10/20. I looked at the logprobs:

  • On frames where the truth is flap, flap leads wait by +3.24 nats
  • On frames where the truth is wait, flap still leads by +0.59 nats

The second line is the whole story. When it should say wait, it says flap anyway, just less emphatically.

Split those two numbers in half and you get something you can compute on your own data:

offset     = (margin₊ + margin₋) ÷ 2
separation = (margin₊ − margin₋) ÷ 2

That's not a heuristic. It's an addition and a subtraction. Plugging in:

offset     = (3.24 + 0.59) ÷ 2 = 1.915 nats
separation = (3.24 − 0.59) ÷ 2 = 1.325 nats

The test, in one line: when offset exceeds separation, argmax will collapse to one side. You're shifting the whole score axis by 1.915 while the two class centers sit only 1.325 from the origin — both clouds get pushed across zero together. Measured result: 20/20 flap. The inequality called it.

The offset also translates into plain language. Run it through a sigmoid:

1 ÷ (1 + e^−1.915) = 87.2%

Before seeing any image at all, this model is already 87% inclined to say flap. It has a verbal tic.

Meanwhile separation = 1.325 nats says that after seeing the image it genuinely does pull the two classes apart — roughly a 79/21 swing in odds. The signal is real. It was just shoved entirely onto one side of the line by the tic.

So the fix isn't a different model, it's putting the line where it belongs. I calibrated a threshold on 20 frames and tested on 30 fresh ones: 27/30, AUC 0.949. Same model, same calls, not one word of the prompt changed.

3. The second faceplant: maybe what it loves is the letter A

Back in the above-or-below round, I produced an uglier pair of numbers:

Option mapping Correct
A = above 11 / 12
A = below 3 / 12

Same frames, same question, only the two options swapped positions. 92% down to 25%.

That's computable too. Average the "picked A" rate across both runs:

P(pick A) = [acc(A=X) + (1 − acc(A=Y))] ÷ 2
          = [11/12 + 9/12] ÷ 2 = 83.3%

However far that sits from 0.5 is your letter bias. Here it's 33 points off.

Mean accuracy across both runs: 58.3%
P(pick A):                      83.3%

The second number is further from chance than the first, which tells you position is driving the output more than content is.

This is why swapping the option order and re-running has to be the default move for any scored-choice model. Run it once and you cannot distinguish a model reading the question from a model reciting "A."

4. Switch to a commercial cloud API and the bias flips direction

The tempting conclusion here is: small local model, weak, of course it does this — a purpose-built commercial decision service wouldn't have this problem. That is the kind of sentence that feels obviously true and costs nothing to say, which is exactly the kind worth checking.

I actually said out loud that the commercial API "does calibration internally." He asked how I knew.

I didn't. That sentence was extrapolated from "its normalized probabilities feel nice to work with," which is not evidence of anything. Retracted, then measured.

Same 30 frames as numeric state, sent to a commercial cloud decision API, raw argmax:

Overall           21 / 30   (70%)
15 frames = wait: 15 / 15
15 frames = flap:  6 / 15

70% looks like a working model. Break it apart and it's perfect on one class and mostly wrong on the other — it's betting on wait, the exact opposite direction from the local model. Move the decision threshold to around 0.3 and it hits 28/30.

Both have a verbal tic. One says flap, the other says wait. The direction isn't predictable; the existence is. Assume any single-forward-pass scorer has a prior until you've measured it. Calibration belongs in the connective layer, and you have to redo it per engine — but every engine benefits.

5. How to debunk somebody else's numbers

That 21/30 is a weapon, so take it with you.

When a benchmark reports "70% binary accuracy," you have learned nothing. 70% could be 70% on each class, or 100% on one and 40% on the other. In the second case the model isn't deciding at all — it's betting, and a test set that happens to be balanced hands it 50 points for free.

The test:

Given any binary or multiple-choice score, ask two things first:
① What's the per-class recall? A gap over 20 points reads as bias, not ability.
② What's the distribution of argmax outputs? All on one side, and the accuracy number is void.

Neither question requires you to rerun anything. Both answers are already in their results file. Mine was 15/15 against 6/15 — sixty points of recall gap. "70% accuracy" and "bets one way, gets a balanced test set" produce an identical number. Only a confusion matrix separates them.

This is also why reporting AUC is the more honest choice. AUC only looks at ranking, so it doesn't care where you put the decision line. My model's argmax accuracy was 50% and its AUC 0.949; publish both and a reader instantly knows the ability exists and the threshold is crooked. Publishing only the first buries a working model. Publishing only the second oversells it.

6. What's wrong with this experiment

Numbers without stated limits should be assumed untrustworthy. Mine, stated:

  • Tiny sample. 20 calibration frames, 30 test frames. The confidence interval on AUC 0.949 is wide. Read it as "ranking ability clearly present," not as a precise value.
  • Synthetic frames, not real screenshots. I rendered clean composites. Real rendering has background, jitter, particles — the bias almost certainly needs remeasuring there.
  • Threshold calibrated in-distribution. Change resolution, scene, or model version and both the offset and that 0.3 are void. This is not a one-time calibration.
  • Only two models. One local, one cloud, opposite directions. n=2 supports the conjecture that this class of models generally carries a prior. It doesn't prove it.
  • Don't reuse the 19/20 from the above-or-below phrasing. As covered in section 1, that question leaks half the answer. It's here to explain how I found the problem, nothing more.
  • Latency was only measured single-stream at p50 87ms. No concurrency distribution.

7. The checklist

Next time a scored-choice decision model lands on your desk, in this order:

  1. Look at the argmax output distribution before you look at accuracy. If it's all on one side, accuracy carries no information — skip to step 3.
  2. Swap the options and re-run. Compute the averaged pick-A rate; its distance from 50% is your letter bias. This takes ten minutes and saves you every downstream hour of self-doubt.
  3. Export per-class logprob margins, compute offset and separation. If offset exceeds separation, argmax is dead. Stop tuning the prompt and go tune the threshold.
  4. Calibrate on a small held-out slice, test on the rest. Calibration and test sets must not overlap, or the AUC is a prize you mailed to yourself.
  5. Report AUC alongside per-class recall. A single accuracy figure is the least informative thing you can publish about this task.
  6. Redo all of it per engine. Bias direction does not transfer across models. Local loving A says nothing about what the cloud loves.

Step 3 now lives in my probe — it prints offset and separation automatically at the end of every run. Saves the next person (me) from staring at a 50% accuracy score wondering whether to go buy a bigger model.