Same Model: 94% on My Own Questions, 76.7% on a Public Set. Who Handed Me Those 17 Points?

Same model. Same prompt. Same card.

On the 85 questions I wrote myself: 80 correct, 94.1%. On 300 questions pulled from public datasets: 76.7%.

A 17.4 point gap. The model didn't change. The person writing the questions did.

This is the record of how I handed myself those 17 points, plus what this "single forward pass, read the logits" trick actually buys you, and one benefit I was sure I'd find and did not find at all.

Want the concurrency rule straight away? Chapter 3, it's one division — I computed 314ms and measured 311ms. Want to see me flunk my own exam? Chapter 5.

1. The trick: don't let it talk, just look at what it wanted to say

The normal way to get a decision out of an LLM is to have it emit JSON: {"action": "B", "reason": "..."}. It shuffles out one token at a time. Slow.

This trick chokes that off. Set max_tokens=1 in the request, ask for top_logprobs=20, then look only at the logprobs of A, B, C — softmax over those, take the biggest. The model does exactly one forward pass: read the whole prompt, compute once, done. There is no step two.

Why that's fast doesn't need much explaining. Decoding re-reads the full set of activated weights for every token you generate — want 30 tokens, read them 30 times. Reading logits reads them once. My decision prompt is 165 tokens; those go through the weights in one shot, one read amortised across all 165.

None of this is new. It's logit scoring, roughly a decade old. Someone recently packaged it up as an open-source project using a 4B model to read option probabilities, and that's what got me started. Its own unit tests pass on my local card, 11 passed, and then I started writing questions.

2. 85 questions, three models, and a reversal in the last batch

Every question is a Chinese-language scenario I wrote myself, modelled on judgements I actually need: did this automation step succeed, should this event escalate, which bucket does this text go in.

Three contestants: a 35B MoE (a mixture-of-experts model splits its parameters into many "experts" and only wakes a few per token, so despite the 35B label it activates far less than that each pass), a 27B dense model (all hands on deck every time), and the 4B the open-source project ships with. All three over a standard API, identical trick, zero code changes between them.

Four batches, 85 questions:

Batch	35B-A3B	27B dense	4B
Batch 1, 35 questions	35	34	33
Batch 3, 25 questions (traps)	24	24	21
Batch 4, 25 questions (reasoning)	21	22	20
Cumulative 85	80	80	74

The 35B led the first three batches. On batch 4, which needs two-hop reasoning, the 27B took it by one. Dead even at 80 apiece.

So which one when they tie? Latency: 65ms for the 35B, 155ms for the 27B. 2.4x. Not a hard call.

One metric that's easy to skip: flip the option order and ask again, see if the answer changes. In batch 1 the 35B flipped 0 times; the 4B flipped twice. A judge that changes its mind because the options moved is unusable no matter how high its accuracy reads.

The two questions all three got wrong were the most valuable. One was disk capacity — a 150G thing that fits on neither of two drives, and all three picked the 80G one. The other was date arithmetic: which day was "last Wednesday". A single forward pass cannot do arithmetic. Hand that class of judgement to a few lines of code instead.

The ugliest one was the 35B's solo miss: a command finishes with no output at all, followed immediately by echo "confirmed". It called that a success. Confidence 0.993. Fake evidence swallowed whole, with total conviction.

3. There's a division hiding in the concurrency curve

People are driving real-time games with this trick, so I wanted to know: open more lanes and does total throughput climb, or does everyone just split the same pie?

I ran a concurrency curve, each request carrying a unique prefix so nothing hits cache:

Concurrency	Decisions/s	p50
1	13.5	64ms
4	34	112ms
16	51	311ms
32	51	0.5s
64	51	1.1s

Saturated at 16. Past that, throughput doesn't budge and latency climbs linearly — that's queueing, not parallelism.

Which gives you this check:

past saturation, p50 ≈ concurrency ÷ saturated throughput

Against the 16 lane row: 16 ÷ 51 = 314ms, measured 311ms. That's not an empirical rule of thumb. It's a division.

At 32 it predicts 627ms against 500ms measured; at 64, 1.25s against 1.1s. The formula runs about 20% high, consistently. The reason is dull: p50 is a median, and the queueing tail drags the mean up without moving the middle. So use this as an upper bound, not a prediction.

The same numbers tell you whether concurrency is worth bothering with:

max concurrency gain = saturated throughput ÷ single-lane throughput = 51 ÷ 13.5 ≈ 3.8x

Measured 1→16 lanes: 3.7x. Close enough.

This curve also kills a claim I'd seen repeated — that a single forward pass like this is "compute-bound, not bandwidth-bound." Compared to decoding, sure, it's inverted. But if a single lane were already saturating compute, adding concurrency could not possibly help. It helped by 3.7x. Which means most of that 64ms single-lane number is moving activations around and launching kernels, with the compute sitting idle.

So the design rule is two sentences. Throughput work gets 16 lanes. Latency-sensitive work gets one lane, or two to four. There's nothing cleverer in between.

4. I was sure logit reading would be more accurate. It wasn't, at all.

This is the bet I placed hardest and lost cleanest.

My prior: if the model can't generate freely, it can't wander off, so accuracy should go up. That feels reasonable — tighter constraint, less room to err.

So I ran both readouts on the same model: one reading logits, one just emitting an option letter normally. The result is one sentence — logit readout does nothing for accuracy. Same weights, same prompt, same answers.

Which, once you say it out loud, is obvious. Taking the argmax after a softmax and greedily sampling the first token are the same operation. I had mistaken an identity for an optimisation.

So what does it actually buy? Two things, both real:

Latency. What you skip is the entire length of the JSON you would have emitted, not just one token. {"action": "B", "reason": "..."} is dozens of tokens, which is dozens of trips through the weights.

And the output is structurally confined to your candidate set. A generative answer can always hand you an option that doesn't exist; reading logits makes that physically impossible, because anything outside the candidates isn't among the letters you're reading.

Both are worth having. Neither of them is "more accurate."

A caveat belongs right here: I compared accuracy only. I never ran the two readouts against each other on latency under matched conditions. So "dozens of trips through the weights" is derived from the mechanism, not measured by me. Don't quote it as a number.

5. And then my question set flunked

I was pretty pleased with myself the day I got 94.1%.

Then I swapped in three public datasets — two intent classification, one textual inference — 300 sampled from each for a first pass. Every model ran on the same three-card box, driven from the same client, one model at a time, serially. Latency is end-to-end including the LAN round trip:

• Cloud decision API: 84.0%, p50 323ms (excluding 11 failed retries)
• 27B: 77.0%, p50 528ms
• 35B: 76.7%, p50 104ms
• 4B: 68.0%, p50 100ms
• 2B and 0.8B: both around 30%, p50 near 60ms

94.1 down to 76.7. Same model, same prompt.

Of course you ace an exam you wrote yourself. But I genuinely hadn't seen it coming, because I'd written those questions modelled on real scenarios and felt entirely fair about it. Looking back, there are at least three biases, and I noticed none of them at the time:

First, I knew the answer before I wrote the stem. Writing "did this operation succeed" means the success criterion was already in my head, and it leaks into the wording without asking permission.

Second, I instinctively avoided the cases I wasn't sure about myself. If a question took me three minutes, I'd usually swap it out — and that is precisely the question worth testing.

Third, the distribution was my intuition's distribution, not the real one. Public intent datasets have dozens to hundreds of classes plus out-of-scope samples. My questions topped out at three or four options.

That third point has a hard receipt. The cloud API caps at 255 options — I sent 300 and got rejected. Not one of my 85 questions had more than 4 options. I never tested anywhere near the point where it breaks.

Digging into that 255 turned up a rule worth taking away. The cap isn't arbitrary; it comes from the label pool:

max candidates = number of single-token labels available

Every candidate needs a letter label, and that label has to be exactly one token. The local service accepts 255 and rejects 256; the cloud one accepts 255 and rejects 300 — two completely independent implementations hitting the same wall. Better still: labelling them in order, I hit a multi-token label at number 69 and had to skip past it. Skipping still gets you to a full 255, because the candidate indices need to be contiguous, the label spellings don't.

On the textual inference set the local-versus-cloud gap narrows sharply: cloud 74.7%, 35B 74.0%, a 9B at roughly 71% in two precisions, 27B 70.3%, 4B 63.0%. Same models, different task, and both the ranking and the spread change. Which is why any single "local trails cloud by X points" number deserves no trust at all.

The probe lied to me too

Before any of that evaluation ran, my own readout code got me first.

Version one built its lookup with token.strip() — clean up the whitespace, easier matching. What actually happened is that ' B' with a leading space (logprob −9.25) overwrote the real 'B' (logprob −0.0). It printed a ridiculous 6/20 and I nearly went off to blame the model.

What saved me was an order-of-magnitude assertion: greedy sampling was emitting the obviously correct token, and that disagreed with the argmax I'd computed. Those two are supposed to be identical. When they disagree, the bug is mine.

Matching letters against logprobs requires exact token matching. Any normalisation lets a variant overwrite the truth.

There's a sibling of this bug that looks different and is the same disease: the readout layer will copy one label's probability onto a longer label sharing its prefix, and if it reads no valid label at all, it degrades into a uniform distribution and quietly picks the first candidate. That has to raise an error. Dressing up "no evidence" as "a normal decision" is far worse than being wrong outright.

6. Use this to puncture other people's numbers

For any "small local model makes decisions" benchmark, four questions will squeeze most of the water out:

One: who wrote the questions. Self-written versus public, same model, 17.4 points apart. Halve any number where the author set the exam and graded it.

Two: does the latency include the network, and at what concurrency. My 104ms is end-to-end over LAN; someone reporting 20ms is probably timing the forward pass on the GPU. Concurrency is sneakier — same box, same model, 64ms at one lane and 311ms at sixteen. A 4.9x spread. A single latency figure with no concurrency stated lets the author pick whichever they like.

Three: how many candidates. Under 255 is one forward pass. Over it, you're selecting a bucket and then selecting within it, which is an extra round trip — my hierarchical module still costs about 200ms after optimisation, more than double the flat path. Latency measured on 4 candidates will not survive 400.

Four: was logit readout compared against plain generation. It's probably a tie. If someone claims accuracy went up after switching to logits, ask for the same-model generative arm.

7. Where these numbers stop

I'll stand behind every figure above, but here's what they don't cover:

• The 300 questions are a stratified first pass to validate the harness, not the full public test set.
• The 27B ran in its resident configuration with speculative decoding, untuned for this task. Its 528ms is not its ceiling.
• The cloud 84.0% excludes 11 failed retries. Counting them drops it; I didn't compute by how much.
• The 4B's 44ms floor is artificially slow — that box is missing an acceleration library for one linear attention layer and falls back to a reference implementation, which the log says plainly. It should be faster once installed. I didn't test that, so it isn't a conclusion.
• Local runs over LAN, cloud over the public internet. That comparison measures deployment experience, not raw inference speed.
• Most important: model capability, prompt fit, and inference configuration are not separated. So "local trails cloud by 7 to 16 points" answers how much, and cannot answer why.

8. The order to do it in

Next time you evaluate any "small model makes fast judgements" setup, go in this order:

1. Find a public dataset before you write questions. Home-made questions are a smoke test, not a scorecard. Write them if you like, but run the public set afterwards and print both numbers side by side.
2. Flip the option order and ask again. Flip rate exposes instability better than accuracy does.
3. Pull the arithmetic and injection-style questions out and look at them separately. A single forward pass can't do arithmetic, and injection questions were unstable on both large models here. Send that class to a rules layer.
4. Run a concurrency curve and find where throughput flattens. Past that point latency scales with concurrency; use p50 ≈ concurrency ÷ saturated throughput as an upper bound and stop expecting more hardware to fix it.
5. Run both readout arms. Accuracy will likely tie. What you're buying is latency and a guarantee the answer lands inside the candidate set. Set expectations accordingly.
6. Give "no valid label found" its own error path. Degrading to a uniform distribution and silently taking the first candidate produces a log line identical to a healthy decision.

Item 3 cost me tuition: a command that printed nothing, one line of echo "confirmed", and a confidence of 0.993.