Ariaki
/
Back to blog

2026.10.25

7 of 8 requests truncated, and I almost wrote the model off: when picking a local model, which of my verdicts were really testing me?

In one day of choosing between two local Qwen models I reached four verdicts, and every one of them needed correcting: a missing launch flag dumped chain-of-thought into the answer, I compared a fixed model against an unfixed one, 3 of my own 8 test questions were broken, and a per-request thinking switch was silently overridden by the server. Three checks you can run without reading any source: answer share, broken-question rate vs. score gap, and identical token counts across two configs.

llminferencevLLM评测方法论

Two models, the same batch of real prompts, and 7 out of 8 requests ran into the output cap and got cut off. Both models: 7/8.

I came very close to writing in my notes: "Neither model is suitable for structured output."

The fix was a single launch flag. After it, truncations dropped to 0/8, and one of the models finished the batch in half the time.

That was only the first verdict of the day. I reached four conclusions about these two models that day, and all four had to be revised: two were measuring my own configuration, one measured the wrong dimension, and one measured my own exam questions. If you just want the checks, see sections 4 and 5; each is one division or one side-by-side comparison. If you want to see how 3 broken questions hid in an 8-question exam, that's section 4.

1. 7/8 truncated, and the answer was 2.4% of the output

The setup is simple. I run a multi-agent simulation world on my own machine, and every turn needs the model to emit a strict JSON object. I wanted to pick between two models:

  • Qwen3.8-27B, dense, meaning every parameter takes part in every token, FP8 weights
  • Qwen3.6-35B-A3B, MoE (mixture of experts), meaning the network is split into many "experts" and only about 3B parameters wake up for each token, 4-bit AWQ weights

Each got its own 64GB card, both served with vLLM. I pulled 8 real prompts from the simulation's saved state and sent them over.

Both models came back with 7 of 8 at finish=length, which means generation hit the max_tokens cap and was hard-stopped. It looked like "this model loses control as soon as the constraints get long."

I shrank it down to the smallest possible test: "Output only one JSON object: {"ok":true}."

content:            "Here's a thinking process:\n\n1. **Analyze User Input:**..."   251 tok
reasoning_content:  None

It was thinking out loud. This generation of Qwen's chat template opens every assistant turn with a thinking tag, so the model writes a chain of thought before the answer. On the vLLM side, --reasoning-parser qwen3 is what splits that chain of thought into a separate reasoning_content field. I had left the flag out, so the whole chain of thought landed in the answer body.

The official vLLM recipe describes this flag as "not optional in practice." I only went back to read it after someone asked me whether the chat template was configured right.

The diagnostic signature is one line: reasoning_content is None, yet the content contains a chain of thought. When you see lots of finish=length, check this field before you start doubting the model.

With the flag added, and thinking switched off via --default-chat-template-kwargs '{"enable_thinking":false}', the same request:

content: '{"ok":true}'    6 tok

Here is a number you can compute yourself:

answer share = tokens the answer actually needs ÷ tokens the model generated
             = 6 ÷ 251 ≈ 2.4%

The answer needs 6 tokens. Before the fix, the model had written 251 and still hadn't reached it. Almost the whole max_tokens budget went to a monologue you never asked for. Tighten the budget and truncation is guaranteed. That says nothing about model ability; it's just a division.

2. Fixing one side, then running the comparison

I did fix that first problem, but only on the 27B.

As soon as the 27B was fixed, I used the still-unfixed 35B as the baseline and concluded "the 35B truncates more." That sentence was measuring the difference in launch flags and had nothing to do with the models. Once I caught it, I fixed the 35B too.

With both fixed, on the same 8 real prompts:

Model Before fix After fix
Qwen3.8-27B 604.4 s, truncated 7/8 366.8 s, truncated 0/8
Qwen3.6-35B-A3B 163.0 s, truncated 7/8 84.4 s, truncated 0/8

The 35B row got almost twice as fast (163.0 ÷ 84.4 ≈ 1.93x). Most of that earlier time was spent generating reasoning nobody wanted.

This is controlled-experiment basics: every config change you make to the treatment arm, make to the control arm too. Everyone knows that. I still fixed only one side, because after fixing the first one I couldn't wait to see numbers.

3. 46/46 vs. 43/46: I was measuring obedience

With both configs equalized, I ran a more formal suite: 46 cases drawn from the simulation, covering 12 kinds of JSON output contracts, a bit over two hundred individual checks.

Model Cases fully passed Checks Time
Qwen3.6-35B-A3B 46/46 276/276 5.6 min
Qwen3.8-27B 43/46 271/274 27.8 min

All 3 of the 27B's failures were in the same category. Opening them up, it had written an English analysis with Markdown headings, like a report, ending with finish=stop. In other words, it thought it was done. Those three took 123, 113 and 33 seconds, the slowest in that category.

The chain of thought had already been stripped, so this was the model's own behavior: given a task with long constraints that requires reasoning, it tends to write an analysis instead of filling in the form.

So I wrote down verdict number three: the 35B is better.

Then I got one question: "There's no way 3.8's 27B is worse than 3.6's 35B, right?"

I went to look up a third-party number. On the Artificial Analysis Intelligence Index, the 27B scores 52 and the 35B scores 32, a 20-point gap. I haven't reproduced that index myself; I'm using it as an independent source.

Looking back, my 46 cases measured "can it reliably emit valid JSON under a very long prompt." That is format compliance, a measure of obedience. I had used it to declare which model was stronger, which is the wrong dimension.

4. 3 of my own 8 questions were broken

To test "which one is smarter," I wrote 8 questions on the spot. Each needed multi-step reasoning, had a single correct answer, and could be graded by a program, so no second model was needed as a judge.

First round: 27B 4/8, 35B 5/8. The 35B wins again.

That contradicted the third-party index, and I didn't quite believe it. So I verified the reference answers for all 8 questions using Python's fractions module and brute-force enumeration:

  • Question 1: my reference answer was wrong. A bag holds 5 red, 3 blue and 2 green balls; draw 3 without replacement; probability of exactly 2 red and 1 blue. I had written 5/12. The correct value is C(5,2)×C(3,1)÷C(10,3) = 10×3÷120 = 1/4. Both models answered 1/4, and I marked both wrong.
  • Question 3: no solution exists. "Exactly one of A, B and C is lying. A says B is lying. B says C is lying. C says A and B are both lying." Try each case: if A lies, B is honest, so C is lying, and that's two liars. If B lies, C is honest, but A is honest too, which makes C's statement false, a contradiction. If C lies, A is honest, so B is lying, two liars again. No assignment gives "exactly one." I wrote a broken puzzle.
  • Question 7: ambiguous. "On a 4×4 grid, going from top-left to bottom-right, moving only right or down, how many paths are there?" If it's 4×4 cells, the answer is C(8,4) = 70. If it's 4×4 points, it's C(6,3) = 20. I had marked 20; the 27B answered 70. It wasn't wrong; my wording was unclear.

Dropping the unsolvable question 3, fixing question 1, and accepting both answers on question 7 left 7 questions:

before fix   27B 4/8 = 50%    35B 5/8 = 62%
after fix    27B 7/7 = 100%   35B 6/7 = 86%

The ranking flipped. The question the 35B missed was (1+1/2)(1+1/3)…(1+1/99). Each factor is (k+1)/k, adjacent terms cancel all the way down, and you're left with 100/2 = 50. It answered 100.

Here is the second formula you can compute yourself, and I think it's worth more than any score in this section:

broken rate = broken questions ÷ total questions = 3 ÷ 8 = 37.5%
score gap   = questions separating the two models ÷ total questions = 1 ÷ 8 = 12.5%

broken rate ≥ score gap  →  the ranking can't be trusted

My exam was 37.5% broken, and the two models were 12.5% apart. The broken share was three times the gap, so the ranking was about as good as a coin flip, and the coin happened to land the wrong way. Grading models with unverified questions measures your own arithmetic.

In hindsight, question 1 had already sent a signal: both models gave the same answer and both were marked wrong. Two independent models making the identical mistake is much less likely than me writing down a wrong reference answer. That makes a free filter: if both models agree and both get marked wrong, check the reference answer first.

5. Trying to turn on thinking, and the parameter vanished

In that reasoning round, I also wanted to see whether the 27B would do better with thinking on. The server had been launched with --default-chat-template-kwargs '{"enable_thinking":false}', so I passed chat_template_kwargs: {"reasoning_effort":"xhigh"} in the request, hoping to turn it on per request.

After the run, the "thinking off" and "maximum thinking" outputs were byte-for-byte identical:

thinking off   Q1 12.4s 511tok   Q2 15.9s 673tok   Q8 8.0s 317tok
thinking max   Q1 12.8s 511tok   Q2 16.6s 673tok   Q8 7.8s 317tok

Timing wobbled by a few tenths of a second, the token counts matched exactly, and the reasoning field was empty everywhere. In my deployment, that server-side flag is a hard override: the request-level parameter of the same name can't change it, and no error is raised.

So the "thinking on" data from that round was wasted. The only way to change thinking modes was to restart the server, and after restarting I checked the reasoning field again: 156 characters, which is what confirmed thinking was actually on. The 27B's 7/7 above was scored with thinking off.

This is the check I think generalizes best in the whole post:

two configs, same questions, identical completion_tokens on every question  →  the variable you changed didn't take effect

At temperature 0, two genuinely different configs are very unlikely to produce answers of exactly the same length on every single question. If they're all identical, you were running the same config twice. This needs no source code at all; just line up the token counts from both runs.

6. That 4.3x speed gap, and how to take apart someone's "X beats Y"

After all the fixes, on those 8 real prompts the 27B took 366.8 seconds and the 35B took 84.4 seconds, 4.3x faster.

Can you estimate that yourself? During decoding, each new token roughly requires reading the active weights from memory once:

weights read per token ≈ active parameters × bytes per parameter

27B dense, FP8:     27B × 1 byte    ≈ 27 GB
35B-A3B, 4-bit:     3B × 0.5 byte   ≈ 1.5 GB

On paper that's an 18x gap; measured, it's 4.3x. I didn't break that gap down, and at least three things are mixed into it: prefill (the step that reads the whole prompt in first) isn't governed only by active parameters; attention and cache have their own costs; and the two models wrote answers of different lengths on these 8 prompts, so 4.3x is a ratio of total wall-clock time, not of per-token speed. Treat the two estimate lines as order-of-magnitude only.

Still, one rule is directly usable when choosing: for MoE models, speed follows active parameters; reading the total parameter count gets it backwards. The 35B has the bigger name and runs more than 4x faster.

Now back to the final table from that day:

Dimension Qwen3.8-27B Qwen3.6-35B-A3B
Third-party intelligence index 52 32
My reasoning questions (7) 7/7 6/7
JSON contracts (46 cases) 43/46 46/46
Total time, 8 real prompts 366.8 s 84.4 s

Same pair of models: the first two rows say the 27B wins, the last two say the 35B wins. For strict JSON and throughput I ended up with the 35B; for conversation and reasoning, the 27B.

This table is also my template for taking apart anyone else's "X beats Y." Whenever I see a model ranking, I ask three questions:

  1. Which dimension does it measure? Ability, format compliance and speed often pull against each other. Anyone who draws a general conclusion from one dimension doesn't get my trust by default.
  2. Were both sides deployed with the same flags? Especially the thinking switch and the reasoning parser. If one side is missing one, the result measures config.
  3. Were the reference answers verified? If they don't say, compare their likely broken-question rate against their score gap.

7. What I didn't get right

  • Only 7 reasoning questions. 6/7 vs. 7/7 is one question apart; a different set could easily tie. All I can say is "the 35B did not win on ability." I can't say "the 27B is significantly stronger"; that claim rests on the third-party index.
  • I didn't reproduce the third-party index. The 52 and 32 are quoted directly.
  • Accepting both answers on question 7 was a leniency I added after the fact. It favors the 27B, which answered 70. Under the strict reading that only accepts 20, the 27B drops to 6/7. I didn't keep a record of what the 35B answered on that question, so under the strict reading I can't say which model leads.
  • The 4.3x isn't decomposed. I didn't separately measure output length or prefill share, so it's only valid as "these two models on my batch of prompts."
  • I never ran a full round of the 27B with thinking at maximum. Once I confirmed thinking was really on, I only checked those 156 characters.
  • The hard-override behavior is confirmed only on my one vLLM version. Other versions or other engines may behave differently; test it yourself with the token comparison from section 5.

8. Next time you choose a local model, check in this order

  1. If you see lots of finish=length, check whether reasoning_content is empty and whether the answer body contains a chain of thought. Compute the answer share; if it's absurdly low, it's a config problem.
  2. Copy every launch flag you changed on the treatment arm to the control arm, then run.
  3. For questions you wrote yourself, verify the reference answers with a program, enumerate to confirm a solution exists, and reread each one for a second interpretation.
  4. Compute the broken-question rate and compare it with the score gap between the models. If it isn't smaller than the gap, don't rank them.
  5. If both models give the same answer and both are marked wrong, check the reference answer first.
  6. After flipping a switch, line up per-question token counts from both runs. If they're all identical, the switch didn't take effect.
  7. Your verdict table needs at least three rows: ability, format compliance, speed. Pick per task; don't crown one overall winner.