AriakiI added a format constraint to a model and fully expected it to get slower. It got almost 100 ms faster per step.
With a different way of writing the constraint, the same model on the same questions failed to produce a complete JSON object 60 times out of 200. It would write {"x": 608, and then just emit spaces until it ran out of tokens.
The funny part: if you compare them by tokens per second, the number everyone looks at first, the option that saved 98 ms gets ranked as "a third slower." That's in section 3, and it's the one thing I'd like you to take away. If you just want the checklist, it's section 6.
1. The job is tiny: look at a screenshot, name a point
Last month I was testing whether a local model can click things on screen. The setup: give Qwen3.6-35B-A3B (a 35B mixture-of-experts model that only wakes up about 3B parameters per token) a window screenshot plus a one-line task like "Undo the last change," and have it say where the button is. The scenes are ones I drew with Tk: 25 windows, 8 controls each, half text buttons and half bare icons. So every question has ground truth. The program checks which control box the point lands in.
The model's native answer looks like this:
```json
[
{"point_2d": [613, 607], "label": "target"}
]
```
The coordinates are relative, 0 to 1000. To deliver those two numbers it writes a code fence, an array, and a label, about 33 tokens all told. Now and then it just replies There are none. (can't find it), and my regex parser gets nothing.
That's the problem constrained decoding is supposed to solve. In plain terms: before each token is generated, the inference server masks out every candidate that would break the format you gave it, so the model can only pick from legal tokens. Hand it a regex or a JSON Schema and it can only write something that fits that shape.
Intuitively, computing "which tokens are legal" at every step should cost time. When I tried it at the end of October on 40 questions, it went the other way. Today I reran it on all 200, with the four modes interleaved (the order of the four rotates per question, so no mode always lands on a warm cache), plus one excluded warm-up call per mode. The server is vLLM 0.26, temperature=0, thinking off, max_tokens=40.
2. Four ways to ask, 200 questions
| Mode | Median latency | Mean latency | Mean output tokens | Parse failures | Correct clicks |
|---|---|---|---|---|---|
| Free output (model's native format) | 382 ms | 406 ms | 32.4 | 5 | 183 |
Ask for {"x","y"} in the prompt, no constraint |
413 ms | 454 ms | 37.6 | 200 | 0 |
| Regex constraint | 284 ms | 337 ms | 17.9 | 0 | 188 |
| JSON Schema constraint | 311 ms | 365 ms | 25.6 | 60 | 128 |
The second row is a control I added on purpose. Both constrained modes use the same prompt: "output {"x": x, "y": y}." What happens if you change only the prompt and add no constraint? All 200 fail. The model ignores the instruction and writes its usual fenced array, just with the key renamed to x and half the task text stuffed into the label; 71 of them ran straight into the 40-token cap. So the credit for row three belongs to the constraint. The prompt alone did nothing.
The regex looks like this, with spaces fixed and numbers limited to 0 through 1000:
\{"x": (1000|[1-9]?[0-9]{1,2}), "y": (1000|[1-9]?[0-9]{1,2})\}
All the model can write is something like {"x": 871, "y": 560}: 17 or 18 tokens, not one character more.
The JSON Schema row is the second wreck in this post. The schema was about as plain as it gets: an object, two integers from 0 to 1000, both required, no extra fields. But JSON grammar allows arbitrary whitespace, and the constraint engine follows the grammar to the letter, so the model is allowed to add unlimited spaces after a comma. In 60 cases the model got this far:
{"x": 608,
and then kept emitting spaces until it hit 40 tokens and was cut off. Those 60 truncated calls had a median of 424 ms, slower than the ones that finished, so it's pure waste.
Did it get dumber? Look at the 140 calls that did parse: the schema mode clicked correctly 128 times, and the regex mode got 129 of those same 140 right. Basically identical. The model's ability to recognize buttons didn't change. The format was what jammed it. My 40-question run in late October also had 8 of 40 stuck; picking 40 questions at the same sampling positions today, it was 8 again, exactly.
One more sneaky trap. Older vLLM releases took a top-level guided_regex field in the request. On 0.26 it doesn't raise an error. The request returns 200 and you get a 40-token code fence back, because the constraint was never applied. You now have to write structured_outputs: {"regex": ...}. The cheapest way to check whether a constraint is actually live is to look at the output token count: if it's working, the count should sit in a very narrow band (17 to 18 for me).
3. Why the constraint is faster, and how tok/s calls it slower
Put "output tokens" and "total latency" for all 800 calls together and fit a straight line:
latency per call ≈ fixed cost + per-token cost × output tokens
≈ 228 ms + 5.73 ms × n
The fixed cost is mostly processing the screenshot plus the prompt (about 590 tokens) up front, plus the network round trip. Each extra output token costs 5.73 ms. Plug the numbers back in:
Regex: 228 + 5.73 × 17.9 ≈ 331 ms measured mean 337
Free: 228 + 5.73 × 32.4 ≈ 414 ms measured mean 406
Schema: 228 + 5.73 × 25.6 ≈ 375 ms measured mean 365
All three land within 10 ms. In plain terms, whatever extra masking work the constraint engine does each step is invisible at this scale. What it saves is tokens. The 14 or so extra tokens free output spends on fences, brackets and a label come to about 83 ms at 5.73 ms each; the measured gap in means is 69 ms.
Now look at the metric everyone reaches for:
apparent throughput = output tokens ÷ total latency
Regex: 17.9 ÷ 0.337 s ≈ 53 tok/s
Free: 32.4 ÷ 0.406 s ≈ 80 tok/s
By tok/s, the regex version is a third slower. By how long the user actually waits, the regex version is 17% faster. Same data, opposite conclusions.
The reason is that 228 ms fixed cost. The shorter the output, the bigger the share of the denominator it takes, and the worse tok/s looks. So the next time you see "turning on structured output dropped speed by X%," ask first: is that tok/s, or total time per request? Did both sides output the same number of tokens? If the constraint made the output shorter, a drop in tok/s is close to guaranteed and tells you nothing about speed.
There is a real cold start, though. I made three regexes the server had never seen and called each twice. The first one took 2541 ms on its first call and 246 ms on its second; the other two took 388 ms and 372 ms the first time and about 240 ms the second. A new format has to compile a grammar on first use, and after that it's cached. With only three samples, all I'll claim is that the first call is slower, somewhere between a few hundred milliseconds and over two seconds, and after that you're fine. If every request builds a different schema, you pay that every time.
4. Do those 5 extra correct answers count?
Regex 188 versus free output 183 looks like 5 more correct. Using the same method as my earlier post, count only the questions where one is right and the other wrong: regex wins 6, loses 1.
two-sided p of at least k out of n split questions favoring one side
= 2 × P(X ≥ k), X ~ Binomial(n, 0.5)
at least 6 of 7: 2 × (7 + 1) / 128 = 0.125
0.125 isn't enough to call it more accurate. And look closer at those 6: in 4 of them the free mode answered There are none.. Free output said "can't find it" 5 times in total. The regex leaves no room for that, so it forced a point every time, and got 4 of those 5 right and 1 wrong. Most of the extra wins came from being forced to guess.
That is the regex approach's biggest flaw: it deletes "I can't find it" as an option. In my scenes the target always exists, so forcing a guess happens to pay off. On a real interface the target may genuinely not be on screen, and the model will still confidently hand you a coordinate, and your agent will click something strange. If you use this, leave a legal path for "not found" in the regex, for example allowing {"none": true}. I haven't tested that yet.
There's nothing to agonize over on the schema side: regex wins 60, loses 0, and all 60 are the truncated ones.
5. What's wrong with this experiment
- One model, one version. The AWQ-quantized Qwen3.6-35B-A3B on vLLM 0.26. I didn't test whether other models also spew spaces inside a schema, or whether other constraint backends tighten whitespace by default.
- Synthetic scenes where the target always exists. That flatters "always answer with a point," as section 4 says.
max_tokenswas only 40. With a bigger cap, those 60 schema calls would spew longer. By the formula above, at 512 a single stuck call would take about 228 + 5.73 × 512 ≈ 3.2 seconds. That's an extrapolation, not a measurement.- Each question ran once per mode, on an idle server, serially. Under concurrent load both the fixed cost and the per-token cost change, so the line doesn't transfer as-is.
- The constrained and free modes use different prompts. The control row only shows that changing the prompt alone doesn't work. It doesn't fully separate "changing the format" from "adding the constraint."
6. Next time you put a format on model output, do it in this order
- Count how many tokens in the current output are pure format (fences, labels, key names). That's what a constraint can save you.
- Use a regex over a JSON Schema whenever you can, and fix the spaces in the regex. Schemas allow free whitespace by default, and short outputs are the easiest to get burned.
- After you ship it, check that the output token count actually narrowed. If it didn't, the parameter name is wrong and the constraint isn't on.
- Leave a legal exit for "can't do it" in the format, or you'll get a pile of perfectly formatted, made-up answers.
- Compare speed by total time per request, not tok/s. Log the first call separately; that's the grammar compile bill.