AriakiTwo machines, same model family, almost identical error rates: 3.29% on one, 3.32% on the other. Both failed in the same spot, too: right after an agent called a tool and sent the result back to ask for the next turn.
Conveniently, someone on Hugging Face had just published a set of "fixed" chat templates. The README promised 80%+ fewer stalls, a 100% KV cache hit rate, and 80% faster runs on llama.cpp. I tried it on both machines. On the first, the failing argument shapes went from 7 out of 19 to zero. On an isolated copy of the second, the same bad request still came back as a 500, and it took 3 milliseconds to do it.
That 3 milliseconds is the most useful number in this post. If you just want the test, it's a single division in section 4. If you want to know where the README's 80% came from, see section 2.
1. A 3.3% error rate
Quick background on chat templates. What you send an inference server is a messages array. What the model actually reads is one long string full of special tokens. The thing that turns one into the other is a small Jinja program called the chat template, and it usually ships alongside the model weights.
The first machine is an M1 Max running Qwen3.6-35B-A3B, with a small Python shim of mine in front that applies the template. Over five days it handled roughly 4,900 requests, and 162 of them died with the same error:
template render failed: Can only get item pairs from a mapping
By day the 162 broke down as 30, 69, 35, 16, 12. It happened every day, so this wasn't a one-off incident. 95 of them had tool calls in the message history.
(My two sets of notes give the denominator as 4,850 and 4,927. I queried twice that day with slightly different time windows. I'm quoting 3.29%; against 4,850 it's 3.34%. Same ballpark either way.)
The cause wasn't hard to find. In the OpenAI format, tool_call.function.arguments is supposed to be a JSON string. The official template assumes it's a dict and calls |items on it to iterate key-value pairs. A string has no key-value pairs, so Jinja throws before the model has computed a single token.
2. The README's 80%, and a test suite that never touches an engine
The community template is v21.3. It's 16,289 bytes; the official one is 7,764. A line-by-line diff gives a similarity of 0.22, so this is basically a rewrite.
It does add real things: it accepts string-form arguments, tolerates system messages dropped into the middle of a conversation, handles both vendors' fields for reasoning content, truncates oversized tool results, and injects a correction hint into the context after repeated tool errors. I ran the repo's own 9 unit tests, and all 9 passed.
Then I went looking for where the performance numbers came from. The test script is 184 lines of pure Jinja input/output fixtures. I counted the places where it calls llama.cpp, vLLM, MLX, an HTTP client, or any benchmark at all: zero.
A template just assembles a string. The only ways it can affect performance are how long the resulting prompt is, and whether consecutive turns produce byte-identical prefixes (which decides whether the KV cache can be reused). You can only measure either one on a real engine. A test suite that never touches an engine can show the template renders correctly. It can't show anything is 80% faster.
The test is simple. For any performance number, find the code that produced it and check whether it actually called the system being measured. If it didn't, treat the number as the author's expectation and nothing more.
3. Machine one: 19 shapes, 7 failures on the official template
Before swapping the live file, I ran an offline comparison. I fed 19 message shapes through both templates, rendering only, no inference: plain chat, first and follow-up turns with tools, parallel tool calls, a mid-conversation system message, 7 shapes of arguments (dict, JSON string, malformed string, array, scalar, null and so on), structured tool errors, both reasoning-field formats, and an oversized tool result.
- Official template: 7 of 19 failed
- v21.3: 0 of 19 failed
- p95 render time: 0.119 ms → 0.193 ms
The extra 0.07 ms of rendering doesn't matter. What's worth working out is the extra tokens it assembles, because those all go through prefill:
template overhead share = extra tokens from the new template ÷ total prompt tokens
It's a division where the numerator is roughly constant and the denominator is however long your prompt is. I ran 46 real requests against an isolated instance, 23 per arm, and all of them succeeded:
- Short tool-calling prompts: about 96 extra tokens, a median relative increase of 30.4%, and a median paired latency 6.4% slower
- One real 25K-token prompt: 107 to 111 extra tokens, or 0.42% to 0.44%. The two paired runs came out +4.2% and −5.8%, averaging −0.8%. That's noise; there's no measurable difference
So the same template is a visible fixed cost on short requests and effectively free on long ones. Your traffic mix decides which side of that you're on.
On protocol behavior, the v21.3 arm had zero malformed arguments, zero unknown tools, zero raw XML leaks, zero reasoning leaks, and zero repeated calls to the same broken tool. So I switched. I restarted only the shim that applies the template; the model process kept the same PID. In the first 13 requests after the switch, 12 succeeded, 1 was a connection error during the maintenance window, and 0 were template render errors.
The conclusion at this point is plain: the template fixed stability, and it didn't make anything faster. I couldn't reproduce any of the README's numbers. I also didn't find evidence against them, because I never measured KV cache hits (section 6).
4. Machine two: a 500 in 5 milliseconds
The second machine is a laptop with 16GB of VRAM running llama.cpp, serving a distilled model from the same family. Over ten-thousand-odd requests, 3.32% failed. It looked like the same disease, and I almost copied the first machine's conclusion over.
The error text gave it away first:
Failed to parse tool call arguments as JSON
That's a different message from the one on machine one. But error text is only indirect evidence, and I wanted a check that works without reading source. So I sent the running server two minimal requests, both with max_tokens=1, both with one tool call in the history. The only difference was that one had valid arguments and the other had a truncated, broken string:
- Valid history: HTTP 200 in 1,632 ms
- Broken history: HTTP 500 in 5 ms
time to failure ÷ time for a normal request of the same size = 5 ÷ 1632 ≈ 0.3%
Once a request actually reaches the model, it has to run the whole prompt through prefill even if it's only generating 1 token. That costs over a thousand milliseconds here. Five milliseconds isn't even a rounding error on prefill. The request never got as far as the model; it died in some preprocessing step before that.
The next question was whether that preprocessing step was the template. I spun up an isolated copy on the same image version, with a 0.5B model on CPU, and swapped the template to v21.3:
- Valid history: 200
- Broken history: 500 in 3 ms, with the identical error
Changing the template didn't move the failure point or the failure time. At that point you can conclude without looking at any source: the error happens before the template runs, and changing the template can't reach it.
I did read the source for that version afterwards, and it lined up. llama.cpp has a function called func_args_not_string(). When a template advertises that it supports arguments as objects, this function runs before Jinja and forcibly json.parses every string arguments field in the history. If parsing fails, it throws a 500. Both the official template and v21.3 advertise that capability, so both go down this path. There were two upstream issues describing the same cascade, and two PRs that would have made parsing more lenient. Neither had been merged.
The check carries over to any layered service:
time to failure ≪ cheapest normal processing step → it failed before that step
Before changing a layer, use timing to locate which layer is actually failing.
5. 341 errors were really 39 sessions
On machine two, that 3.32% works out to 341 JSON parse failures out of 10,278 requests. (There were 361 errors total; the other 20 had different causes, and template render errors were zero.)
341 sounds like 341 separate incidents. I hashed each request's prompt:
average replays per bad session = error count ÷ distinct prompt hashes = 341 ÷ 39 ≈ 8.7
The 341 errors mapped to only 39 distinct prompts, arriving in 75 bursts, the longest of them 20 in a row.
Here's how it happens. At some point the model emits a tool call but stops halfway through the JSON, and the caller saves that half-object into the history anyway. From then on, every request in that session carries the broken fragment, and every one of them dies with a 500 in preprocessing. Each agent retry adds another error. One bad generation poisons the whole session.
This changes how you prioritize. If you read it as "3.32% of requests fail," you'd expect to hit it about once every 30 calls. Read as "39 poisoned sessions," the fix is obvious: before a request reaches llama.cpp, catch any arguments in the history that won't parse and degrade them, or patch the parser to be lenient. A new template does nothing for either.
While I had the isolated copy up, I also ran a quick 10-request A/B with complex array-of-object tool arguments. Both templates went 10/10 on success and 10/10 on schema validity. But with v21.3, the prompt went from 250 tokens to 450, generated tokens went from 55 to 99, and mean latency went from 781 ms to 1,928 ms. I never figured out why generation got longer too. Absolute numbers from a tiny model don't transfer, but at minimum it shows this template has a real cost.
Final verdict: switch machine one, leave machine two alone.
6. What I didn't get to
- I didn't replay the original 162 failing requests one by one. My call log only keeps readable text and truncates at 32K, so byte-for-byte reconstruction wasn't possible. I covered the same root cause with the 19-shape matrix plus a set of real tool loops, so I can't claim "162 of 162 fixed."
- I didn't measure KV cache hits. Prefix caching was off on machine one's isolated instance. I neither confirmed nor refuted the README's "100% hit rate."
- The isolated instance only had 65,536 tokens of context, and the longest prompt I tested was about 25.8K tokens. I never got near the model's advertised 262K limit.
- Machine two's isolated checks used a 0.5B model on CPU, not the 35B distill it actually serves. That's enough to show the failure happens before inference, but the token and latency comparison only tells you direction. To test whether v21.3 reduces the model emitting half a JSON object in the first place, you'd need an A/B with the real model, and that machine's 16GB of VRAM can't fit a second instance.
- After the switch on machine one, I only have a 13-request sample. I'd scheduled a follow-up check for two hours later, and looking back now I can't find its output. So "the errors stayed at zero" is only something I can say about those 13 requests.
7. Four questions before swapping in a "fixed" template
- Does the error text say something like
template render failed? If not, leave the template alone for now. - Send a broken request with
max_tokens=1and time it. If it returns far faster than a single prefill, the error is in a layer before the model. Then check whether that layer is the template. - Dedupe the errors by prompt hash. If error count ÷ distinct hashes is well above 1, you're looking at poisoned sessions; fix history sanitization first.
- Work out "extra tokens ÷ your typical prompt length." If most of your traffic is short requests, the template costs you something real. For any performance claim, check whether its test code ever called an engine.