AriakiThe first request took 38.52 seconds. I sent the same input under a fresh session ID, and it took just 0.45 seconds.
A wonderfully fast new session? The cache counters told a less exciting story: 0 input tokens cached on the first request, all 3633 cached on the second. Fresh label, old work.
The embarrassing part came earlier. My test had already detected reuse across sessions, and I had marked it green. I wrote “less reuse” where the experiment needed “no reuse.”
If you want to check your own cold-cache test, jump to section 2. For the broken green light, keep reading.
1. My negative control had an open door
This was a local-model experiment for my own use, recorded on 2026-08-16. The machine was an Apple M1 Max with 64 GB of memory, running MTPLX 2.7.1 and a mixed-precision version of Qwen3.8-27B. The behavior below belongs to that version; it makes no claim about later releases.
I wanted to establish how session caching worked before running a full multi-turn test. After reading an input, a model can retain intermediate computation and reuse it when another input starts the same way. That's prefix caching. The saved work is real. I had misunderstood what identified the reusable work.
My first probe sent consecutive requests in one session, then sent the same prompt—the input fed to the model—in another session. The experiment report preserved these observations:
| Request | Request duration | Input tokens | Cached input tokens |
|---|---|---|---|
| A, first turn | 25.51s | 2347 | 0 |
| A, second turn, same session | 2.51s | 2405 | 2382 |
| B, second turn, different session, same input | 1.04s | 2405 | 2346 |
B had plainly reused 2346 tokens. How did I let that pass?
Here's the assertion I used for the negative control:
B.cached <= A.cached
2346 <= 2382 → pass
That only checks whether B reused more than A. B could reuse almost everything and still pass. The door was open; I was checking the nameplate.
If the hypothesis is “a different session will not reuse this prefix,” the passing condition must require zero reuse of that prefix. Shared template content needs separate accounting, too. Otherwise even the location of the expected zero is ambiguous.
That was the point to stop and repair the experiment. More turns would only have added decimal places to a bad premise.
2. A fresh beginning separated the two variables
For the next probe, I put a previously unused random marker at the beginning of the input. Nothing magical about the marker. Its job was to make the prefix new.
I sent C first and confirmed zero cached input tokens. Then I kept the input identical, changed only the session ID, and sent D. The report recorded:
| Request | Request duration | Input tokens | Cached input tokens |
|---|---|---|---|
| C, fresh prefix | 38.52s | 3633 | 0 |
| D, same prefix, fresh session | 0.45s | 3633 | 3633 |
Now the cache accounting was hard to dodge. You can do the arithmetic yourself:
Input cache hit rate = cached input tokens / input tokens
C: 0 / 3633 = 0%
D: 3633 / 3633 = 100%
And the input left uncached:
Uncached input = input tokens − cached input tokens
C: 3633 − 0 = 3633
D: 3633 − 3633 = 0
That zero describes uncached input under the reported counter's semantics. Parsing the request, restoring cached state, and generating output still take work. Don't read it as zero computation for the whole request.
The session diagnostics agreed. Both requests were marked new_session, both reported reference_lease as the restore mode, and their prefix token_hash values matched. Think of the hash as a content fingerprint: the session name changed, the fingerprint didn't, and the engine could borrow the existing computation.
That version reused prefix state by content across sessions. The session ID did not partition this cache. For my own repeated inputs, that's useful. For an experiment equating “new session” with “cold cache,” it's a problem.
I did not start the full multi-turn test, following the experiment's existing gate. First separate the variables. Then compare settings.
3. Before crediting a setting, check what did the work
Turning those durations into a speedup chart would be easy:
Request-duration ratio = 38.52 / 0.45
But the denominator belongs to a request whose entire input was cached. Calling that “faster model computation,” or attributing it to a newly added session parameter, would credit the wrong thing for the avoided work.
This pair did not test what happens with no session ID at all. It therefore cannot establish that IDs are irrelevant on every path. The result is narrower: changing the ID was insufficient to create a cold cache, and a cold-then-warm sequence cannot establish that a parameter caused the speedup.
When reading someone else's benchmark—or your own—look for this pattern:
- The input stays unchanged; the old setting runs first and the new setting runs afterward.
- “New session” or “new request” is the only justification for calling the later run cold.
- The duration drops, but input and cache-hit counts are missing.
Hold the applause. The later run may simply have picked up work left by the earlier one. Asking for a cache-counter column is more useful than arguing about the architecture.
There's another trap here. An uncached prefix, a newly started process, and a freshly loaded model are different kinds of cold. Changing the input helps isolate prefix reuse. It doesn't establish model-loading or first-compilation costs.
This record has limits, too. What remains is an experiment report containing durations, counters, and diagnostic fields. The probe scripts lived in a temporary directory; this article supplies no downloadable raw request trace and does not rerun that engine version. I retain the label “request duration,” without relabeling the numbers as time to first token or pure input-processing time. The pair establishes that cross-session reuse occurred in that test. The available repetition and output-length records cannot support a stable speedup factor or hardware ranking.
4. What to do next time
- Specify which cold condition you mean: uncached prefix, fresh process, or freshly loaded model.
- To test prefix reuse, start with previously unseen input content and check the first request's cache counters. Account for any shared template prefix separately.
- Keep the input identical, change only the session ID, and read the counters again. Test session identity and cache partitioning separately.
- Make the negative control assert that the forbidden reuse did not happen. “Less than the positive control” is insufficient.
- When comparing settings, hold the model, input length, and output conditions fixed. Report cold-cache and warm-cache results separately, interleave the order, and repeat.
- Save the raw requests, timing boundaries, cache fields, and versions. Mark missing evidence explicitly; don't turn a single duration ratio into a claim about model speed.