Going from a 4GiB to a 6GiB prompt cache should have bought 8 to 10 more points of hit rate. Why did it last only 61 minutes?

By my math, taking the prompt cache from 4GiB to 6GiB would raise the hit rate by another 8 to 10 percentage points.

It lasted 61 minutes. Before the 50-minute mark the kernel's OOM killer took it out once. It restarted on its own, ran for another ten minutes or so, and then my own memory guard switched it back to 4GiB.

The math wasn't wrong. What went wrong was one addition, and I could have done it before I flipped the switch. That's section 3, and it takes three numbers. If you only want that addition, go to section 3. If you want to see how a "hit rate" number can fool you, go to section 4.

1. Why bother: 47 seconds vs 3.4

The machine is a gaming laptop with an RTX 3080 Laptop GPU (16GB of VRAM) and 16GB of system RAM. The model is a 14.3GB quantized 35B MoE running on llama.cpp inside WSL2, the Linux VM that ships with Windows.

VRAM and system RAM are separate pools here. Weights and the KV cache live on the GPU. The prompt cache in this post lives in system RAM, and WSL only gets about 12GiB of that.

What feeds it is a multi-agent simulation I run for myself. Several characters take turns sending requests, and each one drags along 30K to 55K tokens of context. Every turn adds a short new chunk, and the tens of thousands of tokens before it stay the same.

A prompt cache does something simple. It keeps the computed state of a long prefix in memory. When the same prefix comes back, the model picks up from where it left off and skips prefilling it again.

How long a miss costs is one division:

time to first token ≈ prompt length ÷ prefill speed

This box prefills at roughly 1,078–1,130 tok/s. 45K tokens ÷ 1,100 ≈ 41 seconds. With the cache at 1GiB, requests over 45K tokens with short outputs had a median wait of 47 seconds. That's about what a full recompute should cost, and that bucket also includes requests longer than 45K.

After I grew the cache to 4GiB, the median for the same bucket dropped to 3.4 seconds. A median that low means most of the requests in that bucket were hits.

That's a 14x gap, which is why this was worth messing with.

2. First cut: 1GiB only held two or three entries

The old setting was --cache-ram 1024. I measured how big a few long prefixes were once they were saved:

• 54K tokens: about 552 MiB
• 40K tokens: about 346 MiB
• 30K tokens: about 417 MiB

Size doesn't scale cleanly with length, so don't assume it's linear. Just use the measured range:

entries that fit ≈ cache-ram ÷ size per entry
1024 ÷ (346 to 552) ≈ 2 to 3 entries

With characters taking turns, each entry got pushed out by the next one right after it was saved. The cache was on, but it barely did anything.

I raised it to 4GiB and changed nothing else. After a full 6 hours, here's how it compared with the 6 hours right before the switch:

Metric	1GiB	4GiB
Forced full recompute	545	311 (−43%)
LRU evictions	535	278 (−48%)
Hit proxy, long prompt + short output	20.0%	60.2%
Median wait, 30K–45K bucket	55 s	32 s

That's a real win. But over the last three hours it was still evicting about 31 entries an hour. The cache held 13 entries totaling 4.11GiB, so it was full. Over those 6 hours, 42 distinct prefixes showed up.

13 entries in 4.11GiB works out to about 324 MiB each on average. At that average, 6144 ÷ 324 ≈ 19 entries.

3. Second cut: the 6GiB math, and the addition I skipped

I didn't pick the new size by gut feel. I used reuse distance. When a request hits a prefix you've seen before, count how many other distinct prefixes came in between the two visits. If the cache holds N entries and evicts whatever was used longest ago, every request with a distance under N is a hit.

Running that over the real requests from those 6 hours: going from 13 entries (4GiB) to 19 (about 6GiB) buys another 8 to 10 points. Going to 25 (about 8GiB) adds roughly 7 more, and the returns start shrinking.

I ruled out 8GiB on the spot. The whole machine has 16GB, and at 4GiB the Windows side already had only about 2.2GiB free. For 6GiB I added two safety nets:

• An 8GiB hard memory limit on the container, so it couldn't eat all of WSL
• A guard that samples memory once a minute. If WSL's available memory stays under 2GiB for 3 samples in a row, or Windows free memory stays under 0.75GiB for 3 in a row, it switches back to the 4GiB container automatically.

Here's how I picked the 8GiB limit. After the switch, when the cache had filled to about 19 entries, the container's resident memory was 7.13GiB. That left almost 1GiB of headroom, and I decided that was enough.

Before 50 minutes were up, this showed up in the kernel log:

llama-server invoked oom-killer ... constraint=CONSTRAINT_MEMCG
Killed process ... (llama-server) anon-rss:8176388kB

CONSTRAINT_MEMCG means it hit the container's own limit. The machine as a whole still had memory left. The last cache state logged before it died: 26 entries, 5,850 MiB. That's still almost 300 MiB short of the 6,144 line.

The process died before the cache was even full. Here's the addition I skipped:

container memory limit ≥ cache-ram + resident memory outside the cache + headroom

How much is resident outside the cache? Subtract it straight from the crash:

anon-rss 8,176,388 kB ≈ 7,985 MiB
7,985 − 5,850 (cache at that moment) ≈ 2,135 MiB ≈ 2.1GiB

That 2.1GiB is the process's own runtime overhead. It doesn't change with the cache size. Plug it back in:

6,144 + 2,135 = 8,279 MiB > 8,192 MiB

Even with zero headroom, it's 87 MiB over.

So the setup had no way to work from the moment I switched. As soon as the cache grew to a point still short of full, the container was guaranteed to die first.

My 7.13GiB was a reading taken while the cache was still filling. At 19 entries the cache was nowhere near 6,144, and even at 26 it was only 5,850. I took a halfway reading and treated it as the full-load number, and that was the entire mistake.

4. 6GiB hit 9.1%, 4GiB hit 51.3%: did a bigger cache make things worse?

When I looked back at it the second time, the table had these two rows:

• The 6GiB stretch: 9.1% hit rate
• The 5 hours after rolling back to 4GiB: 51.3% hit rate

From those two rows alone it's easy to conclude that a bigger cache gave a lower hit rate.

Start with the denominator. The 6GiB stretch lived for about an hour, and only 11 long requests in that window qualified. One hit out of 11 is 9.1%.

weight of a single sample = 100% ÷ n
n = 11 → each request is worth 9.1 points

One more hit makes it 18.2%. Another makes it 27.3%. When a single request can move a number by 9 points, that number tells you nothing.

Then look at what happened inside that hour. It got OOM-killed once. After the restart the cache was empty and had to fill up again, so those ten-odd minutes couldn't have produced hits no matter what.

Then there's the definition. In this second review, a request counted as a hit if the prompt was over 20K tokens, the output was under 400, and the implied prefill speed worked out above 3,000 tok/s. Raw prefill on this machine is around 1,100, so beating 3,000 only happens if the cache was reused.

The 20.0% → 60.2% in section 2 used a different test: whether llama.cpp's log showed a recompute. Both numbers are called "hit rate," but you can't compare one to the other.

So when someone posts a cache hit rate, ask three questions: how a hit was defined, how many requests are in the denominator, and whether the window includes a restart. If any of the three goes unanswered, don't trust the number yet.

5. The guard saved the day, but missed the moment it mattered most

The guard's log looked like this:

09:55:45 wsl_avail=8037948 win_free=576496  bad=1
09:56:55 wsl_avail=7867180 win_free=575744  bad=2
09:58:05 wsl_avail=7409412 win_free=437904  bad=3
09:58:10 ROLLBACK_OK restored remote 4GiB container

What tripped it was the Windows check. While 6GiB was running, free memory on the Windows side slid from about 2.6GiB down to 0.42GiB. After three bad samples it switched back within 5 seconds, cleanly.

Scroll back to the OOM, though, and the guard logged two lines:

ERROR could not read remote WSL MemAvailable

In the seconds when the process was being killed, WSL couldn't even answer a remote memory read. The guard's code treated "couldn't read" as "no data this time," skipped it, and didn't bump the counter. That was the one moment in the whole run that most needed an alarm.

A failed read is a signal in itself. In the guard's logic, a failed read should count as a bad sample. At the very least, it shouldn't be treated as if nothing happened.

One more thing was taking up cache space, and it had nothing to do with the simulation. I have a load balancer in front of a few machines, and every so often it sends a health check a few dozen tokens long. In the 5 hours after the rollback, 357 of the 715 prefills were requests under 100 tokens. At 4GiB, the tiny 31- and 33-token entries took up about 377 MiB of cache, roughly 9%. 26 entries totaling just 5,850 MiB averages only 225 MiB each, and these tiny entries are probably a big part of why the average dropped.

I tried adding cache_prompt:false to the health checks. It didn't help. Reading the source showed why. That flag controls whether this request may reuse existing cache. The host-memory cache layer saves the old slot's whole state the next time the slot gets replaced, no matter who sent it. The name sounds like "don't cache me." What it actually does is "don't use the cache to speed me up."

6. Where this experiment is weak

• 6GiB ran for about 61 minutes total, in two pieces with an OOM restart in between. The "8 to 10 more points" is a prediction from reuse distance and was never tested at all. Nothing confirmed it and nothing ruled it out.
• Both hit rates are proxies. They infer reuse from "long input, short output, fast," and I didn't check request by request against the cache logs.
• The 2.1GiB of non-cache overhead comes from one OOM crash, one model, one llama.cpp version. Change the model, the context length, or the number of parallel slots and that number moves. Measure it yourself.
• The anon-rss in the kernel log only counts anonymous memory, and the container's memory accounting includes other things. So the real non-cache overhead can only be larger than 2.1GiB. My subtraction is a floor.
• One machine, running inside WSL2. The cache_prompt behavior is what I saw in the source of the llama.cpp version I was running. Other versions may have changed it.

7. Next time you grow a prompt cache, go in this order

1. Measure how big a few real prefixes are once saved. cache-ram ÷ size per entry tells you how many fit. Compare that with how many distinct prefixes are active at once.
2. Pick the size with reuse distance. Look at how much each extra entry buys, and stop where the returns flatten.
3. Set the container or process memory limit to cache-ram + resident memory outside the cache + headroom. Get the outside-the-cache part by subtracting current cache size from current resident memory. Don't take a total resident reading from a half-full cache and treat it as the full-load number.
4. Then do the next layer out: how much the host has left. WSL, VMs, and Docker each have their own ceiling.
5. In your memory guard, a failed read counts as a bad sample.
6. Keep short requests like health checks off the cache your long sessions use. If you can't, at least subtract what they take from your capacity math.
7. Before comparing hit rates, line up the definitions, look at the denominators, and check whether either window includes a restart. One sample is worth 100% ÷ n, so if n is only a dozen or so, don't compare.