AriakiI wrote one line in a concurrency controller: per_backend = 8. I didn't think about it. I just typed an 8.
For the rest of the day I built on top of that 8. I fixed three bugs around it, drafted four optimization plans, and made one "correction": I noticed the health check was counting 2 machines as 4, so I cut the concurrency cap from 32 down to 16.
The more correct I made it, the further it got from the truth. That miscounted 32 was the closest thing to the real answer.
Later I replaced the 8 with a measured number. The cap became 28, and end-to-end latency dropped from 20.47 s to 17.50 s. Nice win. Then I deleted the cap altogether.
If you want the formulas, sections 3 and 4 are each one division. If you want a way to see through "utilization is only 41%, concurrency isn't the problem," read section 2.
1. What the cap was for
I run a simulated world: about thirty AI characters live in it and make their own decisions. Each time the world advances one step, every character acts at once, and the next step only starts after all of them finish. That all-must-arrive checkpoint is a barrier. Think of a class handing in homework: the teacher doesn't start the next lesson until the last kid turns theirs in.
The backend is two inference boxes, each running vLLM. Every character action is one model call.
So each time a barrier opens, a crowd of requests hits those two boxes at the same moment. I wrote an adaptive concurrency controller: tighten when latency climbs, loosen when things are calm, never exceed a hard cap:
cap = healthy machines × per_backend
Everything went wrong at per_backend = 8. Nobody had ever measured it.
2. The mean said "not full." The p90 said "full since the first second."
The first time I suspected concurrency, I looked at the average number of in-flight requests: 6.59, against a cap of 16.
6.59 ÷ 16 ≈ 41%. Clearly not saturated. Concurrency can't be the bottleneck. I went off to look at other things.
Later I asked another model to review my plans, and it dismantled that conclusion in one line: I had counted the wrong thing. I counted "activities per step," median 10. But each character makes its own calls, and one activity can involve several characters. Recounted: every barrier launches more than 16 characters at once, typically 29.
So where did 6.59 come from? At the start of a barrier, 29 requests land together and the cap splits them into two waves. After that there's a long stretch of waiting for the slowest few characters to finish, and in-flight count drops to single digits. The mean blends the spike and the idle tail into one harmless-looking number.
The real fingerprint is in the queue-time distribution:
queue time p50 = 0.3 s
queue time p90 = 30 s
The median request barely queues. The p90 waits half a minute. A split that wide means most requests made the first wave, and a minority got pushed into a second wave that only starts once someone in the first wave finishes.
The wave count is just a division:
waves = ⌈ simultaneous requests ÷ concurrency cap ⌉
⌈ 29 ÷ 16 ⌉ = 2
How to see through it: when someone tells you "average utilization is only 40%, concurrency isn't the issue," first ask whether the load is bursty. Under bursty load, mean utilization tells you almost nothing. Look at how far apart queue-time p50 and p90 are. Two orders of magnitude means requests are being chopped into waves by the cap.
3. Break it down before you optimize: all four plans hit the 13.7%
Before I found this, I had spent a whole day measuring: concurrency, prefix cache hit rate, prompt structure, barrier structure. Out came four plans, all of them about slimming the prompt.
The reviewer asked a question I couldn't answer: for the slowest calls, how many seconds go to queueing, prefill, and decode?
Prefill is the model reading the whole prompt once; decode is it emitting tokens one at a time. vLLM exports histograms for all three phases, so no instrumentation needed:
vllm:request_queue_time_seconds queueing
vllm:request_prefill_time_seconds prefill
vllm:request_decode_time_seconds decode
vllm:e2e_request_latency_seconds end to end
A 15-minute window, 528 requests:
| Phase | Per request | Share |
|---|---|---|
| Queue | 3.75 s | 18.5% |
| Prefill | 2.78 s | 13.7% |
| Decode | 13.72 s | 67.8% |
| End to end | 20.47 s |
Slimming the prompt only touches prefill. All four of my plans were aimed at the 13.7%.
max end-to-end gain from an optimization = share of the phase it touches × fraction it removes
13.7% × 50% ≈ 6.9%
Even if I cut the prompt in half and prefill time shrank linearly with it (already optimistic), end-to-end gets about 7% faster. Do that division and all four plans get demoted on the spot.
The queue share, 18.5%, was the line pointing at the real problem: the 8.
4. Replacing the 8 with a measured number
How many requests a server can hold at once has two hard limits: how many copies fit in the KV cache (the scratch space the model keeps for each in-progress conversation), and the max number of parallel sequences set in vLLM's launch flags.
per-box capacity = min( total KV cache capacity ÷ p90 prompt length , max_num_seqs )
= min( 1,490,818 ÷ 100,000 , 160 )
= 14
two boxes = 28
KV cache capacity is a line vLLM prints in its startup log. It isn't exported as a metric, so you have to dig it out of the log. The p90 prompt is my measured 100k tokens.
I also ran the most pessimistic version: assume every request fills the model's whole context window. One box holds about 11.37 of those, two boxes about 22. Even that pessimistic, 22 beats 16. My hand-typed 8 didn't even reach the pessimistic floor; against the measured 14, it underestimated by 43%.
Same window length, same request count, before and after:
| Cap 16 | Cap 28 | |
|---|---|---|
| End to end | 20.47 s | 17.50 s (−14.5%) |
| End to end per box | 32.91 / 12.18 s (2.7×) | 17.90 / 17.05 s (1.05×) |
| Prefix cache hit rate | 58.3% / 54.1% | 71.6% / 63.2% |
| Preemptions | 0 | 0 |
The second row was a bonus. I had filed "the two boxes are unevenly loaded" as a separate problem and planned to fix the routing later. Raising the cap made it disappear on its own.
Once you see it, it's simple: routing is sticky, so a given character's requests always go to the same box to reuse its cache. With a cap of 16, each box had 8 slots, the long-prompt characters piled up queueing on one box, and the other sat idle. At 28 both boxes had room and the queues evened out.
One caveat right here: prefix cache hit rate went up too, so not all of that 14.5% belongs to the cap. The two windows were consecutive time periods, not a simultaneous A/B.
5. So why delete it
28 is much better than 16. But plug it back into the section 2 division:
⌈ 29 ÷ 28 ⌉ = 2
Still two waves. The second wave just shrank from 13 requests to 1.
Then I stared at the function and asked: why does this cap exist at all?
I pulled the simultaneous launch count for all 681 barriers: median 29, max 32. The world only has about thirty characters. Concurrency can never exceed the roster.
So every possible cap is in one of two states:
- Below 32: throttles at least some barriers. Both my 16 and my 28 live here.
- 32 or above: never reached. Dead code.
There is no value that is actually useful.
The other half of the argument is on the server side. In the cap-28 run, KV cache usage peaked at 54.9%, with zero preemptions. At the moments requests did queue, KV usage was only 22%; they were waiting on vLLM's per-step prefill token budget (8192 tokens), with plenty of memory to spare.
In other words, I was working hard on the client to estimate "how much can the server take," and the server already knows. It queues, it reports "requests waiting for capacity" in its metrics, and if it truly can't cope it preempts. The client's estimate can only ever be a degraded copy of that signal.
The final version is two lines:
def fleet_ceiling(self, healthy: int) -> int:
if healthy <= 0:
return self.cfg.min_limit # every box is down; don't fire requests into the void
return self.cfg.max_limit # anything alive: let the server speak for itself
max_limit changed meaning too. It's now a runaway fuse, set to 48, well above the roster's 32, so it's never the thing doing the throttling. The KV-probing code I had written for the previous version, about 100 lines that scraped capacity out of logs, went out along with its 6 tests.
After the change, the controller log went from printing 32 -> 28, clamp: fleet ceiling every time to 32 -> 32.
6. The tests had pinned the bug in place
Part of why that 8 survived a whole day was the tests. The old test said:
check("never exceeds the fleet ceiling", max(seq) <= 16) # 2 × 8 hard-coded into the assertion
Hard-coding a capacity constant into an assertion turns the bug into the spec. Anyone trying to raise the 8 gets a red test first.
After deleting the cap I added the reverse mutation test: deliberately put the healthy × 8 cap back and see whether the tests notice. Six test cases caught it. If anyone ever tries to sneak that design back in, the suite goes red.
One small trap along the way. On the second version (the measured-capacity one), the first round of mutation testing had 2 of 9 mutants survive. The fixture gave both boxes identical parameters, and their p90s landed in the same bucket, so "take the narrowest box" and "take the widest box" produced the same answer. If every sample in your fixture is identical along some dimension, that dimension is untested.
7. Where this experiment is weak
To be honest about it:
- The breakdown and the before/after comparison each use a single 15-minute, 528-request window. I didn't repeat runs to see the variance.
- Cap 16 vs cap 28 compares two consecutive periods, and the prefix cache hit rate changed at the same time. I can't split the 14.5% between the two.
- After deleting the cap, I didn't measure end-to-end latency as a separate run against cap 28. The only evidence it didn't get worse is the controller log reading
32 -> 32, KV peaking at 54.9%, and zero preemptions. Strictly speaking, the gain from the deletion itself is unmeasured. - "The server reports its own saturation" relies on vLLM's queueing and preemption. Swap in a backend without those signals and this conclusion doesn't transfer directly.
- The 14 ignores prefix-cache savings. Measured hit rates were 55% to 66% but swung a lot between phases; counting them risks overcommitting and triggering preemption, so I left them out on purpose.
8. Before you add a client-side concurrency cap
Ask these in order:
- Break latency into queue, prefill, and decode first. Max gain of a plan = share of the phase it touches × fraction it removes. Do the math before you touch code.
- Look at queue-time p50 vs p90, not mean utilization. If p90 is tens of times p50, requests are being chopped into waves.
- For every capacity constant in the code, ask "was this measured?" If nobody can answer, measure it:
min(KV capacity ÷ p90 prompt, max_num_seqs). - Does the workload have a natural upper bound (roster size, batch size, connection count)? If so, any cap below it throttles and any cap above it is dead code.
- Does the server report its own saturation? If so, the client only needs a "server is completely down" fallback.
- When you delete a design, write a mutation test that puts it back, so it can't quietly return.
In the controller source today, per_backend only shows up in one comment, and that comment explains how it was wrong.