AriakiI once wrote this line in my notes: "Four concurrent requests, 86–98 tok/s each, roughly 350 tok/s aggregate, continuous batching works."
The next day I took the server offline for an isolated re-test. Four equal-length requests fired at once. They finished at 5.5 s, 11.1 s, 16.8 s, and 22.5 s. One after another. Aggregate throughput: 91.2 tok/s.
350 versus 91. A 3.8× gap with zero config changes in between. The only thing that changed was how I did the arithmetic.
If you just want the test, go to section 2, it's a completion-time table and one division. If you want to see how the same log line fooled me twice, section 3.
1. Where the 350 came from
The setup is simple. An M1 Max laptop with 64 GB, running a 35B-parameter MoE model that activates 3B per token. (MoE means the model is split into many "experts" and each token only wakes up a handful of them, so you get a 35B footprint with 3B worth of compute.) The engine is a newly released closed-source inference engine for Macs, and one of its headline features is continuous batching: stacking the decode steps of several requests together so one pass over the weights serves multiple users. That's the entire secret behind server-side throughput.
My first test went like this: fire four requests concurrently, and each response comes back with a "decode speed for this request" field. The four numbers ranged from 86 to 98. All succeeded, none were slow, so surely they ran in parallel? Add them up, about 350.
The problem is what that per-request field actually measures. It's the rate from this request's first token to its last token. If four requests are queued, each one gets the whole machine to itself when its turn comes, and naturally hits the full 98. Four requests at 98 each, summed to 392, looks like four-way parallelism. It's actually one runner doing four relay legs.
All four returned HTTP 200. No errors, no timeouts. A 100% success rate proves the queue didn't blow up. It proves nothing about parallelism. I remember this one well because I carried that wrong number into the next day, mentally budgeting how many more background jobs this laptop could absorb.
2. The test: completion times are either a staircase or a cluster
I changed exactly two things. All requests are equal length (greedy sampling, forced to 512 tokens, no early stop). And I record each request's completion timestamp, ignoring whatever speed it reports about itself.
N=1 finish at 5.51s aggregate 93.0 tok/s
N=2 finish at 5.4 / 11.0s aggregate 93.4 tok/s
N=4 finish at 5.5 / 11.1 / 16.8 / 22.5s aggregate 91.2 tok/s
Aggregate throughput is one division:
aggregate = N × tokens per request ÷ time the last request finished
N=4: 4 × 512 ÷ 22.5s = 91.0 tok/s
This number needs nothing from the engine. No logs, no trust in self-reported speeds. A stopwatch is enough.
Then look at the shape. Real parallelism gives you completion times bunched together: all four start at about the same time and end at about the same time, total wall time only slightly longer than a single request (each step reads a bit more KV cache), and aggregate throughput climbs with N. A queue gives you an evenly spaced staircase, step size equal to one request's runtime, and aggregate throughput sits on a flat line no matter what N is.
In the table above the step is about 5.5 s, identical to the N=1 time of 5.51 s. Flat line, staircase, serial. There is no second interpretation.
One aside on why N=4 aggregate (91.2) is 2% lower than N=1 (93.0): queuing has overhead. Each handoff rebuilds state. Serial didn't just fail to gain, it lost a little.
3. The engine had told me all along, and I skipped the line twice
Go read the server's startup log. There's a WARNING to the effect of: this model is a Gated-DeltaNet hybrid architecture, its recurrent state can only hold a single sequence, continuous batching is unavailable for it.
So my --continuous-batching 4 flag was accepted by the engine and then quietly refused at startup. A flag being accepted and a feature being on are two different things. I saw no error and assumed it was on.
The more embarrassing part came about ten days later when a new version shipped, with release notes saying "continuous batching for hybrid models." Great. Installed it, re-ran:
C=1 C=2 C=4 C=8 per-stream
old version, default 90.3 89.5 89.6 88.5 ~98 constant
new version, default 87.5 89.3 88.7 88.3 ~98 constant
new version, CB=8 on 86.7 87.2 86.8 86.3 ~95 constant
Three flat lines. Back to the log: the WARNING had new wording. The new version's continuous batching covers dense hybrid models only; hybrid plus MoE is unvalidated and unsupported. My model is exactly hybrid plus MoE.
And with the flag on, it's 2% slower. One extra layer of paged-KV bookkeeping, then it falls back to serial anyway. A feature flag that is accepted, throws no error, and costs you a little performance, and the only place that exposes it is one WARNING line. I skipped that line both times.
The second lesson is worth more than the first: when release notes say "X is now supported," whether your model falls inside that X is something you verify yourself, and the tool is the completion-time table from section 2.
4. Is a 98 tok/s single stream fast or slow? One division
Queuing settled, one question remained. If concurrency is off the table, is there still a lot of room above 98 tok/s single-stream? My plan at the time was to write a dedicated engine for this one model, targeting 125 tok/s.
That question is also computable from two numbers: how many bytes have to be read from memory per generated token, and how much memory bandwidth this machine actually delivers. Decode is almost entirely moving weights; compute mostly sits idle. So the speed limit is "how fast you can haul bricks ÷ how heavy each brick is."
The first number can't be guessed. I audited the 4-bit weight file layer by layer, excluding the token embedding (a table lookup, one row) and the unused prediction head. The real figure is 1.849 GB read per generated token. My original plan said 1.6 GB, 13% low, because it didn't count the routed experts properly.
The second number is even easier to fool yourself with. The machine is rated at 400 GB/s, but that's the theoretical figure. I wrote a pure read-stream probe in Metal and measured:
buffer 1–4 GiB 358–366 GB/s
buffer 8 GiB 316 GB/s
buffer 16 GiB 267 GB/s
Bandwidth drops as the working set grows; page table and TLB reach is real. The model occupies 21 GiB resident, so the honest denominator is 265–315 GB/s. Using 366 is lying to yourself.
Plug in:
achieved bandwidth = 97.83 tok/s × 1.849 GB/token = 181 GB/s
utilization = 181 ÷ (265 ~ 315) = 57% ~ 68%
MoE decode has sparse, non-contiguous expert access, and in practice tops out around 60–65% of available bandwidth. At 98 tok/s the engine is already sitting on the ceiling. My 125 target back-solves to 231 GB/s, which is 73–87% utilization, physically out of reach for MoE decode. That target was built on 1.6 GB/token and 330 GB/s, one number too low and one too high, and the two stacked into a goal that didn't exist.
I had pre-registered a kill gate for the custom-engine project: if the existing engine already achieves more than 55% of measured bandwidth, close the project. Measured 57–68%. Triggered. Closed. One day of division saved six to eight weeks. It stung a bit, but the math was right there: the winnable margin was single-digit percent.
While I was at it, I decomposed the "new engine is 2.36× faster than llama.cpp" number. The llama.cpp dynamically quantized weights read 2.80 GB per token (it keeps attention layers and the output head at higher precision); the new engine reads 1.849 GB. That's 1.52× fewer bytes alone. Kernel bandwidth utilization: 181 vs 116 GB/s, another 1.56×. 1.52 × 1.56 = 2.37. No magic. Just those two terms.
5. How to tear apart someone else's concurrency number
When you see any "N concurrent, X tok/s aggregate" claim, ask three things:
First, how was the aggregate computed? If it's a sum of each request's self-reported rate, throw it out. When requests don't overlap, that sum turns serial into N× parallel on paper. What you want is N × tokens per request ÷ last completion time.
Second, are the completion times a staircase or a cluster? Give me the end timestamps of four requests. Evenly spaced, step equal to one request's runtime: it's a queue. No expertise required, just look at four numbers.
Third, does aggregate throughput rise with N? A flat line is the diagnostic signature of serial execution. Real batching climbs somewhere between N=1 and N=4. How much depends on the model and KV size, but it is not flat.
One more for yourself: an accepted flag is not an enabled feature. Read the startup log. Both of my crashes would have been prevented by that sentence.
The same method applies to prefix caching. I later sent the same 6,000-token prompt three times in a row: time-to-first-token constant at 6.8 s, cache counter constant at 0. That --prefix-cache flag was also accepted, silent, and inert. Same root cause.
6. Where this test falls short
In the interest of honesty:
- One machine, one model. Every number comes from an M1 Max 64 GB and a 35B-A3B 4-bit quant. Different hardware or model changes every absolute value. The staircase test in section 2 and the division in section 4 don't change.
- 1.849 GB/token comes from auditing the weight file, which assumes all 8 activated experts are read from DRAM every step. The M1 Max has a system-level cache that may absorb some hot experts, so real bytes read could be slightly below 1.849. That would push utilization above 57–68%, making the conclusion more conservative, not less.
- The bandwidth denominator comes from my own probe, pure read stream only, no mixed read/write. The 265–315 range is there to absorb that uncertainty.
- Decode speed degrades with context length, and all of this is short context. On the same machine a 43k-token prompt dropped decode to 34 tok/s. The 98 in section 4 is a short-context ceiling; long conversations won't see it.
- The closed-source engine's warning text is paraphrased. The wording changed between versions and I did not quote either verbatim.
7. Next time you test concurrency, in this order
- Equal-length requests, greedy, forced fixed output length. Otherwise "fast ones finish first" smears the staircase into a cluster.
- Record each request's completion timestamp. Don't trust self-reported rates in the response body.
- Aggregate = N × tokens per request ÷ last completion time. Use only this number.
- Sweep N=1, 2, 4, 8 and check whether aggregate is flat or rising. Flat means queue.
- Evenly spaced completions with step equal to single-request time: confirmed serial, stop looking for other explanations.
- After adding any performance flag, grep the startup log for WARNING and confirm it wasn't refused.
- To know how much single-stream headroom exists: audit bytes per token, measure bandwidth with a large buffer, divide, and reconcile against measured speed. Above 55% utilization, don't write your own engine.