Batch size went from 2 to 16 and throughput went from 65.7 to 67.4. What is batching actually saving?

Same M1 Max, same 35B MoE model. My own inference engine decoding B sequences at once:

B=2    65.7 tok/s
B=4    67.2
B=12   67.3
B=16   67.4

All 16 sequences produced output identical, bit for bit, to what each one produced when run alone. Perfect correctness, flat throughput.

Next to it, MLX running the same model: 65.7 at B=1, 132.8 at B=16. A bit over 2x.

The awkward part: two months earlier I had written a document that used a very tidy byte ledger to prove batching wasn't worth it on this model. My own measurements later knocked out all three of its pillars. This post is about where that ledger went wrong, and how the fix let me read the flat line above at a glance. If you just want the formula, jump to section 3. If you want to know how to take apart someone else's batching numbers, section 5.

1. The ledger I started with

Context: I'm writing an inference engine that serves exactly one model, Qwen3.6-35B-A3B. It's a mixture-of-experts model: 256 "experts", and each token wakes up only 8 of them. On top of that, 30 of its 40 layers use a form of linear attention (Gated DeltaNet), which means every sequence carries its own recurrent state from token to token.

Decode, the phase where tokens come out one at a time, is almost entirely spent hauling weights out of memory. Compute sits idle most of the time. So the textbook case for batching goes: read the weights once, serve B sequences, and each token's share of the bytes drops to 1/B.

I split the bytes into three buckets:

Regular projections and the output head: shared across all B sequences, so total bytes stay flat. This is where the win comes from.
MoE experts: each sequence wakes different experts. At B=16 you expect to touch about 102 of them, so expert reads inflate 12.7x.
The linear-attention recurrent state: one copy per sequence, so traffic grows linearly with B.

Then I ran a probe on a batched matrix-vector kernel and concluded that register pressure exploded at B≥2 and that at B=16 each token actually got 15% slower.

Put together, I called these "the three killers," and wrote the verdict in hard terms: the first two are baked into the model architecture, and no kernel work can fix them.

Around the same time, a closed-source Mac inference engine refused to enable batching for this exact model, with a startup log line saying the batched MoE plus Gated-DeltaNet path was "unvalidated." I took that as corroboration: see, they hit the wall too.

None of the three killers survived.

2. How the three killers fell

The register probe went first. Its 4-bit weight unpacking was wrong: one nibble was never shifted right, and one offset was added in the wrong units. It never got caught because the probe only compared B=1 against B=n, i.e. itself against itself, with the same broken unpacking on both sides. Of course they agreed. A self-comparison proves you didn't break anything. It can't prove the math is right.

I rewrote it with an independent CPU fp32 reference, all 9 configurations matched, and remeasured B=16:

Killer           What the byte ledger said       Measured per-token speedup at B=16
MoE coverage     ceiling around 1.26x            3.59x
Recurrent state  linear blowup, no gain          10.34x
Registers        B=16 is 15% slower              3.35x (weighted)

The MoE row is the interesting one. The coverage blowup really happened: B=16 touched 98 experts, close to my Monte Carlo estimate of 102. By bytes alone, it could get at most 1.26x faster. It measured 3.59x, nearly three times the ceiling.

The ledger itself was fine. What broke was an assumption it never stated: that at B=1, bandwidth is already saturated. It wasn't. Not even close.

3. The formula: batching speedup = byte ratio × bandwidth ratio

The memory bandwidth ceiling on this machine measures 367 GB/s, using buffers of 128 MB and up. Small buffers hit cache; I once got a bogus 646 that way. Divide a few operators' measured B=1 bandwidth by that ceiling:

Linear-attention recurrent state   27.6 GB/s    7.5%
MoE experts                        58.7 GB/s   16.0%

At B=1 this GPU spends most of its time waiting, not moving data. One sequence doesn't hand it enough work to stay busy.

So the batching gain splits into two factors multiplied together:

per-token speedup = byte ratio × bandwidth ratio
byte ratio      = bytes read per token at B=1 ÷ bytes read per token at batch B
bandwidth ratio = measured bandwidth at batch B ÷ measured bandwidth at B=1
                  (upper bound = 367 ÷ measured B=1 bandwidth)

One division times another division. Plug in the measurements:

Recurrent state: one copy per sequence, byte ratio = 1.00
                 bandwidth 27.6 → 285.3 GB/s, bandwidth ratio 10.34
                 1.00 × 10.34 = 10.34, measured 10.34

MoE experts:     B=1   0.1607 ms × 58.7 GB/s ≈ 9.4 MB/token
                 B=16  0.716 ms × 161.4 GB/s ≈ 115.6 MB, ÷16 ≈ 7.2 MB/token
                 byte ratio ≈ 1.30, bandwidth ratio 161.4 ÷ 58.7 = 2.75
                 1.30 × 2.75 ≈ 3.59, measured 3.59

Both match. My original ledger only computed the first factor. For MoE the byte ratio really is only about 1.3, close to the 1.26 I estimated back then, so I got that part right. What I never computed was the 2.75 after it.

It works in reverse, too. The output head (the big matrix that maps the hidden state onto a 248K-token vocabulary) already hits 264.3 GB/s at B=1, 72% of the ceiling. The bandwidth ratio can be at most 367 ÷ 264.3 = 1.39. My implementation also re-read the weights once per sequence, so the byte ratio was 1. Ceiling: 1.39x. Measured: 1.23x, and it didn't budge anywhere from B=2 to B=16.

I had picked it as the first whole-graph slice because it's stateless and easy to change. When I picked it, I looked at how much of the runtime it took, not at how close to full it already ran. It was already fed. Batching had nothing left to feed it.

The one-line rule: the lower an operator's bandwidth utilization at B=1, the more batching is worth. If an operator already runs at 70% on its own, you can batch as much as you like and it won't move much.

4. Back to the flat line

With that formula the opening numbers read easily. Convert to per-token time:

B=2    1000 ÷ 65.7 = 15.2 ms/token
B=16   1000 ÷ 67.4 = 14.8 ms/token

B went up 8x and the cost per token barely moved. That shape only happens when both factors are 1.

The code agreed. This version only made the data path multi-sequence: every sequence got its own recurrent state and its own slice of the KV cache. But each layer was still invoked once per sequence, so every projection and every expert was still a single-token matrix-vector multiply that read the weights B separate times. It was batching on paper. The GPU saw exactly the same work as B=1, just queued B times over.

This path was also slower than the engine's single-stream serving mode at 88.6. That's because serving mode overlaps CPU and GPU work, and this path doesn't.

I had written a gate for this phase: B=16 aggregate throughput has to beat MLX's ~132, and B=1 must not regress. Miss it and you stop. The flat line came in and I stopped. The multi-sequence kernels planned for the linear attention and regular attention layers never got written, because the bottleneck wasn't in them at all.

Honestly, I'm a little relieved that gate existed. Without it I'd probably have followed the plan, written the kernels, and only then found out it was wasted work.

5. Taking apart someone else's batching numbers

When you see "batching made it X times faster" or "batching doesn't help," ask three things.

First, are they looking at per-token time or at GB/s? I got burned here. After moving the output head onto the batched path, the reported bandwidth fell from 264 to 20 GB/s and I immediately declared it "fully serialized." Later, on a different small matrix, a true batched kernel's bandwidth also fell, from 37 to 7.7 GB/s, while each token got 3.35x faster. The numerator of that GB/s figure is a fixed weight size. As B grows and total time grows, the ratio has to shrink. A ratio with a fixed numerator will point the wrong way if you use it to judge concurrency. Only look at how amortized per-token time changes with B.

Second, is this a microbenchmark or the whole graph? A microbenchmark measures a bare kernel. The whole graph runs the full path, including the per-sequence work that can't be shared. I got caught twice: I extrapolated the output head from a 4.96x microbenchmark on a matrix of the same kind and the whole graph measured 1.23x; a shared expert that microbenchmarked at 2.80x landed around 1.7x in the whole graph, a 40% haircut. Ask what that block's B=1 bandwidth utilization is inside the whole graph. Above 60–70%, the microbenchmark multiplier mostly won't show up.

Third, who said it's "not supported"? That "unvalidated" log line only tells you that one engine hadn't done it. On the same machine, MLX's batch_generate runs the same model at a bit over 2x at B=16. Somebody else's WARNING, allowlist, or "not supported" is a lead, not a wall. I used it as one of the pillars of my wall and left it in my docs for two months, until I went back and tagged every "can't be done" with where its evidence came from and found that this one was borrowed.

6. Which numbers here aren't solid

The opening flat line is a rough measurement: the throughput includes the output head and a CPU-side argmax, and context was capped at 4096. Trust the shape, not the absolute values.
The 10.34x, 3.59x, and 3.35x figures come from microbenchmarks on synthetic probe data, not the whole graph. The recurrent-state probe in particular is a naive version I wrote that launches only 32 threadgroups at B=1, so it's underloaded by construction. The production kernel in the engine launches 4096 threadgroups and is much faster at B=1, so the real batching gain there is certainly nowhere near 10x. The 10.34 demonstrates the mechanism, not the size.
The formula doesn't always fit: in the same round of microbenchmarks, the 2048-row and 512-row projections roughly match (bandwidth ratios 4.61 and 3.60, measured 4.96 and 3.48, within 7%), but the 8192-row one doesn't. Bandwidth went 148.2 → 259.0 GB/s, a bandwidth ratio of 1.75, and even the ceiling-based bound is only 2.48, yet it measured 2.62. That kernel re-reads the weights per sequence, so its byte ratio should be 1. I haven't broken down where the extra came from; cache absorbing some of the repeated reads is a possibility.
My own whole-graph estimate of about 157 tok/s at B=16 has never been measured. It's a weighted extrapolation from per-block microbenchmarks. An earlier version of the same method produced 328 and was shot down by whole-graph data on the spot. I don't trust 157 either, until a real batched kernel lands.
Run order bit me: the same configuration run sequentially twice gave 1.64x and 2.05x, because this machine's dynamic clocking penalizes whichever setting runs first. It only settled after I interleaved all settings strictly within one process for 7 rounds and took the median, and the spread was still 14–32%.
The MLX numbers are its own reported generation throughput, 3 rounds per setting, 256 tokens, 8 long-form prompts in rotation. The accounting isn't identical to my engine's, so treat it as an outside reference.
One machine, one model: everything here comes from one M1 Max 64GB and a 4-bit 35B-A3B. Change the hardware or the model and every number changes. The formula doesn't.

7. Before you add batching, work through this in order

Measure each large operator's actual bandwidth at B=1. For the denominator, use a ceiling measured with large buffers, not the spec sheet.
Compute the bandwidth-ratio ceiling: ceiling ÷ B=1 bandwidth. Don't expect batching to help operators below 1.5.
Compute the byte ratio: shared weights divide by B, MoE inflates by expert coverage, and per-sequence state doesn't amortize at all.
Multiply the two, and that's your estimate for that operator. Before weighting by whole-graph time, order the work by utilization from lowest to highest, not by share of runtime.
To tell whether batching took effect, look only at amortized per-token time as B grows. If it doesn't fall, you haven't batched anything, however nice the data path looks.
Put a question mark on any microbenchmark multiplier before it goes into the whole graph, especially for blocks already running at 60–70% at B=1.
Check memory first: this model's KV cache is about 45 KB per token, so a fully provisioned 256K context is 11.8 GB per sequence and B=4 doesn't fit. Either lower the context or move to paged KV (I measured the indirect-addressing cost at 0.1–0.9%).
Write yourself a gate: B=16 aggregate throughput beats the off-the-shelf option, and B=1 doesn't regress. Miss it and stop.