AriakiSame M1 Max. Same attention operator. 512 tokens.
The old implementation inside my own inference engine takes 101.539 ms. A freshly open-sourced implementation takes 1.496 ms. Nine strictly alternating rounds, nine wins, 67.87x.
My first thought: prefill is about to take off.
Then I did one division. The ceiling for the whole prefill was 5.5% faster. I then assumed all seven known kernels in the engine took zero time, and the ceiling was still just 1.69x. Meanwhile another engine on the same machine is 8.7x faster than mine.
Swapping kernels can't close that gap, no matter how many you swap. If you just want the formulas, read sections 2 and 3. If you want to know how to take apart someone else's "X times faster," read section 5.
1. The experiment I almost didn't run
The setup is simple. I'm writing an inference engine that serves exactly one model, Qwen3.6-35B-A3B (a mixture-of-experts model that only wakes up a small slice of its parameters per token), on a 64GB M1 Max. Prefill, the phase where the model reads your whole prompt before it starts answering, currently runs at 96.7 tok/s. On the same machine, an off-the-shelf closed-source engine called BaseRT does 843.74 tok/s. That's an 8.7x gap.
In early September, Perplexity open-sourced an inference implementation called Lily. It targets the exact same model and the exact same 4-bit quantization format. Its source contains a check: GPU family must be at least 10, which means M5-generation chips and newer. The M1 Max reports 7.
My first write-up said "its prefill kernels are useless on older machines."
That conclusion didn't last long. The 843.74 number was measured on this very M1. If prefill on this machine can reach that speed, I should at least stop blaming the hardware so fast.
And it got more embarrassing when I looked back. My own repo already had three commits where the same newer matrix instructions compiled on this M1, ran correctly, and got tuned up to 1.98x. The evidence was sitting in my own house. I just didn't look.
So I went straight to probes:
- Took its attention kernel source unmodified and compiled it with the newer compiler. It worked.
- Built the pipelines on a family-7 device. All 11 built.
- Changed nothing but its own family check, from 10 to 7. The upstream project's test passed all 4 cases on the M1.
- Wrote a separate CPU fp32 reference and compared every output: 0 out-of-tolerance elements out of 98,304. When I deliberately added 1 to one entire row of V, the count jumped to 48,706, which shows the checker actually catches errors.
That family >= 10 is a product support whitelist. It says nothing about what the chip can run.
That's where the 67.87x came from. A clean scripted rerun gave 67.73x, a 0.2% spread between the two runs, so the number itself is solid.
The problem was what I planned to do with it.
2. First division: one operator got faster, how much faster can the chain get?
How much of the whole prefill does this operator account for? I had a phase breakdown for a 512-token prefill, 7250.8 ms in total:
| Segment | What it does | Share | Max whole-prefill speedup if this segment took zero time |
|---|---|---|---|
| gateup | MoE expert gate/up projection | 12.18% | 1.139x |
| gqkv | qkv projection in the linear-attention layers | 9.79% | 1.109x |
| asdpa | attention operator in the full-attention layers | 5.32% | 1.056x |
| gout | output projection in the linear-attention layers | 5.19% | 1.055x |
| grecur | recurrence in the linear-attention layers | 4.30% | 1.045x |
| gz | z projection in the linear-attention layers | 2.79% | 1.029x |
| lmhead | final projection onto the vocabulary | 1.40% | 1.014x |
The 67x operator from the intro is row three. It's 5.32% of the time.
The formula is Amdahl's law, and it's one division:
whole speedup ceiling = 1 ÷ (1 − p + p ÷ s)
p = this segment's share of total time
s = how many times faster this segment got
Plug in: 1 ÷ (1 − 0.0532 + 0.0532 ÷ 67.87) = 1.055
A 67x operator buys you +5.5% overall. Push s from 67 to infinity and you only get 1.056. A segment with a small share can only give back the small share it had, even if you optimize it out of existence.
Honestly, I tripped again right here. After doing the attention math, I casually said "the linear-attention recurrence is probably 4x bigger than this, that's the one worth porting." Then I checked the numbers. The recurrence is 4.30%, even smaller than 5.32%. My mouth was faster than my spreadsheet. That kind of thing can happen several times a day.
Now do all seven at once:
seven segments combined p = 40.97%
all seven take zero time: 1 ÷ (1 − 0.4097) = 1.69x
96.7 tok/s × 1.69 ≈ 163.8 tok/s
compare: 843.74 tok/s
Replace all seven kernels with magic kernels that take no time at all, and the engine reaches 163.8. That's still more than five times short of 843.
So where did the other 59% go?
3. Second division: the 59% that belongs to no kernel
Out of 7250.8 ms, the seven segments add up to 2970.6. That leaves 4280.2 ms not attributed to any compute segment.
My engine does prefill token by token: one token comes in, every layer dispatches its GPU work, then the next token does it all again. I counted the dispatches:
dispatches = layers × tokens × dispatches per layer per token
= 40 × 512 × 23.8 ≈ 486,400
fixed cost per dispatch ≈ unattributed time ÷ dispatches
= 4280 ms ÷ 486,400 ≈ 8.80 µs
Every dispatch makes the CPU encode commands, bind buffers, and insert sync points. 8.8 microseconds sounds like nothing, but multiply it by 486,000 and you get over four seconds. It's like moving a ton of bricks one brick per trip, with a turnstile on every trip. Carrying the brick is fast. The time goes to the turnstile.
Lily does it the other way around. It cuts the prompt into chunks of up to 4096 tokens, dispatches one set of work per layer for the whole chunk, and puts the token loop inside the kernel. For 512 tokens that's roughly 40 layers × 25 dispatches ≈ 1,000 dispatches, and the count doesn't grow with prompt length.
same 8.80 µs per dispatch, at a 1,000-dispatch shape ≈ 9 ms
486,400 ÷ 1,000 ≈ 486x difference in dispatch count
eliminating just that 59%: 1 ÷ (1 − 0.59) = 2.44x
Just fixing the dispatch shape has a higher ceiling than making all seven kernels free (1.69x). Most of the 8.7x gap lives in the loop structure. Kernel quality is the second layer.
So the real value of that 67x attention kernel is proof that kernels built for this chunked-prefill structure run on this machine. It's a map of the route. Pulled out on its own as a patch, it isn't worth much.
4. Third check: look at the slope, not a single point
Once I knew dispatch count was the target, I did three things in a row, and each one nearly fooled me with a single-point number.
One: reorder the loops. Switch from "each token walks through every layer" to "each layer walks through every token." Sounds like batching. Measured dispatch counts:
N=8 8,280 dispatches
N=16 16,520 dispatches
N=32 33,000 dispatches
fit: dispatches = 1030 × N + 40
The slope of 1030 didn't move at all. Per layer per token it actually went from 23.8 to 25.8 dispatches. The traversal order changed, but the inner token loop still lived on the host and still dispatched every iteration. If you only look at 16,520 at N=16, you can't tell whether anything got batched.
The check, written down:
slope ≈ 0 → tokens really moved into the kernel, dispatch tax gone
slope ∝ N → same loops in a different order, tax fully intact
fit a line over at least three values of N; a single count means nothing
Two: actually move the linear-attention recurrence into the kernel. On a standalone harness, the slope went from 1.00 per token to 0.00. At M=512, 512 dispatches became 1, the segment went from 10.152 to 4.157 ms, 2.44x, and the output matched the old implementation bit for bit.
Wired into the full chain, it was only 1.03x to 1.05x. The reason: in the full chain, this segment's baseline was no longer token-by-token. An earlier round of changes had already cut it to 2 dispatches per layer. The 2.44x in the harness was measured against the old token-by-token version, which isn't the full-chain baseline. You can't multiply those two ratios together, and you can't swap one for the other.
Three: batched dispatch. Merge a batch of small elementwise operators into a single dispatch. Whole-graph dispatches went from 17,860 to 15,840, the loop-reordered version got 10.0% faster, and the output stayed bit-identical.
But that version was still slower than the original token-by-token one (971 vs 871 ms). The 10% is real. It hasn't won yet.
Along the way it killed one of my own priors. I assumed the projection segments, the most expensive ones by time, were the biggest dispatchers. Counted, they issue only 10 dispatches per token. The real dispatch hogs were a pile of cheap-looking elementwise operators at 500 per token, half of all dispatches. Share of time and share of dispatches are two different denominators. Before optimizing, make sure you know which one you're pushing on.
5. How to take apart someone else's "X times faster"
Run the same math backwards and it becomes a tool for reading other people's numbers.
Question one: is this an operator number or an end-to-end number? The 67.87x is real, but it's per-operator. For any "the kernel got N times faster," first get that operator's share p of the whole chain, then plug it into 1 ÷ (1 − p + p ÷ s). If you can't get p, the multiplier tells you nothing.
Question two: what's the baseline, and what did the timer include? Lily's official performance report compares against stock MLX on an M5 Max. Averaged over ten context-length points, decode is 1.285x and prefill is 1.032x, and past 32K context its prefill already loses to MLX. The timing leaves out HTTP, tokenization, the chat template, and model loading. It also skips the full-vocabulary logprobs MLX computes by default, so part of the speedup comes from doing less work. BaseRT and oMLX aren't in the comparison at all. So even a bigger whole-model multiplier wouldn't show it beats those production engines, and it definitely wouldn't show it's a better server: it handles only one request at a time with greedy decoding, no streaming, no tool calls.
Question three: is "unsupported" a hardware wall or a whitelist? For a check like family >= 10, first find one example of the same kind of capability actually being used on your target hardware. If you find one, remove the check and run the official tests. This time I skipped that step at first and nearly wrote off an entire route.
6. Which numbers here aren't solid
- The seven segment shares come from an older 512-token phase breakdown (7250.8 ms), not a fresh measurement. And the moment the chunked rewrite lands, every one of those percentages is void and has to be remeasured.
- The 5.5% and the 1.69x are Amdahl extrapolations. I never wired that 67x kernel into the full chain and measured it.
- The 8.80 µs per dispatch comes from "unattributed time ÷ dispatch count," which assumes the whole 59% is dispatch overhead. An independent measurement earlier in the project gave 3.1 µs per dispatch. Same order of magnitude, but nearly 3x apart, so I only trust this unit cost to the order of magnitude.
- Lily's 1,000 dispatches is an estimate from "40 layers × about 25 dispatches." I didn't count it on Lily's code.
- 843.74 is the closed-source engine's number on this machine. I can't see its dispatch count, so "most of the gap is loop structure" is inferred from Lily's source structure and can't be verified against that engine itself.
- The full-graph A/B timing script got contaminated by me partway through: every round allocated a fresh 4MB and appended entries to global tables, which pushed the spread of paired ratios to 27.59x. After the fix it dropped to 1.10. The 1.03x to 1.05x above are post-fix numbers.
7. Next time you see "N times faster," do the math in this order
- Ask whether it's per-operator or end-to-end. If per-operator, get the share p.
- Plug into 1 ÷ (1 − p + p ÷ s), then compute 1 ÷ (1 − p) for s at infinity. If even the latter doesn't reach your target, drop that route.
- Add up p for every known segment and compute 1 ÷ (1 − Σp). That's the "all kernels free" ceiling.
- Subtract the known segments from the total and see how much nobody claims. If a lot is left, go count dispatches.
- Count dispatches at three or more input lengths and fit the slope. If the slope grows with length, fix the structure before swapping kernels.
- After fixing the structure, remeasure every share, then decide which kernel to swap.
- When something says "not supported on your hardware," find one example of the same capability used on the same hardware before you believe it.