Blog

8 posts

All tags 方法论 llm AI benchmark Apple Silicon engineering 性能 Agent

2026.10.16

The README says 1000x faster. My machine measured 0.07x to 26x. How much does swapping a tokenizer actually buy you?

A new high-speed tokenizer advertises roughly 1000x over HuggingFace. On a 6-core Ryzen I ran six workloads and got anything from 0.07x to 26.33x. This post takes apart the README's own table to show the multiplier is mostly set by the denominator, then gives one division for how much text you need before the time saved even covers a single model call.

LLM · benchmarking · 性能9 min read

2026.10.09

Even at 100% acceptance it only gets 0.97x: why did speculative decoding make my own inference engine slower?

I added speculative decoding to the inference engine I wrote for an M1 Max. Output stayed bit-identical, and throughput dropped from 92.0 to 65.26 tok/s. On the same machine, another engine with an acceptance rate of just 0.38 runs at 133.4. Here is one division that tells you, before you write any code, whether speculative decoding will pay off on your engine, plus three checks for picking apart benchmarks that only quote acceptance rate.

推理引擎 · 投机解码 · 性能9 min read

2026.10.07

979 tok/s from a cold process, 850 when it keeps running: I called it a tie with the competition. Why was I actually 15% behind?

My own inference engine on an M1 Max did 979 tok/s on a 4096-token prefill from a fresh process. The competing engine scored 984, and that night I said it was "basically a tie." An hour later I found out the competitor's 984 came from back-to-back runs; given a short rest and a single shot it does 1017–1053. Mine goes the other way: back-to-back, it drops to around 850. The cause is a whole-machine power cap of roughly 60 W. My engine burns an extra 5.6 W on the memory side, so the GPU gets 5.5 W less and runs 9% slower. This post covers one conversion you can compute yourself (sustained ≈ cold × clock ratio), how to tell whether a benchmark number is cold or hot, and why splitting the work to save bytes did nothing while removing duplicate reads did.

推理引擎 · 性能 · 方法论12 min read

2026.10.05

12% behind llama.cpp, and four agents spent 8 hours to gain 0.66%: which operator does the gap between two engines actually live in?

My own inference engine does 830.6 tok/s prefill on an M1 Max; llama.cpp does 927.9. Four AI agents spent 8 hours tuning the MoE path and gained 0.66%. Then I stopped and spent one evening reconciling the two engines operator by operator. MoE turned out to be the one category where I was already faster. The whole gap was in dense projections and transposes. Eight more hours aimed at the right target gained 9.9%. This post covers how to reconcile two engines per operator: a replay-and-diff measurement, a coverage ratio, a bytes-read check for bandwidth-bound kernels, and why a standalone microbenchmark undercounts MoE by 60%.

推理引擎 · 性能 · 方法论12 min read

2026.10.04

A point release took prefill from 804 to 279 tok/s: how do you tell where it got slower without reading the source?

I re-ran a benchmark I had been treating as a yardstick for two months, and the leaderboard flipped: the fastest prefill on my M1 Max is now llama.cpp at 928, and the engine I had been measuring against drops to 281 in its latest release. Bisecting lands on 0.2.3 → 0.2.4, where decode barely moved but prefill fell to a third. This post gives a two-point fit that uses prompt length to separate 'fixed overhead grew' from 'every token got more expensive', plus a few benchmark traps: TTFT-derived throughput is only a lower bound, speculative-decoding speed depends on what text you feed it, and ablation switches that skip a component without changing the runtime.

推理引擎 · 性能 · 方法论11 min read

2026.09.27

Batch size went from 2 to 16 and throughput went from 65.7 to 67.4. What is batching actually saving?

I made my own inference engine decode 16 sequences at once. All 16 outputs matched their single-sequence runs bit for bit, and aggregate throughput was a flat line. On the same machine, MLX doubled. Two months earlier I had used a tidy byte-accounting argument to declare this path dead; later measurements knocked down all three of its pillars. Most of what batching buys comes from feeding bandwidth that sits idle at B=1, and the bytes it saves are secondary. Here is a two-factor formula for estimating what batching can buy before you write any code, plus three questions for taking apart other people's batching numbers.

推理引擎 · 性能 · 方法论11 min read

2026.09.26

One LLM call costs as much as 3.4 million Python loop iterations. Which line are you still optimizing?

I re-measured Jeff Dean's 2007 latency numbers on my own machine, then built a second table for the millisecond layer I actually work in: a warm local model round trip is 79 ms, one Python loop iteration is 23 ns. With that table, cutting 5,000 tokens from a prompt saves less than a single call. Two divisions you can run yourself, plus a way to see through 'microseconds per prefill token' numbers.

性能 · 方法论 · LLM10 min read

2026.09.25

A kernel got 67x faster. How much faster did the whole inference path get?

I got an open-source attention kernel running on my M1 Max, and it beat the old implementation in my own engine by 67.87x. One division later, the whole prefill could only get 5.5% faster. Even if all seven known kernels took zero time, the ceiling was 1.69x. The remaining 59% of the time belonged to no kernel at all. It belonged to dispatch count. Here are three divisions for deciding whether someone's 'X times faster' will ever show up end to end.

推理引擎 · 性能 · 方法论11 min read