Blog

9 posts

All tags 方法论 llm AI benchmark Apple Silicon engineering 性能 Agent

2026.10.17

Three samples, memory didn't budge, so I ruled out a leak. Five weeks later it went from 1.6GB to 22GB. What did those five minutes miss?

A 64GB M1 Max ran out of memory and rebooted. I sampled three times, saw a flat slope, and blamed too many resident processes instead of a leak. Five weeks later an embedding service had climbed almost linearly by 20GB in three and a half days, while a speech-to-text service sat parked at 58GB. This post works out one division: reading resolution over slope tells you the minimum time you have to watch. It also shows that MLX sets its default cache limit to nearly the whole machine, so two processes can put memory on the books at 1.9x what you actually have.

macOS · Apple Silicon · MLX10 min read

2026.10.11

The model card says 1M context, and the math says it fits in 64GB. So why did my Mac get stopped at 256K?

A 118B MoE with its KV cache compressed to 4-bit should need only 13.3 GB for 1M tokens by the formula, 62.6 GB total with weights, which leaves 6 GB spare on my 64GB M1 Max. In practice 128K passed and 256K got killed by my own memory watchdog. Breaking it down, every extra token of context raised peak memory by at least 39.4 KB, 3.1x what the compression formula predicts. This post gives three formulas you can run yourself: KV bytes per token, a usable limit derived from two measured lengths, and generation speed at long context from three measured lengths, plus a checklist for figuring out how much context a model can really take on your machine.

llm · 长上下文 · 内存13 min read

2026.10.09

Even at 100% acceptance it only gets 0.97x: why did speculative decoding make my own inference engine slower?

I added speculative decoding to the inference engine I wrote for an M1 Max. Output stayed bit-identical, and throughput dropped from 92.0 to 65.26 tok/s. On the same machine, another engine with an acceptance rate of just 0.38 runs at 133.4. Here is one division that tells you, before you write any code, whether speculative decoding will pay off on your engine, plus three checks for picking apart benchmarks that only quote acceptance rate.

推理引擎 · 投机解码 · 性能9 min read

2026.10.07

979 tok/s from a cold process, 850 when it keeps running: I called it a tie with the competition. Why was I actually 15% behind?

My own inference engine on an M1 Max did 979 tok/s on a 4096-token prefill from a fresh process. The competing engine scored 984, and that night I said it was "basically a tie." An hour later I found out the competitor's 984 came from back-to-back runs; given a short rest and a single shot it does 1017–1053. Mine goes the other way: back-to-back, it drops to around 850. The cause is a whole-machine power cap of roughly 60 W. My engine burns an extra 5.6 W on the memory side, so the GPU gets 5.5 W less and runs 9% slower. This post covers one conversion you can compute yourself (sustained ≈ cold × clock ratio), how to tell whether a benchmark number is cold or hot, and why splitting the work to save bytes did nothing while removing duplicate reads did.

推理引擎 · 性能 · 方法论12 min read

2026.10.05

12% behind llama.cpp, and four agents spent 8 hours to gain 0.66%: which operator does the gap between two engines actually live in?

My own inference engine does 830.6 tok/s prefill on an M1 Max; llama.cpp does 927.9. Four AI agents spent 8 hours tuning the MoE path and gained 0.66%. Then I stopped and spent one evening reconciling the two engines operator by operator. MoE turned out to be the one category where I was already faster. The whole gap was in dense projections and transposes. Eight more hours aimed at the right target gained 9.9%. This post covers how to reconcile two engines per operator: a replay-and-diff measurement, a coverage ratio, a bytes-read check for bandwidth-bound kernels, and why a standalone microbenchmark undercounts MoE by 60%.

推理引擎 · 性能 · 方法论12 min read

2026.10.04

A point release took prefill from 804 to 279 tok/s: how do you tell where it got slower without reading the source?

I re-ran a benchmark I had been treating as a yardstick for two months, and the leaderboard flipped: the fastest prefill on my M1 Max is now llama.cpp at 928, and the engine I had been measuring against drops to 281 in its latest release. Bisecting lands on 0.2.3 → 0.2.4, where decode barely moved but prefill fell to a third. This post gives a two-point fit that uses prompt length to separate 'fixed overhead grew' from 'every token got more expensive', plus a few benchmark traps: TTFT-derived throughput is only a lower bound, speculative-decoding speed depends on what text you feed it, and ablation switches that skip a component without changing the runtime.

推理引擎 · 性能 · 方法论11 min read

2026.09.28

It Named Hydrogen and Helium and Scored Zero: How Much Does Web Search Actually Add to a Local Model?

I forced a local LLM to search the web before answering 70 factual questions. The report said the score went from 5.93 to 7.67. Reading the answers one by one two months later, I found 7 correct answers that my grader had docked 47 points because the model switched to English. The real gain after regrading is 2.41, and my own "3x latency" claim came from dividing two different kinds of numbers. Three formulas you can run yourself, and a checklist for measuring what retrieval really buys you.

模型评测 · 检索增强 · 方法论12 min read

2026.09.27

Batch size went from 2 to 16 and throughput went from 65.7 to 67.4. What is batching actually saving?

I made my own inference engine decode 16 sequences at once. All 16 outputs matched their single-sequence runs bit for bit, and aggregate throughput was a flat line. On the same machine, MLX doubled. Two months earlier I had used a tidy byte-accounting argument to declare this path dead; later measurements knocked down all three of its pillars. Most of what batching buys comes from feeding bandwidth that sits idle at B=1, and the bytes it saves are secondary. Here is a two-factor formula for estimating what batching can buy before you write any code, plus three questions for taking apart other people's batching numbers.

推理引擎 · 性能 · 方法论11 min read

2026.09.25

A kernel got 67x faster. How much faster did the whole inference path get?

I got an open-source attention kernel running on my M1 Max, and it beat the old implementation in my own engine by 67.87x. One division later, the whole prefill could only get 5.5% faster. Even if all seven known kernels took zero time, the ceiling was 1.69x. The remaining 59% of the time belonged to no kernel at all. It belonged to dispatch count. Here are three divisions for deciding whether someone's 'X times faster' will ever show up end to end.

推理引擎 · 性能 · 方法论11 min read