Ariaki
/

Blog

62 posts
All tags方法论llmAIbenchmarkApple Siliconengineering性能Agent
2026.10.26

Same config, six runs, throughput bouncing between 366 and 561: to see a 5% improvement, do you run more trials or change the ruler?

Three GPUs were already running a large model, and I wanted to know how much a fourth would help. I compared against someone else's four-card number, got 0.90x, and immediately invented a mechanism for it: a longer pipeline cancelling out the extra compute. Then I found the machine was pinned against its power cap, and that the same config run six times in a row swung between 366 and 561 tok/s, a coefficient of variation of 15%. This post gives one division that tells you, before you run anything, how large a difference your measurement can resolve. Then it covers how I switched to a reading that ignores clock speed, and why the highest number that new ruler produced belonged to the worst config.

推理 · benchmark · 投机解码13 min read
2026.10.25

7 of 8 requests truncated, and I almost wrote the model off: when picking a local model, which of my verdicts were really testing me?

In one day of choosing between two local Qwen models I reached four verdicts, and every one of them needed correcting: a missing launch flag dumped chain-of-thought into the answer, I compared a fixed model against an unfixed one, 3 of my own 8 test questions were broken, and a per-request thinking switch was silently overridden by the server. Three checks you can run without reading any source: answer share, broken-question rate vs. score gap, and identical token counts across two configs.

llm · inference · vLLM13 min read
2026.10.24

The same community-fixed chat template fixed one machine and missed the other completely: can you tell which layer an error lives in without reading the source?

Two machines running the same Qwen family model both failed about 3.3% of requests on the turn after a tool call. I put a community rewrite of the chat template on the first one, and the failing argument shapes went from 7 of 19 to 0 of 19. On an isolated copy of the second one, the bad request still returned a 500, in 3 milliseconds. This post covers how to use time-to-failure to locate the layer that broke, how 341 errors collapsed into 39 poisoned sessions, and why the README's "80% faster" claim doesn't hold up.

llm · inference · llama.cpp11 min read
2026.10.23

Three GPUs, all stuck at 33% utilization: where's the bottleneck? I ran three rounds of elimination and two and a half of my conclusions fell over

Three 64GB cards running a 320B MoE at 24 tok/s, and nvidia-smi shows all three at 33%. That night I ruled out "the cards take turns" and "memory bandwidth is maxed", and blamed the inter-GPU link, based on a concurrency curve that flatlined at 49 tok/s at both 4 and 8 streams. Two months later I reread the raw data: the flatline was drawn by the server's default of 4 slots, and the numbers I used to rule out turn-taking are the most direct evidence for it. This post gives two divisions and one multiplication for reading multi-GPU utilization and concurrency curves.

llm · inference · multi-gpu11 min read
2026.10.22

Two checkpoint files, identical down to the byte, and I ran a whole batch on the wrong one. Why did fixing it make 3 more shots drift?

A local video model ships two checkpoints. Both files are exactly 20,970,379,616 bytes, with zero tensor-shape differences; only the md5 differs. My script loaded the wrong one for an entire batch of experiments. The first retest after the fix couldn't detect anything; the second showed the 'wrong' checkpoint was actually better. This post covers two things: a multiplication that tells you before you start that an A/B is doomed to show nothing, and three questions to ask whenever someone posts an A/B result.

视频生成 · 实验设计 · 对照实验13 min read
2026.10.21

One table said 2.58x, then my own chat client felt no different: which cell of a speculative decoding benchmark should you read?

The same concurrency sweep said "don't switch" when read at 64 concurrent requests and "2.58x faster" when read at 8. After switching, my own chat client showed 38 tok/s on prose, while hitting the same server directly gave 74.1. None of the numbers were wrong. What was wrong was which cell I read and where I stopped measuring. This post gives a formula for how much faster a whole turn actually gets, a division for taking apart a model card's speedup claim, and a checklist to run before you trust any speedup number.

推理 · 投机解码 · benchmark13 min read
2026.10.20

Three pruned variants failed the same questions, so I blamed the base model: why did one run against the official API flip the verdict back?

Three expert-pruned builds of DeepSeek-V4-Flash all scored around 7.3 on my question set, and their wrong answers clustered on the same questions. I took that as proof the base model was weak. Then I ran the same set through the official API: 9.21, the highest score in my table. This post explains why that overlap statistic had no power to answer the question, how to tell before running it, and how to read the benchmark table that ships with a pruned checkpoint.

llm · moe · pruning13 min read
2026.10.19

A 15% error rate, but absent from 58% of my searches: how many engines is my self-hosted search actually using?

My self-hosted SearXNG stats page said the Google engine had a 15% error rate, and my agent got a full 10 results on 1171 searches out of 1173 over three days. The raw result counts told a different story: Google wasn't in the room for 58% of those calls. Here's how a single addition reveals which engine went missing, and how back in August I fixed a server that wasn't even on the call path.

自建服务 · 搜索 · 方法论10 min read
2026.10.18

My eval said the LLM routed messages with 90.9% accuracy. On real traffic it lost to doing nothing. What was I actually measuring for two days?

I wanted my AI assistant to decide which ongoing task each of my messages belonged to. I built 30,000+ eval points and the LLM scored 90.9%. One remark later, I re-cut the data: 83.7% of those points involved no choice at all in real use. Re-tested at real proportions, the ungated LLM scored 82.3% and 'always continue the current task' scored 83.8%. This post gives three formulas you can run yourself: how much of a score is free, how many correct changes it takes to pay for one wrong one, and how many times a day a rare-class detector will bother you.

AI Agent · 评测 · 方法论15 min read
2026.10.17

Three samples, memory didn't budge, so I ruled out a leak. Five weeks later it went from 1.6GB to 22GB. What did those five minutes miss?

A 64GB M1 Max ran out of memory and rebooted. I sampled three times, saw a flat slope, and blamed too many resident processes instead of a leak. Five weeks later an embedding service had climbed almost linearly by 20GB in three and a half days, while a speech-to-text service sat parked at 58GB. This post works out one division: reading resolution over slope tells you the minimum time you have to watch. It also shows that MLX sets its default cache limit to nearly the whole machine, so two processes can put memory on the books at 1.9x what you actually have.

macOS · Apple Silicon · MLX10 min read
2026.10.16

The README says 1000x faster. My machine measured 0.07x to 26x. How much does swapping a tokenizer actually buy you?

A new high-speed tokenizer advertises roughly 1000x over HuggingFace. On a 6-core Ryzen I ran six workloads and got anything from 0.07x to 26.33x. This post takes apart the README's own table to show the multiplier is mostly set by the denominator, then gives one division for how much text you need before the time saved even covers a single model call.

LLM · benchmarking · 性能9 min read
2026.10.15

du says 681G, df says 795G, Docker says 74.6GB is reclaimable: when a disk is nearly full, which number do you trust?

One 1TB disk, four tools, four numbers that disagree. I re-measured all of them today. A non-root du misses 113G. Only 18G of a 61.8G uv cache actually turns into free space when you delete it. A 17.3G SQLite file held 26MB of data. Docker reports more reclaimable space than its own upper bound allows. Four divisions, plus a 34G gap from August that I explained wrong.

Linux · 磁盘 · 方法论10 min read
2026.10.14

Swapping a made-up concurrency cap of 16 for a measured 28 cut latency 14.5%. So why did I delete the cap entirely?

I typed per_backend = 8 without thinking, then built on it for a whole day. Four optimization plans all aimed at a phase worth 13.7% of latency. Replacing the 8 with a measured value took end-to-end latency from 20.47 s to 17.50 s. Then I noticed every possible value of that cap is in one of two states: throttling, or unreachable. Three divisions, plus a way to see through 'utilization is only 41%'.

LLM · 推理 · 并发11 min read
2026.10.13

Someone said three GPUs couldn't load the model, and I proved it was a VRAM shortfall: why do two configs crashing on the same line mean VRAM has nothing to do with it?

A community repo reported that DeepSeek-V4-Flash crashes at load time on three 64GB cards and only runs on four. I did the VRAM math and concluded that raising the memory-utilization knob would fix it. Two days later I read the source and found that knob does nothing during model load. The two settings he tested differed by 3.2 GiB and crashed on the same line, which was evidence the crash was resource-independent. I read it backwards. The real culprit was an environment variable that entered the repo 22 hours after the 'three cards don't work' conclusion was written.

GPU · LLM · 推理13 min read
2026.10.12

Going from a 4GiB to a 6GiB prompt cache should have bought 8 to 10 more points of hit rate. Why did it last only 61 minutes?

On a 16GB gaming laptop running a local model, growing llama.cpp's prompt cache from 1GiB to 4GiB cut the median wait on 45K+ token requests from 47 seconds to 3.4. A reuse-distance calculation said 6GiB would add another 8 to 10 points of hit rate. In under 50 minutes the container's memory limit killed it, and at 61 minutes my own guard rolled it back. The math was fine. What I left out was one addition: the container limit has to fit the cache plus the 2.1GiB the process needs for everything else.

llm · llama.cpp · 缓存11 min read
2026.10.11

The model card says 1M context, and the math says it fits in 64GB. So why did my Mac get stopped at 256K?

A 118B MoE with its KV cache compressed to 4-bit should need only 13.3 GB for 1M tokens by the formula, 62.6 GB total with weights, which leaves 6 GB spare on my 64GB M1 Max. In practice 128K passed and 256K got killed by my own memory watchdog. Breaking it down, every extra token of context raised peak memory by at least 39.4 KB, 3.1x what the compression formula predicts. This post gives three formulas you can run yourself: KV bytes per token, a usable limit derived from two measured lengths, and generation speed at long context from three measured lengths, plus a checklist for figuring out how much context a model can really take on your machine.

llm · 长上下文 · 内存13 min read
2026.10.10

Same machine, same 8 concurrent requests, and I've measured 72, 171, 359 and 606 tok/s. Which one is its real speed?

Before benchmarking a large model split across three ex-mining GPUs, I set myself a rule: if 8-way concurrency doesn't deliver at least 3x the single-stream throughput, rethink the whole setup. The first run came back at 2.84x and nearly killed the plan. It turned out that number mostly measured when the requests decided to stop, not how fast the hardware computes. On the same machine at the same 8-way concurrency, changing prompt length, stopping rule and content moves aggregate throughput anywhere from 72 to 606 tok/s. This post gives a wall-clock decomposition you can compute yourself, reconciles all four numbers against it, and lists the questions I now ask of anyone's concurrency benchmark.

推理 · benchmark · 方法论11 min read
2026.10.09

Even at 100% acceptance it only gets 0.97x: why did speculative decoding make my own inference engine slower?

I added speculative decoding to the inference engine I wrote for an M1 Max. Output stayed bit-identical, and throughput dropped from 92.0 to 65.26 tok/s. On the same machine, another engine with an acceptance rate of just 0.38 runs at 133.4. Here is one division that tells you, before you write any code, whether speculative decoding will pay off on your engine, plus three checks for picking apart benchmarks that only quote acceptance rate.

推理引擎 · 投机解码 · 性能9 min read
2026.10.08

6,950 Saved Lessons, 93.5% Never Read Once: Should My AI Assistant's Memory Get Better Retrieval, or Fewer Writes?

I built a lesson bank for my AI assistant. Every night a model distills 'what to do next time' from the day's conversations, and in four months it piled up 6,950 entries. I went through 962 retrieval calls: only 451 entries ever came back. The top 1% of entries took 47% of all hits. The confidence score the model gave each entry correlates with whether it ever gets used at r = -0.002. I was about to add the usage-feedback loop from the paper I'd copied. One division made me drop the plan. This post gives three formulas you can run on your own system: the read/write ratio, the number of searches you'd actually need, and an upper bound on hit rate when part of your logs are gone.

AI Agent · 记忆系统 · 方法论11 min read
2026.10.07

979 tok/s from a cold process, 850 when it keeps running: I called it a tie with the competition. Why was I actually 15% behind?

My own inference engine on an M1 Max did 979 tok/s on a 4096-token prefill from a fresh process. The competing engine scored 984, and that night I said it was "basically a tie." An hour later I found out the competitor's 984 came from back-to-back runs; given a short rest and a single shot it does 1017–1053. Mine goes the other way: back-to-back, it drops to around 850. The cause is a whole-machine power cap of roughly 60 W. My engine burns an extra 5.6 W on the memory side, so the GPU gets 5.5 W less and runs 9% slower. This post covers one conversion you can compute yourself (sustained ≈ cold × clock ratio), how to tell whether a benchmark number is cold or hot, and why splitting the work to save bytes did nothing while removing duplicate reads did.

推理引擎 · 性能 · 方法论12 min read
2026.10.06

The Circuit Breaker Tripped 80 Times and the Board Still Said "Running": Who Was My Self-Healing Script Actually Saving?

Four AI workers started 1,652 times in one night, each living about 60 seconds, while the task board showed "running" the whole time. My diagnosis that morning: the failure counter was stuck between 10 and 12, never reaching the limit of 20, so the breaker never fired. Two days later I read the raw event table. The breaker had tripped 80 times, and every time my own revive script closed it again within about a minute and a half. This post covers how to work out from one counter snapshot how many times it was reset, why this kind of loop phase-locks to the cron interval, and why a revive script that silently exits looks exactly like a healthy one.

AI Agent · 可靠性 · 方法论10 min read
2026.10.05

12% behind llama.cpp, and four agents spent 8 hours to gain 0.66%: which operator does the gap between two engines actually live in?

My own inference engine does 830.6 tok/s prefill on an M1 Max; llama.cpp does 927.9. Four AI agents spent 8 hours tuning the MoE path and gained 0.66%. Then I stopped and spent one evening reconciling the two engines operator by operator. MoE turned out to be the one category where I was already faster. The whole gap was in dense projections and transposes. Eight more hours aimed at the right target gained 9.9%. This post covers how to reconcile two engines per operator: a replay-and-diff measurement, a coverage ratio, a bytes-read check for bandwidth-bound kernels, and why a standalone microbenchmark undercounts MoE by 60%.

推理引擎 · 性能 · 方法论12 min read
2026.10.04

A point release took prefill from 804 to 279 tok/s: how do you tell where it got slower without reading the source?

I re-ran a benchmark I had been treating as a yardstick for two months, and the leaderboard flipped: the fastest prefill on my M1 Max is now llama.cpp at 928, and the engine I had been measuring against drops to 281 in its latest release. Bisecting lands on 0.2.3 → 0.2.4, where decode barely moved but prefill fell to a third. This post gives a two-point fit that uses prompt length to separate 'fixed overhead grew' from 'every token got more expensive', plus a few benchmark traps: TTFT-derived throughput is only a lower bound, speculative-decoding speed depends on what text you feed it, and ablation switches that skip a component without changing the runtime.

推理引擎 · 性能 · 方法论11 min read
2026.10.03

Same config, run twice, 60 games, 0.26 rounds apart: after you tweak a prompt, how many games until you know it worked?

I had a 35B local model play Balatro and judged every prompt change by the average round it reached. First I ran the unchanged config twice to measure the noise, then judged changes against it. One line, "empty joker slots are waste, buy jokers first," moved the average from 6.58 and 6.32 to 9.12, paired 36:7. But if I had looked only at the first 20 of those same 60 games, I would have called it useless. Two calculations you can run yourself: how big an improvement 60 games can see and how many games +1 round needs, and why a run-it-twice A/A test only gives you a lower bound on noise.

LLM · 评测 · 游戏12 min read
2026.10.02

Pointing directly got 186 right, parse-then-pick got 188, the hybrid got 190: which way should a local model click the screen?

I had a 35B local model click buttons in a window with no accessibility tree. Each route has a trade-off: asking for coordinates directly is over 6x faster but slightly less accurate, while parsing first is a bit more accurate but costs about 3 seconds per step. The hybrid router got 190 of 200 real clicks right, but those 2 extra wins prove nothing statistically. The more useful finding was that "both routes agree" is far less reassuring than it looks, because the two routes fail together. Three calculations you can run yourself: whether two routes fail independently, the average latency of a hybrid router, and how much a zero-error run actually tells you.

LLM · Agent · 评测11 min read
2026.10.01

One renamed stat took the same model from 59% to 72%: what were my three AI game runs actually comparing?

I had a 35B local model play Reigns: Three Kingdoms and ran three live sessions comparing thinking, no thinking, and random choice. They came out indistinguishable. Only after turning the logs into 360 offline questions did I find that I had misnamed one of the four stats, so the model had been reasoning about the wrong quantity the whole time. Fixing that one name moved the same question set from 59.2% to 72.1%. Three calculations you can run yourself: how to split an accuracy gain by stat, where the question set tops out, and how many games a live comparison really needs.

LLM · 评测 · 游戏12 min read
2026.09.30

Five agents vs. one, eight hours each, 114 vs. 104: what did the extra four actually buy?

Same inference-engine optimization problem, same external judge, same model. One group ran five agents coordinating on a shared message board; the other ran a single agent. The multi-agent group won by 10%, but both groups independently found the same first two moves, and the whole gap came from one direction the solo agent never reached. The real ceiling was the judge, which could only evaluate one submission at a time. Two formulas you can run yourself (evaluator capacity, token cost per point gained), plus a teardown of a '10 agents discovered a new algorithm' headline.

多 agent · 推理引擎 · 评测15 min read
2026.09.29

0.08 W more, 34 fps to 61: which knob actually saves battery on a handheld?

Same Snapdragon 8 Gen 2 handheld, same game, same scene, ten configurations measured on battery. The Performance power profile was the worst of the lot: highest power draw, lower frame rate. What worked was finding the bottleneck first and then comparing energy per frame. Along the way two sensors lied to me, and one of them almost had me report a 101 fps that never existed.

硬件实测 · 掌机 · 功耗10 min read
2026.09.28

It Named Hydrogen and Helium and Scored Zero: How Much Does Web Search Actually Add to a Local Model?

I forced a local LLM to search the web before answering 70 factual questions. The report said the score went from 5.93 to 7.67. Reading the answers one by one two months later, I found 7 correct answers that my grader had docked 47 points because the model switched to English. The real gain after regrading is 2.41, and my own "3x latency" claim came from dividing two different kinds of numbers. Three formulas you can run yourself, and a checklist for measuring what retrieval really buys you.

模型评测 · 检索增强 · 方法论12 min read
2026.09.27

Batch size went from 2 to 16 and throughput went from 65.7 to 67.4. What is batching actually saving?

I made my own inference engine decode 16 sequences at once. All 16 outputs matched their single-sequence runs bit for bit, and aggregate throughput was a flat line. On the same machine, MLX doubled. Two months earlier I had used a tidy byte-accounting argument to declare this path dead; later measurements knocked down all three of its pillars. Most of what batching buys comes from feeding bandwidth that sits idle at B=1, and the bytes it saves are secondary. Here is a two-factor formula for estimating what batching can buy before you write any code, plus three questions for taking apart other people's batching numbers.

推理引擎 · 性能 · 方法论11 min read
2026.09.26

One LLM call costs as much as 3.4 million Python loop iterations. Which line are you still optimizing?

I re-measured Jeff Dean's 2007 latency numbers on my own machine, then built a second table for the millisecond layer I actually work in: a warm local model round trip is 79 ms, one Python loop iteration is 23 ns. With that table, cutting 5,000 tokens from a prompt saves less than a single call. Two divisions you can run yourself, plus a way to see through 'microseconds per prefill token' numbers.

性能 · 方法论 · LLM10 min read
2026.09.25

A kernel got 67x faster. How much faster did the whole inference path get?

I got an open-source attention kernel running on my M1 Max, and it beat the old implementation in my own engine by 67.87x. One division later, the whole prefill could only get 5.5% faster. Even if all seven known kernels took zero time, the ceiling was 1.69x. The remaining 59% of the time belonged to no kernel at all. It belonged to dispatch count. Here are three divisions for deciding whether someone's 'X times faster' will ever show up end to end.

推理引擎 · 性能 · 方法论11 min read
2026.09.24

I blamed the gateway for a 300 ms API call. Then I went direct and got 481 — why?

I pinned the latency on a hosted AI gateway without ever measuring the direct route. Once I did, the gateway turned out to be a third of it — and wiring the direct route into a game made things worse, 243 to 481 ms. The culprit was a fresh TLS handshake on every request. Two divisions for figuring out where your latency actually goes and whether a gateway's rate limit will choke you.

延迟 · 方法论 · 网络9 min read
2026.09.21

The model gave the same answer on all 20 frames. Accuracy: exactly 50%. Do you throw out the model, or throw out argmax?

A decision model that does one forward pass and reads the probability of two words scored like a coin flip on a binary task. Its AUC was 0.949 — the signal was there the whole time, drowned by a prior. Here are two pieces of arithmetic that measure the bias, one test for debunking somebody else's benchmark, and a checklist.

LLM · evaluation · local-inference8 min read
2026.09.18

5241 tokens of full history vs 2270 compressed — why did the compressed one remember better?

Same characters, same decision point. Feed in an entire lifetime of memory and the planted events move behaviour 38 times. Compress to 43% of the size and it's 49. When I first wrote that down I was missing one control arm — and adding it showed two thirds of the credit had gone to the wrong place. Here's the division you can run yourself, and how the first three versions of my judge lied to me.

llm · evaluation · memory10 min read
2026.09.17

Same GPU: 8-bit beats 4-bit by 10% on single-stream. So why did I ship the slower one?

Four days earlier I benchmarked a 27B dense model on this card and concluded official 8-bit won outright. I reran the same script on a 35B MoE and the result flipped: 8-bit still won single-stream, but lost across the board under concurrency, because its weights cost 11 GiB more and bought away 600,000 tokens of context. Here's the division that converts one into the other, and a test for whether a quantization result transfers at all.

llm · quantization · moe9 min read
2026.09.14

Three case fans at full blast, still 92°C. A zip-tie and a scrap of paper got it to 64°C. What's the number in between?

A passively-cooled datacenter GPU in a desktop case. Piling on more airflow does nothing. Here's how one subtraction and one division give you thermal resistance, why adding a third fan bought zero performance and only quieter operation, and the six times I fooled myself — including the ambient temperature I used across four chapters that turned out to be a hardcoded constant.

hardware · cooling · GPU17 min read
2026.09.09

Four Concurrent Requests, 350 tok/s Aggregate. Why Did the Isolated Re-test Show 91?

I wrote in my notes: "4 concurrent, ~350 tok/s aggregate, continuous batching works." The next day I looked at completion times and saw four requests finishing one after another, 91.2 tok/s aggregate. This post shows how one division and a table of completion timestamps tells real parallelism apart from a tidy queue, and then works out how far a 98 tok/s single stream really is from the physical ceiling.

llm · benchmark · apple-silicon10 min read