Ariaki
/

Blog

21 posts
All tags方法论llmAIbenchmarkApple Siliconengineering性能Agent
2026.10.25

7 of 8 requests truncated, and I almost wrote the model off: when picking a local model, which of my verdicts were really testing me?

In one day of choosing between two local Qwen models I reached four verdicts, and every one of them needed correcting: a missing launch flag dumped chain-of-thought into the answer, I compared a fixed model against an unfixed one, 3 of my own 8 test questions were broken, and a per-request thinking switch was silently overridden by the server. Three checks you can run without reading any source: answer share, broken-question rate vs. score gap, and identical token counts across two configs.

llm · inference · vLLM13 min read
2026.10.24

The same community-fixed chat template fixed one machine and missed the other completely: can you tell which layer an error lives in without reading the source?

Two machines running the same Qwen family model both failed about 3.3% of requests on the turn after a tool call. I put a community rewrite of the chat template on the first one, and the failing argument shapes went from 7 of 19 to 0 of 19. On an isolated copy of the second one, the bad request still returned a 500, in 3 milliseconds. This post covers how to use time-to-failure to locate the layer that broke, how 341 errors collapsed into 39 poisoned sessions, and why the README's "80% faster" claim doesn't hold up.

llm · inference · llama.cpp11 min read
2026.10.23

Three GPUs, all stuck at 33% utilization: where's the bottleneck? I ran three rounds of elimination and two and a half of my conclusions fell over

Three 64GB cards running a 320B MoE at 24 tok/s, and nvidia-smi shows all three at 33%. That night I ruled out "the cards take turns" and "memory bandwidth is maxed", and blamed the inter-GPU link, based on a concurrency curve that flatlined at 49 tok/s at both 4 and 8 streams. Two months later I reread the raw data: the flatline was drawn by the server's default of 4 slots, and the numbers I used to rule out turn-taking are the most direct evidence for it. This post gives two divisions and one multiplication for reading multi-GPU utilization and concurrency curves.

llm · inference · multi-gpu11 min read
2026.10.20

Three pruned variants failed the same questions, so I blamed the base model: why did one run against the official API flip the verdict back?

Three expert-pruned builds of DeepSeek-V4-Flash all scored around 7.3 on my question set, and their wrong answers clustered on the same questions. I took that as proof the base model was weak. Then I ran the same set through the official API: 9.21, the highest score in my table. This post explains why that overlap statistic had no power to answer the question, how to tell before running it, and how to read the benchmark table that ships with a pruned checkpoint.

llm · moe · pruning13 min read
2026.10.16

The README says 1000x faster. My machine measured 0.07x to 26x. How much does swapping a tokenizer actually buy you?

A new high-speed tokenizer advertises roughly 1000x over HuggingFace. On a 6-core Ryzen I ran six workloads and got anything from 0.07x to 26.33x. This post takes apart the README's own table to show the multiplier is mostly set by the denominator, then gives one division for how much text you need before the time saved even covers a single model call.

LLM · benchmarking · 性能9 min read
2026.10.14

Swapping a made-up concurrency cap of 16 for a measured 28 cut latency 14.5%. So why did I delete the cap entirely?

I typed per_backend = 8 without thinking, then built on it for a whole day. Four optimization plans all aimed at a phase worth 13.7% of latency. Replacing the 8 with a measured value took end-to-end latency from 20.47 s to 17.50 s. Then I noticed every possible value of that cap is in one of two states: throttling, or unreachable. Three divisions, plus a way to see through 'utilization is only 41%'.

LLM · 推理 · 并发11 min read
2026.10.13

Someone said three GPUs couldn't load the model, and I proved it was a VRAM shortfall: why do two configs crashing on the same line mean VRAM has nothing to do with it?

A community repo reported that DeepSeek-V4-Flash crashes at load time on three 64GB cards and only runs on four. I did the VRAM math and concluded that raising the memory-utilization knob would fix it. Two days later I read the source and found that knob does nothing during model load. The two settings he tested differed by 3.2 GiB and crashed on the same line, which was evidence the crash was resource-independent. I read it backwards. The real culprit was an environment variable that entered the repo 22 hours after the 'three cards don't work' conclusion was written.

GPU · LLM · 推理13 min read
2026.10.12

Going from a 4GiB to a 6GiB prompt cache should have bought 8 to 10 more points of hit rate. Why did it last only 61 minutes?

On a 16GB gaming laptop running a local model, growing llama.cpp's prompt cache from 1GiB to 4GiB cut the median wait on 45K+ token requests from 47 seconds to 3.4. A reuse-distance calculation said 6GiB would add another 8 to 10 points of hit rate. In under 50 minutes the container's memory limit killed it, and at 61 minutes my own guard rolled it back. The math was fine. What I left out was one addition: the container limit has to fit the cache plus the 2.1GiB the process needs for everything else.

llm · llama.cpp · 缓存11 min read
2026.10.11

The model card says 1M context, and the math says it fits in 64GB. So why did my Mac get stopped at 256K?

A 118B MoE with its KV cache compressed to 4-bit should need only 13.3 GB for 1M tokens by the formula, 62.6 GB total with weights, which leaves 6 GB spare on my 64GB M1 Max. In practice 128K passed and 256K got killed by my own memory watchdog. Breaking it down, every extra token of context raised peak memory by at least 39.4 KB, 3.1x what the compression formula predicts. This post gives three formulas you can run yourself: KV bytes per token, a usable limit derived from two measured lengths, and generation speed at long context from three measured lengths, plus a checklist for figuring out how much context a model can really take on your machine.

llm · 长上下文 · 内存13 min read
2026.10.03

Same config, run twice, 60 games, 0.26 rounds apart: after you tweak a prompt, how many games until you know it worked?

I had a 35B local model play Balatro and judged every prompt change by the average round it reached. First I ran the unchanged config twice to measure the noise, then judged changes against it. One line, "empty joker slots are waste, buy jokers first," moved the average from 6.58 and 6.32 to 9.12, paired 36:7. But if I had looked only at the first 20 of those same 60 games, I would have called it useless. Two calculations you can run yourself: how big an improvement 60 games can see and how many games +1 round needs, and why a run-it-twice A/A test only gives you a lower bound on noise.

LLM · 评测 · 游戏12 min read
2026.10.02

Pointing directly got 186 right, parse-then-pick got 188, the hybrid got 190: which way should a local model click the screen?

I had a 35B local model click buttons in a window with no accessibility tree. Each route has a trade-off: asking for coordinates directly is over 6x faster but slightly less accurate, while parsing first is a bit more accurate but costs about 3 seconds per step. The hybrid router got 190 of 200 real clicks right, but those 2 extra wins prove nothing statistically. The more useful finding was that "both routes agree" is far less reassuring than it looks, because the two routes fail together. Three calculations you can run yourself: whether two routes fail independently, the average latency of a hybrid router, and how much a zero-error run actually tells you.

LLM · Agent · 评测11 min read
2026.10.01

One renamed stat took the same model from 59% to 72%: what were my three AI game runs actually comparing?

I had a 35B local model play Reigns: Three Kingdoms and ran three live sessions comparing thinking, no thinking, and random choice. They came out indistinguishable. Only after turning the logs into 360 offline questions did I find that I had misnamed one of the four stats, so the model had been reasoning about the wrong quantity the whole time. Fixing that one name moved the same question set from 59.2% to 72.1%. Three calculations you can run yourself: how to split an accuracy gain by stat, where the question set tops out, and how many games a live comparison really needs.

LLM · 评测 · 游戏12 min read
2026.09.26

One LLM call costs as much as 3.4 million Python loop iterations. Which line are you still optimizing?

I re-measured Jeff Dean's 2007 latency numbers on my own machine, then built a second table for the millisecond layer I actually work in: a warm local model round trip is 79 ms, one Python loop iteration is 23 ns. With that table, cutting 5,000 tokens from a prompt saves less than a single call. Two divisions you can run yourself, plus a way to see through 'microseconds per prefill token' numbers.

性能 · 方法论 · LLM10 min read
2026.09.21

The model gave the same answer on all 20 frames. Accuracy: exactly 50%. Do you throw out the model, or throw out argmax?

A decision model that does one forward pass and reads the probability of two words scored like a coin flip on a binary task. Its AUC was 0.949 — the signal was there the whole time, drowned by a prior. Here are two pieces of arithmetic that measure the bias, one test for debunking somebody else's benchmark, and a checklist.

LLM · evaluation · local-inference8 min read
2026.09.18

5241 tokens of full history vs 2270 compressed — why did the compressed one remember better?

Same characters, same decision point. Feed in an entire lifetime of memory and the planted events move behaviour 38 times. Compress to 43% of the size and it's 49. When I first wrote that down I was missing one control arm — and adding it showed two thirds of the credit had gone to the wrong place. Here's the division you can run yourself, and how the first three versions of my judge lied to me.

llm · evaluation · memory10 min read
2026.09.17

Same GPU: 8-bit beats 4-bit by 10% on single-stream. So why did I ship the slower one?

Four days earlier I benchmarked a 27B dense model on this card and concluded official 8-bit won outright. I reran the same script on a 35B MoE and the result flipped: 8-bit still won single-stream, but lost across the board under concurrency, because its weights cost 11 GiB more and bought away 600,000 tokens of context. Here's the division that converts one into the other, and a test for whether a quantization result transfers at all.

llm · quantization · moe9 min read
2026.09.09

Four Concurrent Requests, 350 tok/s Aggregate. Why Did the Isolated Re-test Show 91?

I wrote in my notes: "4 concurrent, ~350 tok/s aggregate, continuous batching works." The next day I looked at completion times and saw four requests finishing one after another, 91.2 tok/s aggregate. This post shows how one division and a table of completion timestamps tells real parallelism apart from a tidy queue, and then works out how far a 98 tok/s single stream really is from the physical ceiling.

llm · benchmark · apple-silicon10 min read