Ariaki
/

Blog

9 posts
All tags方法论llmAIbenchmarkApple Siliconengineering性能Agent
2026.10.26

Same config, six runs, throughput bouncing between 366 and 561: to see a 5% improvement, do you run more trials or change the ruler?

Three GPUs were already running a large model, and I wanted to know how much a fourth would help. I compared against someone else's four-card number, got 0.90x, and immediately invented a mechanism for it: a longer pipeline cancelling out the extra compute. Then I found the machine was pinned against its power cap, and that the same config run six times in a row swung between 366 and 561 tok/s, a coefficient of variation of 15%. This post gives one division that tells you, before you run anything, how large a difference your measurement can resolve. Then it covers how I switched to a reading that ignores clock speed, and why the highest number that new ruler produced belonged to the worst config.

推理 · benchmark · 投机解码13 min read
2026.10.23

Three GPUs, all stuck at 33% utilization: where's the bottleneck? I ran three rounds of elimination and two and a half of my conclusions fell over

Three 64GB cards running a 320B MoE at 24 tok/s, and nvidia-smi shows all three at 33%. That night I ruled out "the cards take turns" and "memory bandwidth is maxed", and blamed the inter-GPU link, based on a concurrency curve that flatlined at 49 tok/s at both 4 and 8 streams. Two months later I reread the raw data: the flatline was drawn by the server's default of 4 slots, and the numbers I used to rule out turn-taking are the most direct evidence for it. This post gives two divisions and one multiplication for reading multi-GPU utilization and concurrency curves.

llm · inference · multi-gpu11 min read
2026.10.21

One table said 2.58x, then my own chat client felt no different: which cell of a speculative decoding benchmark should you read?

The same concurrency sweep said "don't switch" when read at 64 concurrent requests and "2.58x faster" when read at 8. After switching, my own chat client showed 38 tok/s on prose, while hitting the same server directly gave 74.1. None of the numbers were wrong. What was wrong was which cell I read and where I stopped measuring. This post gives a formula for how much faster a whole turn actually gets, a division for taking apart a model card's speedup claim, and a checklist to run before you trust any speedup number.

推理 · 投机解码 · benchmark13 min read
2026.10.20

Three pruned variants failed the same questions, so I blamed the base model: why did one run against the official API flip the verdict back?

Three expert-pruned builds of DeepSeek-V4-Flash all scored around 7.3 on my question set, and their wrong answers clustered on the same questions. I took that as proof the base model was weak. Then I ran the same set through the official API: 9.21, the highest score in my table. This post explains why that overlap statistic had no power to answer the question, how to tell before running it, and how to read the benchmark table that ships with a pruned checkpoint.

llm · moe · pruning13 min read
2026.10.10

Same machine, same 8 concurrent requests, and I've measured 72, 171, 359 and 606 tok/s. Which one is its real speed?

Before benchmarking a large model split across three ex-mining GPUs, I set myself a rule: if 8-way concurrency doesn't deliver at least 3x the single-stream throughput, rethink the whole setup. The first run came back at 2.84x and nearly killed the plan. It turned out that number mostly measured when the requests decided to stop, not how fast the hardware computes. On the same machine at the same 8-way concurrency, changing prompt length, stopping rule and content moves aggregate throughput anywhere from 72 to 606 tok/s. This post gives a wall-clock decomposition you can compute yourself, reconciles all four numbers against it, and lists the questions I now ask of anyone's concurrency benchmark.

推理 · benchmark · 方法论11 min read
2026.09.17

Same GPU: 8-bit beats 4-bit by 10% on single-stream. So why did I ship the slower one?

Four days earlier I benchmarked a 27B dense model on this card and concluded official 8-bit won outright. I reran the same script on a 35B MoE and the result flipped: 8-bit still won single-stream, but lost across the board under concurrency, because its weights cost 11 GiB more and bought away 600,000 tokens of context. Here's the division that converts one into the other, and a test for whether a quantization result transfers at all.

llm · quantization · moe9 min read
2026.09.09

Four Concurrent Requests, 350 tok/s Aggregate. Why Did the Isolated Re-test Show 91?

I wrote in my notes: "4 concurrent, ~350 tok/s aggregate, continuous batching works." The next day I looked at completion times and saw four requests finishing one after another, 91.2 tok/s aggregate. This post shows how one division and a table of completion timestamps tells real parallelism apart from a tidy queue, and then works out how far a 98 tok/s single stream really is from the physical ceiling.

llm · benchmark · apple-silicon10 min read