Ariaki

Ariaki

Engineer, builder, tinkerer.

View blogGitHub

Latest writing

05 public

Same config, six runs, throughput bouncing between 366 and 561: to see a 5% improvement, do you run more trials or change the ruler?

Three GPUs were already running a large model, and I wanted to know how much a fourth would help. I compared against someone else's four-card number, got 0.90x, and immediately invented a mechanism for it: a longer pipeline cancelling out the extra compute. Then I found the machine was pinned against its power cap, and that the same config run six times in a row swung between 366 and 561 tok/s, a coefficient of variation of 15%. This post gives one division that tells you, before you run anything, how large a difference your measurement can resolve. Then it covers how I switched to a reading that ignores clock speed, and why the highest number that new ruler produced belonged to the worst config.

7 of 8 requests truncated, and I almost wrote the model off: when picking a local model, which of my verdicts were really testing me?

In one day of choosing between two local Qwen models I reached four verdicts, and every one of them needed correcting: a missing launch flag dumped chain-of-thought into the answer, I compared a fixed model against an unfixed one, 3 of my own 8 test questions were broken, and a per-request thinking switch was silently overridden by the server. Three checks you can run without reading any source: answer share, broken-question rate vs. score gap, and identical token counts across two configs.

The same community-fixed chat template fixed one machine and missed the other completely: can you tell which layer an error lives in without reading the source?

Two machines running the same Qwen family model both failed about 3.3% of requests on the turn after a tool call. I put a community rewrite of the chat template on the first one, and the failing argument shapes went from 7 of 19 to 0 of 19. On an isolated copy of the second one, the bad request still returned a 500, in 3 milliseconds. This post covers how to use time-to-failure to locate the layer that broke, how 341 errors collapsed into 39 poisoned sessions, and why the README's "80% faster" claim doesn't hold up.

Three GPUs, all stuck at 33% utilization: where's the bottleneck? I ran three rounds of elimination and two and a half of my conclusions fell over

Three 64GB cards running a 320B MoE at 24 tok/s, and nvidia-smi shows all three at 33%. That night I ruled out "the cards take turns" and "memory bandwidth is maxed", and blamed the inter-GPU link, based on a concurrency curve that flatlined at 49 tok/s at both 4 and 8 streams. Two months later I reread the raw data: the flatline was drawn by the server's default of 4 slots, and the numbers I used to rule out turn-taking are the most direct evidence for it. This post gives two divisions and one multiplication for reading multi-GPU utilization and concurrency curves.

Two checkpoint files, identical down to the byte, and I ran a whole batch on the wrong one. Why did fixing it make 3 more shots drift?

A local video model ships two checkpoints. Both files are exactly 20,970,379,616 bytes, with zero tensor-shape differences; only the md5 differs. My script loaded the wrong one for an entire batch of experiments. The first retest after the fix couldn't detect anything; the second showed the 'wrong' checkpoint was actually better. This post covers two things: a multiplication that tells you before you start that an A/B is doomed to show nothing, and three questions to ask whenever someone posts an A/B result.