AriakiOn my machine, the fastest possible local model call takes 79 milliseconds. One iteration of a Python loop takes 23 nanoseconds.
Divide one by the other and you get 3.4 million.
That number turned several of my "optimizations" into jokes. The best one: I had long assumed that trimming a system prompt from 20k tokens to 15k would save real time. At measured prices, the savings don't even cover one call. Worse, the number that convinced me long prompts were expensive came from dividing by the wrong thing (Section 4).
If you only want the divisions, jump to Section 3. If you want to know how "µs per token" figures lie, Section 4.
1. Does the 2007 table still hold?
Jeff Dean has a famous list, "Latency numbers every programmer should know": how many nanoseconds for an L1 cache hit, a main memory read, a branch mispredict. It dates from 2007.
I wanted to know whether it survived 19 years, so I re-measured each row in C++ on a Ryzen 5 5600 (Zen 3, 6 cores, 62 GB RAM):
| Operation | 2007 table | Measured | Verdict |
|---|---|---|---|
| L1 cache reference | 0.5 ns | 1.13 ns | Original is idealized |
| L2 cache reference | 3 ns | 3.43 ns | Accurate |
| L3 cache reference | not listed | 15.55 ns | Add it yourself |
| Main memory reference | 50 ns | 97.56 ns | Original is 2x optimistic |
| Branch mispredict | 5 ns | 5.078 ns | 1.6% off after 19 years |
| Uncontended mutex | 15 ns | 5.45 ns | Nearly 3x faster now |
| Read 1 MB sequentially | 64,000 ns | 47,286 ns | ~22 GB/s single core |
I stared at the branch mispredict row for a while. Nineteen years, off by 0.078 ns.
The takeaway is plain: the table still works for order-of-magnitude estimates with two fixes. Double main memory, and add an L3 row. Modern L3s are 32 MB, and skipping that layer skews any estimate involving large arrays.
But accuracy isn't really the issue. The issue is that my actual work never touches this table.
2. The table I should actually memorize
My day is agent pipelines, Python scripts, data plumbing. Those run in milliseconds to seconds, six orders of magnitude above nanoseconds. Using the nanosecond table for millisecond work is like measuring a highway with calipers.
So I applied the same idea one layer up and measured my own unit prices. The model rows come from a mid-sized model running locally behind an OpenAI-compatible gateway, all measured after warmup:
Operation Time
─────────────────────────────────────────────────
Python loop iteration 23 ns
Python dict lookup 54 ns
Python method call 69 ns
SQLite indexed point lookup (100k rows) 7.4 µs
SQLite 1000 separate point lookups 6.75 ms
SQLite same 1000 rows in one query 168 µs
SQLite 1000 inserts, commit each 2.06 ms
SQLite 1000 inserts, one commit 420 µs
Spawn /bin/true 583 µs
Spawn python3 24.9 ms
Localhost TCP connect 36.5 µs
─────────────────────────────────────────────────
Minimal model round trip (1 token, warm) 79 ms
Model decode 41.4 tok/s, ~24 ms/token
Streaming time to first token 185 ms
Put everything on one ruler, with one Python loop iteration = 1:
Python loop iteration 1
dict lookup 2
SQLite point lookup 320
spawn /bin/true 25k
spawn python3 1.08M
one model call 3.4M
A confession while we're here: my original notes listed "subprocess" as 25,000x. That's the /bin/true number. Spawning python3 is 1.08 million x, more than forty times higher. The same notes claimed "one python3 spawn equals 1,000 SQLite lookups." Do the math: 24.9 ms ÷ 7.4 µs ≈ 3,400. Both were copied from memory without actually dividing.
3. Division one: count times price, find the boss
Dean's method is three steps: count how many times each kind of operation happens, multiply by unit price, see which term dominates.
pipeline time ≈ Σ (count of operation × its unit price)
It's one multiplication and one sum, nothing more mysterious than that.
Plug in a very ordinary agent pipeline: 5 model calls, 200 database lookups, 3 python subprocesses.
5 × 79 ms = 395 ms
200 × 7.4 µs = 1.5 ms
3 × 24.9 ms = 75 ms
────────────────────
total ≈ 471 ms
model 84%, subprocess 16%, database 0.3%
Add indexes, rewrite queries, drive that 1.5 ms to zero, and the pipeline gets less than 0.3% faster.
The number you can actually move is the 5. Merge two calls and you save 79 ms, which beats optimizing the database fifty times over.
This bit me once, and it hurt. I run a simulation-world side project where a model writes a yearly recap. The validator had a rule: titles must not start with "Year N". The model loved starting titles with "Year N". Validator rejects, model rewrites, rewrite has the prefix, rejected again... 8 retries in a row, 1.65 million input tokens, 42 minutes.
Every rejected draft was fine once you stripped those few characters. The title held up, every fact was there. Stripping a prefix is a string operation, microseconds. Each retry was a full-price model call.
The ruler above makes the gap obvious. Before rejecting, ask: can this defect be fixed without calling the model? If yes, normalize in place. If no (invented facts, missing content, broken logic), then reject and regenerate.
A related trap: the fixer and the validator must share one rule set. Otherwise the validator flags something the fixer can't remove, and you get a loop where every lap is a full-price call.
The database layer has the same shape, just much smaller:
- 1,000 separate lookups 6.75 ms vs one query 168 µs: 40x
- commit per insert 2.06 ms vs one commit 420 µs: 4.9x
So a database query inside a loop: fix it now. An .append() inside a loop: leave it alone. I measured swapping append for a list comprehension too: 1.15x. Not worth your ten minutes.
4. Division two: what does trimming a prompt actually save?
This is where I got it most wrong.
When I measured prefill (the phase where the model reads the prompt), I got this:
| Prompt length | Total time | Total ÷ tokens |
|---|---|---|
| 643 tok | 138 ms | 215 µs/tok |
| 5,194 tok | 128 ms | 25 µs/tok |
| 20,790 tok | 251 ms | 12 µs/tok |
My reading at the time: "Per-token cost drops from 215 to 12, an 18x drop, so long prompts are the better deal."
Sounds reasonable. Almost all of that 18x is an artifact of the division.
That column is total time divided by token count. Total time includes a fixed cost that has nothing to do with length: sending the request, queueing, returning. I roughly use the 79 ms minimal round trip as that fixed cost (it reads a tiny prompt too; I'm ignoring that here). In the 643-token row, 79 of the 138 ms isn't prompt reading at all. Divide that by 643 and of course the per-token price looks scary.
Subtract the fixed cost first, then divide:
643 tok: (138 − 79) ÷ 643 ≈ 92 µs/tok
5,194 tok: (128 − 79) ÷ 5,194 ≈ 9.4 µs/tok
20,790 tok: (251 − 79) ÷ 20,790 ≈ 8.3 µs/tok
The last two are basically flat. Most of the "18x" was fixed overhead being amortized. Only the 643 row stays high, and with a short prompt the remaining time is tiny, so noise gets magnified.
To ask "how much does trimming save," you want the marginal price: between two lengths, how much extra time each additional token costs.
time saved by trimming ≈ tokens removed × marginal prefill price
marginal price = (long-prompt time − short-prompt time) ÷ (length difference)
With my numbers:
5,194 → 20,790: (251 − 128) ÷ 15,596 ≈ 7.9 µs/tok
643 → 20,790: (251 − 138) ÷ 20,147 ≈ 5.6 µs/tok
trim 5000 tokens ≈ 5000 × (5.6 ~ 7.9) µs ≈ 28 ~ 40 ms
One model round trip is 79 ms. You carve a quarter off your system prompt and save at most half a call.
With the 12 µs average I first used, I got 60 ms and already thought "not worth it." With the marginal price it's even less worth it. The direction held, the number was wrong.
So this is the first question I now ask of anyone's figure: when you see "µs per prefill token," ask whether the fixed cost was subtracted. If not, the short-prompt end can be inflated several times over, and it will fool you into thinking longer is cheaper. The same disease shows up in "ms per request" and "seconds per image": any time a total gets divided by a count, look twice.
5. What went wrong while I was measuring
This section decides whether you should trust the rest.
Cold start is 50x. My first minimal round-trip measurement came back at 3.99 seconds. After 3 warmup calls it was 79 ms. The first call included loading, graph capture, connection setup. Had I plugged 3.99 s into Section 3, the conclusions would have been absurd.
The compiler optimized away the thing I was measuring. My first branch mispredict test reported 0.01 ns. At -O2 the compiler turned the if/else into a branchless conditional move, so there was no branch left to mispredict. The fix was an empty inline-asm barrier in each branch, then disassembling and counting the conditional jumps to confirm they existed.
Two points I can't explain. The 643-token prefill (138 ms) was slower than 5,194 tokens (128 ms). I also tried a prefix-cache hit once: hit 216 ms, miss 128 ms, so the hit was slower. I didn't chase either. It could be that backend's caching and scheduling, or just noise. That's why Section 4 gives a range for the marginal price instead of one number.
Every figure is a single value, no variance. No repeated-run distributions, no confidence intervals. Good for orders of magnitude, not for comparing differences under 10%.
This table belongs to this machine and this model. Change the GPU, the model, or switch to a cloud API and the prices shift. Cloud round trips are usually higher than local, which only makes Section 3 more extreme. If you plan to use it, rerun it in your own environment. The probes are read-only and take minutes.
I also reproduced the quicksort estimate from Dean's original writeup: predicted 7.41 s, measured 6.08 s, off by 22%. That sounds large, but the estimate exists to tell you which term dominates, and for that 22% is plenty.
6. Next time someone says "this is slow, should we optimize?"
- Don't open the profiler yet. List how many times each kind of operation runs: model calls, database queries, subprocesses, network requests.
- Measure those unit prices on your own machine. Warm the model up 3 times before timing it.
- Count × price, sum, sort.
- Optimize only the top item. If it's model calls, the tools are fewer calls: merge them, cache results, fix in place anything that doesn't need the model.
- Before trimming a prompt, compute the marginal price from two lengths, multiply by the tokens you'd cut, and compare against one round trip.
- Whenever you see a "total ÷ count" unit price, ask whether the fixed cost was subtracted.
- When you write a ratio into your own notes, do the division right then. Both of my typos came from skipping that.