Ariaki
/
Back to blog

2026.10.28

PCIe 4.0 x16 vs 2.0 x4 is 15.75x on paper, but loading a model was only 2.6x faster: how should load time count when you pick a GPU for a job?

I wrote a small scheduler for my three home machines that sends each job wherever it will finish first. Once I added model load time, the same 9B inference job belonged on the 4090 at 5,000 tokens and on the mining-card box at 50,000. This post gives two divisions you can do yourself: one tells you which stage your load time is stuck in, the other gives the token count where two machines trade places. Along the way: I had the mining cards on an x16 link in my head, and three slots with different specs all came out at the same x4.

推理硬件PCIe方法论

On one side, an RTX 4090 on PCIe 4.0 x16. On the other, my unlocked mining cards on PCIe 2.0 x4.

On paper the first link has 15.75 times the bandwidth of the second. So moving a model from disk into VRAM should be an order of magnitude faster on the 4090, right?

Measured: the 4090 loads at 1.88 GB/s and the mining card at 0.72 GB/s. That's 2.6x.

It gets better. Once load time is counted, the same 9B inference job should go to the 4090 when it's short and to the mining card when it's long. The crossover is around 20,000 tokens.

If you just want the formulas, they're two divisions in sections 3 and 4. If you want to know why a slot labeled x8 doesn't mean much, read section 2.

1. How this started: my scheduler treated load time as zero

I have three machines at home that run models: a 4090, a box with three 64 GB mining cards, and a 64 GB Mac. With more than one machine, "where should this job go" becomes a real question, so I wrote a small scheduler with a simple rule:

estimated completion time = time spent waiting in queue + time to compute

The lowest number wins. Compute time comes from a speed table. For one 9B model, say, decode runs at 146 tokens per second on a mining card and 129 on the 4090.

But one term was missing. The model has to get from disk into VRAM first. I had it set to zero, on the theory that "it's a few dozen seconds, who cares."

Then it dawned on me that those few dozen seconds aren't the same everywhere. The 4090 sits on PCIe 4.0 x16. The mining cards sit on an old, locked-down link. Reading the same 20 GB of weights could differ by more than the compute does. Short jobs get hit hardest: 30 seconds of compute plus 30 seconds of loading, and loading alone decides who wins.

So the term had to go in. First I measured.

2. The link first: the x16 I had in my head

I'd always assumed those mining cards ran on PCIe 2.0 x16. I checked the live link. All three were at 2.0 x4.

Then I remembered that the three slots on that motherboard aren't the same. Two are x8 slots wired straight to the CPU, and one is an x4 slot hanging off the chipset. Why would a card in an x8 slot only run at x4?

The fact that they matched is itself the answer. Three slots with different capabilities and one identical link means the limit isn't in the slot. It's in the card. Going back to my notes from when I set these cards up:

  • From the factory the link is locked to Gen1, and an unlock patch raises it to Gen2. On my cards that took measured throughput from 0.85 GB/s to 1.68 GB/s
  • Gen3 and up: what I wrote down back then was that the relevant fuse is already blown, so software can't bring it back
  • The lane count is stuck at x4 because a row of small capacitors next to the card's edge connector was removed. Getting x16 back means soldering them back on

So changing slots does nothing, and all three cards can share one load-speed number.

Here's the takeaway: the slot label is a ceiling, and the link the card actually negotiates is what you get. To see the live link:

nvidia-smi --query-gpu=pcie.link.gen.current,pcie.link.width.current --format=csv

Check it while the card is under load. A lot of cards drop the link to save power when idle, so an idle reading can show a lower generation than you really have. Don't panic over that.

3. Then the load: why only 2.6x

I used the real loading function to read a safetensors weight file into VRAM, after dropping the OS page cache. Then I immediately read the same file again, and this time it was already in RAM.

           cold (cache dropped)   warm (file already in RAM)   link only (RAM to VRAM)
4090       1.88 GB/s              4.91 GB/s                    26.36 GB/s
mining     0.72 GB/s              1.43 GB/s                    1.69 GB/s

Start with the right-hand column. On paper PCIe 4.0 x16 is about 31.5 GB/s and 2.0 x4 is about 2.0 GB/s. Both sides measured 84% to 85% of that, so the links themselves are behaving.

The middle column is where it gets interesting. On the 4090, with the file already in memory, loading still only hits 4.91 GB/s. That's under a fifth of the 26 GB/s the link can do. On the 4090 the link isn't the bottleneck at all. The slow part is the loader picking the file apart into tensors and copying them into VRAM one at a time.

The mining card is the opposite. Warm read 1.43, link 1.69: it's running right up against the link. That x4 really is what's holding it back.

The gap between cold and warm reads is the disk's share. That's the first thing in this post you can work out yourself:

seconds per GB spent on disk       ≈ 1 / cold speed − 1 / warm speed
seconds per GB spent everywhere else ≈ 1 / warm speed

Plug in the numbers:

4090:   1/1.88 − 1/4.91 = 0.53 − 0.20 = 0.33 s/GB on disk, 0.20 s/GB elsewhere
mining: 1/0.72 − 1/1.43 = 1.39 − 0.70 = 0.69 s/GB on disk, 0.70 s/GB elsewhere

The 4090 spends over 60% of its load time waiting on the disk. The mining card splits it about evenly, with the disk and the link equally slow.

So "15.75x" turns into "2.6x" for a simple reason: on the fast machine, the stages before the link hold it back first. The link is 15 times wider, but that only matters in the last stage, and the disk and loader stages aren't nearly as far apart.

This is the one I most want you to keep: when someone says "PCIe 4.0 loads twice as fast," ask whether the data was already in RAM. On a cold start that reads from disk, link width often isn't even the main factor. If you take link bandwidth from a cudaMemcpy benchmark and use it to predict load time, on a fast link you can be off by several times.

4. The token count where the winner flips

With load speed in hand, the estimate becomes:

time ≈ tokens to generate / decode speed + weight size / load speed

Three cases for a 9B model, with the weight size given to the scheduler as 20 GiB (about 21.5 GB):

50,000 tokens, no load:
  mining  50000/146 = 342 s          4090  50000/129 = 388 s       → mining wins
50,000 tokens, with load:
  mining  342 + 21.5/0.72 = 372 s    4090  388 + 21.5/1.88 = 399 s → only 7% apart
5,000 tokens, with load:
  mining  34 + 30 = 64 s             4090  39 + 11 = 50 s          → 4090 wins

Without load time, the mining card always wins. With it, short jobs go to the 4090 and long ones to the mining card. Where's the line? Set the two times equal and solve, and you get the second formula:

crossover tokens = weight size × (1/slow load − 1/fast load) / (1/slow decode − 1/fast decode)

"Slow load" is the mining card's 0.72 and "fast load" is the 4090's 1.88. "Slow decode" is the 4090's 129 and "fast decode" is the mining card's 146. Plugging in:

numerator:   21.5 × (1/0.72 − 1/1.88) = 21.5 × 0.857 = 18.4 s
denominator: 1/129 − 1/146 = 0.000903 s/token
crossover ≈ 18.4 / 0.000903 ≈ 20,000 tokens

You can read the formula straight off. The numerator is how much you lose on loading. The denominator is how much you win back per extra token. The mining card loses 18 seconds on loading and wins back under a millisecond per token, so it needs about 20,000 tokens to break even.

My scheduler has one more rule: if two machines are within 10% of each other, it calls a tie and lets other criteria decide. The 50,000-token case above is 7% apart, so it got called a tie and went to the 4090. By that rule the mining card only becomes clearly faster at around 98,000 tokens. I picked that 10% myself. Use a different threshold and the crossover moves.

After making the change, I checked placement with three real jobs. The 50,000-token job with no load size went to the mining card. The same job with a 20G load size went to the 4090 (a tie). The 5,000-token job with a load size went straight to the 4090. All three estimates matched the hand calculation above.

5. Which of these numbers to trust, and which not to

To be honest, all I've verified is that the placement logic is right. I haven't verified that the predictions are accurate:

  • Those three jobs were set up to exit almost immediately, so they wouldn't write fake data into the speed table. That shows the scheduler picks machines by the formula. It doesn't show that 372 or 399 seconds is close to what a real run would take
  • The 146 and 129 decode speeds came from earlier entries in the speed table, one sample per cell, and I didn't remeasure them. The crossover is very sensitive to them: the denominator is a difference of two reciprocals, so the closer the two decode speeds get, the further out the crossover moves
  • What I measured as "loading" is only reading the weights into VRAM. Starting a real inference server also means initialization, compilation and allocating cache, none of which I counted, and those take different amounts of time on different machines too. So the load term in the formula is a lower bound
  • The test files weren't the same size. The 4090 machine had no model file over 4 GB on disk, so I wrote a 4 GB test file there. The mining-card box read a real 10.7 GB weight shard. File size affects read-ahead and caching
  • On the Mac I measured 6.13 GB/s, but I couldn't drop the cache (that needs an admin password), and I only got one run, so it's probably high. A second read right after came in at 16.83, which was obviously cached. All I'll claim for the Mac is that it's clearly faster than the other two. Unified memory doesn't go over PCIe, so the direction holds

6. Next time you pick a machine for a job, go in this order

  1. Check each card's live link generation and width, under load. Ignore what's printed on the slot.
  2. Use the real loading function and measure one cold read and one warm read. The cold-warm gap is the disk's share. If the warm read is far below raw link bandwidth, the loader is what's slowing you down.
  3. On a machine that uses less than half its link, don't expect a wider slot or a faster card to speed up loading. Fix the disk first, or keep the weights in RAM.
  4. Write your time estimate as "tokens / decode speed + size / load speed," with both terms.
  5. When one machine decodes faster and the other loads faster, use the crossover formula to find the dividing token count, and send anything shorter to the faster loader.
  6. For jobs whose model is already resident in a running server, set the load term to zero so you don't count it twice.
  7. Run real jobs against the predictions, and let at least one run to completion and compare actual time with predicted before you trust the speed table. I still owe that step myself.