Someone said three GPUs couldn't load the model, and I proved it was a VRAM shortfall: why do two configs crashing on the same line mean VRAM has nothing to do with it?

Two configs. Their memory budgets differed by 3.2 GiB. Both crashed on the same line of the same file.

My read at the time: see, both runs hit the same VRAM wall, just push the budget a bit higher and it'll go through.

That's backwards. When you're short on a resource, giving it 3.2 GiB more should at least move the crash somewhere. If the crash doesn't budge, the resource probably isn't involved at all. This post is about how I took a piece of counter-evidence and filed it as support, and why the real cause was so well hidden.

If you just want the rule, it's in section 3 and it's a subtraction. If you want to see how three cards finally ran and how to split the layers, jump to section 6.

1. A report that said "three cards don't work"

I have three data-center GPUs that started life as mining SKUs: 8GB from the factory, unlocked in software to 64GB each, and each one hanging off a PCIe 2.0 x4 link. That's a pathetically narrow pipe.

The model I wanted to run is DeepSeek-V4-Flash: 284B total parameters, 43 layers, roughly 155 GiB of weights. No single card can hold that, so it needs pipeline parallelism. You cut the 43 layers into three chunks, one chunk per card, and each card hands its intermediate result to the next one.

Someone in the community had it running on four of the same cards and published a solid repo. It had a table that went roughly like this:

Config	Result
4 cards	Works
3 cards + speculative decoding, memory utilization 0.90	Crashes at load time, during expert-weight repacking
3 cards, no speculative decoding, memory utilization 0.85	Same failure, same place
2 cards	155 GiB doesn't fit in 128 GiB. Dead end

The author's diagnosis: the crash happens during model load, so it's unrelated to speculative decoding and unrelated to the memory-utilization setting.

I was going back and forth on whether to buy a fourth card at the time. My first reaction to that table: he's got the cause wrong.

2. My first pass at the math: VRAM overflow, set it to 0.95

The step that crashes is expert repacking. Each layer of this model has 256 "expert" sub-networks, and any given token only wakes up a few of them. At load time, every expert's weights get shuffled into the layout the GPU kernel prefers.

I read that code. After repacking the first expert, it allocates the output buffer for all 256 experts in one shot, while the original weights are still resident. I worked out the per-layer transient peak from the tensor shapes and got about 6 GiB at the time. (Later, redone against the real tensors, the heaviest moment was 4 GiB: 2 GiB of input and 2 GiB of output alive together.)

Then the VRAM budget. vLLM has a knob called gpu_memory_utilization, meaning "use at most this fraction of the card":

Default even split: heaviest card holds 15 layers ≈ 53.3 GiB
Utilization 0.85: 63.5 × 0.85 = 54.0 GiB, leaves 0.7 GiB
Utilization 0.90: 63.5 × 0.90 = 57.1 GiB, leaves 3.9 GiB
Repack peak needs 6 GiB. Neither setting is enough

The numbers fit perfectly. My write-up said: raise utilization to 0.95, budget 63.5 × 0.95 = 60.3 GiB, leaves 7.0 GiB, done. Don't swap the motherboard, hold off on the fourth card.

I even read the author's "same failure, same place" as backing me up. Same wall both times, right?

3. That knob does nothing during model load

Two days later someone asked me, "Didn't you say changing the ratio would fix it?" I went back and read the worker startup sequence line by line.

vLLM brings a worker up in three steps: init device → load model → figure out how much memory is left for the KV cache. The utilization knob only shows up in steps one and three:

Step one compares "total VRAM × utilization" against current free memory and lets you through if there's enough. At this point not a single byte of weights is loaded, free memory is close to 63.5 GiB, so it always passes. After that it imposes no constraint at all.
Step three uses "total VRAM × utilization − what the model already took" to size the KV cache pool. That can only be computed after the model is loaded, so the order is fixed.

Expert repacking happens in step two. At that moment it can use all the physical free memory on the card, with no cap whatsoever. Moving utilization from 0.85 to 0.95 changes nothing, not one byte, at the moment of the repack.

The author's "unrelated to memory utilization" was correct. My rebuttal was the wrong one.

Looking back, the thing that should have tipped me off was that opening line:

Resource difference between the two runs = 63.5 GiB × (0.90 − 0.85) ≈ 3.2 GiB
Change in crash location = 0

Running out of a resource is usually gradual. With 3.2 GiB more, you'd expect to get a few layers further, or crash somewhere else. Two different configs crashing on the same line of the same file is evidence that the crash doesn't depend on that resource. I read it as "same wall twice." Exactly the wrong direction.

The raw capacity math didn't hold up either:

Weights per card ≈ 54 GiB + repack peak 4 GiB ≈ 58 GiB
Usable physical memory per card ≈ 63.5 GiB
Headroom ≈ 5.5 GiB

It fits. The capacity theory was dead.

4. Ruling out seven suspects with one card

At that point I only had two cards; the third was still in the mail. But I'd realized something: expert repacking is each card bit-shuffling the experts it owns. It doesn't care how many cards there are in total. So one card can reproduce every kernel-level problem that "the card in the three-card setup" would hit.

I built two layers of probes:

Pull the repack kernel straight out of the vLLM source, not one line changed, and run it on real hardware with synthetic data at production scale (256 experts). All passed.
Feed it the actual weight files and run the full preparation path. All passed.

To make sure I was feeding real weights and not filler, I added a gate: take a small slice and count the distinct values. If there's only one, I'm testing fake data and the result doesn't count. Real weights had several hundred.

All seven hypotheses below were eliminated:

Hypothesis	Result
Memory utilization too low	Ruled out, it isn't in effect during load
Three cards physically can't fit it	Ruled out, about 5.5 GiB of headroom
Larger shared memory on this GPU generation causes an out-of-bounds index	Ruled out, all offsets are compile-time constants
Output buffer overrun	Ruled out, indexing is exact
Repack kernel has a bug on this GPU generation	Ruled out, full 256-expert spec passes on real hardware
Some value in the real weights triggers it	Ruled out, real weights pass
Expert count read wrong from config	Not applicable, this path derives it from the tensor shape

I nearly tripped again partway through. A community fork had a change that swapped "total experts" for "experts on this card," with a very convincing comment, and I was ready to call it the culprit on the spot. Then I grepped the call chain. That function belongs to a different quantization format. My model never goes through it.

After clearing all seven, I wrote down: "the remaining suspect is multi-card sharding interaction, which a single card can't test."

That guess was wrong too.

5. The culprit was an environment variable

The answer came from two independent reproduction reports, and they both pointed at the same thing: a PyTorch allocator flag, expandable_segments:True.

With that flag on, every time PyTorch frees memory it hands the physical pages straight back to the driver instead of keeping them in its own cache. Normally that's harmless. But expert repacking allocates a temporary tensor per expert and throws it away right after. On the last card that's roughly fourteen thousand alloc/free round trips.

The memory-scrubbing engine on these cards can't keep up. The driver log fills with scrub timeouts, and then the card lands in a very strange state:

nvidia-smi        → three cards, capacity normal, temperature normal
VRAM readout      → empty
Allocate memory   → fails
cuInit()          → returns 3

The card looks empty and still won't hand out memory. Only a cold reboot fixes it.

All seven of my hypotheses were asking "is there enough capacity" or "is the indexing right." The culprit was on a third axis: how the allocator releases memory. When the capacity math works out and things fail anyway, look at allocation and release frequency, granularity, and whether pages go back to the driver. Peak size alone won't tell you.

So why does that README still say "three cards don't work"? I went through the repo's commit history:

Day 1 00:57  First results doc says "3 cards fail". The flag is on at this point
Day 1 22:43  Author independently hits the same problem on four cards
             and adds the flag to the list of things to disable
             Never goes back to retest three cards

The conclusion predates the discovery of the real cause by 22 hours. "Three cards don't work" got pinned on the card count when it belonged to an environment variable nobody had identified yet.

This is the rule you can take to other people's conclusions:

When you see "N cards don't work" or "config A doesn't work":
  time the conclusion entered the repo < time a known gotcha entered the repo
  and there's no record of a retest afterward
  ⇒ treat the conclusion as a lead, not as evidence

It's cheap to check: git log, then git show <commit>:path to see whether that version of the doc knew about the gotcha.

6. The third card arrived

Once the third card was in, I turned the flag off and used the layer splits from the two reports:

Split (layers per stage)	Last stage	Result
14, 15, 14	14	Crashes, card gets wedged (reported)
15, 15, 13	13	Won't start, KV has 0.76 GiB, needs 1.24 (reported)
15, 16, 12	12	Runs (reported, reproduced on my machine)
16, 16, 11	11	Runs (reported)

Why does the last stage need fewer layers? The speculative-decoding draft module and the output head both live only on the last card. Speculative decoding means a small model guesses the next few tokens and the big model verifies them in one pass; every correct guess is free throughput. That draft module is about 12 GiB, and each layer is about 3.33 GiB, so it's worth about 3.6 layers.

Balancing the load across three cards is one equation:

First two stages L layers each, last stage L − 3.6 layers
2L + (L − 3.6) = 43
L ≈ 15.5, last stage ≈ 11.9
⇒ last stage ≤ 12 layers

Both 15, 16, 12 and 16, 16, 11 sit on that line. There's one more hard constraint: the draft module reads intermediate outputs from layers 40–42, so those three layers must be in the last stage.

What my machine actually measured:

Cold start 180 seconds; model footprint on the three cards 51.64 / 53.76 / 52.86 GiB
Load, expert repack, and CUDA graph capture all passed, no scrub timeouts in the kernel log
KV cache pool of 1,000,964 tokens
Five needles buried in a 199,957-token document: found 5 of 5, all in the right order, prefill at 3,005.5 tok/s
Single-stream generation across four kinds of 512-token output, median 138.23 tok/s. Code 152.22, Chinese prose 87.15. Content type alone makes a nearly 2x difference, because speculative decoding guesses code far more accurately

While I'm here, the utilization math deserves a second pass. What the knob really controls is the KV pool: 63.5 × 0.85 ≈ 54.0 GiB, minus roughly 53 GiB of weights, leaves about 1 GiB for KV, and three cards won't start. Three cards need 0.94–0.95. So "three cards need a higher ratio" is true on its own; it just rescues the KV pool, not the load-time crash. I got half the conclusion right and all of the reasoning wrong.

vLLM computes a maximum context of 959,488 tokens for these three cards. The upstream report tested long-document retrieval itself: 100% at 150K, 87% at 300K, 60% at 450K, 50% at 750K, 30% at 900K. "Fits 959K" and "can find things in 959K" are two separate claims. My 5/5 was at 200K tokens. I didn't test beyond that.

7. Where this experiment is weak

I never turned the flag back on to reproduce the crash on three cards myself. Every crash means a cold reboot, and I didn't want to pay for it. The causal chain is: two independent reports name the flag, plus three cards run once it's off. That's "removing it fixes it." Strictly speaking it's still missing "adding it back breaks it."
The fourteen-thousand alloc/free count and the scrub engine falling behind come from external reports. I didn't count them myself.
Of the four rows in the split table, only 15, 16, 12 ran on my machine. The other three are other people's results.
The two probe layers in section 4 only cover a single card. One card can rule out kernels and data but not post-sharding behavior. And the conclusion I drew from them, "the remaining suspect is sharding interaction," turned out wrong as well.
The evening it first ran, the service crashed on its own while completely idle, with zero requests running or queued. Still unexplained. My first two "zero hardware errors" reports were based on nvidia-smi. Only after switching to the system kernel log did I catch an Xid I'd missed before.
Generation speed was only measured single-stream, 512 tokens, four content types.

8. Next time you see "N cards won't run it"

Find the commit that wrote the conclusion, then find the commits where the author later added disabled flags and known gotchas. If the gotcha came after the conclusion and there's no retest, the conclusion is void.
Two different resource settings crashing on the same line: start from "this has nothing to do with that resource." Running short almost never leaves the crash point exactly where it was.
Before blaming a knob, confirm it's active during the phase where things break. vLLM's memory utilization only governs the pre-start check and the KV pool size. It can't reach model loading.
Capacity math adds up but it still fails: look at the allocator. Release frequency, granularity, whether physical pages go back to the driver. Print PYTORCH_CUDA_ALLOC_CONF first.
nvidia-smi says the card is fine and cuInit() returns nonzero: believe cuInit().
When you spot a change that "explains everything," grep the call chain and confirm your code path actually goes through it.
For pipeline-parallel splits, convert anything that lives only on the last stage (draft module, output head) into layer-equivalents and solve 2L + (L − those layers) = total layers.
Label any conclusion you only derived on paper as "static estimate, not run." Remove the label after it actually runs. I did write that label back then, which made eating my words a little less embarrassing.