AriakiAt 10:17 the machine started logging memory-scrub timeouts. By 11:38 all three cards were demanding a reboot.
Here's the part that makes no sense:
nvidia-smi → three cards, all present, correct capacity, normal temps
cuInit() → returns 3
cuDeviceGetCount() → 0
The management tool says the cards are fine. CUDA says there are no cards at all. And the two workloads that tripped it that morning had nothing to do with each other — one was diffusion image generation, one was LLM inference, neither was training. Their only shared property: both pushed VRAM close to exhaustion.
Want the reusable rule? Chapter 2, it's one division. Want to see how many times I got this wrong? Chapter 5.
1. What those 141 MB actually are
The card is a mining-crippled version of the GA100 die. The factory gives you 8GB. There's an open-source community tool that software-unlocks it to 64GB — it flips registers to re-enable the masked HBM, and it survives reboots. Nothing mystical, nothing like overclocking.
The unlock itself is fine. The problem lives in a patch inside the unlock tool called late-pma. What it does is straightforward: take the topmost reserved memory region and register it into the allocator's usable pool, so nothing goes to waste.
In the factory 8GB address space, that's safe. But unlocking moves the top of VRAM from 8GB to 64GB, and the GPU firmware coprocessor (GSP) plus its heap moves up there with it. So the region being "helpfully reclaimed" is now backing the firmware itself.
The allocator log lays out the ledger:
region[6] base=0xff7300000 limit=0xfffffffff rsvdSize=0x8d00000
WPR meta wprStart=0xff7400000 wprEnd=0xffff00000
It's a subtraction:
region size = limit − base + 1 = 0x8D00000 = 141.0 MiB
firmware WPR2 = wprEnd − wprStart = 0x8B00000 = 139.0 MiB
actually reclaimable = 141.0 − 139.0 = 2.0 MiB
Two megabytes of genuinely free memory, in exchange for exposing 139 MB of firmware to arbitrary writes. The allocator sorts regions by performance, so this one sits dead last — you only ever get handed it after the main pool is completely drained. Write to it and you get a memory access violation, then the scrub engine wedges.
The verification after the fix is also a subtraction, and it lands to the decimal: per-card capacity as seen by the inference framework drops from 63.53 GiB to 63.39 GiB. That's 143.4 MiB, matching region 6's 141.0 MiB. The 141 MB went back, cleanly.
2. My stress test measured nothing
This is what the post is actually about.
With the patch applied I needed to prove it worked, so I ran stress tests: start the inference server, fill the KV cache, see if it crashes. Three rounds, peaks at 3.8%, 75.7%, 84.1%. All PASS.
Then someone asked, "isn't 84% kind of low?"
The problem isn't that 84% is low. The problem is that no value of this metric would have helped. The KV cache pool is a single block the server grabs from the allocator at startup. Once the server is up, the allocator has clocked out. Driving KV to 84% just rearranges furniture inside the block you already own — it can never touch what the allocator still holds.
And even with the right metric, 84% is nowhere close. Here's how deep the landmine is buried:
faulty region share = 141 MiB ÷ 65536 MiB = 0.215%
to reach it, VRAM utilisation must exceed 99.785%
I stopped at 84.1%. That's 15.7 percentage points short, which in absolute terms is 10,290 MiB. I was 10 GB away from the bug.
You can lift this division straight out: if a fault only appears in the final 0.2% of allocations, any stress test measured in "percent used" is worthless unless that percentage exceeds 99.8%. In most stress-test reports, that number is 80, or 90, or at best 95.
Three green rounds. Three wasted rounds.
3. Allocating without writing is not a test
The third probe threw out percentages entirely: hammer the allocator directly, keep asking until the driver refuses.
There's a second trap right here. My first attempt only called cudaMalloc without writing, and it came back clean. A bad pointer only faults when you touch it. Get handed a pointer into the firmware region, never write to it, and it's just a number sitting quietly in a variable.
Adding the write is what made it a real test. The five runs:
| Test | Scale | New faults | New scrub timeouts |
|---|---|---|---|
| Drain + write × 3 cards | 64,576 MiB each, 5.4 MiB left | 0 | 0 |
| 4 MiB fine-grained | 16,144 blocks, all written | 0 | 0 |
| In-process churn × 12 | 58,112 MiB dirtied per round | 0 | 0 |
| Cross-process × 10 | fresh process holds 92%, exits | 0 | 0 |
| Container restart × 3 | peak 55,822 MiB per round | 0 | 0 |
That "5.4 MiB left" in row one is what a finished stress test looks like. The finish line is when the system refuses to hand over any more — 84% isn't even in the parking lot.
One order-of-magnitude aside, which explains why a second candidate fix wasn't load-bearing: measured memory release takes 0.02–0.04 seconds against a 4-second timeout budget. Two orders of magnitude of headroom. A healthy card can't come anywhere near that limit — it only wedges after the region has already been corrupted. So the scrub timeout is a downstream symptom, not the disease.
4. nvidia-smi will lie to you
Back to that opening scene. Three cards alive and well in the management tool, and CUDA reporting zero devices.
The mechanism is simple: nvidia-smi talks to the management interface, and it doesn't need a CUDA context. The driver's CUDA layer can be completely wedged while the management layer keeps answering happily, listing every card with correct capacity and temperature.
Use this directly: the only trustworthy probe for "can this card still do work" is the return value of cuInit() plus cuDeviceGetCount(). Any health check built on nvidia-smi output is theatre in this failure class — it hands you a beautiful green light at exactly the moment you most need the truth.
Same family, also hit that day: dmesg | grep returned zero hits. Does zero mean "not there" or "you can't read it"? Most distributions restrict unprivileged access to the kernel ring buffer by default. Switching to systemd's kernel journal surfaced 228 lines on the same machine. A grep count of 0 always needs "I couldn't read it" ruled out first.
5. Where I got it wrong
Listing the parts I didn't do cleanly, because otherwise the numbers above don't deserve your trust.
I changed two variables at once. To restore service quickly I installed two patches together. Later the upstream author asked directly whether the bug reproduces with only one of them, and I couldn't answer — because I'd changed two things. I had to remove one, recompile, reboot, and re-run all five test groups above just to answer a question that one round should have answered.
I cited a conclusion from a PR that got renamed. The fix PR was originally titled "Skip late PMA extension on the [X] SKU," then renamed to "Leave the high reserved region reserved: it backs WPR2" — and its scope widened from one SKU to two in the process. If you read the old title during that window, you'd correctly conclude it didn't apply to your card, and you'd be wrong. Check the rename and commit history before citing someone's PR.
Neither fix is on the main branch. One got merged into a side branch, the other is still open. git pull gets you neither, and the patch on main is still the version that corrupts memory. So "I'm on latest" means nothing here.
There's a second fault still open. After the fix, a batch of ~150K-token prefills hit a different error, at a virtual address around 117 TiB, unrelated to region 6, with the driver staying healthy throughout. Currently localised to a kernel in the inference framework; the suspicion is a cross-stream memory reuse missing a synchronisation call. Until that's closed, everything above only covers the memory allocator layer.
The table in chapter 3 was run with only one patch installed and the timeout budget at its default. Change the configuration and you need to re-run it.
6. Give back 141 MB, get how much?
The funny part: handing that region back was net positive.
With 143.4 MiB released, the remaining VRAM layout became clean enough that bumping the inference framework's memory utilisation parameter from 0.94 to 0.95 actually worked. KV cache pool capacity:
1,000,964 tokens → 1,192,267 tokens (+19.1%)
An extra 191,303 tokens of KV room. The 141 MB given up bought back far more than it cost.
This deserves its own note: before optimising for some "reclaimable fragment," compute what fraction of the total it represents. 141 MiB is 0.215% of 64 GiB. Introducing a failure mode that forces a full machine reboot, in exchange for 0.2%, loses on every accounting. The real lever was the utilisation parameter — same memory, different usage, 19% more.
7. The checklist
Next time you unlock a hardware limit or reclaim a reserved region, run it in this order:
One, look at which regions the allocator registered into the pool. Don't just check that the total capacity number looks right — check its composition.
Two, your acceptance probe must write to every block. Allocation without writes means bad addresses never fault, and you get a beautiful false green.
Three, before testing a mechanism, ask whether your workload will actually drive it. KV cache versus the memory allocator, log keywords versus message delivery — same error class: a metric that can't physically reach the thing under test.
Four, compute the faulty region's share of the total and use it to derive what percentage your stress test must hit to mean anything. Below that number, any number of rounds gives you zero.
Five, check liveness with cuInit(), not nvidia-smi. The management interface doesn't need a CUDA context, so it keeps reporting green while the CUDA layer is dead.
Six, install one patch at a time. Otherwise when someone asks you for the isolated result, your only answer is to go re-run everything.
Seven, check the rename history before citing an upstream conclusion. Titles change, scopes change, and the version you read may already be stale.