AriakiI wrote a probe to check whether peer-to-peer memory access worked between each pair of three GPUs. The cross-card copy test passed in all 6 directions. Every bit arrived intact.
At that moment, P2P was not enabled on a single pair.
Worse, I tried to catch it with bandwidth and measured 1.627 GB/s. Someone in the community, on a machine where P2P genuinely worked, reported 1.68. A 3% gap. That's no gap at all.
If you just want the test that works, skip to section 4. If you want to see how bandwidth lies in both directions, read sections 2 and 3.
1. A link that wasn't open still got half the answers right
The hardware: three mining-edition datacenter cards. They ship with 8GB, a community tool unlocks them to 64GB each, and each sits on a PCIe 2.0 x4 link. That's a painfully thin pipe.
P2P, in plain terms, means one card reads and writes another card's memory directly, without a detour through the CPU's RAM. When you split a model across cards, it decides whether data between them takes the long way around.
Someone in the community had already unlocked P2P on these cards too. Before applying any patch, I wanted to settle one question first: once it's patched, what exactly will convince me it works?
So I went backwards. I ran the probe in the locked state to see what "not working" looks like. Here's what came back:
| Check | Result |
|---|---|
| Driver-reported capability bit | NO everywhere |
| Enabling peer access | "invalid device" everywhere |
| Cross-card copy (6 directed pairs) | 6/6 pass |
| Kernel on card B reads card A's memory directly | 0/6, illegal address |
| Kernel on card A writes card B's memory directly | 0/6, illegal address |
That copy row nearly fooled me.
The reason turned out to be boring. When the copy API sees no direct path between two cards, it legitimately falls back to staging through host memory: card A to system RAM, then RAM to card B. The data is correct and the call returns success. From the API's point of view, the job got done.
So "copy succeeded" proves the data arrived. It says nothing about which road it took.
I also made a separate mistake in this step. The system tool showed GNS for every pair in the capability column, and my first read was "the hardware doesn't support it." Wrong. That's the bit the current driver reports, and it's precisely what the unlock patch changes. Judging hardware from a driver's self-reported label is the same kind of shortcut as judging the path from a success code.
2. I tried to tell them apart by bandwidth and got 1.627 vs. 1.68
My reasoning was naive: staging means two hops, so it should be noticeably slower.
I measured 1.627 GB/s. The report from the unlocked machine said 1.68 GB/s.
First, what can this link carry at all? It's a single multiplication:
one-way link ceiling = per-lane rate × lane count × encoding efficiency ÷ 8
PCIe 2.0 runs 5 GT/s per lane with 8b/10b encoding: 8 useful bits out of every 10, so 0.8 efficiency. Plugging in:
5 × 4 × 0.8 ÷ 8 = 2.0 GB/s
1.627 ÷ 2.0 = 81%
1.68 ÷ 2.0 = 84%
Both numbers are pressed against the ceiling. So why can't you tell them apart? Draw the two paths:
direct: card A's x4 link → motherboard forwarding → card B's x4 link
staged: card A's x4 link → host RAM → card B's x4 link
Both paths cross A's x4 and B's x4. The middle segment, host RAM, is far wider than 2 GB/s. As long as the driver pipelines the transfer in chunks (one chunk heading into RAM while the previous one is already on its way to B), the two hops overlap and everything is throttled by the x4 link. Same narrowest segment, same result.
Turn that into a ratio and you have the test for whether bandwidth can discriminate between the two paths:
discrimination ratio = staged-path ceiling ÷ direct-path ceiling
= min(link A, host RAM, link B) ÷ min(link A, motherboard forwarding, link B)
The closer it sits to 1, the less bandwidth tells you. On my box both sides cap at 2.0, so the ratio is about 1. Measured: 1.627 ÷ 1.68 = 0.968.
Two caveats I owe you:
- "The driver pipelines in chunks" is inferred from the numbers. If it moved one full hop before starting the next, staging should drop to around 1.0, and it didn't, so that's my read. I didn't read the driver source.
- 1.68 came from someone else's machine and motherboard, and my own 1.627 is a single run. I never measured run-to-run variance. Under those conditions, a 3% gap means nothing.
3. The opposite failure: 6.7 GB/s, zero bytes moved
On my machine, bandwidth was merely unable to tell the paths apart. Another community member hit the reverse, and it's much nastier.
He was testing a different unlock approach from an upstream branch. Everything looked great: the capability bit said yes, the topology tool said OK, 6.7 GB/s one-way, 13.4 bidirectional, 1.7 to 1.9 microseconds latency.
Then he checked the contents. The cross-card copy returned success and the destination memory hadn't changed. A multi-GPU collective sum hung on its very first 4-byte operation. He retracted the report.
The multiplication from section 2 should have flagged this immediately:
6.7 ÷ 2.0 = 3.35x
If measured bandwidth exceeds the link's physical ceiling, the data can't have gone over that link. That check is faster than any elaborate analysis.
That inference rests on one assumption, and I'll mark it: his cards also run at Gen2 x4. This generation's link speed is fused at Gen2 from the factory, and someone in the community ruled out a software fix with a controlled experiment. But I didn't separately confirm each slot's actual width on his machine.
One more thing worth knowing: the vendor's own peer-to-peer bandwidth and latency sample only times the transfer. It never checks the contents. A correct transfer and an empty one produce the same reading.
So in this business, bandwidth lies both ways. When the link is closed, it looks just like an open one (my machine). When the link is fake, it looks better than a real one (his).
4. What actually decides it: make the GPU reach into the other card's memory
The copy can take a detour and bandwidth can't discriminate. What's left is to remove the detour entirely.
Run a tiny kernel on card B that dereferences a pointer into card A's memory and reads one value. Then run a kernel on card A that writes one value into card B's memory. There's no staging fallback here. If P2P is off, there's exactly one outcome: illegal address. Open is open, closed is an error, nothing in between.
That alone isn't enough. A few details decide whether this is an honest ruler:
Use a different expected value for every directed pair. Three cards give 6 directions, 3 checks each, and every check uses a distinct value. The retracted report saw reads of a "local alias": it looked like the peer's memory, but it was a mapping on the card itself. With unique values, that kind of coincidence can't pass.
Zero the destination before copying. Otherwise last round's value can masquerade as this round's success.
Run a negative control first. Write a value into some memory and confirm that reading it back doesn't equal the sentinel. It's crude, and all it proves is that the comparison can't unconditionally say "equal." But probes that always report green have burned me more than once, so it stays. If the comparison can't pass that, the whole probe aborts without a verdict.
Run each check in its own subprocess. I learned this one the hard way. The illegal-address error is sticky at the device level: once you hit it, every later call in that process keeps failing. I first wrote a function to tear down and recreate the context as a reset, and context creation itself kept returning the same error. Run all 18 checks in one process and the first failure poisons the other 17. In a subprocess, the dirt dies with the process, and the parent only reads exit codes.
The final signature looked like this:
pairs=6 checks=18 failures=12
The 12 failures are exactly 6 reads plus 6 writes; all 6 copies passed. For a locked system, that's the right answer.
You can compute this signature in advance. For n cards:
directed pairs = n × (n − 1)
total checks = 3 × n × (n − 1)
failures when locked = 2 × n × (n − 1)
Three cards give 6, 18, 12. If the run doesn't match, suspect the probe before drawing conclusions.
The reading rules are simple. All pass: genuinely open. All fail (or only the copies pass): consistent with locked. Partial pass is the dangerous one: it means some pairs will silently move wrong data. Treat it as failure and list exactly which pairs.
5. What I haven't finished
- The probe has only run in the locked state. I never applied the patch: I worked out the numbers for my workload, and switching to a split that depends on P2P came out as a net loss, not worth the risk of a modified driver. So the "all pass means open" side is proven on someone else's machine; here I've only shown that "closed" gets reported correctly.
- 1.627 is a single run, with no variance data.
- The 6.7-exceeds-the-ceiling argument depends on his cards also being Gen2 x4.
- The chunk-pipelining claim is inferred from the numbers, not from reading driver source.
- My three cards each sit under a different upstream bridge, one of them under the chipset, and the kernel command line disables BAR reallocation. Whether any of that blocks P2P after unlocking, I haven't tested.
- The collective step, actually having the cards do one minimal sum together, I never ran. That's exactly where the retracted report died.
6. The order to do it in
- Compute the one-way link ceiling first: per-lane rate × lane count × encoding efficiency ÷ 8. If measured bandwidth exceeds it, the data didn't cross that link. Stop there.
- Draw the direct and staged paths and compute the discrimination ratio. If it's near 1, don't expect bandwidth to tell them apart.
- Treat driver capability bits and topology tables as hints only. The driver reports them, and they're exactly what an unlock changes.
- A successful copy call means the data arrived. It says nothing about the path.
- Decide with kernels that dereference the peer's pointer directly, a distinct value per directed pair, destination zeroed first.
- Run a negative control before anything else and abort if the ruler fails it.
- One subprocess per check, so one illegal address can't poison every later result.
- Compute the expected signature from n × (n − 1) before comparing against the run.
- Treat partial pass as failure and list the exact pairs.
- Finish with one minimal collective sum. If it completes, the link is usable.