Three GPUs, all stuck at 33% utilization: where's the bottleneck? I ran three rounds of elimination and two and a half of my conclusions fell over

Three 64GB GPUs, a 320B-parameter MoE model, 24 tokens per second. Open nvidia-smi and all three cards sit at 33% GPU utilization, in lockstep.

That night I went through three rounds. First I ruled out "the three cards are taking turns." Then I ruled out "memory bandwidth is saturated." Then I delivered the verdict: the bottleneck is the narrow link between the cards. The evidence was a beautiful flatline: 49.14 tok/s at 4 concurrent streams, 49.00 at 8. Not a hair of growth. I wrote that conclusion into my notes, into the comments of the diagnostic script, and used it to explain a second weird result while I was at it.

Rereading the raw data today, two and a half of those three conclusions fall over. The flatline was drawn by the server quietly defaulting to 4 slots. And the numbers I used to "rule out turn-taking" are the most direct evidence that the cards were taking turns.

If you just want the rules of thumb, read sections 2 and 4, each one is a division. If you want to watch me get fooled by my own flatline, that's section 4.

1. The setup

The hardware is three mining-edition cards built on the same die as the A100, unlocked to 64GB each with a community tool. The only path between them is PCIe Gen2 x4. I measured cross-card copies at 1.627 GB/s, which is pitifully narrow.

The model is GLM-5.3-Flash, a mixture-of-experts model (the network is split into many "experts" and each token only wakes up a few of them). It has 320B total parameters but activates only 18B per token. I ran a 3.01-bit GGUF quant, 112.1 GiB of weights, split by layer across the three cards in llama.cpp at a 33/33/34 ratio.

A single stream generated 24.39 tok/s, which works out to 39.2 ms per token.

And then the picture that makes your fingers itch: all three cards hovering around 33%. The instinct fires immediately. There's 3x headroom sitting there unused. Find the thing that's choking it and you get 3x the speed.

2. Mistake one: "they rise and fall together, so they're not taking turns"

Layer split sounds like a pipeline by name. Card one runs the first third of the layers, hands the intermediate result to card two, which hands it to card three. For one token of one request, only one card is computing at any instant. That should have been the first suspect.

My rebuttal at the time: the three cards are at 30-something percent simultaneously. What I didn't see was one card at 95% with the other two idle. Turn-taking should look like a seesaw.

That rebuttal collapses on one assumption: the sampling window is far longer than the rotation period.

utilization.gpu is defined as "the fraction of the sampling window during which some kernel was running." The samples I saved were one per second. One token takes 39.2 ms, and each card gets roughly 13 ms of that per rotation. In one second each card takes its turn about 25 times, busy a third of the time on each turn. Averaged out, that's 33%, and of course all three cards "rise and fall together."

The number I should have looked at was already sitting in my own notes: what the three cards add up to.

Single-sequence layer split:  Σ per-card utilization ≈ 100%
Each card ≈ 100% × its share of the layers

It's a division. At any instant only one card is busy, so the busy time gets divided among the cards.

Check it against the data. Across those 18 samples, the three-card sum ranged from 96% to 102%, pinned to 100% at nearly every point. The per-card averages were 33.2%, 34.6% and 31.3%, matching the 33/33/34 split.

If the cards were genuinely running in parallel, the sum would head toward 300%. Pinned at 100% has exactly one explanation: at any moment, exactly one card is working. I copied sum=97% and sum=106% line by line into my notes and then concluded the opposite. A little embarrassing.

So "33%" by itself carries zero information about headroom. Three cards, layer split, one stream: it's supposed to read 33%. With four cards it would read 25%.

3. Mistake two: bandwidth, which I got wrong twice

The second suspect was memory bandwidth. My first pass went like this: each token reads 112 GiB, divide by three cards, that's close to the limit.

The error was using total parameters as the per-token read. An MoE only wakes 8 of its 288 experts per token, so what actually gets read is the 18B of active parameters.

Per-token read (bytes) ≈ active params × bits per param
                       ≈ 18B × 3.01 bit ≈ 6.31 GiB (6.78 GB)

112.1 versus 6.31 is an 18x gap. I fixed that one the same night, and I've covered it in another post on this site about quant selection: total parameters decide whether it fits, active parameters decide how fast it runs.

What I missed that night was the second layer of the error. I went on to write "aggregate 165 GB/s, 55 GB/s per card, only 3.5% of the theoretical 1555." Dividing by three assumes all three cards are reading at once. Section 2 already showed they take turns: each card reads its third of the weights during its third of the time, and while it's reading, it's reading at the full 165 GB/s.

The cleaner way is to compute the ceiling directly and ignore how many cards it's split across:

Single-stream ceiling ≈ per-card memory bandwidth ÷ bytes read per token
                      = 1555 GB/s ÷ 6.78 GB ≈ 229 tok/s
Measured 24.39 → about 10.6% of the ceiling

A serial split doesn't change the ceiling: each card reads its slice at full speed, and the three slices add up to the time it takes to read the whole set once. The direction of the conclusion is the same as that night, bandwidth is nowhere near saturated, but the 3.5% figure is wrong. It's closer to 10.6%.

4. Mistake three: the flatline

With the first two suspects eliminated, the remaining conclusion felt inevitable. Here's that night's concurrency sweep:

Streams   per-stream tok/s   aggregate tok/s   wall time
   1          25.35              22.93            8.4s
   2          20.76              37.48           10.2s
   4          13.37              49.14           15.6s
   8          13.16              49.00           31.3s   ← aggregate capped, per-stream stopped falling

My reading: aggregate throughput is capped, compute is at 33%, memory is at 6–7%, neither is full, so some fixed-capacity resource must be full, and by elimination that leaves the cross-card link. Later, speculative decoding made this model drop from 24.39 to 13.78 tok/s (with a 66–83% acceptance rate, no less), and I pinned that on the link too: speculation verifies more tokens, so it makes more trips over the slow link.

Rereading it today, the thing I should have spotted first is the wall time in the last row: 31.3 seconds, exactly double the 15.6 seconds in the 4-stream row.

Then look at the write times of those eight response files. Four finished in the same second. The other four finished in the same second 15 seconds later. Two batches, four each.

In the llama.cpp server source, the slot-count flag -np defaults to auto when you don't pass it, and auto means 4 slots. My launch script didn't pass it.

So at 8 concurrent requests the server only served 4 at a time, and the other 4 waited at the door until the first 4 finished. Of course per-stream speed "stopped falling": at every instant only 4 streams were running. Of course aggregate "capped": it was 4 streams' worth of work divided by twice the time. That's a multiplication:

When request count C exceeds slot count S:
wall time ≈ ⌈C ÷ S⌉ × time for one batch of S
          = ⌈8 ÷ 4⌉ × 15.6s = 31.2s      measured 31.3s

It lines up exactly. That flatline measured the slot count. It has nothing to do with the link.

So is the link the bottleneck or not? You can estimate it. During generation with a layer split, each hop carries one token's intermediate vector, which is "hidden dimension × bytes per number." I didn't look up this model's exact dimension, so let's overestimate: 16,384 dimensions in fp32 is still only 64 KB:

Per-hop time ≈ 64 KB ÷ 1.627 GB/s ≈ 40 µs
Two hops     ≈ 80 µs, 0.2% of 39.2 ms

Even with a generous overestimate, link time barely registers in single-stream generation. My most confident conclusion from that night now has no direct evidence holding it up. The speculative-decoding explanation loses its footing too, since verifying a few more tokens still moves data in the KB range.

The real curve is: 22.93 at 1 stream, 37.48 at 2, 49.14 at 4. Four streams is 2.14x a single stream and still climbing. Whether it caps after 4, this data can't say.

I've written a post on this site about using a completion-time table to tell "real parallelism" from "a very tidy queue." I fell into the same hole here. The only difference is that the queue came from too few slots instead of the engine refusing to parallelize.

5. Using these three on other people's numbers

These are the three claims I see most often in multi-GPU inference threads.

"GPU utilization is only 30%, there's huge room to optimize." First ask how it's split. With layer split and a single stream, utilization should be 100% divided by the card count, which tells you nothing. Then ask what the cards sum to: pinned near 100% means they're taking turns; clearly above 100% means several cards really are working at once. Utilization measures "time during which a kernel is running." A kernel that uses 1% of the compute units for a full second still reports 100%.

"Throughput caps at N concurrent requests, so the hardware is maxed." First find the server's concurrency limit: -np in llama.cpp, max_num_seqs in vLLM. A flatline that doesn't state that number shouldn't be trusted by default. The fastest tell is whether the wall time at the capped step exactly doubles. If it doubles, it's queueing.

"Faster PCIe will speed up generation by X." A few days ago I watched a YouTube video where the creator packed 32 of these same cards into one machine and projected generation speeds linearly across PCIe Gen1 through Gen4, with no method shown. By the division in section 4, a layer split only moves KB-scale data per hop during generation. Link generation mostly affects weight loading and prefill on long prompts. Extrapolating generation speed linearly from it has no basis. The creator is also sitting on 32 cards and the video description is full of affiliate links, which belongs in the calculation as well.

6. What this post didn't finish

• I never reran with -np 8. That's the single most important missing step, and it would directly answer whether throughput caps after 4 streams. The 112 GiB of weights have since been deleted, and in quality evaluation 97% of this model's answers degenerated into an endless string of question marks partway through, so it isn't worth re-downloading. Section 4 can therefore knock down the old conclusion but can't offer a new ceiling.
• From 2 to 4 streams, per-token time jumps from about 48 ms to 73–78 ms, and I can't explain it. One possibility is that different requests in an MoE wake different experts, so batching doesn't save on reading expert weights. That's a guess. I haven't tested it.
• The 64 KB is a deliberately generous upper bound, not a measurement. I didn't look up the model's hidden dimension, and I never captured live PCIe traffic (the metric set I had nvidia-smi dmon collect doesn't include PCIe rx/tx; you have to add -s t, and I didn't).
• Why speculative decoding made it 44% slower is now a blank. The old explanation is withdrawn and there's no new one. That speculative implementation was on an unmerged PR branch at the time, so a problem in the implementation itself can't be ruled out either.
• Sampling was once per second, 18 points total. The section 2 conclusion only needs "the window is much longer than the rotation period," and these samples are enough for that. Seeing the actual shape of the rotation would need a higher-frequency tool.
• The single stream only reaches 10.6% of the ceiling, and this post doesn't explain where the other 89% goes. It might be the efficiency of 3-bit dequant kernels on this GPU generation. Untested.

7. Next time multi-GPU utilization looks low, check in this order

1. Ask how it's split. With layer split and one stream, utilization ≈ 100% ÷ card count. That's the normal value. Don't rush to optimize.
2. Add up per-card utilization. ≈100% means turn-taking; clearly above 100% means parallelism.
3. Compare the sampling interval with the per-token time. If the interval is much longer, "rising and falling together" proves nothing.
4. Compute the bandwidth ceiling: per-card bandwidth ÷ (active params × bits per param). For MoE use active parameters, not total, and don't divide by the card count again.
5. Before running a concurrency sweep, write down the server's slot count, and make your highest step at least double it.
6. When you see a cap, check whether the wall time at that step exactly doubled, then check whether responses finished in batches.
7. Before blaming the link, compute bytes per hop ÷ link bandwidth and compare it with per-token time. If it's under 1%, go find another suspect.