Raising the power cap from 200 W to 250 W bought 1.4%. Undervolting at the same 200 W bought 11.4%. Which knob should you turn on a power-capped GPU?

I raised a card's power limit from 200 W to 250 W. A full-load matrix-multiply benchmark went from 162.5 TFLOPS to 164.83. Fifty extra watts, 1.4% faster.

Then I put the limit back at 200 W, changed a single clock offset, and the same benchmark read 181.0. Power draw was 199.3 W, basically unchanged. That's 11.4% faster.

Then I hooked it back into a long-running simulation job I'd had going for weeks, computed the end-to-end gain, and reported +28%. But staring at the throughput chart, I couldn't see anything close to a fifth. I recomputed. The real number was +7.2%. The missing 20 points came from a timezone.

This post is about what happened between those three numbers. If you just want the rules of thumb, sections 2 and 4 each boil down to one division. If you want to see how two different tools fooled me, jump to section 5.

1. First, figure out which wall you're hitting

The card is a mining-edition variant of a datacenter GPU. It's passively cooled, and I built my own air duct for it (an earlier post covers how that duct took it from 92 °C to 64 °C). With the duct done, it sits at 81–83 °C at 200 W.

When a GPU drops clocks, it's usually one of two things: it's too hot, or it's out of power budget. A lot of people see low clocks and reach for more fans, or reflexively crank the power limit up. Those two moves treat two different diseases.

NVIDIA cards will tell you which one you have:

nvidia-smi -q -d PERFORMANCE

What I saw that day:

SW Power Cap        : Active
SW Thermal Slowdown : Not Active

The power cap was active, thermal throttling wasn't. A 10-second sample showed draw between 195 and 207 W, pinned to 200 every single second. In plain terms: the card wasn't hot, it was hungry. It runs as fast as the watts allow, and no faster.

The embarrassing part: for several rounds I'd been interpreting this card through conclusions from the old duct, back when heat really was the limit. Three cooling revisions later, the thing holding it back had changed, and I was still treating a fever.

2. Why raising the power cap only bought 1.4%

If the card is hungry, feeding it should make it work harder. Measured:

Power limit	Actual average draw	bf16 compute
200 W	199.3 W	162.5 TF
250 W	203.9 W	164.83 TF

The limit went up by 50 W. The card drew 4.6 W more. Peaks occasionally touched 256 W, but the average just wouldn't move.

Before deciding whether this is worth it, compute a utilization ratio:

power-limit utilization = actual average draw ÷ power limit setting
203.9 ÷ 250 = 81.6%

Well below 100% means that on its current voltage-frequency curve, the card can't actually use the extra budget. The other 46 W is a check nobody cashes. For any given clock the card asks for a fixed voltage. If the curve doesn't change, the clock doesn't go up, no matter how many watts you allow.

I later ran an alternating A/B of 250 W vs 200 W on my real workload: 250 W was 2–3% faster, same direction in all 8 comparisons, but it drew 15% more power, ran 3 °C hotter, and the share of time spent throttled went from 0 to 64–100%. The 250 W setting varied by 2.4% between its own two runs, so the "2–3% faster" magnitude isn't precise. Trust the direction, don't copy the number.

The conclusion is boring: the power limit stays at 200.

3. Undervolting: more clock for the same watts

There's an open-source project dedicated to tuning this class of card. Its approach is to apply an offset to the whole voltage-frequency curve. Bluntly, it tells the card "at the same voltage, you may run X MHz faster than the factory curve says." Factory curves usually leave a conservative voltage margin, and the offset eats into it. In the overclocking world this is usually called a curve undervolt.

On a power-capped card the effect is especially direct: each clock needs less voltage, so each watt buys more clock, and the power cap is still the same cap.

Power limit at 200 W, all services stopped so the benchmark had the card to itself, 15 seconds per step:

Offset	bf16 compute	Power	Core clock	vs stock	Max temp
+0	162.5 TF	199.2 W	1215	—	81-82
+100	174.3 TF	199.0 W	1303	+7.3%	77
+150	178.1 TF	199.0 W	1333	+9.6%	79
+200	181.0 TF	199.3 W	1355	+11.4%	80
+250	184.3 TF	198.8 W	1380	+13.4%	81
+300	188.9 TF	197.7 W	1412	+16.2%	81

Same 200 W, +300 gives 16.2% more, and runs 1 °C cooler. Clock climbs from 1215 to 1412 while power doesn't budge.

Put sections 2 and 3 side by side: on the same card, adding 50 W bought 1.4%, and undervolting in place bought 11.4%. On a card where the power cap is the active limit, the knob to turn is the voltage curve.

3.1 Why I shipped +200 and not +300

The +350 step crashed outright with Xid 13. An Xid is an error code the NVIDIA driver writes to the kernel log; 13 is a graphics engine exception. The process dies, the card survives. So this card's wall is at +350.

Here's the math I used to pick a point:

safety margin (steps) = (crash step − chosen step) ÷ step size
+300: (350 − 300) ÷ 50 = 1 step
+250: (350 − 250) ÷ 50 = 2 steps
+200: (350 − 200) ÷ 50 = 3 steps

+300 gives 4.8 more points, with one step of margin left. The author of that open-source project is explicit about his own reference card: +300 passed four full-VRAM sweeps, served a full day of benchmarks, then faulted during an hour-long soak. Three of his profiles ended the same way, and he ended up shipping +200 himself.

With undervolting, the scariest failure is the one that doesn't crash: no error, just an occasional wrong number. For a simulation that runs for weeks, quietly wrong is much more expensive than crashed. Trading 2 points for a whole extra step of margin seemed worth it to me.

With +200 applied, I ran the real workload for 225 minutes. The Xid count in the kernel log never moved, power bounced between 179 and 235 W (the cap is 200; instantaneous samples overshoot), the core sat at 82–83 °C, and the limiting factor was the power cap the whole time.

4. The microbenchmark said 11.4%. End to end it was 7.2%

This is where I crashed and burned.

4.1 That +28%

I sliced two time windows out of the request logs: one before the offset took effect, one after, and compared tokens generated per second. I got +28% and was pretty pleased.

Then I stared at the chart for a while, couldn't see anything like 20%, and started to get nervous.

It was the timezone. The offset went live at 20:32 local time. Daylight saving was in effect, so local time was UTC−4, which makes the switch 00:32 UTC. The log database stores UTC, but my filter said 02:33 UTC, a full two hours late. The "after" window only caught the last 13 minutes, and those 13 minutes happened to coincide with a batch of heavy requests: about 1169 tokens per request on average during that stretch, versus roughly 350 normally. That's what inflated the number.

This trap was already in my notes: the database is UTC, system logs and file timestamps are local time. I had even said "pin down the timeline first" at the start of that same round. Having it written down doesn't mean I follow it. I still don't have a good fix; the best I've done is make every comparison script echo the converted start and end times so I look at them once.

4.2 The correct numbers

Same method, 15-minute buckets, this card only:

	Buckets	Requests per bucket	Generated tokens/s	Avg input length	Avg output length
Stock	15	374.2	143.9	27277	355
+200	9	383.0	154.2	28131	369

end-to-end gain = 154.2 ÷ 143.9 − 1 = 7.2%

The workloads in the two windows match: inputs 3.1% longer, outputs 3.9% longer, both within noise.

Why couldn't you see it on the chart? Per-bucket throughput bounces between 103 and 203. The natural bucket-to-bucket swing is an order of magnitude bigger than the 10.3 tok/s gain. You will never eyeball a 7% change on a curve like that; you have to aggregate. So "doesn't look faster" and "actually 7.2% faster" were both true. The only wrong number was my 28.

4.3 Estimating end-to-end from a microbenchmark

pass-through ratio = end-to-end gain ÷ microbenchmark gain
7.2 ÷ 11.4 ≈ 0.63

The microbenchmark is pure matrix multiply at full load, so it scales with clock. Real inference also involves attention, kernel launches, scheduling, networking and queueing, and not all of that scales with core clock. So only about 60% of the gain makes it through, which seems reasonable to me.

You can use this number in reverse: next time someone says "undervolting/overclocking made my card 15% faster," ask which measurement. If it's a matmul microbenchmark, plug in a pass-through ratio of around 0.6 and you're looking at roughly 9% end to end. My 0.63 is one point from one card and one workload. A more compute-bound workload will pass through more; one dominated by memory traffic or scheduling will pass through less.

5. Two tools, each lied to me once

5.1 The qualification tool swallowed a crash

The open-source project ships a "qualify" command: it climbs offsets from low to high, sweeps at each step, backs off one full step to produce a recommended value, and writes a "qualified" receipt. I ran it. The terminal output was spotless: recommended +250, four full-VRAM sweeps passed, tens of thousands of matrix multiplies with zero errors.

Then I checked the kernel log and found an Xid 13 from the climb at +350, with the tool's own test binary as the process name.

The cause is how the script is written: the output of the ladder step is captured by the outer script with $(...), and only the last line, "last good step," surfaces. Which step crashed and how is all swallowed. In the terminal you just see it calmly moving on to "confirming the ship point."

So after running any tool like this, ignore its stdout and check separately:

journalctl -k | grep -i xid

Two more side effects it doesn't mention. The qualify command hard-codes the power limit to 250 W, and the ladder subcommand sets power again on its own, so when it finishes the card is left with a 300 W limit and a +250 offset. It doesn't restore anything. There's also a boot-check service that gets enabled on install, which also hard-codes 250 W and has no ordering constraint relative to my own boot service that sets 200 W, so whichever one wins on a given boot is luck. I disabled it.

5.2 My monitoring script counted zero for 225 minutes

For the long soak I wrote a monitor that logged new Xids, VRAM temperature and the current offset every minute, and would automatically roll back to stock if anything looked wrong.

The first version came up with all three fields broken:

• Xid=0: it ran as a systemd service, and in that environment dmesg can't read the kernel buffer, so it silently returned nothing and the count was 0. The real baseline was 42.
• hbm=C: I grepped for the wrong field name for memory temperature, so it captured an empty value followed by the unit.
• off= empty: there's a Success token in front of the number in the tool's output, and I parsed the wrong position.

The first one is the dangerous one. "Zero new Xids" and "not counting at all" look exactly the same. If the other two fields hadn't been obviously empty, I'd have happily watched it count zeros all afternoon. After fixing it, I added a self-check line at the start of the script that prints the current total Xid count. Seeing 42 there is how I know the check is alive.

6. Debunking a cross-card comparison

The author of that open-source project has a card of the same model. His stock result is 184.3 TF. Mine is 162.5. At a glance mine looks like 88% of his, and it's easy to conclude "I got a bad chip."

162.5 ÷ 184.3 = 0.88

But his card doesn't have a 200 W cap. On my side, a +250 offset gives exactly 184.3, identical to his stock number. The whole gap is the operating point. The silicon is fine.

So whenever you see "same card, mine is X% slower than someone else's," line up three things first: same power limit? Same clock offset? Full-load benchmark or end-to-end? If any of the three is different, that X% is unreadable. A related trap is "my card is temperature-limited, yours is power-limited," where the same setting can have opposite effects on the two cards.

7. What isn't done, stated up front

• One card, one workload. The 63% pass-through ratio will move with a different model or concurrency level.
• 225 minutes is not long-term stability. The plan was 240 minutes; I stopped 15 minutes early because the machine had to be moved. The reference card's history says some configurations only fail after an hour or more. I don't have days or weeks of data. And the "quietly computes a wrong value" failure mode is invisible to an Xid counter; I currently have no way to verify it isn't happening.
• The end-to-end samples are lopsided. 15 buckets for stock, only 9 for +200, with a stretch of downtime for baseline testing in between. Trust 7.2% for direction and magnitude; don't read much into the decimal.
• Two experiments, both about 11% more clock, very different results. At 250 W the core clock rose 11.3% and long-prompt prefill rose 2.4%; with the undervolt the clock rose 11.5% and end-to-end rose 7.2%. The measurements differ (median prefill speed per request vs total throughput per 15-minute bucket), and I never ran a like-for-like comparison, so I'm not drawing a conclusion here.
• The qualification receipt doesn't match the operating point. It was produced at 250 W and I'm running at 200 W. Voltage margin shifts with the operating point, so strictly speaking I should re-qualify at 200 W. I haven't.
• The machine was later moved to another room with a different ambient temperature, and I haven't re-measured the cooling, so "81–83 °C at 200 W" only holds for before the move.

8. Do it in this order next time

1. Run nvidia-smi -q -d PERFORMANCE and see whether SW Power Cap or SW Thermal Slowdown is Active. If temperature is the limit, fix cooling first. Only continue if power is the limit.
2. Before raising the power limit, run a full load under the new limit and compute actual average draw ÷ limit. If that's well below 100%, don't bother. Raising it just adds heat.
3. On a power-capped card, try a clock offset. Keep the power limit fixed, step up gradually, and record compute and power at every step.
4. Once you find the crash step, back off and compute margin with (crash step − chosen step) ÷ step size. Leave at least two steps. For long-running jobs, leave three.
5. After any tuning or qualification tool, run journalctl -k | grep -i xid yourself and ignore its terminal output. Also check whether it changed your power limit or offset and left them that way.
6. In a soak-test monitor, print the absolute value of the current counter as the first line. Only trust the string of zeros after you've seen a plausible non-zero baseline.
7. When computing end-to-end gains, echo your window start and end times converted into the log database's timezone before slicing. Then form an expectation with microbenchmark gain × pass-through ratio. If the measured result is way above that, suspect the window before you celebrate.