AriakiI fixed a bug. Same 24 shots. The number of shots where the character drifted went from 5 to 8.
Before the fix, I'd been loading the wrong checkpoint the whole time. It and the right one are both 20,970,379,616 bytes. Not a single byte apart.
The worse part was the round in between. I swapped in the correct checkpoint, reran, burned two hours, and got "no difference." That round could never have shown a difference. One multiplication before launch would have told me so. Section 2 is that multiplication, if you just want the tool; section 4 is how to use it on other people's A/B numbers.
1. How can two files be the same size?
Quick background. I was using a local video model on my own GPU to shoot a 24-shot short, and measuring one thing: does the same character, from shot 1 to shot 24, slowly drift away from who they're supposed to be, face and setting included? Each finished shot goes to a vision model for a binary call, drifted or not. Drifted shots divided by 24 is the drift rate. Lower is better.
The model ships two checkpoints (think: same architecture, two finished sets of weights trained for different jobs):
- First-last-frame checkpoint: you give it an opening frame and it continues from there
- Reference checkpoint: you give it a few reference photos of the character and it shoots that person
I was testing the reference path, so I should have been loading the reference checkpoint. Someone looked at my setup and asked: have you been treating these two as the same thing this whole time? What's the actual difference?
I checked, and my stomach dropped. The day I downloaded the reference checkpoint I'd actually noticed the two files were the same size. I thought "same spec, makes sense" and moved on. Now I laid the two files side by side:
| First-last-frame | Reference | |
|---|---|---|
| File size | 20,970,379,616 bytes | 20,970,379,616 bytes |
| Tensor count | 932 | 932 |
| Shape differences | 0 | 0 |
| Metadata | empty | empty |
| md5 | different | different |
The first four rows match. Only the last row tells them apart.
Then the part that really hurt: the node that loads the weights doesn't check which variant you gave it. Load the wrong one and it still loads, still renders, throws zero errors.
The actual bug was in my own batch code. When a job needed reference images, it switched the generation node into reference mode, but the weight-loading line never changed with it. The filename was hard-coded to the first-last-frame checkpoint. So every "reference experiment" that went through that queue had never once used the reference weights.
Five lines above that spot sat a comment I'd written myself, roughly: "sixth time today I've caught the same disease: a config silently ignored, no error." This was the seventh. Three lines below the comment.
The takeaway here is short: to decide whether two weight files are the same, size, tensor count, shapes and metadata don't count. Only a hash counts. Loading fine and rendering fine don't count either.
2. The first retest: two hours to measure nothing
I fixed the code and restarted the batch worker (that step almost wasted a run too: the process had started at noon and I edited the code that evening, so it was still running the old code from memory). Then I reran the best configuration so far with the correct reference checkpoint.
That configuration was "hybrid anchoring." Every 6 shots form an act. The first shot of each act is generated from the reference photos to lock the character's look. The other 5 shots in the act each start from the last frame of the previous shot to keep the motion continuous. On the wrong checkpoint it scored 7/24 = 29%, the best of five versions.
Before launch I wrote the criterion down: landing between 6 and 8 drifted shots (24%–34%) means "no material difference."
Result: 8/24 = 33%. Inside the band. The hypothesis "29% is a ceiling caused by the wrong checkpoint" was falsified.
Then I pulled the generation mode of all 24 records straight from the database, and saw the problem:
Shots 1, 7, 13, 19: reference mode ← only these 4 act openers
Other 20 shots: chain mode ← unaffected by which checkpoint
Swapping the checkpoint touched only 4/24 = 17% of the shots. Split out:
| Drift | |
|---|---|
| 4 act-opening shots in reference mode | 1/4 = 25% |
| 20 chain-mode shots | 7/20 = 35% |
Most of that 33% came from 20 shots the fix never touched.
That's the multiplication from the top. Whether a controlled experiment can detect anything is computable before you run it:
Expected shift (count) ≈ true effect of the variable × samples the variable actually touches
Detectable ⇔ expected shift > noise band
Before you have data you don't know the true effect, so use the ceiling: every touched sample flips. That's 4 shots, max. My noise band was ±2 shots. A ceiling of 4 only clears it in the extreme case where all 4 act openers flip the same direction. Normal outcomes can't reach it.
Plug in the real effect measured later in section 3 (12.5 percentage points): 12.5% × 4 ≈ 0.5 shots. Half a shot, against a ±2-shot noise band. This round was a guaranteed "no difference" from the moment I designed it.
One more, even shorter formula, for whether your samples are fine-grained enough:
Resolution = 1 ÷ sample count
With 24 shots, one shot is 4.2 percentage points. Looking only at the 4 act openers, one shot is 25 points. "Act openers: 1/4 = 25%" jumps to 0% or 50% if a single shot changes. Drawing any conclusion about the reference checkpoint from that is a coin flip.
Both formulas are grade-school arithmetic, and I did neither before launch. That's where two GPU-hours went.
3. Round two: push coverage to 100%, and the "wrong" one wins
Since the problem was touching only 17% of shots, make every shot go through reference mode. I set act length to 1 shot, so all 24 shots are "act openers." Coverage 100%. The two arms differ only in the checkpoint; everything else is identical.
Same drill before launch:
| Round 1 (hybrid) | Round 2 (all-reference) | |
|---|---|---|
| Coverage | 17% (4/24) | 100% (24/24) |
| Effect ceiling | 4 shots | 24 shots |
| Noise band | ±2 shots | ±2 shots |
| Verdict | basically undetectable | worth running |
Criterion pre-registered again: a gap of 3 or more shots counts as signal; 2 or fewer means "no material difference."
Result:
| Arm | Checkpoint | Drift |
|---|---|---|
| A | Reference (the "right" one) | 8/24 = 33% |
| B | First-last-frame (the "wrong" one) | 5/24 = 21% |
A 3-shot gap, 12.5 percentage points, sitting exactly on my pre-registered line. The drifted shots barely overlap between arms; only 2 are shared.
My first instinct was to suspect myself. I'd already caught seven silently-ignored configs that day, and the criterion itself said "anomalous result, check parameters first." So I compared all 48 records column by column: LoRA, step count, strength, resolution, frame count, sampler, scheduler, reference image set, each with exactly one value; the opening-frame field had 0 non-empty entries in both arms. The only thing that changed was the one weight file. I also read both arms' submissions back from the generation queue itself instead of trusting the batch script's own log.
That habit saved me once in this very run. Arm B's worker crashed midway, and when it auto-restarted it dropped the environment variable that picked the checkpoint. Its default happened to be the reference checkpoint. If I hadn't read the process's actual environment, arm B would have quietly become "reference vs reference," and I'd have spent time analyzing why the two arms looked similar.
So the bug was real and fixing it was correct. On this machine, for this short, fixing it made things worse. The five earlier "wrong checkpoint" runs weren't inflated; they'd been using the checkpoint that performs better here.
4. When someone posts an A/B result, ask these three first
I generated a number that deserved debunking myself: "29% is the ceiling of the anchoring strategy." It sounded well-supported. Three out of five versions got stuck at 29%. Here's what took it apart, and it works on anyone's "we changed X and got Y% better."
One: what fraction of samples did X actually touch?
A change that acts on 17% of samples can leave at most a 17%-sized mark on the total. If someone reports only the total and not the coverage, both "no difference" and "improved" deserve a question mark.
Two: run the same config twice. How far apart are they?
I tried this under the hybrid config: same config, same seed, rerun, and the output md5 still differed. That one time I nearly explained run-to-run jitter as "the new strategy already kicks in on the first segment," with a convincing story to go with it, while the source code plainly said the strategy only applies to continuation segments. An A/B without a measured noise floor can't tell you whether a 1-point gap or a 10-point gap is signal.
Three: do the two numbers differ in only that one variable?
I tripped on this in the same dataset. Aggregating two versions, I found a beautiful "layer-3 cliff": shots 0–2 positions from the act opener drifted 12%, shots 3+ drifted 62% (a 48-shot sample). I predicted that shrinking acts to 3 shots would push drift below 20%. Measured: 29%. Prediction failed. Distance-from-opener was collinear with scene type: shots near the opener happened to be mostly the stable "person standing behind a counter" kind, and the far ones happened to be mostly doorway and screen-close-up shots, which are hard.
Split by scene instead, and the ranking holds across three completely different anchoring strategies:
| Scene | Drift (three versions combined) |
|---|---|
| Counter (person in a fixed spot) | 9/42 = 21% |
| Doorway (person + door + rain + street) | 8/18 = 44% |
| Screen close-up (subject is a picture inside the picture) | 8/12 = 67% |
A 3x spread. Anchoring strategy wobbles between 29% and 46% no matter how I change it, and swapping checkpoints stayed inside 21%–33%. What decides drift is what you shoot; how you chain shots is the smaller lever. What saved me that time was having the prediction written to a file in advance, with "below 20% counts as success" fixed. Without that, 29% ties the best version, and I'd very likely have called it "a tie, so this works too."
5. Where this conclusion doesn't hold up yet
Before anyone quotes it, these need to be on the table:
- Only 4 steps. I used accelerated sampling, 4 steps per shot. The official docs are clear that 4 is the minimum, with 6–8 recommended. The reference checkpoint may well need more steps to shine. So the correct statement is: "for this short, this reference set, at 4 steps, the first-last-frame checkpoint drifted 3/24 less." It does not mean the first-last-frame checkpoint is better than the reference one.
- n = 1, and right on the line. The 3-shot gap equals my threshold exactly. I never measured the noise floor separately under the all-reference config; I measured it once under hybrid, and reruns of the same config do change.
- One metric. Drift rate only asks whether identity and setting stay consistent. It ignores image quality, motion range, and whether the story holds together.
- A signal that contradicts something I said earlier. I'd written that "reference mode doesn't consume the previous shot, so it shouldn't accumulate drift." Yet arm A drifted 2/12 = 17% in the first 12 shots and 6/12 = 50% in the last 12; arm B went the other way, 3/12 = 25% then 2/12 = 17%. Each half is only 12 shots, so one shot is 8.3 points. Logged, not concluded.
- Arm B took three attempts. The first hit an illegal GPU memory access on shot 4. The second reported a self-contradictory out-of-memory on shot 1 (asked for 125 MiB while reporting 61 GiB free), which turned out to be a dead driver. After a full reboot the third attempt finished 24/24. The failed attempts didn't leak into the results, but long high-load batches are themselves an unstable factor.
One meta-lesson that has nothing to do with the tech. The day before I found the wrong checkpoint, I'd made up a claim that "the reference mode needs a separate 144GB set of weights" and burned 4 GPU-hours on it. When I corrected that, I slid straight to the other extreme and said "the two modes share the same weights." The truth was in the middle: shared architecture and loader node, two different checkpoints. Overturning a wrong conclusion needs the same checking as writing it.
Next, in order of cost-effectiveness: first rerun 8 shots with the same seed under the all-reference config to measure the noise floor; it's the cheapest step and directly decides whether this 3-shot gap is trustworthy. Then run 6 and 8 steps to see whether the reference checkpoint overtakes.
6. Next time you run a controlled experiment, in this order
- Confirm what's loaded is what you think is loaded. Weights, config, environment variables: read them all back from downstream. Hash against the file, queue against the submission, process environment against the variable. Logs are the program's own story; they don't count.
- After editing code, confirm the running process started after the edit. If the process start time is earlier than the file's modification time, it's running old code.
- Count how many samples the variable touches. Use "expected shift ≈ effect × touched samples" to get a ceiling. If even the ceiling can't clear the noise band, don't run it; redesign to raise coverage first.
- Use "resolution = 1 ÷ sample count" to check whether your samples are fine enough. A subgroup where one sample is 25 points shouldn't carry a conclusion.
- Measure the noise floor first. Run the same config twice and see how far apart they land. That number is your noise band.
- Write the criterion and the prediction to a file before launch. Thresholds, enumerated outcomes, stopping rule. No edits after the run.
- Anomalous result? Compare parameters column by column first. Only trust it once you've confirmed a single variable differs.
- Ship the scope with the conclusion. Which samples, which parameters, how many runs. Missing any of those, don't quote it bare.
I used to think fixing a bug and getting negative returns only happened in other people's code.