AriakiBack in August I wrote a tidy little line in my notes: the company's own team, using its own model, made a five-minute short from 195 generations and kept 46 shots. Hit rate 23%. Roughly one keeper per four rolls.
This week I sampled the film at 4 frames per second, 1,231 frames in total, and traced each frame back to the generation it came from. Same public data. The answer came out to 1 in 1.45.
That is almost a 3x gap, and both mistakes were mine: one in the denominator, one in the numerator. So this post is about a narrow question: when someone (including you) quotes "AI video takes N tries per usable shot," how do you check whether the number holds up?
If you just want the checks, read sections 4 and 7. If you want to see where I went wrong, sections 2 and 3.
1. The material: a short film plus its entire generation log
In August an AI video company released a 5 min 8 s sci-fi short and said every prompt and asset was now public. The web UI didn't actually let you open anything, but the project page fetches an unauthenticated JSON endpoint that returns everything: the prompt for every generation, reference images, duration, resolution, model name, creation time, and a direct link to each output video.
I pulled 195 generation records back then (the platform calls them job sets; one submission is one record). Median prompt length was 6,914 characters, and the longest ran past 20,000. These people do not write short prompts.
This week I downloaded every output video as well: 197 files, 2.4 GB, none of them broken. I also grabbed the 1080p version of the finished film, 307.8 seconds long.
So I had the finished product and, in theory, all the raw material it was cut from.
2. Mistake one: the denominator included footage generated after the film shipped
The 195 records live in two folders: 105 in the main folder and 90 in a subfolder called "regenerations." In August I just added them together.
One look at the timestamps this week and it fell apart:
| Records | Created (UTC) | Model | Resolution | |
|---|---|---|---|---|
| Main folder | 105 records, 107 outputs | Jun 26 – Jun 29 | v2.0 | 4K ×98, 1080p ×7 |
| regenerations | 90 records | Aug 11, 16:16 – 16:50 | newer | all 720p |
The film's publication record was created at 12:00 UTC on Aug 11, and the launch tweet went out at 13:50 the same day (tweet IDs encode their timestamp; shift right by 22 bits and you can read it off).
The 90 regenerations were batch-run in a 34-minute window four hours after the film went public. 89 of the 90 reuse a main-folder prompt word for word, and one metadata field is literally named something like "new model comparison." In plain terms: they re-ran the old prompts on the new model as a side-by-side. None of it had anything to do with making the film.
The timestamps are direct evidence on their own. Footage generated after release cannot be in the release. I still checked with frame matching (method in section 4): at a 0.85 similarity threshold, zero regeneration segments show up in the film.
Check one: before counting how much footage went into something, drop every record created after the thing was published. There's nothing clever about this step. I just skipped it.
That takes the denominator from 195 down to 107.
3. Mistake two: the numerator counted shots, the denominator counted generations
The "46 shots" from August came from running a shot-detection tool over the finished film. The problem is that one generation is not one shot.
I grepped the 105 main prompts for HARD CUT: 26 of them contain it, 95 times in total. Things like:
1.6s HARD CUT 1.6s–3.4s ... 3.4s HARD CUT 3.4s–5.8s
That is: lay out four or five shots in text inside a single 10-second generation and have the model render them in one go. I sampled 6 of these outputs and ran shot detection (scene threshold 0.3). Prompts with 3 to 7 hard cuts produced 2 to 4 detected cuts. I also sampled 5 prompts that explicitly said "single continuous shot, no cuts" and found 0 cuts in each. The model mostly does what it's told.
So one shot in the film may be a small slice of one generation, and one generation can feed several shots. Dividing generations by shots compares two different units.
Worse, the 46 itself isn't stable. I reran ffmpeg's scene detection on the film at several thresholds (cuts closer than 0.5 s merged into one):
| Scene-change threshold | 0.25 | 0.3 | 0.4 | 0.5 |
|---|---|---|---|---|
| Shots detected | 115 | 93 | 45 | 23 |
| Median shot length (s) | 2.0 | 2.33 | 4.08 | 6.71 |
The same film has 23 shots or 115, depending on where you set the knob. My August 46 happens to sit near 0.4, and I wrote it down as fact. I also couldn't reproduce the other line from that note, "median shot 3.9 s, 97% under 5.2 s," because I never recorded which tool or settings produced it. That claim is retracted.
Check two: when you see "N shots," ask what tool cut them and at what threshold. A shot count without a threshold can be off by 5x.
4. A different ruler: trace each frame to where it came from
If counting shots is unreliable, stop counting them. Ask instead which generation each frame of the film came from.
The method is crude:
- Sample both the film and all 197 outputs at 4 fps, shrink each frame to a 64×27 grayscale thumbnail, subtract the mean, normalize
- Correlate every film frame against every output frame and keep the best match
- If the correlation is above 0.85, attribute that film frame to that generation
- A generation only counts as "in the film" if it matches at least 2 consecutive frames (0.5 s)
After steps 1–4, 787 of the 1,231 film frames match something in the main folder, forming 95 segments. Of the 107 outputs, 74 appear in the film.
So:
Rolls per keeper (lower bound) = generations made before release ÷ generations that appear in the release
= 107 ÷ 74 ≈ 1.45
My August method = all records ÷ detected shots = 195 ÷ 46 ≈ 4.24
The two mistakes separate cleanly. Fix only the denominator and 195 ÷ 46 becomes 107 ÷ 46 ≈ 2.33. Fix the numerator too and you get 107 ÷ 74 ≈ 1.45. The 88-record drop in the denominator works out like this: remove the 90 regenerations, and the 105 main records turn out to hold 107 outputs.
I picked 0.85 myself, so I swept it too (without the 0.5 s continuity filter here):
| Correlation threshold | 0.8 | 0.85 | 0.9 | 0.95 |
|---|---|---|---|---|
| Film frames matched | 69.9% | 63.9% | 51.3% | 29.2% |
| Distinct generations seen | 95 (incl. 2 regenerations) | 88 | 70 | 44 |
At 0.8, regenerations start sneaking in, and the timestamps say they can't be in the film. That means 0.8 is already letting false positives through. At 0.95 the match is so strict that a bit of color grading or reframing knocks frames out. 0.85 is the point I could find that admits nothing impossible without dropping too much. Even at the strictest 0.95, the ratio is 107 ÷ 44 ≈ 2.4, still nowhere near 4.
5. The expensive part is seconds
1.45 rolls per keeper sounds like AI video is cheap. Look at seconds before you get excited.
Of the 74 generations that made the cut, each contributes only a small part of its runtime: the median fraction used is 19.4%. A quarter of the generations that made it in had less than 8.8% of their footage used.
Seconds utilization = seconds of the film traced to a generation ÷ total seconds generated before release
= 188.5 ÷ 1048 ≈ 18%
Flip it around: every 1 second on screen cost roughly 5.6 seconds of generated footage. The traceable segments in the film have a median length of just 1.5 s, 94.7% are 5 s or shorter, and the longest is 8 s.
This also explains why the per-roll ratio is so low. They keep almost every generation, but only the best second or two of each. Those second-by-second action timelines in the prompts are probably there to make that best second or two land somewhere predictable.
If you're budgeting an AI short, counting rolls will badly undercount. Counting seconds is what matches the bill. Most platforms charge by generated seconds, so multiply your target runtime by 5 to 6 to get the seconds you'll be paying for. That multiplier comes from this one film and will move with genre and team, but it's a far more useful order of magnitude than "1 in 4."
6. What this method misses
Until these are fixed, treat every number above as a lower bound:
- 36% of film frames have no source. 444 of 1,231 frames don't match anything. The two longest gaps both fall between 192 s and 225 s, 28 seconds combined. I tried a 70% center crop and a horizontal flip as rescues; they recovered only 31 and 22 more frames, so most of it isn't simple reframing. It could be heavy post work, or generations that aren't in this public folder.
- The public folder is called "snapshot." The name suggests a curated copy, and failed generations may have been deleted. So 107 is a lower bound on pre-release generations, and 1.45 is a lower bound on rolls per keeper. The true value can only be higher, and the public data can't tell me by how much.
- Correlation is a blunt ruler. A 64×27 grayscale thumbnail can't tell apart two similarly composed shots. For the groups where one prompt was run several times (93 distinct prompts across 107 outputs), I can't tell which run the film used.
- 4 fps misses flash cuts shorter than half a second, and my 0.5 s continuity rule filters them out on top of that.
- The August shot stats had no recorded tool or settings. I couldn't reproduce 3.9 s or 97%, and both are retracted.
7. Next time you see "AI video takes N tries," ask in this order
- Was every record in the denominator created before the product shipped? Drop the ones that weren't. Post-release comparison reruns, reshoots, and promo material often sit in the same folder as the real thing.
- Is the numerator counting shots or generations? If prompts script multiple shots, one generation can produce several, and the units need to match.
- What tool and threshold produced the shot count? Without a threshold, the same film can have anywhere from 23 to 115 shots.
- If you have the raw outputs, trace frames back directly. Small grayscale thumbnails and correlation are enough. Sweep the threshold and watch for where time-impossible footage starts matching.
- Finish by converting to seconds. Final runtime ÷ generated runtime is the ratio you actually pay for. For this film it's 18%.
That "1 in 4" line came out of my keyboard very smoothly in August. The first step to debunking it was a single glance at a timestamp, and I put that off for two and a half months.