AriakiSame set of characters, same decision point, two context strategies.
The full one is 5241 tokens: all 240 events from six terms, verbatim. The compressed one is 2270: current term in full, one engine-written summary for the middle stretch, and only three categories kept verbatim from the distant past. After 216 real model calls, the compressed arm wins 49 to 38 on the only metric I care about — did an old memory actually change what the character decided to do.
43% of the size, 29% more hits. I was about to write that into the design doc.
Good thing I didn't. The compressed arm had one extra thing the full arm didn't: it pulled due promises out of the event stream and into a standalone table. So was that the compression, or was that the table? I added a third arm that differs by exactly that one feature. Answer: of the 15-point gap, the table is worth 5 and the compression is worth 10. I'd been about to credit all 15 to compression.
Want the criterion straight away, it's in §1 and it's a division. Want to see how the first three versions of my judge fooled me, that's §3.
1. One division: net hits per thousand tokens
The setting first. This is a prototype for an AI upbringing simulation — six kids, each with a profile and a core motivation, each forming three "here's what I want to do this term" intents on their own. The question is whether compressing away their distant past changes how they behave.
The method is planting. Each kid gets three events planted in terms one and two: a wound (being dismissed in public), a promise (agreeing to do something for someone), and an identity moment (the first time someone noticed a strength). Then three terms later, at term seven, I ask them what they want to do. A second model judges, intent by intent, whether that intent was shaped by the planted event.
Raw hit counts will lie to you, because you can always buy more hits with more context, at the cost of money and latency. So the number to look at is this:
memory efficiency = (planted hits − unplanted-control hits) ÷ (prompt tokens ÷ 1000)
No heuristics involved. It's a division. Three arms, 6 kids × 6 seeds = 36 intents each:
| Arm | prompt tokens (median) | wound | promise | identity | net hits | per 1K tokens | fabricated citations |
|---|---|---|---|---|---|---|---|
| Full | 5241 | 6 | 11 | 23 | 38 | 7.25 | 73 |
| Full + promise table | 5354 | 5 | 16 | 17 | 37 | 6.91 | 42 |
| Layered compression | 2270 | 5 | 26 | 18 | 49 | 21.59 | 13 |
(The three middle columns are hits out of 36. Net hits already has the unplanted control subtracted — 2 for the full arm, 0 for layered.)
Almost 3x. And look at the last column: in the full arm the model invented 73 event IDs that don't exist, most often by dropping a letter — writing ev-5-25 where the real ID is ev-S5-25. The more you give it, the more it makes up. That column comes back in §4.
2. The control arm I nearly skipped
The middle row of that table was added after the fact.
Originally there were two arms, full and layered, and the gap was a clean 38 to 49. But layered differs from full in two ways, not one: it compresses history, and it lifts due promises out of the stream into a table. The promise column is also where the gap is widest — 11 versus 26, bigger than the other two columns combined.
Which invites an obvious suspicion: maybe the model just finally saw the table, and compression has nothing to do with it.
The fix is equally obvious. Add an arm that differs by exactly one thing: full event stream as before, plus the same promise table. It scored 16.
Three numbers in a line:
promise hits: 11 (full) → 16 (full + table) → 26 (layered)
credit to the table = 16 − 11 = 5
credit to the compression = 26 − 16 = 10
Both factors are real, roughly 1:2. But without that middle arm I'd have assigned all 15 to compression and written down a conclusion nobody could reproduce.
Generalisable version: for every factor you suspect, add an arm that differs by that one factor; the contribution is the difference between adjacent arms. Three arms cost 50% more than two, and two arms give you a number you cannot take apart.
That middle arm falsified something else along the way. It has the most tokens (5354), the lowest net hits (37), and the worst efficiency (6.91). Adding a table does not compensate for a diluted context — in a stream that's already long, your shiny new table gets partly drowned too.
3. Four versions of the judge; the first three lied
Every number above comes from round five. I don't trust a single number from rounds one through four, because the judge itself was broken. This is the part I most want to keep.
Round one: count citations. Full was 4501 tokens with the planted event cited once; layered was 2071 tokens with four citations. I was pleased. Looked like the result was already in.
Round two: add an unplanted twin for every arm. Identical setup, except the three planted events are replaced with ordinary filler, and report hits minus baseline. That's when it fell apart. One kid's profile already said "prove I'm not just the girl who draws" — her identity hits were 3/6 with the plant and 5/6 without it. Negative after subtraction. So round one's "4 versus 1" was mostly the character sheet talking, not memory.
The same round caught the keyword criterion tuning its own result: I added two words ("mediate", "conflict") to one kid's expected wound shape and the full arm's hits went from 2 to 4. If two extra words in the rubric move the result, it isn't a rubric.
Round three: swap in a judge model. Have a second model rule, intent by intent, on whether this intent was shaped by that event. I calibrated on hand-written examples first — 12 out of 20, all passed, looked fine. Then it hit real data and ten baselines blew past the line, including one kid whose wound scored 6/6 in the unplanted control. Pure false positives. Reading them back, the judge was treating "same topic" as "influenced": the event says her legs went soft during a run, the intent says "prove I'm not just the girl who draws", verdict true.
The fix was to take those five real misjudgements produced by the system under test and put them into the calibration set as hard negatives, then rewrite the judge prompt so the only acceptable standard is "names something specific that only this event could have supplied". Calibration went to 19/20. The one it still misses is a paraphrased positive — the judge errs strict, and since hit and base use the same judge, strictness is symmetric and doesn't bias the difference.
Round four added a gate. Run the unplanted baseline first; if any planted event scores above 2/6 in any strategy, abort before the real run. It immediately caught one: I'd written a kid's wound as "the board he built got snapped in half" while his profile motivation is "why does a circuit board make that noise" — the wording collided, and the unplanted baseline was 5/6. Reworded to "told in front of the class that he doesn't fit in", baseline 0/6.
Four rounds in which the thing that kept changing was the ruler. The system under test never changed once. Hand-written calibration sets are always too clean — a judge that goes 12/20 on your examples will still call "same topic" an influence on real data.
4. How to pick apart someone else's memory benchmark
Three questions, now with teeth:
One: is there an unplanted control arm? Without it, nobody knows how much of that hit rate is the character sheet, common sense, or "they'd have chosen that anyway". I measured a negative once. Only the control arm made it visible.
Two: does the criterion measure citations or behaviour? They're far apart. In round three the full arm cited one identity event 18 times and the judge ruled only 10 of those actually influenced the intent; the layered arm cited it 6 times and the judge ruled 6. Citation counts can be padded; changed behaviour can't. Any memory evaluation that only reports recall or "number of items retrieved" is inflated by construction.
Three: does the judge's calibration set contain real misjudgements from the system under test? If not, it isn't calibrated.
Free bonus signal: check whether they report fabricated IDs at all. 73, 42, 13 across my three arms — same model, longer context, more invention. An evaluation that doesn't report that number probably never looked.
5. The column I can't take apart, and other defects
None of this is settled until the following are fixed.
The wound column is low across all three arms (6/5/5), and I can't tell you why. Dismissed in public three years ago, apologised for two years ago, not mentioned in this term's intents — that could be compression losing it, or it could be a normal person letting it go. Absolute hit counts can't separate those. The criterion has to become counterfactual: run the same plant in an apologised and a not-apologised version and look at the difference, not the level. Not done yet.
There are only three intent slots, and promises crowd out identity. Layered: promise 26, identity 18. Full: promise 11, identity 23. The total still favours layered, but the identity column can't be read on its own — it's losing a slot to promises. Which also means the totals in that table carry a ceiling effect.
The noise band is wider than I'd like. With 6 kids × 6 seeds I don't take anything within ±2/36 seriously. By that standard the wound column's 6/5/5 is entirely noise; only the promise column (11 vs 26) is separated cleanly. The 7.25-vs-21.59 headline holds because promises carry it. If you only care about traumatic memory, this dataset can't answer you.
The event stream is synthetic. Forty events per kid per term across six terms, all generated except the plants. Round five does read them out of the real memory store (written term by term, terms advanced properly, promises going through real due-date logic), but the event content is still invented, and a real playthrough won't produce this distribution.
One model, one judge, and they're relatives. Characters run on a local 35B open-weights MoE; the judge is a 27B dense model from the same family. Whether the conclusion survives a model swap is untested.
"Weakly coupled" isn't a rule yet. I require plants to point in the same direction as the character's motivation without being derivable from it — otherwise you either collide (baseline explodes) or the model ignores it entirely. One kid's plant pointed the right way and still scored near zero: her three intent slots were fully occupied by family matters. Plant design is still a craft here, with no reproducible rule behind it.
6. Next time, in this order
- Build the unplanted twin arms first and run the baseline. If any plant baselines above 2/6, stop and fix the plant or the judge before running anything real.
- Judge behaviour, not citations. Keep citation counts as a side signal, never as the conclusion.
- Put real misjudgements from the system under test into the judge's calibration set. Hand-written negatives are too clean.
- For every factor you suspect, add an arm differing by that one factor. Subtract adjacent arms to get its contribution.
- Add a "net hits per 1K tokens" column. Don't report raw hits.
- Write down the column you can't take apart, and say why you can't.
By the way, the full-context arm invented 73 event IDs that never existed. Bad memory, excellent imagination.