AriakiIn 2023 a New York attorney with thirty years of practice filed a brief written by ChatGPT. It cited six cases. All six were fabricated. He did have doubts — he asked the model repeatedly whether the cases were real, and every time it told him they were. The judge fined him five thousand dollars, and the opinion was not gentle about it.
That same year, countless people who cannot program at all used the same class of tool to build small working utilities they now use every day.
Both sides did pure wishing: say a sentence, get a thing, never look inside. So where's the difference?
One division
Your ceiling for safely using AI equals your acceptance bandwidth.
acceptance bandwidth = oracle strength ÷ cost per check
An oracle is the thing that decides right from wrong: do you have some method, independent of the AI, for judging whether what it just handed you actually holds up? Cost per check is what running that method costs you in time, money, and expertise.
The attorney's oracle strength was actually enormous — a case either exists or it doesn't, and there's nothing to argue about. He crashed on the cost term. He believed the cost of checking was zero (ask the model), when the real cost was a thirty-second database lookup. He didn't know he needed to look it up, so he filled in the denominator with zero, computed an infinite bandwidth, and drove straight into a wall.
The small-utility builders are the mirror image. Their oracle looks weak — it's just "does it run, is the output right" — but the cost of checking really is near zero. They are the perfect oracle themselves. Open it, look, done. The bandwidth holds up fine.
So these two groups were never competing on the same track.
The domain ladder: how fast AI penetrates a field has nothing to do with how hard the field is
Push that division one step further and you get something I find more useful than the formula itself:
How fast AI advances in a field is roughly independent of the field's intellectual difficulty, and proportional to how cheap its oracle is.
Ranked from cheap to expensive:
Go — the win condition is the oracle. Cost is essentially zero, and one machine can play thousands of games against itself per second. So this was the first field taken decisively, ten years ago.
Code — compilers, type checkers, unit tests, all free oracles the machine can run itself. Which is why this is where the frenzy is right now. Watch the causality here. Code got eaten deeply by AI because "is this code correct" has an automatic referee that costs seven cents. Ease of writing had nothing to do with it.
Formal mathematics — proof checkers like Lean are absolutely reliable oracles. Machine-verified, immune to persuasion. It's no accident that OpenAI's math results came out along this axis. You can have a model emit a hundred candidate proofs, run the checker over each one, and kill the bad ones on the spot. The hundred candidates were never the valuable part. The checker was.
Medicine, law, investing, corporate strategy — the oracle is expensive, and it arrives late. A wrong diagnosis, a contract with a trap in it, a losing strategy: feedback comes back in weeks to years. AI accidents in these fields share one signature — at the moment of failure there is no sound at all. A compiler goes red immediately. A judge goes red in six months.
Taste — the oracle is free, but it cannot be outsourced. You are the only referee, and nobody can stand in for you. So AI helps enormously here and will never replace the person doing the nodding.
This ladder explains something a lot of people find backwards: AI races ahead in "hard" domains like proof search and flops in "easy" ones like writing a graceful business email. Math has a checker. Graceful doesn't.
The number most worth citing, and what it's actually saying
METR ran a randomized controlled trial in 2025: 16 experienced open-source developers, 246 real issues, randomly assigned to "AI allowed" or "AI forbidden," working in repositories they already knew well.
The AI group was 19% slower.
That number has been reshared endlessly, and most of the resharing stops at "see, AI is useless." I think the real information sits in two other numbers:
- Before the experiment, these developers expected AI to make them 24% faster
- After the experiment, having personally lived through the slowdown, they still believed they had been 20% faster
Forty-plus points wrong going in, forty points wrong coming out. They finished the work and still didn't know they'd been slowed down.
Why is it specifically experienced developers in their own repositories who got slower? My reading: the part that got slower is precisely the acceptance. These people hold absurdly high standards inside a codebase they've maintained for years — style, edge cases, consistency with existing abstractions, whether this plants a landmine for future maintainers. The generation half really is free and instant, but afterward they have to read every line and fix things in several places, and the reading plus fixing outran just writing it themselves.
Put differently, this experiment measured something other than whether AI works. It measured what happens when acceptance cost gets high enough to eat the entire gain from free generation.
A note on how to take apart numbers like this one: METR listed the boundary conditions themselves. Developers had only tens of hours of Cursor experience and may not have cleared the learning curve; the sample consists of people willing to join a study, which is a selection effect; the design measures one person on one issue, and says nothing about workflows that sample thousands of trajectories. All of that is in the original write-up. The posts citing this number as ironclad proof that AI is useless mention none of it.
Beginners really do win, but the attribution is wrong
There's a popular claim: people who understand nothing about technology do better, because they don't know what "impossible" means and therefore make outrageous requests.
I think that attribution is wrong.
The advantage of "unconstrained by conventional wisdom" has already been neutralized in the AI era, because AI is itself the most complete carrier of conventional knowledge there is. Ask it whether something can be done and it will hand you the industry's standard boundaries verbatim. The beginner gets the same answer the expert gets. Worse: when the model says "that can't be done," the expert knows it might just be the model being conservative, while the beginner has no choice but to believe it.
The expert's curse got inherited by AI, and the beginner didn't get the antidote.
But beginners genuinely do win on a huge class of tasks, and the real reason is different: the acceptance threshold on those tasks happens to be zero. A personal utility, a one-off script, reshaping a pile of Excel — the oracle is "I open it and look and it's right." On these, the person who wanted the thing is the perfect oracle, checking costs nothing, bandwidth is unbounded.
Experts actively lose money on this class of task. They can't help thinking about architecture, extensibility, what happens when a feature gets added in three months, and turn a five-minute job into half a day. That isn't stupidity. Their muscle memory comes from high-acceptance-threshold environments and it's firing in the wrong place.
The key point is that this class of task is enormous. The overwhelming majority of "I want a thing" in the world lives here. So "non-technical people are building things with AI" will keep happening at scale. It's a real phenomenon. The cause is just not the one everybody names.
Where the oracle breaks
Where's the line? Watch for when these questions start showing up:
- Is this still correct when a thousand people use it at once
- Is this numerical precision good enough
- Could this code delete the database some day
- Does this conclusion still hold on a different batch of data
Once acceptance requires answering questions like these, "open it and look" stops working. The beginner's state past this line isn't "slower" —
they don't know they're already wrong, and they will go on not knowing until some very expensive moment.
That's a categorically different thing from being slow. When you're slow, you know you're slow. Past the oracle break, your self-assessment feels identical to before, maybe better, because AI never hesitates. The METR numbers already told us this: even experienced engineers perceived "19% slower" as "20% faster." The instrument doing the perceiving is itself broken.
Three times I fell over this line myself
I do prediction-market quant work as a hobby — AI generates strategies, I run backtests, it writes the analysis. The defining feature of this setting is that the oracle is both expensive and delayed: a fake strategy takes months to travel from "the backtest looks beautiful" to "real money is gone." All three of the following are real. On the first two, if my oracle hadn't been hard enough, money would have left.
Fall one: every model stalled at 51%. I had AI run 6 models × 3 sequence architectures to predict 5-minute direction. Every one landed at 51–52% accuracy. Not one higher, not one lower. My instinct at the time was to reach for a bigger model.
But that shape is itself the answer. If model capacity were the bottleneck, accuracy should climb monotonically with capacity. Eighteen combinations lined up flat means the ceiling lives in the data, not the model. The thing to change was the information source.
That judgment came purely from domain intuition. Someone with only programming ability keeps swapping models here, and collects a little hit of accomplishment each time from "I tried another architecture."
Fall two: a strategy with apparently positive EV turned out to be the market's own pricing wearing a costume. The high-confidence buckets AI screened out showed expected return of +0.008 to +0.013 per trade on paper. Looked ready for real money.
I ran two oracles. Bootstrap resampling by day put the 95% confidence interval across zero. Then I checked it against the market's own odds: 99% overlap. Meaning this "strategy" was predicting something the market had already priced, and I'd used a complicated model to restate public information and then pay fees for the privilege.
Twice. Two different model architectures. Identical disease.
Fall three: a fee formula off by 3.3× nearly killed a real strategy. This one ran the other direction. My original fee formula skipped a share-conversion step and underestimated cost by 3.3× in one price band. After correction, the same batch of 5,447 trades went from +1.36 cents expected return per trade down to +0.56, and the 5% lower bound on the daily bootstrap fell from +0.57 to -0.21 — across zero. The verdict flipped from "there's an edge" to "no edge demonstrated once real costs are charged."
The reverse also happened: a different strategy that a bad approximation had scored as unprofitable came back to life under the correct formula. A wrong accounting convention doesn't only put you on the wrong train. It also makes you shoot the right thing yourself.
And one trap I only found afterward, which no amount of programming skill would ever surface: that platform settles against an on-chain oracle, not the exchange whose data I trained on. The two sources agree on direction only 96.6% of the time — roughly eleven hundred of thirty-odd thousand samples carry a flipped label. However well the model learns, it's learning the wrong thing.
This trap has nothing to do with code quality, model selection, or engineering ability. It has to do with whether I knew to go ask what the settlement source was.
The flaws in this experiment, stated plainly: everything above comes from historical replay on a single market over a single time window, with no cross-market replication. The bootstrap aggregates by day, and counts independent samples by market rather than by trade, which is a convention you can legitimately attack. And strictly speaking, the "flat line means the bottleneck is in the data" inference in fall one only rules out insufficient model capacity. It doesn't rule out my feature engineering having failed across the board. I lean toward the former. I haven't proven the latter dead.
Four questions to ask before accepting anything from AI
- What's the oracle this time? Name the specific thing that decides right from wrong. If you can't name it, what you're doing is gambling, not using a tool.
- What does one run of the oracle cost? Thirty seconds or three months. If the answer is "I asked the AI and it said it was right," your oracle is zero.
- How long until feedback arrives? The compiler goes red instantly, the judge goes red in six months, a fake strategy might go red a year later. The longer the delay, the more the oracle has to exist before you start. Retrofitting it afterward is too late.
- Which side of my oracle break does this task sit on? Inside it, turn AI loose — the speed is pure profit. Outside it, go acquire the oracle first, or get a different person to do the checking.
If you want to raise your own ceiling in the AI era, the direction isn't learning more generation tricks. It's getting some domain's oracle into your own hands — the knowledge of what conditions make a given conclusion fail. The generation side is already approaching free. The money and the risk have piled up downstream, on the act of nodding.
The attorney didn't need a better prompt. He needed somebody to tell him that citation checking has an oracle you can run in thirty seconds, and that the oracle does not live inside ChatGPT.
References: METR, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (2025-07); Mata v. Avianca, Inc., 678 F. Supp. 3d 443 (S.D.N.Y. 2023).