AriakiI spent two days testing whether an LLM could take every message I send my AI assistant and file it under the right task.
It scored 90.9%. The simplest rule-based baseline scored 84.6%. I was already thinking about how to wire it in.
Then someone asked a simple question: if you reply inside a thread, aren't you obviously continuing that thread?
I went back and counted. Of my 30,355 test questions, 83.7% had the answer printed right there in the question.
Re-tested against real traffic, the ungated LLM scored 82.3%. A strategy with zero lines of code, "always continue the current task," scored 83.8%.
If you just want the yardsticks, read sections 3 and 5. If you want to know what else was quietly inflating my 90.9%, read section 6.
1. What I was trying to build
All my conversations with the assistant live in one chat app, one thread per piece of work. On a typical day I have a median of 7 things open at once, and on 44% of days it's 8 or more.
The trouble is I keep losing track of which thread I'm in. So I wanted a "secretary" layer: I say something anywhere, it figures out which piece of work the message belongs to, and forwards it to the right session.
The nice part is that this needs no labeling. Every time I opened a thread or typed into one, I was effectively labeling the data: this message belongs to this task. A few months of that is tens of thousands of free labels.
The eval was straightforward. Take each message, use the 12 most recently active tasks at that moment as candidates, ask the model to pick one, score it right or wrong.
Follow the most recent focus (rule) 84.6%
LLM judgment 90.9%
LLM + confidence gate 98.6%
The LLM beat the rule by over 6 points, and the gated version was close to perfect. It looked convincing.
2. One sentence that deleted 84% of the test
The sentence was: if you reply inside a thread, you're continuing that thread. Only a message posted directly to the channel might be starting something new.
That signal sits right in the data. Every message records whether it was posted in a thread or at channel level:
In-thread messages 31,699 99.7%
Channel-level 99 0.3%
More extreme than the remark suggested. Posting straight to the channel barely happens.
Re-slicing 31,148 human messages along that line:
Continue in the same thread 80.93% location already decides it, no judgment needed
Open a brand-new thread 2.77% that's "create", not routing
Switch back to an existing one 16.30% the only bucket with a real choice
My "pick 1 of 12" eval turned every single message into a multiple-choice question. But 80.93% of messages were just the next line in the same thread. Where they go is obvious, secretary or no secretary. The only place a secretary could actually save me effort is that 16.30%, the moments when I'd otherwise have to scroll back to another thread myself.
That 84.6% rule baseline, "follow the most recent focus," just sends each message wherever the previous one went. Almost all of its score is free. At an actual switch it scores zero by definition, because a switch means leaving the previous focus.
Which gives the first formula:
free score ≈ share of items that need no decision
ability actually tested ≈ total score − free score
It's a subtraction. In my 30,355 questions, 83.7% needed no decision. 84.6 − 83.7 = 0.9, so the rule baseline earned almost nothing on the questions that mattered, which matches "always wrong at a switch." The LLM's 6.3-point lead over the rule came entirely from the remaining 16.3% of questions, and that's before subtracting the leaks in section 6.
The question to ask before building any eval: if this step did nothing at all, would the system still be correct? If yes, it isn't a decision point. Testing it is handing yourself a prize.
3. Re-testing at real proportions
So I rebuilt the test: 1,199 messages sampled at real proportions, with continuations, switches and new tasks mixed in, and each approach run end to end.
Always continue current 83.8% zero model calls
Rule: if idle > 1h, switch back to the 84.1% zero
most recently used other task
LLM, no gate 82.3% 1 call per message
LLM + gate (only redirect at conf ≥0.8) 86.1% 1 call per message
LLM, two-stage 55.2% 2 calls per message
"Always continue" scores exactly the share of continuations in the test set, since it's 100% right on those and 100% wrong everywhere else. In this test slice continuations are 83.8%, a bit above the 80.93% across all data. More on that in section 7.
The ungated LLM lost to doing nothing. The per-bucket view shows why:
continue switch new
Always continue 100.0% 0.0% 0.0%
LLM + gate 98.9% 11.8% 75.0%
The ungated version dropped 6.8 points on continuations, down to 93.2%. It sees an ordinary "go ahead" and gets pulled toward some candidate task that looks related.
That leads to the second formula, which works out how many correct redirects it takes to pay for one wrong one:
net gain of automation ≈ share needing a decision × accuracy on that share
− share needing no decision × mis-redirect rate on that share
Plug in the numbers: continuations are 83.8%, the ungated mis-redirect rate is 6.8%, so that term alone costs 83.8 × 6.8% ≈ 5.7 points. It has to earn 5.7 points back on the other 16.2% of traffic just to break even. Its actual total is 82.3, which back-calculates to roughly 4.2 points earned on switches and new tasks, so it nets out 1.5 points behind.
With the gate, the mis-redirect rate on continuations falls to 1.1%, costing only 83.8 × 1.1% ≈ 0.9 points. That's what finally leaves a small positive margin.
There's one cost the formula leaves out: mis-redirects are silent. A message lands in the wrong task with no error and no warning. Weeks later I open that task, find a stray sentence that doesn't belong, and by then I have no memory of what I meant. Asking me one extra question costs close to nothing. That asymmetry makes the continuation mis-redirect rate even more expensive than its weight in the formula.
Does the gate's +2.25 points actually count?
Gated 86.1% vs. always-continue 83.8%. Their 95% intervals are [80.5, 89.9] and [76.8, 88.4], heavily overlapping. The usual reflex would be to call it "not significant."
But both approaches ran on the same questions, and only 63 of the 1,199 (5.3%) got different answers. The right comparison is per item: the interval on the paired difference is [+0.96, +4.58], which doesn't cross zero. It's significant.
The marginal intervals are wide because difficulty varies a lot between sessions, and both approaches rise and fall together. When comparing two approaches, look at the paired difference, not whether two intervals overlap.
Significant still doesn't mean worth it, though. +2.25 points costs one extra model call per message, roughly 0.9 seconds more of waiting. And it only manages 11.8% on switches, the one place a model is genuinely needed.
Why the two-stage version collapsed
The two-stage idea: first ask "is this still the same piece of work?", pass it straight through if yes, otherwise go choose. Sounds safer.
In practice that first binary question scored only 59.5%, while the single-shot judgment scored 91.9% on the same bucket. Once stage one gets it wrong, stage two can't recover. Splitting a task in two only pays off when each step is much easier than the original, and here step one turned out harder.
4. The 2.77% of new tasks: how often a day will it bug you?
Continuations and switches weren't worth it. That left one bucket that looked promising: new tasks. The gated version hit 75% there, from zero. It was the only big jump on the whole table.
If the secretary could spot "this is a new thing" and ask me "want me to open a thread for it?", that seemed genuinely useful.
New-task examples were scarce (just 24 in the test set), so I oversampled to 250 and tested separately. After oversampling, precision was 86.3%, which looks usable.
But that precision was computed on oversampled questions. On real traffic you need the third formula:
false interruptions per day = messages per day × (1 − base rate) × false-positive rate
true catches per day = messages per day × base rate × recall
My actual traffic: about 130 messages a day, roughly 3.5 of which are genuinely new work, a base rate of 2.72%. 97% of messages are "not new," so even a low false-positive rate multiplied by that base produces a pile of false alarms.
caught/day false asks/day caught÷false model calls
LLM only, threshold 0.85 2.8 27.2 0.10 130/day
LLM only, threshold 0.99 2.1 7.7 0.27 130/day
Rule + LLM, threshold 0.99 2.0 4.3 0.47 78/day
Do nothing 0 0 — 0
The best row asks me 4.3 times a day in order to save me 2 clicks. Its real-world precision is 2.0 ÷ (2.0 + 4.3) ≈ 32%, so it's wrong two times out of three. And "open a new thread" is one tap of my finger.
The "rule" in the third row is worth a word. The false-positive rate started at 209.8 per thousand. Reading the false positives, they were all like this:
"ok" "go ahead" "your call" "didn't get that"
There's no content in these at all. The model is forced to guess, and guesses "new task." Contentless short messages like these are 33.8% of everything I send. I added one deterministic rule: after stripping prefixes, if the message is very short or contains no content word of three or more characters, call it "continue" without asking the model. False positives fell from 209.8 to 32.9, a 6x drop, and model calls fell by about 40%.
Image messages can't be filtered that way. I was about to, then checked 31 image messages: 30 of them really were new work.
The rule rescued usability. It can't make the feature worth having.
5. When someone reports "routing accuracy X%", how to take it apart
These numbers are everywhere: intent classification accuracy, ticket triage accuracy, multi-agent task dispatch accuracy. I now ask four questions.
One: what does doing nothing score? Find the score of "always pick the most common answer" first. Here it's 80.93%. An 85% system may only be doing about 4 points of real work.
Two: does each test item actually require a choice in real use? Every data point can be dressed up with a candidate set. That doesn't mean the system ever has to choose among them. In my data, 99.7% of messages carry their own location. I invented the multiple-choice question myself.
Three: are the hints in your prompt correct on the slice being tested? This one bit me personally. When I re-tested the switch bucket, my prompt included "Note: the user's previous message went to [N]." Across all traffic that's the strongest clue there is, right more than 80% of the time. But a switch is defined as the user leaving N. On that slice, the hint points the wrong way 100% of the time.
Same 400 switch questions, prompt changes only:
With that hint 43.0%
Hint removed 46.5% +3.5
Changed to "they're leaving N" 55.2% +12.2
N removed from the candidates 64.5% +21.5
Rule baseline (zero calls) 50.2%
With that hint in place, I nearly wrote down "LLMs can't do routing, they lose to a rule." Remove it and the conclusion flips. This kind of mistake throws no error and crashes nothing. It just hands you a conclusion pointing the opposite way.
Four: what does the high score depend on? That 64.5% requires knowing in advance that the question is a switch, so you can drop N. In real use the secretary doesn't know that; it has to decide whether it's a switch first. So 64.5% is a score earned with information you won't have, and it only works as a ceiling. Whenever you report a subset score, say in the same sentence what prior information it needs and where that information would come from in real use.
6. What else was inflating my 90.9%
Section 2 was about the question itself being wrong. Beyond that, the old eval had several leaks. I had a second model do an adversarial review, then reproduced each finding myself:
- The "representative title" for each candidate task came from the future. Each task used the first session title in a list as its label, and that list was sorted by final message count. So questions included thread titles that hadn't been written yet: 15.20% of questions were affected, with titles showing up as much as 59 days early. The reviewing model estimated this leak at 1.34%. My measurement was 15.20%, 11 times larger. It only checked the samples that had been kept; I checked every candidate the prompt actually saw.
- The labels themselves were fabricated. To merge multiple threads about the same work, I clustered by title similarity. Chinese text was split into overlapping two-character chunks, so a generic title like "look into this" became 5 tokens, easily cleared the threshold, and 13 completely unrelated web pages were merged into one "task." A wrong merge manufactures correct answers out of nothing, so I switched to merging only on an identical link, or an identical title of at least 8 characters.
- Missing confidence was treated as full confidence. Some answers came back without a confidence value and the code defaulted it to 1.0, which inflated that 98.6% gated score.
- The gate threshold was tuned on the test set. Studying from the exam paper.
- The same approach was implemented differently in two scripts. One counted a failed call as wrong; the other silently fell back to "follow focus." The 91.1% and 90.9% I'd reported at different times weren't scores for the same method.
The sorting-key one is the most portable lesson: any code that "takes the first one" or "takes a representative" deserves a question: who decided this order, and when?
7. Where these conclusions are shaky
- One person, one assistant, one usage pattern. I habitually open one thread per piece of work, which is why 99.7% of messages carry their own location. Change the habit and the proportions change. The formulas in sections 2, 3 and 4 transfer directly; the numbers don't.
- The labels are my own behavior. Whichever thread I typed in counts as the right task, but I also post in the wrong thread sometimes. There's no way to subtract that from the data.
- Continuations are 83.8% of the test slice but 80.93% overall, almost 3 points apart. "Always continue" scores exactly that share, so those 3 points shift every comparison. I never pinned down whether it's a time-period effect or sampling.
- The 4.2 points in section 3 are back-calculated from the total score and the continuation drop. I didn't directly measure the ungated version on the switch and new buckets.
- The gate thresholds of 0.8 and 0.99 were chosen on a calibration period, then run once on the test period. A different time window might move the best threshold.
- The model was a local 27B. A stronger model might do far better than 11.8% on switches, and the section 3 math could change. But the mis-redirect formula holds for any model: as long as it still mis-redirects on 80% of traffic, it has to earn that back on the other 20% first.
- The new-task bucket is 250 items, oversampled. The base rate of 3.5 a day was counted from all data; the caught-per-day and false-asks-per-day figures are back-calculated from it. I never wired it in and ran it for a week.
8. Next time you evaluate a routing or triage feature, go in this order
- Check for location signals first. Where the message was sent from, which window was open, what it replied to. If that's in the data, measure how much it already decides.
- Split traffic into three buckets: location-decided, needs a choice, needs a new item. Write down each share.
- Only build test items from "needs a choice" and "needs a new item." Track a separate mis-redirect rate on the location-decided bucket and keep it out of the headline score.
- The first approach you run is always "do nothing." Its score is the location-decided share, and every other approach gets compared to it.
- Run competing approaches on the same items and look at the paired per-item difference, not whether two confidence intervals overlap.
- After oversampling a rare class, always back-calculate "false interruptions per day and catches per day" at the real base rate, and write both numbers side by side.
- Filter contentless inputs with a deterministic rule first, then hand the rest to the model.
- For every hint in your prompt, compute separately how often it's correct on the slice being tested.
- For any "take the first one" code, check when its sort key was produced.
My secretary's first version ended up as: continue the current task by default, never auto-redirect, never ask whether to open something new. Two days of evaluation bought a version that does nothing.