AriakiA vulnerability-localization model found the right file in three consecutive runs. I switched the same repository to its patched version. It reported the same file three more times.
Awkward. The opening half makes a great local AI demo. Add the rest, and I cannot claim it knows when to stop accusing the code.
My decision: keep investigating it as a candidate-file ranker, but keep it away from the final clean-or-vulnerable decision. For a way to read benchmark claims, jump to section 2; the reproduction checklist is in section 4.
1. The file was right. Why am I still unconvinced?
These are historical measurements from 2026-07-22. I did not rerun the model for this article. The subject was Cisco Foundation AI's Antares-1B, running BF16 weights through MLX on an M1 Max with 64GB of memory. MLX is a machine-learning framework for Apple Silicon. The answers here came from the local model; a remote frontier model did not do the thinking for it.
The task came from the public VLoc Bench, which asks a model to locate vulnerable files inside a repository. I selected a single recursion-related case from express-xss-sanitizer. Recursion means a function keeps calling itself. Without a suitable boundary, deeply nested or cyclic input can exhaust the call stack.
The available tools were fairly ordinary: search the repository, read files, and submit a finding. I connected the model repository's own agent loop to the local backend and kept the official sampling temperature of 0.3. The model could inspect the code. It could not edit it.
On the vulnerable snapshot, all three runs named lib/sanitize.js. The explanation matched the target defect, too: traversal through objects and arrays lacked depth or cycle detection. So far, so good.
Then I tried the patched snapshot. Three runs. Same file again.
| Snapshot of the same case | Local result | What this side establishes |
|---|---|---|
| Version containing the target vulnerability | 3/3 located the target file | Localization worked on this case |
| Version with the target vulnerability patched | 3/3 still submitted that file | The control failed to exit cleanly on this case |
One run was especially awkward. It read MAX_DEPTH=100 and the depth counter, acknowledged that deep input would throw an error, then submitted the file as vulnerable anyway. The final action contradicted its own explanation.
That makes me suspect an "I was asked to find something, so I must return a file" bias. But I did not run a causal experiment on how training might produce that bias. The defensible observation is narrower: the final submission did not change with the evidence between the vulnerable and patched snapshots.
Two calculations are enough to describe this tiny sample:
- Target-file hit proportion = correct localizations / runs on the vulnerable snapshot = 3 / 3.
- Correct-exit proportion on the patched snapshot = correct reports of no target vulnerability / runs on the patched snapshot = 0 / 3.
Keep both denominators. Report only the first and the entire problem in the second half disappears from view.
2. Before reading File F1, ask what the task guarantees
The public report contains a number that is easy to use for the wrong purpose: a File F1 of 0.209. Under the benchmark definition checked at the time, that is file-level macro-F1 over 500 tasks guaranteed to contain a vulnerability. Macro averaging means scoring the tasks individually and then averaging those scores.
F1 combines how accurate the reported files are with how many target files the model recovers. Written out, F1 = 2 × P × R / (P + R). Here P is precision, the fraction of reported files that are correct, and R is recall, the fraction of target files recovered.
Do not read 0.209 directly as vulnerability recall. It cannot tell you whether the model will flag code whose target defect is already fixed. Tasks guaranteed to contain a vulnerability never ask that question.
My small paired control makes the missing question visible: 3/3 hits on the vulnerable snapshot, 0/3 correct exits on the patched snapshot. A good result at locating a file cannot answer the separate question of whether the model should report anything at all.
When someone demonstrates a vulnerability model, I now look for three things: a patched control, identical prompts and tools on both sides, and a final action that permits "nothing found." Without those, the demonstration's evidence stops at localization. Do not quietly extend it into a safety verdict.
Please do not make the reverse leap with my result, either. Three false reports here do not establish that the model fails everywhere. Repeated runs on the same case add observations of that case; they do not add coverage of repositories, languages, or vulnerability classes. Fixing the known target defect also does not prove that the whole repository is free of every possible security problem.
3. My speed measurements took a wrong turn, too
If you want to try the model locally, another tempting piece of evidence deserves a closer look: the speed table.
My initial cold smoke test generated only 21 tokens and recorded a decode rate of 12.39 tok/s. A token is roughly a fragment of generated text; decode throughput measures how quickly the model emits those fragments during generation. That number sent me in the wrong direction. I nearly blamed another model sitting in memory.
For the follow-up, I fixed the input at 1,932 tokens and the output at 256 tokens, warmed up the model, and ran each condition 7 times. The hashes of the generated text were identical across both conditions, meaning the output bytes matched. In one condition the small model had the machine to itself. In the other, a roughly 22GB model remained resident but idle, with no requests sent to it.
| Condition | Median decode throughput |
|---|---|
| Exclusive use | 76.280 tok/s |
| Another model resident and idle | 75.939 tok/s |
The measured gain works out to (76.280 − 75.939) / 75.939 ≈ 0.45%. That is a small difference in this test, giving me no good reason to blame the idle model for the speed problem. It says nothing about what would happen if both models were actively computing.
The earlier short, cold run included fixed first-execution overhead. Using it to predict warmed-up sustained generation would be badly misleading. Going the other way is misleading too: around 76 tok/s cannot tell you how long you will wait from starting the task to receiving a vulnerability judgment. Reading code, tool round trips, and a growing input transcript all take time.
In the clean-window rerun of the complete agent workflow, the vulnerable snapshot took 14.8–36.0 seconds and the patched snapshot took 16.3–24.9 seconds. Speed improved; the quality finding did not change. Both sides still submitted the same file in 3/3 runs. Getting a false alarm sooner did not remove the work of checking it.
4. Before installing it, make the test a pair
The narrow use I am willing to investigate is helping reduce the reading area when there is already an alert or a known vulnerability lead, with the final judgment verified separately. This experiment has not established that the model is more cost-effective than ordinary search or another ranking method. Even "candidate ranker" remains a proposed use worth testing, rather than a demonstrated win.
Here is what is still missing: this covers a single case in a single repository, with no paired evaluation across languages or vulnerability classes, no estimate of the overall false-positive rate, and no comparison of human review time saved. The speed A/B covers only an idle resident model. Do not use it to plan simultaneous inference workloads.
Sources: the model repository, benchmark repository, and technical report. Local numbers come from the dated reproduction notes. These links identify the model and tasks; they are not raw logs of my experiment.
For your reproduction:
- Obtain vulnerable and patched snapshots of the same case, and check which target defect the patch addresses.
- Fix the model, prompt, sampling parameters, and tool environment. Give both snapshots the same opportunity to be inspected.
- Preserve an explicit "no target vulnerability found" submission. Record the final tool action, rather than judging only how convincing the explanation sounds.
- Count target-file hits and correct exits on patched snapshots separately, then expand across repositories and vulnerability classes.
- Report cold-start behavior, warmed-up generation throughput, and complete workflow time separately. Include the time spent reviewing false alarms when deciding whether the tool helps.