AriakiPointing directly got 186 right, parse-then-pick got 188, the hybrid got 190: which way should a local model click the screen?
I had a 35B local model click buttons in a window with no accessibility tree. Each route has a trade-off: asking for coordinates directly is over 6x faster but slightly less accurate, while parsing first is a bit more accurate but costs about 3 seconds per step. The hybrid router got 190 of 200 real clicks right, but those 2 extra wins prove nothing statistically. The more useful finding was that "both routes agree" is far less reassuring than it looks, because the two routes fail together. Three calculations you can run yourself: whether two routes fail independently, the average latency of a hybrid router, and how much a zero-error run actually tells you.
LLM · Agent · 评测11 min read