A 15% error rate, but absent from 58% of my searches: how many engines is my self-hosted search actually using?

Over three days my AI agent ran 1173 web searches. It asks for 10 results each time. On 1171 of them it got a full 10.

Over the same window, the SearXNG stats page put the Google engine's error rate at 15%.

Both numbers look fine. But the logs hold a third number: how many raw results the backend handed back before my agent trimmed them. That number was exactly 25 on 685 calls. That's 58%. With all three engines answering, it should sit around 45.

So on more than half of my searches, Google never showed up, and neither the upstream metric nor the downstream one told me.

This is the second time I've tripped over the same self-hosted search box. The first time, in August, was worse: I fixed a search server, announced it was fixed, and then found out the server I'd fixed wasn't on the call path at all. It's the same bug in both cases, and section 3 covers it. If you just want the diagnostic method, read section 2 and the checklist at the end.

1. Where the 25 comes from

Quick context. SearXNG is an open-source metasearch engine: you give it a query, it fans out to Google, Bing, Yandex and friends at the same time, merges and dedupes what comes back, and hands you the combined list. My agent's web search runs on it.

Each response is JSON with a results array. My agent sorts that by score, keeps the top 10, and writes one log line:

SearXNG search '...': 10 results (from 25 raw, limit 10)

That from 25 raw is the count before truncation.

I built a histogram of the raw counts over those three days. The top entries:

raw	count
25	685
24	68
22	68
45	42
44	25
35	25

One value taking 58% of all calls is suspicious. After a merge and dedup, counts should scatter. They shouldn't line up this neatly.

So I probed the live instance ten times and checked which engines contributed. The answer was very regular: google cse gave 20, yandex gave 15, bing gave 7 to 10. That adds up to 42 to 45.

Now the numbers reconcile:

full-load raw ≈ sum of one page from each engine that answered
              = 20 (google cse) + 15 (yandex) + 10 (bing) = 45
without google cse: 15 + 10 = 25

It's an addition, nothing more. That big spike at 25 in the histogram is the fingerprint of one specific state: google cse is missing and only yandex and bing are answering.

It's more useful run the other way. If you know how many results each engine returns per page, the histogram of raw counts tells you which engine went missing and when. You don't need SearXNG's admin side. The caller's own logs are enough. Your instance's per-engine page sizes may differ from mine; one probe will tell you.

2. 15% or 58%: which one is real?

The SearXNG container logs spell out the cause:

google cse: Our systems have detected unusual traffic from your network. (suspended_time=180)

Google decided my traffic looked odd and sent back a 429. When SearXNG gets that, it suspends the engine for 180 seconds, and for those three minutes no query goes to it at all.

So where does the 15% come from? My reading: the error rate only counts requests the engine actually sent. The request that tripped the suspension counts as one error. For the next 180 seconds it sends nothing, so it can't rack up any more errors. Time spent suspended doesn't show up in the error rate at all.

Those two numbers are measuring different things:

error rate   = failed requests ÷ requests it actually sent              → 15%
absence rate ≈ calls whose raw count dropped to the "minus that engine" level
               ÷ total calls = 685 ÷ 1173 ≈ 58%

The second one is what I care about. My question is "was it there when I searched?" I don't really care what fraction of its own requests succeeded.

Splitting by load makes it clearer. I bucketed calls by hour and called any hour with 50 or more calls "busy":

	calls	raw=25	share
busy hours (≥50/hour)	973	651	67%
quiet hours	200	34	17%

In busy hours Google was missing two times out of three. In quiet hours it was missing about one time in six. Busy hours are mostly me running several agents doing research at once. The harder I search, the more Google gets annoyed and the more often the engine gets benched. And busy hours are exactly when I most need good results.

Then there's the sneakiest layer. The downstream "10 results" hides all of it. Twenty-five raw results is still plenty to fill 10 slots, so the agent gets a full set every time and from where it sits, search looks healthy. I haven't measured how much the top 10 degrades without Google. More on that in section 4.

There's a general rule in here that applies to the health numbers of any aggregating system:

An error rate tells you whether things worked when they were tried. It won't tell you whether they got tried when they should have been. Any system with circuit breakers, suspensions, or fallbacks needs a separate absence number, and ideally you count it from the caller's side.

3. August: a green instance, and I fixed the wrong box

Back to August, because I'd already seen this bug once.

At the time, I thought the search backend "kept going down." First I checked the instance: the container had been up for four weeks with no restarts, it had never been OOM-killed, it returned HTTP 200, and responses took 2.35 seconds. All green.

The real problem was in the engine layer. Here's what /stats/errors showed:

engine	error rate	type
mojeek	95%	403
duckduckgo	75%	CAPTCHA
brave	70%	429
baidu	15%	CAPTCHA
startpage	5%	CAPTCHA
google	0%	—

Google: 0% errors and always zero results. After that I wrote down a rule for reading these:

• The engine appears in the response's unresponsive_engines list → it's been blocked by anti-bot measures
• It's missing from that list, the error rate is 0%, and it still returns zero results every time → the parser is out of date

The second case is the nastiest. You get HTTP 200, no exceptions, nothing in the logs, and it just can't parse anything. That time, the root cause was that my image was two months behind upstream. The search engine had changed its page layout and the old parser couldn't read it anymore. I started a throwaway container on the new version and ran the same query against both: google went from 0 results to 44. That settled it on the spot.

Two smaller findings from the same session:

• A residential proxy did nothing for the blocked engines. brave, mojeek, and startpage returned exactly the same responses direct and through the proxy. They're fingerprinting the whole connection (TLS, HTTP/2, behavior), so a different exit IP won't get you past them.
• Response time went from 2.35s to 0.90s, and most of that came from disabling the blocked engines. A blocked engine waits out the full timeout every time before giving up, and the merged response has to wait for the slowest engine.

After the fix, the same 10 test queries went from a median of 36 results to 44, with 0 of 10 coming back empty. I happily reported it fixed.

Then I got one follow-up question: "Is it actually fixed?" I had the agent run its search tool for real. It returned {"web": []}. Empty.

It turned out this machine had two SearXNG instances: one on the NAS and one in a local container. The agent reads its URL from .env, which pointed at the local one. I had checked the systemd unit, the drop-in override, and the process environment. All three said NAS, so I assumed that was the live value. But .env takes precedence over all three. Every place I'd looked was showing a value that got overridden.

I spent a morning doing a lovely repair job on a server that wasn't on the call path.

The embarrassing part: "check the value actually in effect at runtime, not what the config file says" was a principle I had written down that same morning, from the Google parser episode. I applied it strictly to the SearXNG config and never applied it to the agent's config.

Looking back, both incidents are the same bug. In August, "instance healthy" hid "engine dead." This time, "always returns 10" hid "Google gone half the time." A metric can only vouch for the layer it's measured at.

4. What I didn't finish

Some things I haven't measured, so I'm not committing to these conclusions yet:

• I didn't measure the quality loss. I know 58% of calls were missing Google's 20 results. I haven't tested how much worse the top 10 got. Bing plus Yandex might be good enough, or they might be a lot worse.
• The busy-hour cutoff is my own pick. "50 calls per hour" was just a way to separate the two regimes. A different cutoff would move the percentages, though I'd expect the direction to hold.
• I lined up the two log sources by hand. The container logs are in UTC and the agent logs are in local time. After converting time zones, the suspension peaks roughly line up with the call peaks, but I only eyeballed it. I didn't match events one by one.
• I haven't reconciled suspension time against absences. Over three days the container logged that Google 429 79 times. 79 × 180 seconds ≈ 4 hours. Calls come in bursts, and I haven't checked line by line whether those 4 hours actually cover all 685 calls. There may also be a stacking behavior in the suspension logic that I don't understand yet.
• The Chinese-language engines are all down right now. In today's probes, baidu, quark, and sogou showed up in the unresponsive list every time, and the stats page shows 90% CAPTCHA errors for both baidu and sogou. (They're the main domestic search engines for Chinese-language queries.) So Chinese queries are currently running on those three English-language engines too, and I haven't measured that impact either.

5. Next time your self-hosted search looks off, check in this order

1. Confirm you're fixing the instance the caller actually uses. Read the caller's code to see which value it ends up reading. systemd and the process environment may be showing a value that gets overridden.
2. Run the caller's own search tool and look at what comes back. Curling the backend doesn't count.
3. Probe once, note how many results each engine returns per page, and add them up to get the full-load raw count.
4. Pull a histogram of raw counts from the caller's logs, find the spike, and use the addition from step 3 to work out who's missing.
5. Split the absence rate into busy and quiet hours. An engine that only goes missing when you're busy is almost certainly being rate-limited and suspended.
6. If an engine shows 0% errors but always returns zero results, suspect a stale parser first and diff against a fresh throwaway instance.
7. Disable engines that are hitting CAPTCHAs or 429s outright. Don't count on proxies.
8. Run at least two Chinese-language engines if you need them. Relying on one means a single suspension leaves you with nothing for that whole window.