Your DNS resolver is dropping queries and calling it a timeout

I found 39,662 failed DNS lookups in a log file. They had been accumulating at
roughly 50 an hour for as long as the log went back, and every one read the same way:

1/1 upstreams failed ... 192.168.8.1:53=timeout after 4s

I had been blaming the wrong component for months. The interesting part is not the
fix — it is one setting — but why the standard diagnostic could never have found it.

The misdiagnosis

The log belonged to my VPN client, so for a long time this was "a VPN problem." It
wasn't. The VPN was simply the only component on the box that logged its DNS
failures. Everything else — the container runtime, the schedulers, the scripts —
retried silently or died silently.

That is worth sitting with. The component that reports an error is not usually the
component at fault. It is usually the only one that bothered to tell you.

The cause: a rate limit that answers with nothing

My router runs AdGuard Home as its resolver. AdGuard has a per-client rate limit,
default ratelimit: 20 queries per second.

Here is the part that makes it invisible:

Over the limit, AdGuard discards the query with no response at all. Not
SERVFAIL. Not REFUSED. Nothing.

A refusal would have been a signpost — you look it up, you find the limit, you fix it
in ten minutes. Silence is indistinguishable from a network fault, so every symptom
pointed at the network layer: the cable, the VPN, the MTU, conntrack. I had ruled
those out one by one, with evidence, and got nowhere, because the resolver was
answering correctly according to its configuration.

A dropped packet and a working rate limiter look identical from the client.

Why dig can never reproduce it

This is the bit I most want people to take away.

The obvious test is to run some queries and see if they fail:

for i in $(seq 1 150); do dig @192.168.8.1 example.com +short; done

150 queries, zero failures, 0.02s each. A clean bill of health, and completely
meaningless — because dig waits for each reply before sending the next one. Serial
dig in a loop tops out around 1 query per second. The limit is 20. You cannot
approach it by trying harder in sequence; you will never get near it.

The failure only exists under concurrency, so the test has to be parallel:

rm -f /tmp/f
for i in $(seq 1 40); do
  ( dig +time=3 +tries=1 @192.168.8.1 p$i-$RANDOM.example.com +short >/dev/null \
      || echo x >> /tmp/f ) &
done; wait
wc -l < /tmp/f

Note the random subdomain per query — otherwise the cache answers and you measure
nothing.

test	result
150 queries, one at a time	0 lost
40 in parallel	24 of 40 lost, repeatable
40 in parallel, reverse/PTR lookups	24 of 40 lost

Two thirds of a burst, gone. Every single time.

The lesson generalises well beyond DNS: if a system fails under concurrency, a
serial test will certify it as healthy forever. Match the shape of your test to the
shape of the load, not to what is convenient to type.

What was actually being lost

Breaking down two days of failures by name was the point at which this stopped being
an academic curiosity:

• 72% were in-addr.arpa reverse lookups for my own LAN, peaking around 50/min during the 04:00 backups
• 7,802 market-data queries belonging to a trading system
• 1,728 imap.gmail.com
• 166 calls to a transactional email API

So a "background annoyance in a VPN log" was plausibly costing live data fetches and
outbound mail. Bursts are exactly what backup jobs, page loads and API clients
generate — the traffic that matters is the traffic that arrives in parallel.

And a detail worth knowing if you ever chase a user-facing report of this:
Chrome renders a dropped lookup as NXDOMAIN, which is indistinguishable from
the record not existing. The user tells you "that hostname is wrong." It isn't.

The fix, and the trap in applying it

The fix is one line — ratelimit: 0, correct for a LAN-only resolver with no
exposure to the internet.

The trap: setting it in the web UI did not apply it.

I set the field to 0 in AdGuard's admin UI and saved. AdGuard rewrote its
config.yaml minutes later — and the file still said 20. The burst test still
lost 20 of 40. Worse, I could not even query the running config to check, because
that build ties its API auth to the router's UI session and returns 401 from
localhost.

What worked was stopping the service, editing the file, and starting it again —
AdGuard rewrites config.yaml from memory on exit, so editing a running instance's
config file achieves nothing.

Result:

	before	after
40 parallel lookups	20–26 lost, repeatable	0, 0, 1
20 serial lookups	0 lost	0 lost

Note the second row. The serial test read identically before and after a change that
eliminated a two-thirds packet loss. A test that passes in both the broken and the
fixed state is not a test.

One correction, because it cost me time

In my first pass over the loss data I reported query.finance.yahoo.com as a casualty.
That hostname does not exist — it is NXDOMAIN at Yahoo's own authoritative servers.
The real client was calling query1.finance.yahoo.com, which resolves fine.

I had found a real signal and attached a wrong label to it, then reasoned from the
label. Worth checking that the thing you are alarmed about exists before you build a
theory on it.

What I'd take from this

• The component that logs the error is rarely the component at fault. It is the one that was honest.
• Silence is a design choice with debugging consequences. A rate limiter that returns REFUSED is trivially diagnosable. One that drops is indistinguishable from a broken network.
• Serial tests certify concurrent failures as healthy. If load arrives in parallel, test in parallel.
• Verify the file, not the form. A UI that accepts a setting has told you nothing about whether the daemon applied it.

---

If you run a homelab and want the rest of this kind of thing — the diagnostics, the
dead ends, and the bits where the first answer was wrong — my notes are on GitHub at
casareanderson, and I write most of it up as I go.