The linter ecosystem is actively migrating silent failure into an undetectable idiom

fei — Fri, 28 Aug 2026 06:17:57 +0000

In a well-known family of LLM evaluation bugs, the failure doesn't look like a failure. A judge model goes down, and the benchmark keeps publishing scores — because somewhere in the code, a handler like this turned an outage into a legitimate-looking zero:

try:
    score = await llm_judge(prompt)
except Exception:
    return 0.0

I hit this pattern repeatedly while fixing correctness bugs across mainstream AI/ML open source. So I did what you do when the same defect family keeps appearing: I built a detector for it — failroute, a static analyzer that flags failures converted into success-looking values. It's on PyPI, it runs in CI, and every number in this post is reproducible from the repository.

But the interesting finding of the last week is not my tool. It's about the ecosystem it lives in.

The idiom that learned to hide

Python has two ways to write "ignore failures here". The old one:

try:
    os.remove(path)
except FileNotFoundError:
    pass

Every linter has an opinion about that shape. Ruff's S110 flags try-except-pass. Bandit's B110 does the same. The message is always some variant of "consider logging the exception".

The modern one:

with contextlib.suppress(FileNotFoundError):
    os.remove(path)

Semantically identical — the failure is discarded either way. And here is the part that surprised me: no shipped linter flags it. Not ruff, not Bandit, not bugbear, not pylint. The only related rule is bugbear's B022, which fires on contextlib.suppress() with no arguments — the case where nothing is even suppressed.

It gets better. Ruff's SIM105 rule exists specifically to recommend this rewrite: "Use contextlib.suppress(...) instead of try-except-pass." A project that runs autofix on SIM105 converts every flagged handler into a suppress block — and moves all of them out of every existing detector's view in one automated sweep. The silence is unchanged. The syntax learned to hide.

I want to be careful about the claim here, because "linter X doesn't catch Y" is easy to assert and hard to defend. This one is defensible: contextlib.suppress(...) with valid arguments is a call expression, not an ExceptHandler, and every exception-handling rule in the shipped linters dispatches on handlers. As of today, the detection gap is real, shipped, and — in SIM105's case — actively widened by autofix.

Does it matter in real code?

contextlib.suppress is everywhere in modern Python. In the source packages of eight real AI/eval repositories (garak, inspect_ai, pydantic-ai, uqlm, trl, smolagents, deepteam, fickling), I counted 77 suppress blocks, sitting alongside 403 silent-fallback / masked-exception handlers that syntactic rules also cannot express — 613 findings total, against 67 for ruff's exception rules. Each finding in the benchmark is a routing decision a reviewer should have made explicitly: is discarding this failure correct here? Sometimes yes — that's what the opt-out marker is for. Often it's the judge-outage pattern wearing a cleaner syntax.

None of this means suppress is bad. contextlib.suppress(CancelledError) is idiomatic cancellation absorption and failroute doesn't flag it, for exactly the reason handler rules exempt except asyncio.CancelledError. The point is narrower: a routing decision that used to be visible for review is migrating into a form where review tooling cannot see it, and the migration is automated.

What a detector for this needs (and what I learned building one)

Syntactic detection of suppress is trivial — it's one AST pattern. The hard part is the same as for any semantic defect family: precision, because a linter that cries wolf gets disabled.

failroute's answer is a hand-labelled corpus. Every rule must match fixtures whose ground truth was written from the semantics of each case, independently of tool output, with precision and recall pinned at 1.0. The corpus has already paid for itself twice:

The suppress fixtures were added before the detector existed. The benchmark ran red — exactly the five expected misses — which is what makes "labels independent of tool output" a testable claim rather than a slogan.
The corpus then caught a design mistake before release: the first draft flagged contextlib.suppress(asyncio.CancelledError). But absorbing cancellation is idiomatic control flow, and the handler rules already exempt except asyncio.CancelledError. Same semantics, two syntaxes, two answers — a bug. v0.5.1 unified the ignore lists across both syntaxes.

The broader lesson for anyone building tooling with AI assistance: the AI is good at proposing detector logic and finding the gap; the deterministic gate (a corpus that must first fail, then pass) is what turns a plausible-sounding rule into a defensible one. The full loop, including the failure cases, is documented in the repository's process doc.

Try it

pip install failroute
failroute --repo .        # text/JSON/SARIF, exit code CI-friendly

There's a pre-commit hook, a GitHub Action that uploads findings to code scanning, and a self-scan of the repository itself in CI (expected alerts: zero). If you maintain an eval or red-team framework, I'd genuinely like to know whether it finds anything real in your codebase — and if it finds something that isn't real, I want to hear about that too, because that's the corpus growing.

DEV Community: fei

The linter ecosystem is actively migrating silent failure into an undetectable idiom

The idiom that learned to hide

Does it matter in real code?

What a detector for this needs (and what I learned building one)

Try it