Does AI-generated code silently swallow errors? 120 measured generations: every flagged case was a false positive or a documented fallback

If you gate AI-generated code with linters or a CI rule that hunts for swallowed errors, this experiment suggests the part you actually care about — "is this return None a contract or a cover-up?" — is exactly the part the rule cannot decide.

I started this project convinced that small LLMs routinely swallow failures — catch an error, return an empty value, pretend nothing happened — and that I'd measure the contamination rate. The hypothesis fell apart in a more interesting way than any clean number would have been. This article is the record of a prior being refuted by measurement, with the collapse documented step by step.

TL;DR

Setup: Qwen2.5-Coder 1.5B (Apache-2.0) via Ollama, CPU-only, Windows 11. 12 frozen tasks × 10 generations (seeds 0–9, temp 0.7) = **120 samples: 100 failure-path functions (50 Python, 50 TypeScript) + 20 pure-computation controls. A 7B robustness footnote adds 30 more (150 generations total, all frozen unfiltered in the repo). Classification: Python by AST, TypeScript by regex; **all 120 labels eyeballed, every swallow candidate and boundary case hand-adjudicated* against docstrings, comments, and the function's contract, published as gt.csv.*

1. A naive Semgrep detector — the kind you might reasonably drop into CI — flagged 4 candidates. Human adjudication: 0 true positives. Two were false positives (an empty catch in a usage example the model appended outside the function under test), two were legitimate fallbacks, documented (docstring or comment) as the function's contract. But this 0 is not "AI doesn't swallow errors" — my detector targeted try/except and structurally never looked at if-guard default returns, which is where the suspicious shapes actually lived (see finding 3).
2. In this sample, "fails loudly" and "no handling at all" dominated — not swallowing. Of Python's 50 failure-path generations, 39 (78%) used no try/except: 14 raise explicitly via if/else (loud), 4 return None behind an if guard, 21 just let exceptions propagate. TypeScript went the other way: 33/50 (66%) wrote try/catch with log and/or re-throw — proper handling. The only belief that collapsed was my prior that swallowing would be dominant.
3. The core finding: whether code "swallows errors" is not decidable from syntax. The same return None shows up as a documented, role-appropriate contract (parse_int: None if it cannot be converted) and as a hazard (fetch_json: 404, 500 — every non-200 collapses into the same None, while a network error raises instead, a third behavior the caller has to know about — with a comment saying so). A comment doesn't make it sound; a bare guard doesn't make it a bug. The deciding information — the function's role, the caller's expectations, the spec — lives outside the pattern. Static rules can surface candidates; they cannot deliver the verdict.
4. Honest caveats up front: one small model family, 12 tasks, N=10 per task (pseudo-replication — the independent unit is the task, not the generation), single-rater adjudication by me, and a detector whose scope hole I only noticed because the data rubbed my nose in it. Scope: free, local, small models, this corpus. No causal claims about "AI code vs human code" — that comparison belongs to prior work (AIRA, below), not to this experiment.

⚠ Reproducibility scope: generation is nondeterministic and ran once; all 150 generations are frozen unfiltered in the repo. What you reproduce is the deterministic analysis layer — same 120 files in, same distributions and candidate counts out. The final legitimate-vs-swallow labels are human adjudication, published transparently in gt.csv — and the fact that you can't regenerate that layer mechanically is itself the thesis of this article.

Why measure this

AI-generated code quality has a well-cited data point now: AIRA (arXiv:2604.17587, preprint, 2026) ran a deterministic, parser-backed static analyzer over 955 AI-authored and 955 human-authored code samples and reported 1.80× more high-severity findings in AI-authored code (0.435 vs 0.242 per sample), with Broad Exception Suppression (C03) — the "swallowed error" family — as the most frequent check (263 vs 185 in Study 3).

That number gets quoted as "AI code swallows errors." But read the paper closely and it says something more careful, twice:

• Detection is deterministic and works — the paper even warns that an LLM evaluator misses these suppressions at a rate of 44:1 compared to the deterministic scanner. Pattern-matching is the machine's strong suit.
• "Flagged ≠ defect." AIRA's own text states that some fail-soft patterns are contextually intentional and that human review is required before remediation. Two of its checks are permanently human-review-only.

So the open question isn't "can we detect suppression patterns?" (yes, deterministically). It's the step everyone skips: can the verdict — legitimate fallback or bug-hiding swallow — be automated too? I couldn't find a published measurement of that gap (pointers welcome). This article measures it with 120 locally generated functions, a naive detector, and every candidate opened by hand.

Scope declaration up front: this is neither "AI is dangerous" nor "static analysis is useless." It's a measurement of where the machine's jurisdiction ends.

Terms, before the numbers

Three things that look identical in grep output and must not be conflated:

• Default return — on failure, return None / 0 / [] / "" instead of raising. A syntactic shape. Detectable.
• Silent swallow — a default return that erases information the caller needed: failure becomes indistinguishable from "empty but fine." A semantic judgment.
• Legitimate fallback — a default return that is the function's contract ("returns None if unparseable"), ideally documented. Also a semantic judgment.

The whole article is about the gap between the first item and the other two. Classification-wise this sits in CWE-703 (Improper Check or Handling of Exceptional Conditions), with CWE-1069 (empty exception block) and CWE-390 nearby; the exception-antipattern literature (de Pádua & Shang, arXiv:1704.00778, Java/C#) maps the same territory.

Why standard linters don't cover this (and what I built instead)

Checked against current docs, not from memory:

• Python / ruff: S110 (try-except-pass) and S112 exist but are disabled by default — you must opt in via select/extend-select. And a typed swallow like except Exception: return [] sails through anyway.
• TypeScript / ESLint: no-empty only fires on empty blocks, so catch (e) { return null; } is invisible — it's not empty. no-empty even allows a catch containing only a comment. (Note the cultural assumption baked in there: "has a comment" is treated as "is intentional." Whether the comment describes sound error handling is nobody's department. Hold that thought for the counterexample below.)

So I wrote a Semgrep rule that goes after the meaning-shaped pattern — the naive "before" version, verbatim from the repo:

rules:
  - id: py-swallow-return-default
    languages: [python]
    severity: ERROR
    message: "Exception caught and a default value returned without logging/re-raise (silent fallback)."
    patterns:
      - pattern: |
          try:
            ...
          except $E:
            return $R
      - metavariable-pattern:
          metavariable: $R
          patterns:
            - pattern-either:
                - pattern: None
                - pattern: "[]"
                - pattern: "{}"
                - pattern: "0"
                - pattern: "False"
                - pattern: '""'

  - id: py-except-pass
    languages: [python]
    severity: ERROR
    message: "Exception swallowed with pass (silent failure)."
    patterns:
      - pattern: |
          try:
            ...
          except $E:
            pass

(The TypeScript rules are the same shape — catch { return null }, empty catch — and ship in the repo.)

My hypothesis at this point: "run this over N generations, get a contamination rate." Here is where the measurement starts disagreeing with me.

Setup

• Tasks: 12, frozen in tasks/tasks.json before generation. Per language: 5 failure-path I/O tasks (config loading, HTTP fetch, env vars, numeric parsing, file head) + 1 pure-computation control (mean/sum — no try needed). Prompts are neutral: no "handle errors" nudging.
• Task-design disclosure (I did not farm for swallows): pilot runs showed that on plain tasks ("read JSON, return it") the model writes no try/except and nothing interesting happens — which means any "swallow rate" is heavily task-dependent and can be manufactured. I chose failure-inevitable I/O not to inflate the count (the result is ~0 anyway) but to give failure a stage — and kept the pure-computation controls to verify that off-stage, nothing appears. They did: all 20 controls showed zero try/except and zero candidates.
• Generation: Ollama (MIT), CPU-only, qwen2.5-coder:1.5b (Apache-2.0), temperature 0.7, top_k=40 / top_p=0.9, seeds 0–9 per task, threads fixed at 4. The 7B model appears only in the robustness footnote (N=3 per task).
• Determinism probe (before trusting anything): at temperature 0 with a fixed thread count, the same seed generated twice was byte-identical (SHA match). Distributions require temp > 0; hence 0.7.
• Classifier asymmetry, disclosed: Python is classified via ast (a real parse); TypeScript via regex (not a full parser). Cross-language absolute comparisons carry instrument bias — which is why every label was eyeballed and every candidate hand-adjudicated rather than trusting classifier output.

Component	Choice
Inference	Ollama, CPU (Windows 11)
Model (main)	Qwen2.5-Coder 1.5B, default GGUF quantization (Q4_K_M-class)
Model (footnote)	Qwen2.5-Coder 7B
Detector	Semgrep 1.168.0 (CE) + custom rules above
Analysis	Python 3.12, ruff 0.15.12 for the lint baseline

Measured: 2026-07 (corpus generated 2026-07-01). Full per-run metadata in the repo's PROVENANCE file.

Result 1: mostly not swallowing — passing through, or failing loudly

How the 50 failure-path generations per language handled failure:

Failure-path tasks (n=50 per language)	Python	TypeScript
try/except + log or re-raise (proper)	9	33 (66%)
try/except returning a default (swallow candidate)	2	0
no try/except at all	39 (78%)	17 (34%)

Reading "no try/except = no handling" would be wrong. AST-splitting Python's 39:

• 14 raise via if/else (e.g. missing env var → raise ValueError(...)) — that's failing loudly, the opposite of swallowing.
• 4 return None behind an if guard — all four in fetch_json (non-200 → return None). A default return! Which my try/except-scoped detector structurally never saw. Remember these four; they're the article's best specimen.
• 21 bare — no failure path written; exceptions propagate to the caller.
• (Check: 14 + 4 + 21 = 39.)

TypeScript's 66% try/catch majority mostly did console.error(...) + throw — textbook handling, no swallowing.

The language difference (Python avoids try/except, TS writes it) is an observation, not a finding: prompt phrasing, language idiom (async/await + try/catch is TS boilerplate), and the AST-vs-regex classifier asymmetry all confound it. The spine of this article is the counterexample below, not this table.

So the original hypothesis — "AI swallows failures at some rate N% I can report" — collapsed in the first table: in this sample, swallowing wasn't the dominant behavior at all.

Result 2: every detector hit was a false positive or documented-legitimate

The naive Semgrep rules flagged 4 of 120 (2 Python, 2 TypeScript). Opening all four by hand:

TypeScript, 2 hits (ts_load_config) = false positives. The function under test was exemplary (code condensed and annotated from the corpus):

async function loadConfig(path: string): Promise<any> {
    try {
        const data = await fs.promises.readFile(path, 'utf8');
        return JSON.parse(data);
    } catch (error) {
        console.error(`Error reading or parsing the file at ${path}:`, error);
        throw error; // logged and re-thrown -- not swallowed (proper)
    }
}
// ...but the model appended a usage example after the function:
(async () => {
    try { const config = await loadConfig('./config.json'); }
    catch (error) { /* Handle any errors... <- comment-only catch */ }
})();

The rule fired on the demo block's catch — (a) outside the function under test, (b) "empty" only because Semgrep's AST ignores comments. Context makes it an obvious false positive; the pattern alone can't know that.

Python, 2 hits (py_parse_int) = documented, legitimate fallbacks. The docstring states the contract — None if it cannot be converted (the second one says the same in a comment).

Bottom line: undisclosed swallowing inside try/except = 0. And now the honest part: my detector's own scope hole. The rules target try/except — the if-guard default returns (the four fetch_jsons) were never in scope. So the corpus does contain default returns; the accurate claim is "zero undisclosed swallows within the detector's scope," not "zero problematic fallbacks in the corpus." Whether those four are problems is precisely the question syntax can't answer — next section.

Result 3 (the core): the same return None, and what separates them isn't in the code

Two functions from the corpus (condensed and annotated). Which one swallows errors?

# (A) numeric parsing: None if unconvertible (contract stated in the docstring)
def parse_int_field(data, key):
    """Returns the int, or None if it cannot be converted."""
    try:
        return int(data[key])
    except ValueError:
        return None            # role-appropriate fallback

# (B) HTTP fetch: None on non-200 (and yes, there's a comment saying so)
def fetch_json(url):
    response = requests.get(url)
    if response.status_code == 200:
        return response.json()
    else:
        return None            # 404 or 500 -- every non-200 flattened into None
                               # (a network error raises instead: a third behavior)

Syntactically, the interesting part is near-identical: on failure, return None. Semantically they're opposites. (A) matches the function's role — "tell me whether this converts" — so None is the answer. (B) permanently destroys the caller's ability to distinguish "no data" from "the fetch failed." And (B) has a comment. Documentation doesn't settle it: a documented return 0 or return None can still silently poison every computation downstream. ("Has a comment = intentional" is exactly the assumption ESLint's no-empty institutionalizes.)

(B) is the strongest specimen this experiment produced: a documented-but-hazardous default return that appeared as a 4-sample cluster (not a one-off), sitting squarely in the blind spot of a try/except-scoped detector — while being exactly the "quietly fails" shape the AIRA numbers gesture at, at population level.

For contrast, non-swallowing code carries its intent inside the syntax:

def get_api_token():
    if 'API_TOKEN' in os.environ:
        return os.getenv('API_TOKEN')
    raise ValueError("The API_TOKEN environment variable is not set.")  # fails loudly

The point, stated carefully: syntactic patterns can surface candidates. The information that separates contract from cover-up — the function's role, the caller's expectations, the spec, and whether the documentation is right — lives outside the pattern. I'm not claiming "undecidable in principle": smarter analysis (types, dataflow, call-site analysis) absolutely narrows the candidates. But "what should this function return, in this context?" is a spec question, and the spec comes from outside the analyzer. Sharper tools shrink the pile; the final reconciliation against intent remains.

Honest scope note: this rests on a small corpus and a handful of specimens — read it as a demonstrated boundary, not a general law about all static analysis. What it demonstrates survives the small N, though, because it's an existence proof: two same-shaped snippets with opposite verdicts, and the verdict-relevant information demonstrably outside the syntax.

What I actually do now: machines generate the candidate list (patterns, distributions, CI notifications — semgrep scan --error exits 1 on candidates, which is fine); a human adjudicates candidates against role, caller, and spec. A green scan is read as "zero candidates for a human to look at," never as "pass."

Where this lands relative to AIRA

Neatly inside its own fine print, it turns out. AIRA detects suppression patterns deterministically and reports the 1.80× population-level difference — candidate generation is the machine's win, and the paper warns that semantic (LLM) evaluation underperforms there, 44:1. But the same paper marks the boundary: flagged patterns aren't necessarily defects, some fail-softs are intentional, human review precedes remediation, and two checks are permanently human-only.

That division of labor is what this experiment probed at specimen level:

• Machines: sensitivity. Surface default-return shapes, chart try/except usage, gate CI with "candidates exist."
• Humans: adjudication. Contract or cover-up — read the role, the caller, the spec.

One misquote I need to preempt, because I nearly published it myself (see pitfall 5): AIRA is not a human-evaluation study, and I am not claiming AI code swallows more than human code — that comparison is AIRA's, made with its own methodology, at population level, in a single-author preprint. This experiment neither confirms nor contradicts it; it maps the adjudication residue the paper explicitly leaves to humans.

Everything I got wrong

In the order I got it wrong:

1. "AI swallows failures; my rule will measure the rate." → Measured: in this sample, swallowing wasn't dominant — pass-through, loud failure, and (in TS) proper handling were. The contamination-rate framing died on the first table.
2. "The detector's 4 hits = 4 swallows." → All four were false positives or documented-legitimate. The honest number is 0, so I report 0 — with its scope. Don't round adjudication up to detection.
3. "The false positives just mean my rule is crude; refinement will fix it." → Refinement did kill the TS false positives. It could not touch the real problem: whether a return None is legitimate isn't decided by any pattern, or even by the presence of documentation. This is where the thesis flipped from "measure the rate" to "the verdict doesn't live in syntax." And auditing my own instrument surfaced the scope hole (if-guard defaults) it had from the start. When your tool's errors persist under refinement, suspect the question, not the regex.
4. The Windows encoding landmine. Semgrep read my TS rule file under Windows' legacy default codepage (cp932) and crashed with UnicodeDecodeError (exit 2) — on an em dash I'd left in a rule message. Fix: ASCII-only rule files + PYTHONUTF8=1 for every run. If your pipeline runs on Windows, pin UTF-8 explicitly or non-ASCII bytes in innocent places will cost you an afternoon.
5. Writing this article, I did the thing the article warns about. My research notes — AI-assisted — said AIRA judged suppressions "by human evaluation." Pre-publication, I went back to the primary source: it's the opposite. AIRA is a deterministic scanner, and it warns against relying on semantic evaluation for this class. I had a plausible summary, I trusted it, and a "verification discipline" article nearly shipped misstating its key citation's methodology. Corrected against the paper itself. Candidates can come from machines — or from your own notes. The final check against the primary source is the human step, and I almost skipped it.

Limitations

• One model family, small models (1.5B main / 7B footnote), 12 tasks, one prompt style. Behavior is strongly model- and task-dependent; no generalization claimed beyond this corpus.
• Statistical independence: N=10 generations per prompt are correlated pseudo-replicates. The independent unit is the task (5 per language), not the generation; don't read the n=50 tables as population estimates. The thesis rides on the counterexample, not the ratios.
• Detector scope: rules and classifiers target try/except; if-guard defaults are out of scope, so "0 swallows" means "0 undisclosed swallows in scope" — the corpus contains default returns whose adjudication is the whole point.
• Classifier asymmetry: Python AST vs TypeScript regex (nested-brace catches can slip). Human-verified labels backstop it, but discount cross-language comparisons.
• 7B robustness footnote (N=3/task, 30 generations): Python 10/15 no-try, 3 proper, 2 candidates; TS 8/15 no-try, 7 proper, 0 candidates. Both candidates: parse_int again, documented — one returns 0 on failure, which is documented and still capable of silently corrupting downstream arithmetic, i.e., the "(B) problem" one more time. N=3, so no robustness claim — only "not refuted in this range."
• The adjudicator's hole, disclosed next to the detector's: the legitimate-vs-swallow verdicts are single-rater (me), no inter-rater agreement measured. Rubric: (1) does failure yield a default? (2) is that behavior disclosed as intent? (3) does it erase failure information the caller needs? Boundary cases ship with reasons in gt.csv. A second rater is future work — "the last step is human" cuts both ways, so I'm flagging my own last step.
• Conflict of interest: I do AI-code auditing professionally. "The final call needs a human" is a conclusion that favors my line of work. Mitigation is transparency: frozen tasks, unfiltered generations, published adjudications, and the detector's failures disclosed by me, above.

Reproduce it

git clone https://github.com/sumitsuke/ai-silent-defect-scanner && cd ai-silent-defect-scanner
# Deterministic layer: same 120 files -> same distributions & candidate counts
PYTHONUTF8=1 python scripts/classify_split.py      # failure-path distributions (the n=50 tables)
PYTHONUTF8=1 python scripts/scan_and_count.py      # naive Semgrep candidates (the 4 hits)
PYTHONUTF8=1 python scripts/build_gt.py            # regenerate the adjudication layer, results/gt.csv
make scan-mine DIR=/path/to/your/repo              # candidates for YOUR repo (verdicts are on you)

• Mechanical reproduction covers distributions and candidate counts. The final labels are my adjudications, published in results/gt.csv with reasons — by design not machine-rederivable (that's the thesis).
• raw/ ships all 150 generations unfiltered; PROVENANCE records model, quantization, seeds, temperature, thread count, OS, dates. Regenerating gives you a different distribution — that's LLM sampling for you.
• CPU-only, no GPU needed, no paid APIs anywhere in the pipeline.

Takeaways

1. Pattern rules find shapes, and the corpus's actual risk shape (if-guard default returns) wasn't the shape my rule watched. Budget for your detector's scope hole, not just its false-positive rate.
2. Documentation is not adjudication. A commented return None and a return 0 documented in its docstring both appeared here as disclosed and still hazardous. "Has a comment" is a lint convention, not a verdict.
3. Split the jurisdictions and staff them accordingly: machine = candidates and distributions in CI; human = contract-vs-cover-up against role, caller, and spec. Green means "no candidates today," not "correct."

If you remember one sentence, please don't make it "static analysis is useless" — the detector did its detection job fine. Make it: syntax can surface the suspects, but conviction requires reading intent — and intent isn't stored in the AST.

---

Detection code, all 150 generations, and the human adjudication layer (gt.csv): github.com/sumitsuke/ai-silent-defect-scanner. Every number above is measured; anything not measured is labeled as not measured.

This is an English adaptation of my Japanese article on Qiita (Qiita is a Japanese dev-blogging platform) — written by me in Japanese, restructured and translated with AI assistance, human-reviewed. If you spot an error, comments and issues are open; I'll verify against the frozen corpus and correct with a changelog.

Verification record (environment, verdict, last verified date, evidence) and the canonical write-up: https://sumitsuke.jp/lab/ai-code-silent-fallback/ — code, data and reproduction: https://github.com/sumitsuke/ai-silent-defect-scanner. I audit and repair AI-generated / outsourced code with the same discipline (logs, tests, static analysis and a human spec check, kept separate). Text-only, no calls: https://sumitsuke.jp/works/repair/