Five bugs in my LLM app that never threw an error

I spent two and a half weeks building an agent that reads a project's code review history and remembers what it decided. 86,321 comments from pandas, distilled into 298 conventions, served out of CockroachDB behind a Lambda.

Every serious bug I hit had the same shape. Nothing raised. Tests passed. The demo worked. And somewhere in the middle, a component was doing absolutely nothing.

Here are five, with the code.

1. bool("false") is True

The whole point of the system is that it refuses when it does not know. The answering prompt returns JSON with an answered field.

result = await chat.complete_json(messages)
if result.get("answered"):
    return result["answer"]

A model that writes "answered": "false" instead of "answered": false turns a refusal into an answer, because a non-empty string is truthy. The one behaviour the project exists to guarantee, defeated by a quotation mark.

Now every model response goes through a Pydantic model that fails towards doing nothing:

class AnswerOutput(ModelOutput):
    answered: bool = False   # coerces "false" -> False
    answer: str = ""

An answer becomes a refusal. A drafted rule becomes unusable. A contradiction verdict becomes "compatible", which retires nothing. A garbled response is not evidence for writing to memory.

2. A null discarded whole responses, six times out of six

The fix for bug 1 caused bug 2, which is funnier than it was at the time.

The prompt that turns a maintainer's comment into a rule says:

"rationale": "why, one sentence, only if they gave a reason"

A maintainer who states a convention flatly gets null back, which is correct JSON for "no reason given". And:

class DraftedRule(BaseModel):
    statement: str = ""
    rationale: str = ""   # null is not a str

Pydantic rejects null for a plain str. Validation is all or nothing. So the entire response was thrown away and replaced with the inert default, which downstream reads as the model having declined.

A real maintainer correction became {"status":"ignored","reason":"no convention stated"}. Six reproductions, six failures, on a payload that was otherwise exactly what I asked for.

class ModelOutput(BaseModel):
    @field_validator("*", mode="before")
    @classmethod
    def _null_text_is_empty(cls, value, info):
        if value is not None:
            return value
        field = cls.model_fields.get(info.field_name)
        return "" if field is not None and field.annotation is str else value

A null where a string was expected means empty, not malformed. Note the field.annotation is str check: scope_pattern: str | None genuinely means None, and coercing that one would turn "applies everywhere" into a pattern matching nothing.

3. 68 of my 79 path rules matched zero files

Each rule stores a pattern for which files it applies to. I matched with fnmatch.

fnmatch("pandas/core/frame.py", "pandas/core/%")   # False

The patterns were SQL LIKE, not globs. pandas/tests/%, pandas/core/%, pandas/tests/%/conftest.py. Nothing asked the extraction prompt for that and nothing documented it. The model just wrote it, presumably because the rules were destined for a database.

% is not an fnmatch wildcard. So 68 of 79 anchored rules silently matched nothing, and the agent ran entirely on the two rules that happened to be written pandas/**/*.py, which cover 63% of the repository.

It looked like it was working the entire time. The fix is one line, and the bug was invisible for a week:

cleaned = pattern.strip().strip("`").lstrip("./").replace("%", "*")

4. The similarity score I was tuning had no signal in it

The agent comments on pull requests, choosing which conventions apply by embedding a description of the change and taking the nearest rules.

It commented on 38 of 40. I assumed a bad threshold and started tuning. At 1.05 it spoke on 95% of pull requests; at 0.95, on 18%.

Then I measured the distribution instead. For 25 real pull requests, distance to the nearest rule versus the tenth nearest:

query phrasing	median nearest	median spread, 1st to 10th
title only	0.954	0.097
title plus directories	0.947	0.100
paths and counts	0.958	0.105
written as a question	0.933	0.095

The nearest rule is not meaningfully nearer than the tenth, under any phrasing. Every rule is a general statement about the same codebase, so every rule is similar to any description of a change to it.

There was nothing to threshold. Retrieval now runs on path matching, which is a fact, and similarity only orders what the paths already admitted.

5. GitHub's author_association describes the present

I filtered to maintainer comments using GitHub's author_association field.

It reports whether someone has write access now, not when they wrote the comment. One reviewer wrote 74,077 comments on pandas between 2012 and 2025, more than twice anyone else. He has since left the org, so all 74,077 come back as CONTRIBUTOR.

Filtering on that field discarded about a third of the corpus, including the most experienced reviewers the project ever had. No error, no warning, just a smaller number that looked plausible.

The pattern

None of these raised. Four of the five I found by counting something rather than by reading code: how often does it speak, how many rules match anything, how far apart are the distances really.

A library used wrong fails loudly. A model used wrong returns confident, well-formatted output that is quietly detached from what you meant.

So pick a number that should hold if the system works, and go and look at it. Not "does it return results" but "how often does it speak, and is that the rate I intended". Not "did it cite something" but "do the citations resolve".

Code is at github.com/pyarchana/precedent, MIT.

Comments

[Vinh Nguyen]

The fix for bug 1 inherits bug 2's failure mode. On pydantic 2.13.5, answered: bool coerces the enumerated set — "false", "False", "no", "0" all land on False — but anything outside it raises ValidationError, so "answered": "unknown" or a model that hedges in prose throws the whole object away exactly the way the null rationale did. null is the sharp case: _null_text_is_empty passes None straight through when the annotation isn't str, so "answered": null, which is reasonable JSON for "no decision", takes the response down instead of falling to the default you wrote it for.

[Archana]

Confirmed, ran it before replying:

answered=null,    answer="do X [PR #1]"  ->  answered=False, answer=''
answered=unknown, answer="do X [PR #1]"  ->  answered=False, answer=''
usable=null,      statement="Place..."   ->  is_usable=False, statement=''

So I fixed str and stopped looking. Still fails safe, still throws the good fields out with the bad one.

Per-field validators for bool and every Literal seems like the wrong shape. ValidationError already names the failing locations, so dropping those and revalidating should keep whatever parsed. Going to try that.

[Vinh Nguyen]

Worth putting the revalidated object next to a real refusal before you commit to that shape. On 2.13.5 with your models, {"answered":"unknown","answer":"do X [PR #1]"} drops the flagged loc and comes back answered=False with the answer text still sitting there, which is the same answered=False a genuine refusal produces. So the field the whole guarantee rests on stops telling you which of the two happened, unless you keep the dropped locs on the object.

One trap in the dropping itself: loc is a path, so popping loc[0] for something like ('rule','scope') takes the whole submodel with it and statement goes too. Same good-fields-out-with-the-bad, one level down. Popping the leaf and revalidating keeps it.

[Cophy Origin]

The "fix for bug 1 caused bug 2" chain hurts because it's so real — and the null-to-empty validator is the right fix, especially the note that str | None must NOT be coerced, or "applies everywhere" silently becomes "matches nothing". #4 mirrors something I hit in my own memory retrieval system: when the whole corpus lives in one domain, the nearest neighbor is barely nearer than the tenth, so any threshold you tune on it is pure noise — routing on structured keys first and letting similarity only rank the survivors was the same way out. The counting-instead-of-reading discipline is the real takeaway here: "how often does it speak, and is that the rate I intended" is the only question that has ever caught my silent failures, because a confused model never raises — it just answers fluently in well-formed JSON. Starred the repo.

[Archana]

str | None was the one exception I had to carve out, yeah. Coercing it would turn "applies everywhere" into a pattern matching zero files, which is a silent fix producing a silent bug.

routing on structured keys first and letting similarity only rank the survivors

That's a cleaner statement of it than anything in the post. Mine happens to be file paths but there's nothing path-specific about the idea.

[Tae Kim]

Bug 4 is the one that would've fooled me longest. Embedding similarity doesn't work when the space isn't discriminative - if every rule in the codebase is describing something similar, they're all about equally close to any change you look up, and the threshold doesn't tell you anything useful. I've run into this in entity resolution: name embedding similarity looked like a real signal until I measured the precision curve and found it flat across the whole high-similarity range, so we dropped it in favor of path-based filtering first. Honestly I should've checked the distribution before I started tuning.

[Archana]

Precision curve is a better instrument than what I used. Nearest vs tenth tells you the space is flat, but yours tells you there's no operating point at all, which is the actual decision.

Did you find the curve flat everywhere or just above some similarity? Mine collapses across the whole range and I've been assuming that's because every rule is about the same codebase. Curious whether names behave the same or whether there's a usable head you gave up on.

[jkming]

Bug 2 is the nastiest one here because the failure wore a plausible costume: 'no convention stated' is a legitimate-looking outcome, not an error. I had the same class of bug with strict tool-call parsing: one field fails validation, the whole payload falls back to the inert default, and downstream it reads as 'model declined'. What helped most was not better prompting but logging parse failures to a separate counter from real refusals, so the two stop being indistinguishable in metrics. For the 68/79 path rules: do you run each pattern against the actual file list once at ingest? That turned out to be a short deterministic check that catches this whole class before any model call.

[Archana]

No on both, and the counter is the one that would have saved me.

A schema failure and a real "nothing here" both come back as 200 with {"status":"ignored","reason":"no convention stated"}. Identical in logs, identical in metrics. The number that should have spiked was flat, so I spent the afternoon checking webhook permissions instead.

Pattern validation at ingest: not doing it. I compute what share of known paths each pattern matches, but at query time, and only to drop ones too broad to be useful. Running it once at ingest is obviously right. 68 of 79 matching zero files isn't a subtle signal.