DEV Community: Archana

Five bugs in my LLM app that never threw an error

Archana — Sun, 30 Aug 2026 02:31:58 +0000

I spent two and a half weeks building an agent that reads a project's code review history and remembers what it decided. 86,321 comments from pandas, distilled into 298 conventions, served out of CockroachDB behind a Lambda.

Every serious bug I hit had the same shape. Nothing raised. Tests passed. The demo worked. And somewhere in the middle, a component was doing absolutely nothing.

Here are five, with the code.

1. `bool("false")` is `True`

The whole point of the system is that it refuses when it does not know. The answering prompt returns JSON with an answered field.

result = await chat.complete_json(messages)
if result.get("answered"):
    return result["answer"]

A model that writes "answered": "false" instead of "answered": false turns a refusal into an answer, because a non-empty string is truthy. The one behaviour the project exists to guarantee, defeated by a quotation mark.

Now every model response goes through a Pydantic model that fails towards doing nothing:

class AnswerOutput(ModelOutput):
    answered: bool = False   # coerces "false" -> False
    answer: str = ""

An answer becomes a refusal. A drafted rule becomes unusable. A contradiction verdict becomes "compatible", which retires nothing. A garbled response is not evidence for writing to memory.

2. A `null` discarded whole responses, six times out of six

The fix for bug 1 caused bug 2, which is funnier than it was at the time.

The prompt that turns a maintainer's comment into a rule says:

"rationale": "why, one sentence, only if they gave a reason"

A maintainer who states a convention flatly gets null back, which is correct JSON for "no reason given". And:

class DraftedRule(BaseModel):
    statement: str = ""
    rationale: str = ""   # null is not a str

Pydantic rejects null for a plain str. Validation is all or nothing. So the entire response was thrown away and replaced with the inert default, which downstream reads as the model having declined.

A real maintainer correction became {"status":"ignored","reason":"no convention stated"}. Six reproductions, six failures, on a payload that was otherwise exactly what I asked for.

class ModelOutput(BaseModel):
    @field_validator("*", mode="before")
    @classmethod
    def _null_text_is_empty(cls, value, info):
        if value is not None:
            return value
        field = cls.model_fields.get(info.field_name)
        return "" if field is not None and field.annotation is str else value

A null where a string was expected means empty, not malformed. Note the field.annotation is str check: scope_pattern: str | None genuinely means None, and coercing that one would turn "applies everywhere" into a pattern matching nothing.

3. 68 of my 79 path rules matched zero files

Each rule stores a pattern for which files it applies to. I matched with fnmatch.

fnmatch("pandas/core/frame.py", "pandas/core/%")   # False

The patterns were SQL LIKE, not globs. pandas/tests/%, pandas/core/%, pandas/tests/%/conftest.py. Nothing asked the extraction prompt for that and nothing documented it. The model just wrote it, presumably because the rules were destined for a database.

% is not an fnmatch wildcard. So 68 of 79 anchored rules silently matched nothing, and the agent ran entirely on the two rules that happened to be written pandas/**/*.py, which cover 63% of the repository.

It looked like it was working the entire time. The fix is one line, and the bug was invisible for a week:

cleaned = pattern.strip().strip("`").lstrip("./").replace("%", "*")

4. The similarity score I was tuning had no signal in it

The agent comments on pull requests, choosing which conventions apply by embedding a description of the change and taking the nearest rules.

It commented on 38 of 40. I assumed a bad threshold and started tuning. At 1.05 it spoke on 95% of pull requests; at 0.95, on 18%.

Then I measured the distribution instead. For 25 real pull requests, distance to the nearest rule versus the tenth nearest:

query phrasing	median nearest	median spread, 1st to 10th
title only	0.954	0.097
title plus directories	0.947	0.100
paths and counts	0.958	0.105
written as a question	0.933	0.095

The nearest rule is not meaningfully nearer than the tenth, under any phrasing. Every rule is a general statement about the same codebase, so every rule is similar to any description of a change to it.

There was nothing to threshold. Retrieval now runs on path matching, which is a fact, and similarity only orders what the paths already admitted.

5. GitHub's `author_association` describes the present

I filtered to maintainer comments using GitHub's author_association field.

It reports whether someone has write access now, not when they wrote the comment. One reviewer wrote 74,077 comments on pandas between 2012 and 2025, more than twice anyone else. He has since left the org, so all 74,077 come back as CONTRIBUTOR.

Filtering on that field discarded about a third of the corpus, including the most experienced reviewers the project ever had. No error, no warning, just a smaller number that looked plausible.

The pattern

None of these raised. Four of the five I found by counting something rather than by reading code: how often does it speak, how many rules match anything, how far apart are the distances really.

A library used wrong fails loudly. A model used wrong returns confident, well-formatted output that is quietly detached from what you meant.

So pick a number that should hold if the system works, and go and look at it. Not "does it return results" but "how often does it speak, and is that the rate I intended". Not "did it cite something" but "do the citations resolve".

Code is at github.com/pyarchana/precedent, MIT.

Google's Workspace Intelligence Might Actually Fix the Thing That Bugs Me Most About AI Tools

Archana — Thu, 23 Apr 2026 03:00:57 +0000

This is a submission for the Google Cloud NEXT Writing Challenge

Okay so I've been watching the Google Cloud NEXT '26 announcements drop today and there's a lot. TPUs, agentic platforms, a $750M partner fund. But the thing that actually made me stop scrolling was Workspace Intelligence :)

Not because it's the most technically impressive thing on the list. It's not. But it might be the one that changes how I (and probably you) actually work day to day.

Let me explain.

The thing that's always bothered me about AI in Google Docs

Every time I open Docs and try to use Gemini to help me write something, I spend the first few minutes basically re-introducing myself. Here's the project. Here's the tone we're going for. Here's who the audience is. Here's the doc we drafted last month that this is building on.

It's exhausting :) The AI is smart but it has no idea who I am or what I'm working on. Every prompt is a cold start.

That's exactly what Workspace Intelligence is supposed to fix.

What it actually is

Google is calling it a "semantic layer" across all your Workspace apps. Gmail, Docs, Sheets, Slides, Drive, Chat, Calendar. The idea is that instead of each app being its own isolated thing, there's now a shared understanding of your work running underneath all of them :)

It learns what projects you're actively working on, who you're collaborating with, what your writing sounds like, and what's actually on your plate right now. Then every time you use Gemini anywhere in Workspace, it's pulling from that context instead of starting fresh.

Practically speaking, here's what that looks like across the apps:

Gmail gets an AI Inbox that doesn't just summarize. It suggests what you should actually do, and drafts replies that sound like you, not like a generic assistant :)

Google Chat gets an "Ask Gemini" mode that Google is framing as a "unified command line for your work." You say what you need, it goes off and does it across Workspace. Schedules the meeting, pulls the file, writes the brief.

Docs can now edit multiple images at once for visual consistency, build infographics from your own business data, and handle comment threads. Including making edits to the doc based on the feedback in those comments :)

Slides can build a full deck from a single prompt, and it'll actually use your company's real templates and visual styles. Not a generic blue gradient.

Sheets lets you build and edit spreadsheets by just... talking to it. And it can pull data from your emails, files, and chat history to populate things :)

Drive is getting "Projects," a smarter way to organize everything around active work rather than just being a folder system that you dig through.

The bit I find most interesting

There's a line in Google's announcement that stood out to me: Workspace Intelligence learns your "unique work style, voice, and formatting preferences" so outputs sound "authentically like you."

I'm genuinely curious how well that works in practice :) Voice matching is one of those things that sounds great in a keynote and then either impresses you or makes you cringe when you see the actual output. I haven't been able to test it yet, but it's the thing I'm most excited (and slightly skeptical) about.

What about the "let AI read all my stuff" concern?

Yeah, this is the obvious question. Google is essentially asking you to let a system reason across your email, your calendar, your chat history, and your files. That's a lot of trust.

Their answer: customer data isn't used for ads, isn't reviewed by humans, and isn't used to train models outside Workspace without your permission :) Admins can control which data sources it touches, so if your org doesn't want it reading Gmail, they can turn that off. Client-side encryption and sovereign data controls (US and EU, with more coming) are part of the rollout too.

Whether that's enough will depend on your org's risk appetite. But at least they're not burying it in the fine print.

Where I land on this

I think the individual features are nice, but the real bet here is the compounding effect :) The more Workspace Intelligence understands your context, the more useful every individual tool becomes. That's a different kind of value than "we added a summarize button."

Google's basically trying to turn Workspace from a collection of apps you switch between into something that actually understands what you're trying to get done. Whether that plays out the way they're describing it, or whether it turns into a very well-marketed context window, is something we'll only know once people start using it for real.

But I'm paying attention. And honestly? I think this one's worth your time to follow :)

Are you using Google Workspace day to day? Curious whether this kind of persistent context layer actually sounds useful to you, or if it feels like AI for AI's sake. Let me know below :)