Phase 7b — Bolting On the AI: An LLMCategorizer Behind the Same Door (with Caching, a Fallback, and Six Model Names)

The seam I built in Phase 7a finally earned its keep.

I dropped a Gemini-backed categorizer in behind the exact same interface — main.py never noticed — with a config toggle, description-level caching so I'm not paying for the same "Starbucks" twice, and a rules fallback for when the model is down, slow, or wrong.

Here's the build, the six model names it took to get one working call, and the silent except block that hid a completely broken LLM for an embarrassingly long time.

Index

• Where we left off
• The plan: everything behind the same contract
• Step 1 — The toggle: a factory and an env var
• Step 2 — The stub: prove the wiring before the call
• Step 3 — The real call: constrained prompt, validation, fallback
• Step 4 — Caching: don't pay for the same Starbucks twice
• One loose end: the CSV finally learned about categories
• The war story: six model names and a comma
• Thinking like an attacker
• Learning shortcut vs. production
• Key habits to keep
• Next up: Phase 8

Where we left off

Phase 7a ended with a promise. I had a nullable category column, a dumb-but-honest rules engine, and — the part I actually cared about — a Categorizer Protocol with the rules implementation hiding behind it. I closed that post like this:

Now the fun part: an LLMCategorizer that satisfies the exact same Categorizer contract — so main.py never knows the difference — with a config toggle, per-merchant caching, and a rules fallback for when the model is slow, down, or just wrong. Phase 7b is where I find out if I actually earned it.

This is that. Spoiler: the seam held. The account quota is a different story.

The plan: everything behind the same contract

The temptation, again, was to reach straight for the API call and stuff it into categorize(). I didn't. I built outward from the safe end, one behavior-neutral step at a time:

Step	What	Why this order
1	A CATEGORIZER toggle + factory	One place decides rules-vs-LLM, defaults to rules — the app behaves identically until I flip it
2	An LLMCategorizer stub that returns None	Prove the toggle + wiring end-to-end with zero risk and zero cost
3	The real Gemini call, with validation + fallback	The risky external dependency goes in last, behind the seam
4	Description-level caching	Stop paying for the same description twice

The principle from 7a still drives it: build the boundary, then plug in the flaky thing. Every step until Step 3 leaves the app doing exactly what it did before. That's not caution for its own sake — it means when something finally breaks, I know it broke in the one thing I just changed.

Step 1 — The toggle: a factory and an env var

Right now the last line of categorization.py hard-codes the choice:

default_categorizer: Categorizer = RulesCategorizer()

That's fine for one implementation. But if I later scatter if CATEGORIZER == "llm" checks across main.py, I've made a mess. So I gave the decision a single owner — a factory: a function whose only job is to construct and hand back the right object based on config.

import os
from dotenv import load_dotenv

load_dotenv()  # read .env before we read any env vars


def get_categorizer() -> Categorizer:
    """Pick the categorizer based on the CATEGORIZER env var. Defaults to rules."""
    kind = os.getenv("CATEGORIZER", "rules").strip().lower()
    if kind == "llm":
        return LLMCategorizer()
    return RulesCategorizer()


default_categorizer: Categorizer = get_categorizer()

Two ideas clicked here:

Idea	Why it matters
os.getenv("X", "rules")	This is an optional setting with a safe default — same shape as my DATABASE_URL read. Contrast with os.environ["X"], which I use for required secrets so the app fails loud if they're missing
Keep the name default_categorizer	main.py still does from categorization import default_categorizer. Rebinding it to get_categorizer()'s result means not one line of main.py changes — the call site stays blissfully ignorant

Env unset → "rules" → RulesCategorizer. The app behaves exactly like Phase 7a. Behavior-neutral seam, done.

Step 2 — The stub: prove the wiring before the call

Before I wrote a single line of Gemini code, I added a stub — a class with the right shape (it satisfies the Categorizer Protocol) but no real logic:

class LLMCategorizer:
    """LLM-backed categorizer. STUB for now: returns None until the real call is wired in."""

    def categorize(self, description: str) -> str | None:
        return None

Why bother stubbing when I could just write the real thing? Isolation. If I drop a live API call straight in and something misbehaves, I can't tell whether it's the wiring (env toggle → factory → app) or the call itself. The stub lets me flip CATEGORIZER=llm, restart, add a "swiggy" expense, and confirm it comes back Uncategorized — which proves the llm path is live and reaching my class. Zero cost, zero network.

And returning None is the correct stub behavior, not a placeholder shrug: my whole design treats None as "no category → NULL → Uncategorized." So a stub that categorizes nothing degrades to the exact same UX as the rules engine finding no match. Nothing breaks; the app just stops guessing while the stub is in.

Step 3 — The real call: constrained prompt, validation, fallback

This is the step the whole phase was building toward. Three non-negotiables went in together.

1. A constrained prompt. I don't let the model free-associate. It picks exactly one of my five existing categories, or the literal word None:

class LLMCategorizer:
    """LLM-backed categorizer using Google Gemini, constrained to the rules categories."""

    def __init__(self):
        api_key = os.environ["GEMINI_API_KEY"]  # fail loud if llm is on but no key
        self._client = genai.Client(api_key=api_key)
        self._allowed = set(CATEGORY_RULES.keys())
        self._fallback = RulesCategorizer()

    def categorize(self, description: str) -> str | None:
        categories = ", ".join(sorted(self._allowed))
        prompt = (
            "You are an expense categorizer. "
            f"Choose exactly ONE category for the expense from this list: {categories}. "
            "Reply with only the category name and nothing else. "
            "If it does not clearly fit any category, reply with the single word: None.\n"
            f"Expense description: {description}"
        )
        try:
            response = self._client.models.generate_content(
                model=GEMINI_MODEL,
                contents=prompt,
            )
            answer = (response.text or "").strip().strip(".").strip()
            for category in self._allowed:
                if answer.lower() == category.lower():
                    return category
            return None
        except Exception as e:
            print(f"[categorizer] LLM ERROR -> falling back to rules: {e!r}")
            return self._fallback.categorize(description)

Why constrain it to five categories instead of letting it invent its own? Because free-form output is data poison. Left alone, an LLM will happily return "Dining," "Restaurants," "Food & Drink," and "Meals" for four expenses that are all Food — and my future "spend by category" report splinters into meaningless buckets. A controlled vocabulary is what makes category-based reporting possible, and it keeps the LLM apples-to-apples comparable with the rules engine. Same five-word language, clean swap.

2. Never trust the output. LLMs drift: "Food.", " food", "**Food**". So I normalize (strip whitespace and a trailing period, compare case-insensitively) and only accept an answer that exactly matches one of my five. Anything else becomes None — the model declining to guess, which is the behavior I want.

3. Wrap the call and fall back to rules. Any failure — no network, quota exceeded, timeout, garbage response — must not break expense creation. On any exception I quietly return the rules answer.

Concept	What it does
Client built in __init__	Auth + connection set up once per instance, reused for every call — the factory returns a single LLMCategorizer(), so it's built once
os.environ["GEMINI_API_KEY"]	Fail loud: if llm is on but the key's missing, refuse to start. Only read when the factory actually builds the LLM path, so rules mode never needs it
self._allowed = set(CATEGORY_RULES.keys())	One source of truth — add a sixth rules category and the LLM's allowed list updates for free
except Exception → rules	Normally a code smell. Here it's deliberate: categorization is a non-critical enhancement that must never block the core POST

A bare except Exception catching everything is usually lazy. This one is a conscious trade-off — I'd rather an LLM outage silently degrade to rules than ever take down expense creation. But "silent" turned out to be the word that bit me. More on that in the war story.

Step 4 — Caching: don't pay for the same Starbucks twice

Every categorize() call hits Gemini, even for a description I've seen ten times. On a free tier, that's wasted latency and wasted quota. The fix is memoization — remember what an input mapped to, and on a repeat, return the stored answer with no API call:

def __init__(self):
    ...
    self._cache: dict[str, str | None] = {}

def categorize(self, description: str) -> str | None:
    key = description.strip().lower()
    if key in self._cache:
        return self._cache[key]

    # ... build prompt, call Gemini ...
    try:
        # ... get validated result ...
        self._cache[key] = result   # cache ONLY successful LLM outcomes
        return result
    except Exception as e:
        print(f"[categorizer] LLM ERROR -> falling back to rules: {e!r}")
        return self._fallback.categorize(description)  # never cached

Concept	What it does
A dict as cache	Key = normalized description, value = resolved category
key = description.strip().lower()	"Uber", "uber ", "UBER" all hit the same entry — trivial differences shouldn't cost a call
In-memory	Lives on the instance, clears on restart. Acceptable for now; a DB/Redis cache is a later step
Cache only on success	The subtle one — see below

That last row is the one worth internalizing. The cache write lives inside the try, on the success path only. If the API errors and I fall back to rules, I must not store that. Picture Gemini down for a minute: I'd cache the fallback answer for "Starbucks" and keep serving it even after Gemini recovers. A poisoned cache outlives the outage. So fallbacks are never cached — ever.

The proof it worked, straight from the logs — I typed kFc, watched it call Gemini once, then added kFc again:

[categorizer] key='kfc' cache=['kfc']
[categorizer] CACHE HIT
INSERT INTO expenses ... (1.0, 'kFc', 'Food', '2026-08-09', 2)

Three things confirmed in four lines: the LLM genuinely ran (KFC isn't a rules keyword, yet it returned Food), normalization works (kFc → 'kfc'), and the repeat was served from cache with zero API calls.

One loose end: the CSV finally learned about categories

Small but real: my Phase 6 CSV export still wrote ["id", "description", "amount", "spent_on"] — it never knew the category column existed. A CSV export has two places that must agree: the header row and the data rows, and they're positional. Add a column to one, add it at the same index in the other, or every cell below shifts and the file is silently wrong:

writer.writerow(["id", "description", "category", "amount", "spent_on"])
# ...
writer.writerow([e.id, e.description, e.category, e.amount, e.spent_on])

A None category writes as a blank cell, which is the honest representation of "uncategorized." Now the category flows all the way from the model to the download.

The war story: six model names and a comma

Phase 7a's bugs were all about the four flavors of "nothing" — null vs "" vs omitted vs UTC-today. Phase 7b had its own theme: everything that isn't your code lying to you — config, the reload watcher, the environment, and a swallowed exception all conspired before I'd made a single real API call successfully.

Gotcha 1 — the comma that lived in my .env

I flipped the toggle on, restarted, added a "swiggy" expense... and it still came back Food. Was .env not being read? I stopped guessing and printed the value with !r:

print(f"[categorizer] CATEGORIZER={os.getenv('CATEGORIZER')!r}")
# [categorizer] CATEGORIZER='llm, '

There it is: 'llm, ' — a trailing comma and a space, baked into the value. In a .env file, everything after the = is literal, so my stray punctuation became part of the string. "llm, " == "llm" is False, the factory fell through to rules, and I got Food.

The lesson isn't "don't type commas." It's !r earns its keep. Without repr(), the log would've read CATEGORIZER=llm and I'd have sworn it was correct — the quotes are the only reason I saw the hidden ,. I later hardened the read with .strip().lower() so a stray space can't do this again (a comma still would — env values are exact).

Gotcha 2 — --reload doesn't watch .env

Before I found the comma, I'd been editing .env and waiting for uvicorn to pick it up. It never did. --reload only re-imports when a .py file changes. default_categorizer is bound once, at import, when get_categorizer() runs — so editing .env and saving does nothing until a true cold restart (Ctrl+C, then relaunch). The running process was holding the categorizer it built at its last real startup. Now "changed .env?" always means "cold restart," not "save and hope."

(VS Code chimed in here too, offering to load .env into the terminal via python.terminal.useEnvFile. Irrelevant — my code calls load_dotenv() and reads the file itself at runtime. Enabling it wouldn't have fixed anything. Dismissed.)

Gotcha 3 — "No module named 'google'" — the environment, not the folder

Installed google-genai, restarted, and: No module named 'google'. My first instinct was "wrong folder." Wrong instinct. pip install doesn't install into whatever directory you're standing in — it installs into whichever virtualenv is activated, landing in that venv's site-packages. The error meant the Python running uvicorn wasn't the same environment where the package landed. A reinstall in a properly-activated terminal fixed it instantly.

The mental model I locked in: a venv is a sealed box of packages; activating it points this terminal's pip/python at that box. No module named X = box mismatch. The one-line diagnostic is pip show <pkg> run in the terminal that runs the server — if it's not found there, the boxes don't match. (And a running process won't see a newly installed package until you restart it — same cold-restart discipline as Gotcha 2.)

Gotcha 4 — the silent except that hid a dead LLM

This is the big one, and it's the reason I now distrust quiet error handling. My fallback was working too well. Every test word I'd been using — uber, netflix, lunch — is also a rules keyword. So when the Gemini call failed and hit except → rules, I still got the "right" category and had no idea the LLM had never actually run. The safety net was masking a completely broken call.

The fix was to make the except talk instead of swallow:

except Exception as e:
    print(f"[categorizer] LLM ERROR -> falling back to rules: {e!r}")
    return self._fallback.categorize(description)

And the way to prove the LLM was genuinely working was to test with a word the rules engine doesn't know — Starbucks, KFC. If those come back categorized, only the model could have done it. Rules would've said Uncategorized.

Gotcha 5 — six model names

Once the except could speak, it immediately told me why nothing worked — a parade of failures across model names:

Model I tried	What the API said
gemini-2.5-flash	404 — "no longer available to new users"
gemini-2.5-flash-lite	404 — same gating for new keys
gemini-2.0-flash	429 RESOURCE_EXHAUSTED, limit: 0 — free tier is zero on this model
gemini-flash-latest	Finally — a clean generate_content

The 429 was the sneaky one: limit: 0 doesn't mean "you used it up," it means "your key was never granted free-tier allowance for this model." That's an account/billing fact, not a code bug. The move that ended the guessing was to stop assuming model names and ask the key what it can do — client.models.list() prints every model available to you. Discover capabilities; don't hard-code hope.

I settled on the gemini-flash-latest alias. It's a moving target (it'll drift to newer models over time, which is bad for reproducibility), but a working alias beats a pinned name that 404s. Pinning a specific stable model is a known, deferred trade-off.

The through-line: before I'd made one successful API call, four different non-code layers had already lied to me — config, the reload watcher, the environment, and my own swallowed exception. The bug is rarely where you first look, and a silent except guarantees you'll look everywhere else first.

Thinking like an attacker

Same habit as always — actively try to break my own feature before reality does:

• LLM returns "Food." with a period? .strip(".") + exact-match validation → still maps to Food.
• LLM returns something off-menu like "Groceries"? Not in my five → None → Uncategorized. It can't invent a category and poison my data.
• API is down mid-request? except → rules fallback → the expense still saves. Proven live when the 429 hit: the row saved as Uncategorized, the app didn't crash, the log named the reason.
• Same "Starbucks" typed ten times? One API call, nine cache hits.
• Gemini flaps during an outage? Fallback answers are never cached, so a recovered model isn't shadowed by a stale rules guess.

Learning shortcut vs. production

I did (learning)	Production would
In-memory dict cache	Persistent cache (DB/Redis) that survives restarts, with TTL
gemini-flash-latest alias	A pinned model version for reproducible behavior + cost
Free-tier key, limit: 0 and all	Billing enabled (flash-lite is fractions of a cent) or a managed quota
except Exception catch-all	Catch specific API errors; alert on sustained fallback, not just log it
print(...) for the error	Structured logging with levels and a metric on fallback rate

None of these are wrong for where I am. They're known — written down, not pretended away.

Key habits to keep

• Build the seam before the risky dependency. The Categorizer Protocol meant the LLM was a drop-in — main.py never changed. Phase 7a earned this; Phase 7b spent it.
• Stub before you call. Prove the wiring with a None-returning placeholder before you introduce the network, the auth, and the quota all at once.
• Never let except be silent. A swallowed error and a convenient fallback will hide a totally broken feature. Make it log.
• Test with inputs your fallback can't fake. If rules and LLM would both say "Food," you've proven nothing. Use a word only the LLM could know.
• Ask the system what it can do. models.list() beat four rounds of guessing model names.
• !r when you print an env value. The quotes are how you catch the hidden comma.
• Cache success, never failure. A poisoned cache outlives the outage that caused it.

Next up: Phase 8

The column holds a category. Rules fill it offline. The LLM fills it when it's on — behind the same door, validated, cached, and with a fallback that's now loud about failing. The human can still override all of it.

What's nagging me: the cache dies on every restart, the model name is a drifting alias, and my free tier can't actually afford to be the default. So Phase 8 is probably where I make the LLM path sustainable — a persistent cache, a pinned model, and a real answer to the quota question — or where I finally let categories drive the thing I've been building toward all along: reporting that actually means something.

The seam held. Turns out the hard part was never the code behind it.

See you in the next one.

— silentcarry

Comments

[Alex Shev]

I like the "same door" framing. The AI part gets much less risky when it is hidden behind the same contract as the deterministic categorizer. The fallback also matters more than people admit: it forces the system to define what "good enough to continue" means when the model answer is slow, ambiguous, or missing.