Phase 7a — Getting Opinionated: Rules-Based Auto-Categorization (and a Seam for the AI Later)

My expense app finally has a point of view on what I'm spending money on. No AI yet — just honest keyword rules, a nullable column, and one interface that means I can bolt an LLM on later without ripping anything out. Here's the build, three "empty value" bugs that bit me, and the habits that kept it clean.

---

Index

1. Where we left off
2. The plan: rules first, AI behind the same door
3. Step 1 — A nullable column (and why nullable matters)
4. Step 2 — The migration: generate → review → apply
5. Step 3 — A dumb-but-working categorize()
6. Step 4 — Wiring it into create (with override precedence)
7. Step 5 — The seam: extracting behind a Categorizer interface
8. Step 6 — The UI loop: show, add, edit
9. 🐛 The war story: three ways "empty" lied to me
10. Thinking like an attacker
11. Learning shortcut vs. production
12. Key habits to keep
13. Next up: Phase 7b

---

Where we left off

Phase 6 gave me the receipts — date-range reports and CSV export. I ended that post with a promise:

Next up: Phase 7, where categories finally enter the schema and the app starts to get opinionated about what I'm spending on.

This is that. But it turned into a bigger beast than one post, so I'm splitting it:

• Phase 7a (this post): the schema, a rules-based categorizer, the interface seam, and the full UI loop.
• Phase 7b (next): the actual LLM — an LLMCategorizer that slots in behind the same interface, with caching and a rules fallback.

Doing rules first isn't a cop-out. It's the whole strategy.

---

The plan: rules first, AI behind the same door

The temptation with "AI categorization" is to reach straight for the API key. I didn't. Here's the order I actually built in, and why:

Step	What	Why this order
1	Nullable category column	The app needs somewhere to store a category before it can fill one
2	Rules categorize()	A working, free, offline fallback — and a baseline to test against
3	Extract behind an interface	So the LLM can slot in later without touching call sites
4	UI loop (show / add / edit)	Give the human final say, no matter how smart the auto-fill gets
5	(Phase 7b) LLM implementation	The risky external dependency goes in last, behind the seam

The principle: build the boundary before you plug in the flaky thing. Rules are boring and reliable. The LLM will be clever and occasionally down, slow, or wrong. If both live behind the same contract, swapping between them — or falling back — is a one-line change.

---

Step 1 — A nullable column (and why nullable matters)

One new line on the Expense model:

category: Mapped[str | None] = mapped_column(String(50), nullable=True)

The interesting bit is the | None / nullable=True. Every expense already in my database was created before this column existed. If I made it nullable=False, the migration would try to force a value into all those existing rows and fail — or demand a default I don't actually want.

Nullable is honest: old rows are genuinely uncategorized. "No category" is a real state, not an error. And it maps perfectly onto the rules engine returning "I don't know" — more on that below.

New syntax I picked up here:

Syntax	Meaning
Mapped[str]	A required string
`Mapped[str \	None]`
String(50)	Length-bounded, like my description at 255

---

Step 2 — The migration: generate → review → apply

Three deliberate moves, not one:

# 1. Generate (does NOT touch the DB — just writes a script)
alembic revision --autogenerate -m "add category column to expenses"

# 2. Review — open the file, confirm it does EXACTLY one thing
#    upgrade() -> add_column('expenses', 'category', String(50), nullable=True)
#    downgrade() -> drop_column('expenses', 'category')   # reversible!

# 3. Apply (this one DOES modify the database)
alembic upgrade head

Autogenerate is good, not infallible. It occasionally invents spurious type tweaks, and SQLite has its quirks with certain operations. So I read the file every single time before applying — confirming both that upgrade() does only what I asked and that downgrade() cleanly reverses it.

Backup habit: SQLite is a single file, so a backup is a cp:

cp expenses.db expenses.backup.db
alembic upgrade head

Verified the column actually landed:

python -c "import sqlite3; print(sqlite3.connect('expenses.db').execute('PRAGMA table_info(expenses)').fetchall())"
# ...a 'category' row, VARCHAR(50), nullable. Existing rows: None. 

(Aside: my .gitignore had *.db but not *.bak. Named the backup expenses.backup.db so the existing rule caught it, then added *.bak anyway as housekeeping.)

---

Step 3 — A dumb-but-working categorize()

No AI. Just keywords:

CATEGORY_RULES = {
    "Food": ["swiggy", "zomato", "restaurant", "cafe", "coffee", "pizza",
             "lunch", "dinner", "breakfast", "meal", "food", "grocery", "canteen"],
    "Transport": ["uber", "ola", "cab", "fuel", "petrol", "metro"],
    "Shopping": ["amazon", "flipkart", "myntra", "mall"],
    "Utilities": ["electricity", "water", "gas", "internet", "wifi", "recharge"],
    "Entertainment": ["netflix", "spotify", "movie", "bookmyshow"],
}

def categorize(description: str) -> str | None:
    text = description.lower()
    for category, keywords in CATEGORY_RULES.items():
        if any(keyword in text for keyword in keywords):
            return category
    return None

The concepts I actually learned writing this:

Concept	What it does
.lower()	Case-insensitive matching — "UBER", "Uber", "uber" all hit
keyword in text	Substring test — "uber" in "uber to airport" is True
any(... for ...)	Returns True on the first match, then stops
return None	No match = no guess — maps onto the nullable column

That last line is the design decision I'm proudest of. When the rules don't recognize something, they don't guess wildly — they return None, and the expense stays honestly uncategorized. Tested in isolation before wiring anything:

python -c "from main import categorize; print(categorize('Uber to airport'), '|', categorize('Swiggy dinner'), '|', categorize('Random mystery charge'))"
# Transport | Food | None

---

Step 4 — Wiring it into create (with override precedence)

The rule: if the user explicitly sends a category, respect it. Only auto-fill when they don't.

data = payload.model_dump()
if data.get("category") is None:
    data["category"] = categorize(data["description"])
expense = Expense(**data, user_id=current_user.id)

data.get("category") returns None if the key is missing or sent as null — my "did the user leave it blank?" test. Blank → auto-categorize. Provided → keep theirs.

This is also where the first landmine was waiting (see the war story). But conceptually: the client now has three honest options — omit it (auto), send a value (override), or send null (explicitly uncategorized).

---

Step 5 — The seam: extracting behind a Categorizer interface

This is the step that makes Phase 7b painless. Right now categorize() lives inside a web route — core domain logic welded to the HTTP layer. That's a smell, and it blocks the LLM work. So I pulled it into its own module behind a contract:

# backend/categorization.py
from typing import Protocol

class Categorizer(Protocol):
    """The contract every categorizer must satisfy."""
    def categorize(self, description: str) -> str | None:
        ...

class RulesCategorizer:
    """Keyword-rules implementation of the Categorizer contract."""
    def __init__(self, rules: dict[str, list[str]] = CATEGORY_RULES):
        self._rules = rules

    def categorize(self, description: str) -> str | None:
        text = description.lower()
        for category, keywords in self._rules.items():
            if any(keyword in text for keyword in keywords):
                return category
        return None

# The single instance the rest of the app imports.
default_categorizer: Categorizer = RulesCategorizer()

Three ideas clicked here:

• A Protocol is a contract. "Anything called a Categorizer must have categorize(description) -> str | None." A class satisfies it just by having that method — no inheritance (structural typing). My future LLMCategorizer will satisfy the same contract for free.
• A class carries state. A bare function can't remember things. The LLM version will need to — an API client, and eventually a per-merchant cache. So a class, with self._rules on the instance.
• One entry point. Everything imports default_categorizer and calls .categorize(...), blissfully ignorant of how it works. Swapping rules → LLM becomes a one-line change in one place.

Then main.py just does:

from categorization import default_categorizer
# ...
data["category"] = default_categorizer.categorize(data["description"])

Refactor discipline: I shipped the extraction as a pure, behavior-neutral commit — same inputs, same outputs — and only then, in a separate commit, widened the Food keywords. Never mix a refactor with a behavior change in the same commit. When something breaks later, you want git bisect to land on one or the other, not a tangle of both.

---

Step 6 — The UI loop: show, add, edit

Auto-categorization is useless if the human can't see or override it. Three small React slices:

Show — a category pill per row, with a loud fallback for the ones the rules missed:

{expense.category ? (
  <span className="mt-1 self-start rounded-full bg-slate-100 px-2 py-0.5 text-xs font-medium text-slate-600">
    {expense.category}
  </span>
) : (
  <span className="mt-1 self-start rounded-full bg-red-100 border border-dashed border-slate-300 px-2 py-0.5 text-xs italic text-slate-400">
    Uncategorized
  </span>
)}

I made the "Uncategorized" pill red on purpose — it's a gentle nudge that says "this one still needs you."

Add — an optional category input on the create form, plus defaulting the date to today.

Edit — the same override on the inline edit form, so those red flags are actionable: click, type a category (or clear it), save.

Both forms share one trick that turned out to be load-bearing — which brings me to the bugs.

---

🐛 The war story: three ways "empty" lied to me

Phase 6 had one signature bug (a CSV that only wrote its last row, traced to an indentation slip). Phase 7a had a theme: every single bug this session was about the difference between null, empty string, omitted, and the wrong day. "Nothing" is not one thing. It's at least four, and they don't behave the same.

Gotcha 1 — Nullable ≠ optional (the silent 422)

I declared the create field like this and thought I was done:

# BROKEN — required, but allowed to be null
category: str | None

Omit category in the request body → 422 Field required. In Pydantic v2, | None only says "null is an allowed value." What makes a field skippable is a default.

# FIXED — truly optional
category: str | None = Field(default=None, max_length=50)

| None = "null is allowed." = None = "you can omit it." You need both for "optional and nullable." Two different questions; I'd only answered one.

Gotcha 2 — Empty string sneaks past the categorizer

My backend auto-fills only when category is None:

if data.get("category") is None:
    data["category"] = categorize(data["description"])

But an HTML text input never gives you None. Empty it, and it hands you "". So the naive form body...

// BROKEN — always sends a string, even when blank
body: JSON.stringify({ description, amount, spent_on: spentOn, category })

...sends category: "". And "" is None → False. So the backend treats an empty box as a deliberate override, skips categorize() entirely, and every blank-form expense silently lands uncategorized. My shiny new auto-categorization looked completely broken — because of a frontend empty string.

// FIXED — blank (or whitespace) becomes real null
body: JSON.stringify({
  description, amount, spent_on: spentOn,
  category: category.trim() || null,
})

"".trim() is falsy → || null kicks in → the backend sees None → auto-categorizes. .trim() also stops someone "overriding" with three spaces. I used the exact same line on the edit form, where clearing the box is how you send a category back to null.

Gotcha 3 — "Today" was yesterday (a UTC off-by-one)

I wanted the add form to default the date to today. The one-liner everyone reaches for:

// BROKEN — this is today in UTC
const today = new Date().toISOString().slice(0, 10);

toISOString() returns UTC. I'm in IST (UTC+5:30). So between midnight and ~5:30 AM my time, UTC is still on yesterday's date, and the form would confidently pre-fill the wrong day. A bug that only appears before breakfast is the worst kind.

// FIXED — build the string from LOCAL parts
function getTodayString() {
  const now = new Date();
  const year = now.getFullYear();
  const month = String(now.getMonth() + 1).padStart(2, "0"); // 0-indexed!
  const day = String(now.getDate()).padStart(2, "0");
  return `${year}-${month}-${day}`;
}

Same getMonth() + 1 / padStart pattern I'd already written for the CSV filename in Phase 6 — reused, not reinvented.

The through-line: null, empty string, omitted, and UTC-vs-local are four different flavors of "nothing/edge," and conflating any two of them is a bug. Now I check which kind of empty I'm actually dealing with.

---

Thinking like an attacker

The habit from earlier phases that keeps paying off — actively try to break my own feature:

• Whitespace-only category? .trim() || null → treated as blank. ✅
• value={null} on a controlled input? React silently switches the input to uncontrolled and warns. Coalesced the edit prefill with expense.category || "". ✅
• Editing description shouldn't nuke the category. The update uses model_dump(exclude_unset=True), so omitted fields are left untouched. Sending only {"description": "..."} preserves the category. ✅
• The substring trap I haven't fixed yet: "gas" is a keyword under Utilities — and it's a substring of "Ve*gas*". So "Vegas trip" categorizes as Utilities. This is the fundamental ceiling of keyword rules, and it's exactly the wall the LLM in Phase 7b is meant to break through. Logged, not hidden.

---

Learning shortcut vs. production

I did (learning)	Production would
Hardcoded CATEGORY_RULES dict	Config/DB-driven, editable without a deploy
Free-text category strings	A constrained enum or a categories table
cp expenses.db expenses.backup.db	Automated, tested DB backups + a restore drill
No re-categorize on description edit	Probably offer "re-run categorization" as an explicit action
Substring matching	Word-boundary matching, or skip straight to the model

None of these are wrong for where I am. They're just known — written down, not pretended away.

---

Key habits to keep

• Build the seam before the risky dependency. The Categorizer interface exists so the LLM is a plug-in, not a rewrite.
• One pure refactor per commit. Never bundle "moved code" with "changed behavior."
• Review every migration before applying it. Autogenerate is a draft, not a decree.
• Know which "empty" you mean. null ≠ "" ≠ omitted ≠ UTC-today. Four bugs wearing one costume.
• Give the human the final say. However good auto-fill gets, the override is the point.
• Log the limitations you're not fixing yet. "Vegas → Utilities" is a bug I chose to defer, not one I missed.

---

Next up: Phase 7b

The column holds a category. The rules fill it. The interface is ready. The human can override it everywhere.

Now the fun part: an LLMCategorizer that satisfies the exact same Categorizer contract — so main.py never knows the difference — with a config toggle, per-merchant caching so I'm not paying for the same "Starbucks" twice, and a rules fallback for when the model is slow, down, or just wrong. The whole point of the seam was to make that a drop-in. Phase 7b is where I find out if I actually earned it.

See you in the next one. — silentcarry