Each Fix to My Classification Prompt Exposed the Next Problem

This comes from my first n8n project with an LLM in it: an intake classifier for an automation agency. It's a practice brief, so the agency and the client are fictional, and I built it with an AI collaborator. The workflow and my full process notes are in the GitHub repo.

A client fills in a form, Gemini reads the project description through a Basic LLM Chain with a structured output parser, and the submission gets sorted into a category that decides which emails go out and what gets logged to Google Sheets. This post is only about the classification prompt, and about a handful of test descriptions that kept sending me back into it.

Where it started

Three of the five categories are real project types: simple_automation, ai_workflow and custom_integration. The other two, ambiguous and invalid, are for submissions that don't sort cleanly. I'm leaving those for another post.

My first definition of simple_automation allowed exactly one action off a trigger. That seemed reasonable until I tested it with a description I'd picked as the easiest case:

when a client submits the form on my website, notify me and log it to a sheet

It came back ambiguous. Notifying me and logging to a sheet is two actions, and my definition only allowed one.

Problem 1: I'd defined the category by counting

Raising the limit to two actions would have been a patch. The real problem was that counting actions doesn't separate the three categories at all. What separates them is whether there's conditional logic, whether data is actually transformed, and whether AI reasoning is involved. I rewrote all three definitions around those, and the simple_automation one now says so directly:

simple_automation: One or more actions fire in a fixed, uniform way off a
single trigger. [...] No conditional branching (no "if X then Y, else Z"),
no computed or derived values, and no AI/LLM reasoning involved. The number
of destinations touched does not matter [...]

Problem 2: the model had to commit before it could compare

The next test was "route incoming leads smartly based on their details". I read the reasoning field on that one, and it said it saw two valid readings and picked one anyway.

Working out why, I realized it was my prompt's doing. It evaluated the categories in sequence, so the model had to commit to one before it ever checked whether another also fit. Genuine overlaps were being settled by the model's best guess instead of reaching ambiguous. I rebuilt the evaluation so each category is assessed independently and the model counts how many plausibly fit:

Do not decide sequentially. Instead:
[...]
3. Count how many categories plausibly fit:
   - Exactly one fits -> assign that category.
   - Two or more fit, and the description does not specify enough to pick
     between them -> assign 'ambiguous'.

I also added a line to the prompt saying that "more common in practice" is a guess about unstated intent, which is exactly what ambiguous exists to catch.

Problem 3: the schema asked for the answer before the reasoning

My schema also listed category before reasoning, so the model was structurally being asked for its answer before it had written the explanation that was supposed to inform it. I swapped them:

{
  "type": "object",
  "properties": {
    "reasoning": { "type": "string" },
    "category": {
      "type": "string",
      "enum": ["simple_automation", "ai_workflow", "custom_integration", "ambiguous", "invalid"]
    }
  },
  "required": ["reasoning", "category"]
}

(Trimmed to the two fields that matter here.) I changed the order and kept testing. I have no record of testing that change by itself, so I can't say how much of the later improvement comes from it.

Problem 4: the fix overcorrected

With the new wording, the model started treating "automatically" as a possible AI signal. That word just means "not manual", and it appears in nearly every automation request, so it would push ordinary requests toward ambiguous.

I tightened the definitions so each category only counts as plausible if the text contains a specific word or phrase that grounds it, and I named "automatically" and similar words as things that don't count. I also added a rule aimed at the habit I'd just watched:

If you find yourself justifying a category with "could imply" or "might
involve" without pointing to a specific word or phrase that drove that
reading, do not count it

Cleaning up

The old [TIE-BREAKING] section was still in the prompt. After the evaluation rewrite it repeated the same logic less precisely, so I removed it and reran both the original failing case and a genuinely ambiguous one to confirm each landed where it should.

Checking the boundaries

I wrote two more tests aimed at the seams between categories instead of one category at a time.

The first stacked signals on purpose: conditional branching ("if it's not feedback, stop"), several systems (Gmail, a data table, Slack), and clear AI work (sentiment extraction, summarizing, drafting a reply). The reasoning field weighed the branching signals against the AI ones and landed on ai_workflow, using my rule that custom_integration applies when the logic is driven by fixed rules, not AI reasoning. That rule is my call. A description with both could reasonably be argued the other way, so the result shows the prompt follows my rule, not that the rule is correct.

The second was the opposite check: budget-based lead routing across HubSpot, Slack and Mailchimp, with no AI language at all. I wanted to see whether multi-system complexity alone would get pulled into ai_workflow. It came back custom_integration, with the reasoning ruling out both other categories using the exclusion wording.

What this doesn't show

Every test above is a single description I wrote myself. That was enough to find four problems in the prompt, and not enough to say the prompt is reliable. A real set of varied inputs is the next thing it needs.