Last post introduced the decision log: every rejected LLM output in my n8n review queue gets one tag — **prompt, **fixture, **model, or **data* — and I tally them weekly. This week I actually sat down with three weeks of tallies. The pattern did what the SOP promised: it turned a vibe into a work order. Here's exactly what the tally said, and what I did about it.*
What the tally looked like
Three weeks of review queues, 31 rejected outputs, one tag each:
| Week | prompt | fixture | model | data | total |
|---|---|---|---|---|---|
| W1 | 6 | 2 | 1 | 1 | 10 |
| W2 | 4 | 4 | 2 | 0 | 10 |
| W3 | 7 | 1 | 2 | 1 | 11 |
| Total | 17 | 7 | 5 | 2 | 31 |
Prompt: 55%. Fixture: 23%. Model: 16%. Data: 6%.
The SOP said one tag over 50% is where the bug lives. Here it is in the open: more than half my rejections were my own prompt, not the model, not the data. "The model got dumber this week" — the most expensive sentence in LLM operations — was wrong three weeks running.
Reading the prompt cluster: drift, not design
Seventeen prompt tags don't mean the prompt was bad. They mean the prompt drifted. The tell: W1's prompt rejections were mostly phrasing complaints (hedging stacks, "In summary" monotony); W3's were mostly scope complaints (answers that were technically correct but addressed a nearby question). Different failure, same tag, one week apart. That's drift — the workflow's inputs moved (a new source field, a slightly different ticket format) and the prompt didn't move with them.
The work order: a prompt diff ritual before every version bump. The SOP already runs the review queue before each bump; now it also shows me the prompt's diff since the last tally, and I read it like a code review. Two of W3's seven prompt tags would have been caught by reading my own diff — the new field I added without updating the instruction that consumed it.
Reading the fixture cluster: W2's poisoned fixture
W2 is the interesting one: fixture tags jumped from 2 to 4 in a single week, all four pointing at the same fixture — the "angry customer" ticket sample in the regression suite. The gate's escalation path had started routing real outputs to human review that looked nothing like production failures. They looked exactly like that one fixture.
The work order: I rebuilt the fixture from a real ticket, re-ran the suite, and the false escalations stopped. Fixture clusters are cheap to diagnose and cheaper to fix — but only if you tag them. Without the tag, "the gate is too sensitive" would have become a config-tweaking afternoon. With the tag, it was a twenty-minute fixture rebuild.
The model tags: noise, until they aren't
Five model tags across three weeks, scattered across different prompts, fixtures, and days. No pattern, no cluster. That's the correct response to scattered model tags: nothing. Watch the trend, don't react to the point.
The rule I wrote on the wall: two consecutive weeks of model-tag growth is a model problem; one week is weather. Tuning temperature or switching models on a single bad week is how you trade a known system for an unknown one. The tally exists to keep my hands off the model knob until the data earns it.
What the tally changed in the process
One process change, deliberately small:
- Monday, fifteen minutes. The tally review happens Monday morning, before the week starts. One work order max — the biggest cluster gets one concrete fix; everything else stays in the log. The constraint matters: a tally without a work-order cap becomes a guilt spreadsheet.
That's it. No new dashboard, no new script. The script from two posts ago owns the mechanical contract; the ten-minute SOP owns the prose; the weekly tally owns the direction of blame. Three layers, each with exactly one job.
Next in this series: the tally says my prompt drifts every three weeks. So I'm writing the thing that stops the drift — a prompt contract test that fails the build when the prompt and its fixtures disagree. The review queue keeps teaching the script.
I'm building AI Automation Lab — n8n workflow templates for AI automation — and keeping a free lab on GitHub. I also send occasional notes on n8n reliability patterns: get them here.
Top comments (0)