Stop Guessing Which AI Model to Use: Build a Two-Week Routing Log From Your Own Tasks

Every time a new coding model drops, my feed fills with benchmark screenshots, and I catch myself doing the worst possible evaluation: vibes. "This one feels smarter." Two days later I've forgotten which model actually fixed my bug and which one confidently broke my tests.

After a few rounds of this, I stopped trying to rank models globally and started answering a smaller, more useful question: for the tasks I actually do, when is a free model enough, and when is it worth reaching for something stronger? The answer came from a boring artifact: a routing log I kept for two weeks. This post is the log format, the decision rules it produced, and a tiny script so you can run the same experiment.

Why benchmarks don't settle this for you

Public benchmarks measure tasks that are clean, self-contained, and graded automatically. My real tasks are none of those things: half-remembered legacy code, a stack trace missing the interesting frame, a migration where the schema drifted from the docs. Whether a model is "good enough" depends on the task shape, not just the model. So the unit of measurement has to be my own task history.

The routing log

One row per AI-assisted task. Five columns, nothing fancier:

date | task_shape | model_tier | outcome | rework_minutes
2026-08-03 | regex-refactor | free | pass | 0
2026-08-04 | flaky-test-diagnosis | free | fail->escalated | 25
2026-08-05 | sql-migration-review | paid | pass | 5

• task_shape — a short tag you invent as you go (mine converged on: boilerplate, single-file-bug, cross-file-bug, design-question, unfamiliar-lib, review-my-diff).
• model_tier — just free or paid/strong. Don't log model names at first; the tier comparison is what you're after, and names churn weekly anyway.
• outcome — pass (shipped with normal review), fail->escalated (free model's answer was wrong or unhelpful and I re-did it with a stronger model or by hand), fail->manual.
• rework_minutes — time spent detecting and fixing the model's mistakes. This is the column everyone skips and the only one that matters.

The script that makes it honest

Memory lies; a prompt that nags you doesn't. I keep this as airoute on my PATH (Python, no dependencies). Run it right after finishing an AI-assisted task — it appends a row and, once you have enough data, prints which task shapes are costing you rework on the free tier:

#!/usr/bin/env python3
"""airoute: log AI task outcomes and summarize rework by task shape.
Usage:
  airoute add <task_shape> <free|paid> <pass|escalated|manual> <rework_min>
  airoute report
"""
import csv, sys, os
from collections import defaultdict

LOG = os.path.expanduser("~/.airoute.csv")

def add(shape, tier, outcome, rework):
    exists = os.path.exists(LOG)
    with open(LOG, "a", newline="") as f:
        w = csv.writer(f)
        if not exists:
            w.writerow(["shape", "tier", "outcome", "rework_min"])
        w.writerow([shape, tier, outcome, rework])

def report():
    stats = defaultdict(lambda: {"n": 0, "esc": 0, "rework": 0})
    with open(LOG) as f:
        for row in csv.DictReader(f):
            key = (row["shape"], row["tier"])
            stats[key]["n"] += 1
            stats[key]["esc"] += row["outcome"] != "pass"
            stats[key]["rework"] += int(row["rework_min"])
    print(f"{'shape':<22}{'tier':<6}{'n':>4}{'escal%':>8}{'rework/task':>13}")
    for (shape, tier), s in sorted(stats.items()):
        print(f"{shape:<22}{tier:<6}{s['n']:>4}"
              f"{100*s['esc']/s['n']:>7.0f}%"
              f"{s['rework']/s['n']:>11.1f}m")

if __name__ == "__main__":
    if len(sys.argv) >= 6 and sys.argv[1] == "add":
        add(sys.argv[2], sys.argv[3], sys.argv[4], sys.argv[5])
    elif len(sys.argv) == 2 and sys.argv[1] == "report":
        report()
    else:
        print(__doc__)

Two weeks and ~40 rows is enough for the patterns to stop being noise.

What my log actually showed

This is my data, not a universal law — that's the point of running it yourself:

task_shape	free-tier escalation rate	median rework/task
boilerplate, single-file bugs	~10%	~4 min
review-my-diff	~15%	~6 min
cross-file bugs, unfamiliar libs	~45%	~25 min
design questions	~30%	(hard to measure)

The routing rule that fell out:

1. Free tier first for boilerplate, contained bugs, and diff review. The escalation rate is low enough that the occasional miss is cheaper than paying for every prompt.
2. Skip the free tier entirely for cross-file debugging and unfamiliar libraries. Starting there was the expensive path — 25 minutes of rework to discover I needed the stronger model anyway.
3. Design questions go to whichever model I'm already talking to, but the output is a proposal, never a decision.

The surprise was row 2. I'd assumed "try free, escalate if needed" was strictly cheaper. For messy tasks it isn't — you pay the free attempt and the escalation and the rework.

Where the free tier comes from

For this to work as a permanent habit rather than a trial-period trick, the free tier has to actually be free and actually be available. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is one option that fits the constraint here — it offers free model access and a free server option, which is what I used as the free tier in the log above. The useful property for this workflow isn't any single model it exposes; it's that I could run the high-volume, low-risk half of my routing policy without a meter running in the back of my head. If you're curious, the free tier is the honest way to replicate this experiment — but any reliably-free access works, and the script doesn't care which one you point it at.

Limitations, and who shouldn't bother

• Two weeks is a small sample. Treat the numbers as routing hints, not statistics. Re-run the report monthly; model quality shifts under you.
• The log measures my repo, my prompts, my patience. Your escalation table will look different, especially if your work is mostly greenfield (free models do better there in my experience) or mostly legacy spelunking.
• Anything touching secrets, credentials, or production data doesn't get routed at all — it doesn't go into a third-party tool on any tier without your org's explicit approval. If your employer has an AI usage policy, that policy wins over this entire article.
• If you do AI-assisted work only a few times a week, the logging overhead costs more than the routing saves. This pays off when you're making the free-vs-strong choice many times a day.

The bigger lesson wasn't about any model: "feels smarter" is unmeasurable, but "rework minutes per task shape" is one CSV away. If you've been model-hopping on vibes, I'd genuinely like to hear what your routing table turns out to be — especially where it disagrees with mine.