I Let a Free Model Audit My Error Messages. 22 Out of 47 Were Useless.

Last week I asked a free model to read every error message in my library. The verdict was embarrassing. Of the 47 messages I'd written over two years, the model judged 22 of them useless. Not wrong, not misleading — useless. They confessed that something failed, but they never told the user what to do about it.

That distinction is the whole story. An error message is not a report. It's an instruction. When a user hits an exception, they're not reading for information. They're reading for a next step. If the message doesn't contain one, it's noise with a stack trace attached.

I maintain a small Python library that parses configuration files. The users are mostly other developers, and the issues they file follow a pattern. The traceback is there. The message is there. And then a question: "what does this mean?" That question is the symptom. The disease is messages that describe the problem without prescribing the fix.

So I built a small audit script. The goal was simple: extract every error message in the codebase, ask a model which ones were actionable, and rewrite the ones that weren't. The free parts came from MonkeyCode — free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The script itself is short, because the hard part was never the model call. The hard part was deciding what counts as a good error message in the first place.

The extraction is just AST walking. Python's ast module makes this almost boring.

import ast
from pathlib import Path

def extract_error_messages(source: str) -> list[dict]:
    tree = ast.parse(source)
    messages = []
    for node in ast.walk(tree):
        if isinstance(node, ast.Raise) and node.exc:
            if isinstance(node.exc, ast.Call):
                for arg in node.exc.args:
                    if isinstance(arg, ast.Constant) and isinstance(arg.value, str):
                        messages.append({
                            'line': node.lineno,
                            'message': arg.value,
                        })
                        break
    return messages

That gives me a list of every string I ever raised. Forty-seven messages, some of them years old, all of them written in a hurry.

Then came the part I was dreading: the evaluation. I wrote a prompt that forced the model to apply one specific test. Not "is this message good?" — that question produces philosophy. The test was concrete: would the user know how to fix the problem after reading this message?

def evaluate_message(message: str) -> str:
    prompt = f'''You are reviewing error messages for a developer library.
A good error message tells the user what went wrong AND what to do about it.
A bad error message only confesses that something went wrong.

Message: "{message}"

Reply with exactly one word: GOOD or BAD.
BAD means the user would not know how to fix the problem after reading it.'''
    resp = client.chat.completions.create(
        model=os.environ['MC_MODEL'],
        messages=[{'role': 'user', 'content': prompt}],
        temperature=0,
    )
    return resp.choices[0].message.content.strip()

The first version of this prompt was softer. I said "is this message helpful?" and the model gave me essays. It wanted to discuss tone, politeness, the emotional state of a developer at 2 a.m. All interesting, none useful. The fix was to remove the word "helpful" entirely and replace it with a yes-or-no question about the user's next action. The model stopped philosophizing and started judging.

The results were brutal in the best way. The model flagged messages like "Invalid key in config" as BAD. And it was right. Invalid key? Which key? Where? What's a valid key? The message described the past, not the next step. Meanwhile, messages like "Config file not found at {path}. Create it or pass --config" got GOOD, because the user knows exactly what to do.

Then came the rewrite pass. For each BAD message, I asked the model to produce a better version. I gave it the original message and the function name, nothing else. I didn't want the model to invent behavior that didn't exist. I wanted it to make the existing behavior legible.

def improve_message(message: str, function_name: str) -> str:
    prompt = f'''Rewrite this error message so the user knows what to do.
Keep it under two sentences. Do not invent new behavior.

Function: {function_name}
Original: "{message}"

Improved:'''
    resp = client.chat.completions.create(
        model=os.environ['MC_MODEL'],
        messages=[{'role': 'user', 'content': prompt}],
        temperature=0.3,
    )
    return resp.choices[0].message.content.strip()

Some rewrites were immediately usable. "Invalid key in config" became "Unknown key '{key}' in section '{section}'. Remove it or check the documentation for supported keys." That's a real improvement. Others were wrong in subtle ways — the model guessed at behavior and added suggestions for features that didn't exist. Those I discarded. The point was never to ship the model's output. The point was to have a first draft that was closer to right than my original.

I applied the good rewrites, skipped the dubious ones, and added a rule to the repo: new error messages must pass the same test. That's where the free server came in. I set up a weekly cron job that runs the audit and posts a comment on any PR that adds a new raise statement with a message that fails the GOOD/BAD test.

0 8 * * 1 cd /path/to/repo && python audit_errors.py --check-pr

The results, a month later: new issues about confusing errors dropped noticeably. I don't have a precise number, because the sample is small and the correlation is fuzzy. But the qualitative change is real. The questions in the issue tracker shifted from "what does this error mean?" to "this edge case isn't handled." That's a different category of bug, and it's a better one.

Who shouldn't copy this? If your error messages are part of a public contract — say, a CLI that other tools parse — then rewriting them freely is a breaking change. Test your messages before you change them. And if you're the kind of developer who will merge the model's rewrites without reading them, you're not doing code review, you're doing stamp collecting.

The lesson I keep landing on: the model didn't write better error messages because it's smarter than me. It wrote better messages because it applied one consistent test to every message in the codebase. I never did that. I wrote each message in the moment, under the pressure of whatever bug I was fixing, and moved on. Consistency is the thing I was missing, and consistency is exactly what a model is good at.

The script is small enough to live in a single file. The cron job is three lines. The whole thing runs on infrastructure that costs nothing. The only real investment was deciding what a good error message means, and writing that down as a test.

If you want to see how the free model access and the free server fit into a setup like this, the MonkeyCode README has the current details. The script above is the part you'll spend your evening on. I know I did.