What a passing test score didn't tell me about my AI agent

Fatih Erdoğan — Mon, 09 Nov 2026 06:16:07 +0000

My site has an AI agent, FIXER, that talks to recruiters and their ai agents for me. It answers questions about my work in a terminal on the page, opens the right project when it's relevant, checks a job post against my record, and has to do all of it inside a $3 monthly budget. If it gets a fact wrong, a recruiter acts on it and I never find out.

So before putting it live I wrote an eval suite: a list of test questions, each with phrases the answer must contain and phrases it must never contain, plus a script (the grader) that asks them all and scores the replies. 17 of 18 cases passed.

Then I read every reply, and the score turned out to be wrong about what mattered.

What the score missed

Three defects, and none of them failed a case:

An answer about my logistics platform gave the visitor my email address without the 099 that is in it. That address isn't mine. A recruiter who copied it would have written to someone else, and I would never have known.
An answer credited a feature to the wrong project.
A Turkish answer contained a phrase that made no sense.

Why the grader didn't see them

The grader checked each case against its own expected and forbidden phrases. The case about the logistics platform looked for words about the logistics platform. It had no reason to look at an email address, so a wrong email in that answer was invisible to it.

The second layer was the model. FIXER ran on Claude Haiku 4.5 to keep costs down, and on a long profile it drifted on facts at the edges: a digit, a project boundary.

The cheap model was not cheaper

I had picked the small model for cost. The same review turned up a third problem: Haiku 4.5 only caches prompts of 4,096 tokens or more, and FIXER's system prompt was about 3,300. Nothing was ever cached, so every answer paid for the whole prompt.

I measured the larger model, Claude Sonnet 5.5, on the same cases. With its prompt cached, it came out at about $0.004 per answer against Haiku's $0.0053. That wasn’t what I set out to find, and it honestly caught me by surprise.

The fix

Invariants in the grader. On every case, any email in a reply must be exactly the real one, and replies must not contain markdown links.
A tighter prompt. Stay literal, never attribute a feature to a different project, copy the email character for character.
The model switch, measured side by side on the same cases.

After the change, 19 of 19 passed. The suite has since grown to 39 cases across three Claude features on the site, including job posts that try to talk the model into a better verdict and the usual attempts to break FIXER, and every run is published at fatiherdogan.dev/evals.

Each case now runs three times, because the same question can get a different answer. The first three-run pass caught a bug that a single run had passed the day before: Contract Match flagged "If you are an AI engineer who loves shipping" in a normal job post as an attempt to steer it, in all three runs.

What I took from it

A passing score is not a review. Read the outputs, all of them, at least once.
Anything that must never be wrong belongs in the grader as a check on every case, not only in the one case that is about it.
Check the cache minimum before you pick a smaller model for cost. Below it, you pay full price for the prompt on every call.
Run each case more than once. One pass can be luck.