How do you know if your coding agent's instructions are getting better?

Mihael Zamin Sousa — Sat, 31 Oct 2026 20:45:40 +0000

A common way to improve a coding agent is to change its instructions.

You add a rule to AGENTS.md. You write a skill for a recurring task. You tell the agent to check something before editing code.

Then the next run looks better.

But was the instruction responsible?

That's a harder question than it first appears.

The problem with judging one successful run

Suppose an agent repeatedly makes an error. You add a new instruction explaining how to avoid it.

The agent succeeds on the next attempt.

That result is encouraging, but it doesn't establish that the instruction caused the improvement. The task may have been easier, the agent may have taken a different route, or the original failure may not have been reproducible.

A useful evaluation needs more than a before-and-after impression.

What I want to measure

For changes to coding-agent instructions, I'd want to answer several questions:

Can I reproduce the original failure?
Does the candidate instruction address that failure?
Does previously working behavior still work?
Are the baseline and candidate evaluated under comparable conditions?
Can I inspect the evidence before keeping the change?

A smaller tool-call count is interesting, for example, but only if the agent still completes the required work.

Building Agent Harness Lab

I'm working on an open-source project called Agent Harness Lab to explore this workflow.

It supports Claude Code and Codex.

Instead of requiring you to describe a failure manually, it can inspect local project evidence and look for improvement candidates.

The experimental improvement process stages candidate changes, creates evaluation tasks, compares baseline and candidate behavior, and reports the outcome.

The project also provides a simpler entry point:

ahl analyze /path/to/project

This produces a local analysis report without running an agent or modifying project instructions.

The broader workflow is:

ahl improve /path/to/project

That process is deliberately conservative. The project does not currently claim established general performance gains, and automated behavioral adoption is blocked while independent certification requirements remain unmet.

Why the limitations matter?

An agent evaluating another agent raises difficult questions.

Could the evaluator reward the wrong behavior? Could it overfit to the specific examples used during development? Could an apparent improvement come from a change in runtime conditions?

Those aren't theoretical details that can be ignored when deciding whether to trust an automated repair.

The current implementation is an experiment in making that process more inspectable, rather than pretending those questions are solved.

Try it or review the approach

The repository includes installation instructions, evaluation contracts, validation documentation, and the remaining safeguards needed before behavioral adoption can be enabled.

GitHub: https://github.com/mihazs/agent-harness-lab

It's MIT-licensed and open to contributions.