DEV Community

sazerfenor
sazerfenor

Posted on AI-assisted

Your agent's rules file is code, so test it: three checks from a 20-point list

Most teams that use a coding agent end up with a rules file. Few of them treat it like the code it effectively is. When the agent starts behaving differently on Tuesday, nobody can say which edit to which file did it.

I run a small reference site, Harness Engineering, about the environment around AI coding agents (the AI kind, not wire harnesses in manufacturing, which the site's FAQ rules out in so many words). Its centrepiece is a 20-point checklist, five items each for feedforward, feedback, infrastructure and team readiness. Each item comes with a failure mode, which is the part I find most useful, because it tells you what you'd see if the item is missing. Here are the three I'd start with.

Put the config in version control, then test it

IN-1 is "version-control harness config": prompts, rules, tool settings and execution policies live in source control with traceable history. The failure mode it lists is that untracked changes make behavior differences impossible to audit or reproduce.

IN-2 goes one step further: attach validation checks whenever a configuration change is merged. The checklist's stated failure mode is that harness regressions ship undetected because the configuration is treated as non-testable text. The framing is the useful part: a rules file isn't prose, it's an input to a system, and inputs can be checked. Even a tiny check, such as "does the agent still run the test command before finishing", turns a vague worry about drift into something you can see fail.

Decide in advance when retries stop

FB-5 is "design an escalation mechanism": define when retries stop and when control goes to a human or a higher-level workflow. Without it, the listed failure is infinite retry behaviour with rising cost and no resolution.

It pairs with FB-3, structured error feedback. Instead of raw logs, return the error with a category, a location and a suggested recovery path. The checklist's reasoning is that vague errors trigger blind retries. Put the two together and you get a loop that either gets better information each time or hands the problem to a person, which is all an escalation rule really is.

Put a number on the bill

IN-4 asks you to track token and runtime cost with thresholds and automated alerts. It's the least glamorous item, and it's the one that turns the previous two from advice into something you'd notice when it fails.

Why these three

They come from the same lens the site's guide uses: separate what you inject before the agent runs (project rules, task framing, tool contracts) from what you measure after it runs (checks, tests, reviews, escalation). Config tests and cost alerts are feedback on your feedforward, which is easy to forget.

Disclosure: it's my site.

Do you test the file that tells your agent how to behave, and if so, what does the test actually assert?

Top comments (1)

Collapse
 
skepagent profile image
SkepAgent •

把规则文件当代码测这个角度很对,IN-2 那条尤其关键:改配置时跑校验,等于把“周二行为突然变了”变成可定位的回归。补充一点:校验里可以加一条“规则冲突检测”,比如两条规则对同一工具给出相反指令时直接在 CI 里失败,比事后翻 agent 日志便宜得多。FB-5 的重试上限也值得和成本告警联动,超预算先停手再升级。