A prompt change fixes one issue, but how do you check it hasn’t caused another?
Do you rerun saved test cases or check a few conversations manually? Curious what’s worked for your team.
Mortis
Posted on AI-assisted
When you change an AI agent’s prompt, how do you check that you fixed one problem without creating another?
For further actions, you may consider blocking this person and/or reporting abuse
Top comments (2)
For the prompt-change question, I would start with the conversation that exposed the bug and a few nearby variations that should produce different behavior. Otherwise a prompt can learn the exact wording of one example without fixing the underlying decision.
Then replay the old and new prompts against the same saved tool results and compare the expected decisions, not just whether the prose looks better. I would keep a separate holdout of older successful cases, including cases where the agent should ask a question or stop. Any changed outcome there deserves inspection before treating the original bug's disappearance as a successful fix.
To ahmetozel's replay setup I'd add repeats, because one run per case can't tell a regression from sampling noise. Say every case in a 50-case suite passes 80% of the time under both prompts, so the change did nothing. Run each case once per prompt and you'll still see about 8 cases go from pass to fail, anywhere from 4 to 12 in most trials, and some will look exactly like the new problem you were worried about. Running each case several times per prompt and comparing pass rates case by case shows which drops are real. The cases that flip back and forth are worth a look anyway, since they fail at random for users too.