I've been building AI agents for a while, and I keep hitting the same wall.
It's not the model hallucinating. It's not the API being slow. It's that the agent works perfectly in my test environment, with my 20-message context, and then fails in production with a 200-message context, a different tool chain, and a user who asks something I didn't anticipate.
The failure modes are different from traditional software:
- The agent doesn't crash. It just makes a slightly worse decision.
- The tests pass. The context window just got too big.
- The demo works. The real user asks a question that's 5% different, and the whole chain breaks.
I wrote about this before ("retry loops are expensive"), but I'm curious about the broader problem. How do you test an agent for the "messy context" it'll see in production, without shipping it to production first?
What does your test suite look like for agents? Do you have a "context window full of noise" test, or is it still mostly happy-path?
Top comments (1)
"Context window full of noise" is the exact test I've been missing.
Most agent tests use a clean context: 5 messages, 3 tools, 1 clear goal. But production is 50 messages, 10 tools, 3 conflicting goals, and a user who pasted a screenshot of a spreadsheet instead of text.
The only way I've found to simulate that is to record real production sessions and replay them with the same noise. It's not perfect, but it catches the "agent gets confused by irrelevant information" failure mode.