A few months ago we started building seriously with AI agents for the first time.
Every time something went wrong our first instinct was the same. Wrong prompt. Wrong model. Temperature too high. We spent a lot of time tuning things that did not need tuning.
It took longer than I would like to admit to figure out that the model was almost never the problem.
The problem was what we were sending it.
This is what we learned, mostly the hard way. Part 1 covers the first three failure modes. Part 2 covers the rest.
What a production AI agent system actually looks like
A production AI agent is not a chat interface.
It reads files. It calls tools. It processes logs, stack traces, CI output, API responses. All of that content gets assembled at runtime and sent to the model. The model reasons over whatever it receives.
The failure modes we kept running into were almost all in that assembly step, not in the model's reasoning. The model was doing its job. We were giving it bad inputs.
Failure 1: Sending the whole log
One of the first problems we ran into was debugging CI failures. Whenever a build failed, we passed the entire log to the model.
Sometimes that meant 40,000–60,000 lines of output. Tens of thousands of tokens, most of which were dependency downloads, repeated warnings, verbose build steps, and other information that had nothing to do with the actual failure.
The model usually found something. It just wasn't always the thing we cared about.
This wasn't a hallucination problem. The root cause was almost always somewhere in the log. We were simply asking the model to find a handful of relevant lines buried inside tens of thousands of irrelevant ones. Attention mechanisms are powerful, but they are not grep, and as the amount of irrelevant context grows, the model has to spend more of its attention on information that doesn't help solve the problem.
The obvious downside was cost. Every unnecessary token increased latency and API spend. The less obvious downside was quality. Flooding the model with noise made it easier to miss the signal.
What worked much better was preparing the context before it ever reached the model. We extracted the failing sections, collapsed repeated output, removed everything that wasn't relevant, and sent only the information needed to answer the question.
Instead of sending 50,000+ tokens of raw logs, we were often sending 500–1,000 tokens of curated context. Responses became faster, cheaper, and noticeably more accurate because the model spent its attention on the failure instead of the surrounding noise.
Looking back, it feels obvious. At the time, it wasn't. Most of the examples we had seen simply passed raw artifacts directly to the model, so that's exactly what we built first.
Failure 2: The one that catches teams off guard
Teams rarely intend to log secrets. But operational data often contains more sensitive information than people realise.
API keys can appear in error messages. Database connection strings can end up in failed connection logs. HTTP request traces may include Authorization headers. Third-party libraries and SDKs sometimes log request or response payloads during failures. None of this is usually deliberate. It is simply a by-product of debugging distributed systems.
The problem starts when those same logs become input to an AI agent. What was originally written for an engineer to inspect may now be forwarded to an external model without anyone reviewing its contents first.
This is the failure mode that worries us most because it is invisible. No error. No alert. The agent produces a useful response. The CI failure gets debugged. Nobody knows sensitive information may have left the boundary it was originally intended to stay within.
The attack surface is not just source code, where most secret scanning tools focus. It is operational data: logs, error messages, tool responses, CI output, and runtime artifacts. Content written for humans to debug systems, not for models to consume.
This is why secret redaction is the first step in XContext's pipeline before anything else runs. Detect and redact before summarisation, not after, because a summariser can preserve sensitive information even if the original artifact is later discarded.
Failure 3: Staleness
Agent reads deployment configuration early in a session. Session runs for two hours. Configuration changes. Agent keeps working from the original.
When it makes a wrong decision the reasoning looks completely correct. Because it is correct given what the agent knew. That information just stopped being accurate ninety minutes ago.
This one is frustrating because you inspect the agent's reasoning and it looks fine. The failure is in the data, not the logic.
We do not have a complete solution for this yet. XContext tracks ingestion time as metadata on every stored context object so at least you know how old something is. Proper staleness policies are something we are still working out. If you have solved this well I would genuinely like to know in the comments.
Where this is going
Three more failure modes in Part 2, including the one that is hardest to debug after the fact and the one I wish we had built correctly from the start.
Part 2 publishes next week.
If any of these first three sound familiar, drop a comment. Would like to know which failure mode is hitting teams most right now.
Top comments (0)