I had my own infrastructure context server running behind my editor, the way I do every day, and noticed it was telling me to re-run the analysis on almost every response. Not occasionally. Almost every request.
My first guess was that I had set a threshold too low. I had not. The refresh hint was doing exactly what it was configured to do — warn after six hours, with a twenty-four hour cache behind it. Those numbers were fine. What had actually happened was that the session had been running long enough for the context layer to quietly empty itself out, while still answering every question in a well-formed, confident-looking way.
That bug is worth writing about, not because the fix is interesting, but because I have now shipped the same class of bug four separate times, and I do not think it is specific to my tool. If you give an AI assistant a live connection to your infrastructure, you have not built a data source. You have built a long-running process. Long-running processes fail in ways one-shot fetches never do, and the AI-shaped version of that failure is unusually quiet.
A context layer is a process, not a fetch
The mental model most people have — mine included, for longer than I would like — is that infrastructure context is a lookup. The assistant needs to know your DynamoDB partition key, something goes and gets it, the answer comes back. Under that model the only interesting question is whether the data is correct.
That is not what actually runs. What runs is a server that boots once, reads your cloud account once, and then stays alive for as long as your editor does — hours, sometimes days — answering from what it has while incrementally updating parts of itself. It watches your source files. It rebuilds a graph on save. It re-checks its cache before every tool call. All of that incremental machinery exists because the alternative is hitting your AWS account on every question an assistant asks, and assistants ask a lot of questions.
Every one of those incremental paths is an opportunity for the long-lived state to drift away from what a fresh start would have produced. Here is the same failure, four times, from one repository's history.
The analyzer that stopped running. An issue filed against the project reads: LambdaMissingTriggerDLQAnalyzer is in the analyzer list used by the full analysis, and absent from the one used by the file-save refresh. The consequence, quoted from the issue: "After the initial infrawise analyze, every file-save triggers runCodeRefresh. From that point forward, all trigger-DLQ findings permanently disappear from the live MCP server's finding set." One missing line in a second list. Save a file once, and a category of finding is gone for the rest of the session.
The server that answered from its boot snapshot. For a while, running a fresh analysis in a second terminal changed nothing for the editor session already open. The fix commit puts it plainly: "a session served its boot snapshot indefinitely and re-running analyze appeared to do nothing."
The caches that expired at different rates. The analysis is stored as several entries — the graph, the findings, the cloud metadata, the record of what was read and when. These used to expire on different clocks: the graph at twenty-four hours, the metadata and provenance at one. A session that ran past an hour served a graph whose companion entries had silently gone null.
The most recent one. Same shape, longer fuse. The file-save refresh rewrites the graph but reads no cloud API, so it rewrites three cache entries and not the other two. That resets the clock on three of them. Past the twenty-four hour mark, a session where you keep editing code holds the graph alive indefinitely while the metadata underneath it expires — after which the rebuilt graph contains no cloud facts at all, and the timestamp saying when the cloud was last read goes null. And a null read-time is treated as "cannot vouch for this", which pins the refresh hint to true on every subsequent response. Which is the behaviour I opened with.
Four bugs, one shape: the incremental path diverged from the full path, and nothing noticed because the divergence produced a valid-looking answer.
Why nobody catches these
Every one of those failures degrades toward a well-formed response.
A missing analyzer does not throw. It produces a findings array that is shorter. An empty findings array is exactly what a healthy account looks like. Expired metadata does not throw either — it produces a graph with fewer nodes, and a graph with fewer nodes is exactly what a smaller account looks like. A null timestamp does not throw. It produces a response where one field is null, in a payload where plenty of fields are legitimately null.
None of that trips an alarm, because from the outside a degraded context layer and a simple environment are byte-for-byte indistinguishable. The tool call succeeded. The JSON parsed. The schema validated.
And then the failure gets laundered through an LLM, which is where it stops being a data problem and becomes a correctness problem. An assistant handed an empty findings list does not say "this list is suspiciously empty for a repository this size." It says your infrastructure looks clean. Handed a graph with no tables in it, it does not ask where the tables went — it writes the query anyway, guessing at a schema, in exactly the confident register it uses when it does know. The whole reason to wire real infrastructure into an assistant is to replace guessing with facts. A silently emptied context layer removes the facts and leaves the confidence.
This is the part that makes it worse than having no integration at all. An assistant with no infrastructure access hedges: I do not have visibility into your tables, you may want to check. That hedge sends you to the console. An assistant with a broken infrastructure integration does not hedge, because as far as it can tell the integration answered.
What to actually check
If you run any long-lived context server — mine, someone else's, or one you wrote — these are the questions worth asking of it. They are the ones I got wrong.
Does the incremental path produce the same thing as a cold start? This is the single highest-value check, and it is testable: build state the slow way, build it the fast way, compare. Two of the four bugs above are exactly this and nothing more. If your refresh path maintains its own list of anything — analyzers, extractors, sources — that list will drift from the authoritative one, because nothing forces someone adding an entry to one to add it to both.
Does related state expire together? If your cached analysis is several pieces, they describe one moment and they must live and die as one. Independent expiry does not produce an error, it produces a chimera: half of Tuesday's read, half of nothing. The fix in my case was to re-stamp the entries the refresh path does not rewrite, so all of them stay on one clock.
Can it distinguish "nothing there" from "did not look"? Every layer between the cloud API and the model needs to preserve this, and by default none of them do — a failed call and an empty account both reduce to [] the moment someone writes a catch that returns a default. If the answer reaching the model is an empty array either way, the model will report an empty account. Carry the failure alongside the data, all the way to the response.
Does a response say when its facts were read? Not when the object was assembled or when the cache file was written — those move for reasons that have nothing to do with your cloud. Rebuilding a graph from cached metadata produces a brand-new object full of old facts, and dating it by its construction time is a lie in the most convincing possible format.
Have you run one session for a full day? Every bug in this article requires uptime to reproduce. A test suite starts fresh, does its thing, and exits in milliseconds. None of these would ever appear there. The most recent one needs twenty-four hours of continuous uptime plus file edits — a combination that occurs constantly in real use and never once in CI.
The uncomfortable conclusion
Feeding an assistant real infrastructure data raises the ceiling on how good its answers can be. It also raises the floor on how bad a silent failure is, because you have removed the reader's reason to double-check. Every bug above made an assistant more confident and less correct at the same time, and the increase in confidence is the part you cannot see.
So the interesting engineering in an AI-aware infrastructure layer is not the extraction. Reading DynamoDB key schemas is a solved problem, and any competent developer can wire up the AWS SDK in an afternoon. The interesting engineering is everything that keeps the layer honest about its own state over a workday: making the fast path agree with the slow path, keeping related state on one clock, refusing to let a failure decay into an empty list, and dating every claim by when it was actually read.
I build this in the open at GitHub — it ships on npm — and the commit history is a fairly honest record of getting each of these wrong first. If you are building something similar, the four checks above will save you the versions I shipped.
Key takeaways
- Treat your context layer as a long-running process, not a data source. Its worst failures require uptime to reproduce and will never appear in a test suite that starts fresh.
- Test that the incremental path matches a cold start. Any refresh path holding its own list of analyzers, extractors, or sources will drift from the authoritative one.
- Expire related cache entries as one unit. Independent TTLs on pieces of a single snapshot produce a half-and-half state that looks valid and is not.
- Never let a failed read decay into an empty result. An empty list and a failed call are the same value; only one of them means "nothing is there."
- Date every fact by when it was read, not when it was assembled. Rebuilding from cache creates a new object holding old facts, and timestamping it by construction is the most convincing lie available.
Top comments (0)