The bug that isn't a bug
On Tuesday you attach a dead-letter queue to orders-queue. On Wednesday a batch of messages disappears and you ...
For further actions, you may consider blocking this person and/or reporting abuse
The "expired → null, never stale-with-a-warning" call is the right one, and I'd generalize it further: this isn't just a caching problem, it's a "does the tool know what it doesn't know" problem.
I hit the same failure shape from a different angle building an MCP codebase-intelligence server — a symbol lookup would silently resolve to a shadow definition in a different part of the repo, and the wrong answer looked exactly as confident as a right one (same schema, same format, no signal anything was off). No TTL involved, but the root cause is identical to your metadata/graph TTL mismatch: two sources of truth that can silently diverge, and nothing downstream can tell.
The
freshnessobject is a good pattern precisely because it turns an invisible failure mode into a visible field the caller can act on. Curious if you've thought about the same idea for provenance rather than just age — e.g. flagging when a graph node was reconstructed from a stale sub-source vs a fresh one, not just "the whole graph is N seconds old."Does the tool know what it doesn't know" is the better framing, and the shadow-definition case is the nastier version of it because there's no timestamp to hang a warning on at all. Infrawise has two half-steps toward the per-node provenance you're describing: CDK-sourced stack outputs carry their own
stale: trueplus astaleReasonwhen the template they came from is an orphan the manifest no longer references, and resources that exist only as an unresolvable code reference (QueueUrl: process.env.QUEUE_URL) stay in the graph markedplaceholder: trueand are excluded from findings entirely, so it won't claim "this queue has no DLQ" about a queue whose config it never read. Both are the same instinct as the freshness object: make the gap a field instead of an omission. What I haven't done is generalize that into a per-node source watermark, so a node reconstructed from a stale sub-source is currently indistinguishable from a fresh one once it's in the graph. Filed it as github.com/Sidd27/infrawise/issues... along with letting a caller state an age tolerance per call, since "what does this architecture look like" and "does queue X have a DLQ right now" clearly shouldn't share one staleness budget. Thanks for the push on this one.Good to see this turned into #101 and #102 — and now that both are up, it looks like there might be a third bucket worth naming alongside "stale" and "failed extraction": wrong-source. I hit that version building an MCP codebase-intelligence server — a symbol lookup would successfully resolve, with no error and no missing data, just to the wrong definition (a same-named shadow in experiments/ instead of the real one in src/). Not absent, not aged, just resolved against the wrong evidence.
Doesn't block #101/#102 landing in order — just flagging it in case the per-source records from #101 end up being a natural place to also record "which of N candidate matches did this actually resolve to," not just "did the source succeed."
Wrong-source is the right third bucket, and your comment sent me to check my own resolution paths — one of them has exactly the bug you're describing. Two of the three refuse to guess already: short-name table qualification detects when "orders" maps to two schema-qualified tables and deliberately declines to bind, falling back to a placeholder rather than picking one, and
get_table_schemareturns every match instead of choosing. Butanalyze_functionlooks up by name with a plain.find()and takes the first hit, even though function node IDs are file-scoped, so a repo withhandlerin bothsrc/andexperiments/gets whichever one the AST scan reached first, with no signal that a second candidate existed. Successfully resolved, no error, no missing data, wrong definition. So you're right that this doesn't fit either bucket: #101 records whether a source could be read and #102 records how old it was, and neither would have caught a lookup that read a fresh, complete source and picked the wrong row out of it. The per-source records do look like the natural place to hang "N candidates, resolved to this one, here's why" — with the honest default being that N > 1 without a tiebreaker returns all of them rather than the first. Filing it separately, and thanks for making me go look.The "N > 1 without a tiebreaker returns all" default is the right call — it shifts the burden of ambiguity to the caller, where it belongs, instead of hiding it in a silent first-match.
I ended up at the same place with the symbol resolver: when there are multiple candidates, returning all of them with their source paths forces the caller to either pick explicitly or surface the ambiguity. The cases where .find() feels safe are exactly the cases where a second candidate shows up six months later and nobody notices.
A second candidate shows up six months later and nobody notices" is exactly what made me go looking rather than just patching the one case. analyze_function now returns every match with its file plus ambiguous: true when more than one hit, shipped in 0.23.1. The sweep afterwards was the useful part, though it turned up a different bucket than wrong-source: get_table_schema had no source mapping at all, so a failed postgres extraction still answered found: false for every postgres table — an explicit "no such table" about a database that was never read — and get_stream_details only checked whether Kinesis had been read, so an MSK permission error returned kafkaClusters: [] with nothing to indicate half the tool went blind. Neither is a wrong-source resolution, they're the unread-source case, but the shape is the same one you named: a confident answer whose evidence nobody checked. Both are fixed, and the fail-closed work from #101 landed too — an unreadable source now answers with the source name and the error instead of an empty list. The thing I did not anticipate is that "disabled in config" needed handling as carefully as "failed": it's the far more common reason a list comes back empty, and it was equally silent. Thanks for the nudge, it was worth more than the one bug it started with.
Strong framing. I’d make freshness part of each tool call’s contract, not only overview metadata. Different questions tolerate different ages: an architecture overview may accept 24 hours, while “does queue X have a DLQ right now?” should request a much smaller
maxAgeSecondsor force a live read. Return per-source watermarks and completeness together with account, region, and effective principal; a fresh snapshot taken with narrower permissions can look exactly like a missing resource. For negative claims, fail closed unless every required source is complete within the requested age. Useful tests: skewed source timestamps, an omitted region, expired credentials, and a resource changing mid-refresh.The per-call
maxAgeSecondspoint lands, and the permissions one sent me back to the code where it turned out worse than I'd have guessed. Infrawise runs every extractor through a helper that catches adapter failures, logs a warning to the terminal, and returns undefined so one bad service never aborts the run. Which means an AccessDenied onsqs:ListQueuesproduces a graph with zero queues, identical in every respect to an account that genuinely has none, and the DLQ analyzer then finds nothing to flag. The absence of a finding reads as a clean bill of health. Nothing records the effective principal either, so there's no way to detect it after the fact. That's a bug rather than a missing feature and I've filed it as github.com/Sidd27/infrawise/issues... — record per-source outcomes and caller identity, then fail closed on negative claims so an unreadable source answers "unknown, the SQS adapter failed" instead of silence. Your test list went in verbatim; the mid-refresh mutation and omitted-region cases are the ones I wouldn't have thought to write. Appreciate it.Thanks for sharing this, very informative
This is a really good point, especially the part about stale data looking more trustworthy than missing data. I’ve been thinking about similar problems while working on CodeCan.net—sometimes having old information available can be more confusing than having no information at all. Making the data age visible is a simple idea, but it can make a big difference when an AI is making decisions based on that context.
The DLQ example is the exact failure mode that makes me distrust grounded agent answers more than vague ones. A stale ARN with real table names is worse than a guess, because the confidence is real even when the facts expired. Returning null past TTL instead of a soft warning is the right call. Callers ignore warnings under deadline pressure, they cannot ignore a missing tool result. The part I would push further is putting the snapshot age in the agent reply itself, so the human sees as of Monday next to the claim before they act on it.
Actually I m also thinking on the same line but I thought maybe its overkill but now I see as more people are inline with this tough I might surface the age directly in tool response with hint so that if needed agent can surface that.
Making freshness explicit instead of silently serving stale context feels like an important design choice. I’m curious whether you’ve considered letting downstream decision logic declare its own freshness and completeness requirements, so stale or unavailable evidence produces an unresolved decision rather than simply being consumed as context.
Half of that shipped this week; the other half turned out to hinge on who the downstream is. A caller can now declare its own freshness requirement per call — maxAgeSeconds — and the response reports whether the data met it. What I deliberately didn't do is refuse: for a coding assistant a labelled answer beats an unresolved one, since a twenty-hour-old snapshot is fine for an architecture question, and refusing moves a judgement into the tool that only the caller can make. Decision logic is different, and there it behaves as you describe — analyzers fire only on explicit evidence, never on absence, so a flag reading false means the setting is off while undefined means it was never read, and only the first produces a finding. Where your framing lands hardest is the CI gate: completeness is arguably already declared, since enabling a service is the statement that you require it, yet a run where an enabled source failed still exits zero. It warns, but doesn't refuse to certify. That's the unresolved-decision case sitting in plain sight, and it needs no new configuration — just the honesty to say a green build on infrastructure that was never read isn't green.