I am building a tool around one question:
which old instructions in your AI's memory can you no longer see?
The slogan I wrote for it is bolder t...
For further actions, you may consider blocking this person and/or reporting abuse
Pointing it at itself is the honest move. The two failure shapes you named look like different problems but are the same missing variable: the detector reads token match, not predicate structure. Sentence about a category vs member of that category is the class-vs-instance confusion that also breaks most retrieval reranking of policy docs. Text similarity cannot tell you which sentence is doing use vs mention.
The over-fire is the cheap mistake. Under-fire is the one that costs. Systems in this class should be calibrated toward false positives on purpose, because a noisy flag costs a human ten seconds of review and a missed drift costs whatever the drift was supposed to prevent. Asymmetric cost, asymmetric bias.
The mechanism that would have caught the second failure without needing "deprecated" keywords: contradiction detection across the governing scope. Not "does this line look stale" but "does this line contradict another line that is nearer to the current work." Contradiction is authority evidence without needing anyone to have written a lifecycle marker.
the asymmetric cost line is the part im taking with me. a noisy flag costs a human ten seconds and a missed drift costs whatever the drift was going to cause, so biasing toward false positives on purpose is just honest accounting. i had been treating a false positive as a bug to eliminate when its actually the cheaper error to live with.
and contradiction across the governing scope is exactly where i landed too. not "does this line look stale" but "does this line disagree with a line thats closer to the current work." thats authority evidence you can get without anyone writing a lifecycle marker, which is the whole problem with the keyword approach.
thats the layer im building toward, and im keeping it honest about where it stands: the proposer names a possible contradiction and cites the specific other line, then a deterministic check confirms the cited line actually exists and actually overlaps in scope before it ever becomes a finding. the design is frozen and the pieces are built. it has not beaten the baseline in a fair run yet, so im not claiming it works. but you described the target better than my own notes did.
"It has not beaten the baseline in a fair run yet, so im not claiming it works" is a sentence that usually gets deleted before publishing. Design frozen, verdict withheld: that is the detector's own discipline applied to the detector, and it reads exactly as trustworthy as it should.
One measurement problem waiting for you at the fair run. Precision is cheap: read the flags, count the good ones. Recall is not measurable at all without a corpus of known-stale lines, and nobody has one, because a drift you can locate is a drift you already fixed. The affordable way out is to manufacture ground truth: plant backdated contradictions with known coordinates and measure how many the detector surfaces. Injection is the only labeled corpus sold at this price. Side benefit: the plants double as a liveness check. The day the detector stops finding them, the detector died; the corpus did not get clean.
The self-audit tells me the product boundary is probably the product: not "this memory is safe," but "here is the authority map, here are the gates, here is what still needs human judgment." Flagging your own "find the old instructions" slogan while missing the June 2026 prose-level drift is exactly the failure pair that separates keyword detection from authority reasoning. As a founder/engineer, I'd measure the next version by whether it shortens a careful review of a messy CLAUDE.md or AGENTS.md without hiding uncertainty, not by whether it produces a clean green result.
Marcus this is the exact frame i needed someone outside my own head to say out loud. the boundary is the product. i kept almost apologizing for it only being a map, and you just named why the map is the thing. and youre right about the metric. a clean green result would actually be the tell that its lying to you. the honest measure is whether it makes a careful pass through a messy CLAUDE.md or AGENTS.md faster without ever hiding where its unsure. thats the yardstick im keeping now. its also why i left the semantic gap in as a failing test instead of smoothing it over. i want the uncertainty visible, not buried under a checkmark. the slogan flag missing the real june drift was the whole lesson in one run. keyword detection saw the words. authority reasoning would have asked what actually replaced what. that gap is the next build and im not going to pretend its closed before it is. appreciate you reading it close enough to hand me the right measuring stick.
Oftentimes the most insidious bugs won’t throw an error and you’ll have to dig through the codebase for problems that the compiler thinks is okay. Looking forward to more from you!
Tightening the detector to require words like superseded or deprecated fixes the slogan bug cleanly, but doesn't it walk right back into the prose case you flagged as the harder one? The false positive and the false negative feel like the same root cause to me, both come from reading vocabulary instead of the authority relationship, and this fix leans even harder on vocabulary. The genuinely dangerous stale instruction, that June plan written like normal thinking out loud, is exactly the one that will never contain the magic word. So the keyword tightening buys precision on the easy cases at the cost of never reaching the case that motivated the whole tool. Not a knock, you already name this as the next layer, but it does make me think the semantic proposer isn't a someday feature so much as the actual core, with the keyword pass being the placeholder. Really glad you shipped the failure instead of the clean rerun.
You’re right that both failures share the root the detector reads vocabulary, and the June case will never contain the magic word. No argument there.The order was deliberate though. The semantic layer alone is a model reading prose and deciding what’s stale a second opinion with confidence. I’ve watched that narrate its way past its own evidence too many times to build on it first. So I built the layer that demands receipts before the layer that proposes claims. Right now that means the system only confirms what announces itself half an architecture doing a whole architecture’s job. Where you’re dead on: the proposer isn’t a someday feature, it’s the other half of the core, and the tool doesn’t reach the case that motivated it until both halves exist. But it ships into the discipline, not instead of it proposer finds the candidate, deterministic layer confirms against evidence in the file, claim and receipts reported separately. “Trust the keywords” was too weak. “Trust the model’s read” is the same failure in better clothes. The pair is the tool.
The relevance is not authority frame is the part I'm carrying out of this one. It's the same problem with stale docs in a codebase - the outdated page is technically there and findable, but it should have zero votes on what actually gets built.
Thats the cleanest version of it. the stale doc is right there in the repo, findable, and often still technically correct, and it should still have zero votes on what gets built today. relevance got it into the search results. it never earned the authority to decide anything.
and the fix isnt deleting the old page. its re-deriving, at the moment of the decision, whether that page still gets a vote on the current work. most systems never ask. they just let the thing that showed up act like the thing thats in charge.
The "slogan as test case" move is exactly right â using your own marketing language as a false positive control is a clean way to catch keyword-matching detectors. The detector flagged "find the old instructions your AI should stop obeying" as an old instruction because it saw "old instruction" and "stop obeying," which are exactly the words a stale rule would contain. But it's not a stale rule; it's a description of one. The gap is exactly what you named: pattern matching vs. authority relations.
The failure taxonomy you worked out â over-fires (slogans flagged) vs. under-fires (genuine staleness missed) â maps onto a well-known problem in information retrieval: recall vs. precision. The detector optimizes for recall (catch everything that might be stale) and pays in precision (catch things that aren't). But the actual problem is harder than pure recall: you need to distinguish "mentions topic X" from "is a member of category X," which requires understanding whether the text has the right authority relationship, not just the right vocabulary.
Path B (training a classifier) sidesteps the vocabulary problem by learning the authority signals, but it introduces a different risk: a classifier trained on your labeled examples will have the same blind spots as your labeling intuition. If you consistently miss genuine staleness that's written in neutral prose, the classifier will learn to do the same thing. Path A (semantic proposer + human confirmation) is probably the right intermediate step â it reduces the review burden without encoding the labeler's biases as a ground truth.
On the "leave receipts for failure" point: this is the right instinct for agent memory management. But receipts only help if they're structured enough to be queryable â "the system failed because it followed a rule that was superseded three months ago" is useful; "the system followed something old" is not. Structured failure logs + semantic retrieval = actionable memory auditing. Have you thought about what the schema for a "failure receipt" looks like beyond a prose note?
the recall vs precision mapping is right, and the classifier warning is the one i needed to hear out loud. if i train it on my own labels it inherits my own blind spot, so the neutral prose staleness i keep missing becomes ground truth that its now confident about. thats worse than the original bug because it looks rigorous. that pushed me away from "train a classifier" and toward the proposer plus deterministic confirm path, for exactly the reason you said.
on the failure receipt schema, real answer, because prose notes were not going to cut it. the receipts are signed and chained, and each one carries the structured pieces: the item it fired on, the relation type it proposed, the exact evidence span it cited, the verdict, and the reason it was rejected or confirmed. so you can query "this failed because it obeyed a rule that a later line superseded" instead of "the system followed something old." the whole point is that a failure has to be replayable and diffable later, not just remembered.
the question about queryable structure was the sharpest thing in the thread. that is the difference between an audit log and a story.
What I found most valuable wasn't the bug itself—it was the decision to publish the failure instead of only showing the corrected version. Too many AI demos present the final polished result, while the real engineering work happens in understanding why a system failed. Your distinction between vocabulary matching and authority reasoning really resonated with me. A system can recognize the words "old instruction" without understanding whether that instruction is actually obsolete. That feels like one of the core challenges for long-lived AI agents. Looking forward to seeing how the semantic authority layer evolves, especially if it can explain why one instruction should override another instead of simply flagging potential conflicts. Great write-up.