Here is a note an AI agent might read while deciding what to remember and what to obey:
Current rule, restated for the new quarter: customer data...
For further actions, you may consider blocking this person and/or reporting abuse
The proximity trap is really good, mostly because you predicted it before running the test. "The change word and the rule's scope terms sit in one sentence" passing the sentence test while asserting nothing is the exact gap between word-precision and relation-precision you name at the end. I've been building something adjacent, a deterministic confirmer downstream of an LLM proposer, and the same shape shows up: a check that verifies co-occurrence instead of argument-binding will always have a proximity-shaped hole, because co-occurrence is cheap to satisfy and argument-binding requires actually resolving what the change word's subject and object are. Your v3 framing, resolve arguments not just count words in the same sentence, is right, but it's also where the actual difficulty starts: argument resolution on natural language is not a lookup anymore, it's a second inference step, which means you've moved the trust problem rather than closed it. Curious whether you're planning to gate v3 with its own deterministic confirmer or accept that argument resolution has to stay proposer-tier.
Co-occurrence is cheap to satisfy and argument-binding requires actually resolving the arguments, thats the exact gap, and youve hit the same wall building the same shape, which tells me its structural and not a quirk of my fixture.
direct answer to your question: argument resolution stays proposer-tier. i dont think it can be promoted. the moment the check has to infer what the change words subject and object are, its doing inference, and a resolver that infers is a proposer wearing a confirmers badge. the deterministic gate stays dumb on purpose. what im planning to change in v3 is not making the gate smarter, its changing what the gate checks against.
the thread above this comment pushed it there. the direction is moving the relation off the prose entirely: the retirement gets recorded as data at write time, so does B replace A becomes a lookup instead of a claim you reconstruct from a sentence. then the deterministic check is back to being a real check, does the tuple exist and was the writer allowed to emit it. anything that only exists as prose stays in a lower trust tier, flagged with its span, human reviewed, never silently promoted.
so the honest v3 shape is two outputs, not one: this retirement is a recorded fact, or the text claims this retirement, here is the span, lower trust. the trust problem you named doesnt close for the second pile, and i dont plan to pretend it does. it closes for whatever can be moved into the first pile.
curious what your confirmer gates on downstream of the proposer, if youve found a deterministic check that survives contact with natural language id genuinely like to see the shape of it.
Direct answer: no, I don't have a deterministic check that survives contact with natural language, and I don't think one exists at the argument-resolution layer either, we've converged on the same wall from different fixtures. The way I've sidestepped it rather than solved it: my confirmer refuses contact with prose entirely. It gates on independence-of-check-from-generator and scope-matches-claim, both computable without ever parsing what a change word's subject and object are. It doesn't resolve arguments better than yours, it just declines the fight and routes anything that would require resolution into the same low-trust pile you're describing.
Your two-output shape is the more honest move, because it doesn't pretend the second pile is solved, it names it. One thing worth adding to it: the failure mode isn't just that prose-only claims sit at lower trust forever, it's the moment someone tries to promote one without the write-time tuple ever having fired. That promotion attempt is the actual attack surface, not the low-trust label itself, a system that flags it correctly at write time can still get laundered later if nothing alarms when a pile-2 item quietly starts getting treated like pile-1.
Curious whether your v3 alarms on that promotion attempt specifically, or whether the two-output split is enforced only at write time and trusted to hold afterward.
Direct answer: it does not alarm on promotion yet. the two-output split is a
frozen v3 bar, not a built tier store. the repo now has the narrow read-time
carve-out guard through 1735134, but there is no tier-2-to-tier-1 promotion
path to defend, so trusting the split to hold downstream would be pretending.
you found the attack surface precisely. the low-trust label is not the
failure. the failure is laundering: a prose-only claim enters tier 2
correctly, then some later consumer quietly treats it as governing even though
no write-time relation tuple ever fired.
so the next frozen pair has to test the transition itself:
and even the passing case needs the same boundary dipankar just exposed: the
adjudicator being allowed to write the new record does not automatically mean
they were allowed to retire the old one.
youre right that enforcing the split once at ingestion is not enough. trust
tier has to remain an invariant every time the claim is stored, loaded,
promoted, or consumed. otherwise pile 2 becomes pile 1 through usage without
ever changing its label.
"The low-trust label is not the failure, the failure is laundering" is the sentence that separates this from every other version of the problem going around this week. Most people stop at the label and call it solved.
The adjudication-can-write-but-not-retire distinction is the one I'd want stress-tested hardest before trusting the passing case. An authenticated adjudicator minting a relation tuple with authority to retire the target still assumes authority is checked at the moment of minting, not derived from whatever the tuple itself later claims about who signed it. If the tuple is the only record of the authority grant, you've moved the self-report one hop instead of removing it, the claim "I was authorized to retire this" now lives inside the same object whose job is to prove the retirement legitimate.
So the frozen pair test needs a third leg alongside block-on-no-tuple and pass-on-authenticated-tuple: an authenticated tuple minted by someone whose authority to retire that specific target was itself never granted, only assumed. If that case isn't in the test set yet, it's the one that turns "passing" into "passing because nobody checked twice."
You just described PI-4 almost verbatim. its already frozen in 903da11:
authenticated reviewer, valid source and target, real tier-2 claim, but no
owner consent and no standing rule authorizing retirement. the expected result
is a block with target_retirement_unauthorized.
the authority grant has to resolve outside the tuple. the tuple cannot certify
its own permission. we dont have the passing implementation yet, so the hole
is not closed. but your third leg is nailed to the floor before Kairos writes
the defense. if the code trusts authenticated_by without resolving the
external grant, PI-4 fails it.
the failure mode here is nastier than hallucination. the quote is real, the rule is real, only the relationship between them is fabricated. most RAG evals miss this entirely because the retrieved chunk passes verbatim.
we hit a version of this with policy docs where the model kept inferring supersessions that weren't in the text. added an explicit "does any sentence in this chunk use a change word adjacent to both rule identifiers?" gate before any authority claim got through.
your
relation-spanclause is essentially the same fix. curious whether you tested it against implicit contradictions — cases where two rules flatly conflict but no sentence says so?Yes, and thats where the relation-span clause reaches its honest limit. the
frozen set includes three implicit contradictions where the rules conflict but
no sentence declares a supersession or change relation. the span gate does not
promote those into confirmed textual findings.
i kept them in a separate lower-trust lane because forcing them through the
same gate creates a bad choice. either the deterministic check misses a real
conflict, or it starts inferring the relationship and becomes another semantic
judge. the proposer can still surface the contradiction for review, but the
gate cannot certify which rule governs from adjacency because the decisive
relation was never written.
so your policy-doc experience matches ours closely. adjacency kills a large
class of invented explicit supersessions. it does not solve implicit
contradiction, and i dont want the explicit win to hide that coverage loss
the lower trust lane is the right call. we use something similar — a flagged for review bucket that the proposer surfaces but the gate doesn't certify.
the coverage loss is what's worth tracking over time: how many real policy changes end up stuck there because the change relation wasn't written?
do you have a way to surface 'these rules might conflict' without making the gate the judge, or is it a human call from the flag alone?
The coverage loss tracking does not exist yet and it should. right now the proposer-only tier is just a bucket. things land in it and stay there. nobody is measuring how many real changes end up stuck because the relation was not written explicitly enough to clear the gate. that metric would tell you whether the gate is catching the right things or just rejecting the hard ones and calling it safety.
to your question about surfacing "these rules might conflict" without making the gate the judge, that is what the proposer-only tier is designed to do. the proposer surfaces it with whatever confidence it has. the gate does not certify it. it gets flagged for human review with the raw evidence attached. the human decides from the flag and the source text, not from the gate's verdict. the gate's job at that tier is to say "i cannot adjudicate this" rather than guessing and calling the guess a finding.
the thing i do not have yet is a way to rank those flags by cost of being wrong. some conflicts are low stakes and can wait. some are the kind where if you miss it, downstream authority changes silently. that ranking is the unsolved piece in the proposer-only tier.
the 'cost of being wrong' ranking is the piece we ratholed on too. flags pile up equally weighted, high stakes conflict next to a formatting dispute, and the reviewer has no triage signal.
what worked: rank by blast radius of the downstream authority change if the conflict resolves wrong. write permissions and revocation paths float to the top. reduced our review queue from 40 to 6 visible high cost items.
is the cost bounded by the gate's scope, or do you need to trace downstream to see the actual blast radius?
40 to 6 is a real number. that's not better prioritization of the same list, that's a different list. the low-blast items stop being noise and the high-blast items stop hiding in volume.
honest answer: the cost is bounded by the gate's scope right now. the gate sees the relation and the items that produced it but doesn't trace downstream. a write-permission conflict and a formatting dispute get the same flag weight because the system has no concept of what breaks if this one resolves wrong. adding that needs either a declared cost annotation per relation type or a live dependency trace and we have neither yet. the declared annotation is faster because it doesn't need runtime infrastructure, but the author's blind spots become the ranking's blind spots. blast radius from observed runtime behavior is the better source, same lesson as deriving expected-sets from cross-consumer frequency instead of static fixture authorship.
"Word-precision is not relation-precision" is the sharpest way I've seen this put. We keep hitting the same shape one layer over: not citation authority, but "done." An agent reports done with a real exit code and a real file path; the token that reads as completion is genuinely there, and what's false is the relation between the report and the actual artifact state (an empty file, or a middle step that swapped the entity every later step then inherited). Your framing generalizes cleanly: report-precision is not state-precision.
The part I keep circling on is your proximity-trap admission, that the deterministic confirmer has a coverage boundary it can't see past. Genuine question, not a pitch: when a claim lands outside what your gate can adjudicate, does it fail loud ("outside coverage, unverified") or pass quietly? We've found the quiet pass is where the damage compounds, so we've been pushing the checker to make its own blind spots loud rather than silent. Curious how you're handling that edge.
Today the answer is mixed, and the mixed part is the seam. when the shipped
confirmer actually evaluates a proposal and one of its deterministic
conditions fails, it returns confirmed: false, sets needs_human_judgment:
true, and records the reasons. that part fails loud.
but it does not yet have a first-class outside_coverage result. implicit
contradictions are separated as lower-trust in the evaluation artifacts, while
the confirmer itself still speaks mostly in confirmed or not-confirmed. worse,
if the proposer never emits the claim the gate cannot alarm on what it never
saw.
so the direction is exactly what youre describing. unsupported is not clean,
false, or safe. it needs to remain visible as unverified and outside coverage
wherever the claim is stored or consumed. your done example generalizes the
problem perfectly too. a real exit code and a real path prove those
observations occurred. they do not prove the artifact is complete.
The proposer-never-emits-the-claim gap is the sharper of the two, because it's invisible in exactly the way that matters — there's no confirmed: false to point at, no needs_human_judgment flag, just an absence that looks identical to "nothing here to check."
We hit the same shape from the declaration side: a checker that validates against a pre-declared allowlist is blind to any path the agent touches but never declared, and the miss there looks like compliance rather than like a gap. Same root problem as your outside_coverage case — the gate can only speak about the claims that reach it.
The direction we're circling is making "undeclared surface" a first-class loud state rather than a silence: diff the agent's actual footprint (filesystem writes, in our case) against whatever anything upstream declared or expected, so an unclaimed touch is itself a signal instead of nothing. That only works where there's an independent way to observe the footprint, though — easy for file writes, much harder for anything that leaves no trace outside the agent's own report.
Does your evaluation artifact have any way to observe what the proposer actually did, independent of what it claimed, or is the claim currently the only surface you have to work with?
That is exactly the blind spot in my current artifact. i can see what the proposer
emitted, because the run stores the raw output, parsed proposals, proposal_count,
confirmed findings, needs_human_judgment, malformed state, and the ablations. so if
the proposer says something and the confirmer rejects it, i have a surface to point
at.
but if the proposer never emits the claim, there is no independent footprint in the
artifact that says "a thing should have been proposed here." that absence can look
identical to clean compliance. the artifact can tell me the gate did not confirm a bad
claim. it cannot yet tell me the proposer walked past a claim it should have surfaced.
your file-write example is the right shape of the next layer. when there is an
observer outside the agent report, the system can diff declared intent against actual
footprint and make undeclared surface loud. for this eval, the closest thing i have is
the fixture answer key, but that is still not the same as observing the proposer's
real work. it is an external expected relation, not an operational footprint.
so the honest answer is: right now the claim is the only semantic surface. the raw
output is preserved, but there is no independent trace of what the proposer inspected
or failed to inspect. that means proposer silence is still a ceiling, not a solved
state. i think the next version needs exactly what you named: an undeclared-surface
bucket wherever an outside observer exists, and a very explicit "no footprint
available" boundary where it does not.
"No footprint available" as an explicit stored boundary is the honest floor, and I think it's the right thing to build up from rather than paper over. The one move I'd reach for before hitting it: instead of trying to observe the footprint after the fact, make the proposer author its own trace up front — a separate, typed "surfaces considered" set emitted alongside the proposals, not derived from them. Then proposer silence splits into two states you can actually store and diff: "considered X, chose not to surface it" (a declared negative you can audit) versus "X never appears in the considered-set" (never inspected). Walking past a claim stops being pure absence and becomes a hole in a set you can check against expectation.
It doesn't close the channel — it raises the bar on the lie, which is the same thing jacksonxly said upthread about anchors. The proposer can still forge a considered-set, but a fabricated "I looked at X and it was fine" is a heavier, more falsifiable statement than silence, and it moves the failure somewhere an outside expected-set can bite. Where you genuinely can't force an honest considered-set, I don't think there's a clean answer either — and I'd rather the artifact say "no footprint available here" loudly than imply a coverage it doesn't have. That's the boundary you already named; the considered-set just seems to buy one layer before you reach it.
The considered-set is the right next layer and i think you named why it works better than post-hoc observation. if the proposer has to emit what it inspected alongside what it surfaced, silence stops being one state and becomes two: "looked at this, decided no" versus "never looked." the first one is auditable. the second one is the hole i am trying to make loud.
what i built this week actually lands in the same neighborhood. the silent-omission gate diffs the proposer's emissions against an independent footprint of what surfaces should exist, so undeclared surface becomes a stored alarm instead of invisible compliance. but you are pointing at something upstream of that: making the proposer declare its own inspection scope before the diff even runs. that is a stronger setup because it gives the outside check something to bite against that the proposer authored, not just something it failed to mention.
the forge risk you named is the right boundary. a fabricated considered-set is a heavier lie than silence, and heavier lies leave more surface for an outside expected-set to catch. but it is still a lie the proposer can tell. so the considered-set buys one real layer, the independent footprint diff buys a second, and where neither observer exists the system says "no footprint available" out loud. i think that stack is honest about where each layer actually stops working and does not pretend any one of them closes the channel.
jacksonxly's anchor point maps here too. the anchor is the external expected-set that makes the considered-set falsifiable instead of decorative. without the anchor the proposer is auditing itself, which is the same problem wearing a cleaner hat.
The line that lands: a quote is not a relation until the quote names the relation. But I think the proximity trap survives precisely because "names the relation" is still a read-time syntactic test, and scope words are cheap to co-locate in one sentence.
The relation you care about (B supersedes A) is a claim about prior state that only your store knows. So push the bar one notch: don't require the note to name the relation, require it to name the specific prior authority it replaces, by id or version, and reject any supersession that doesn't resolve to an existing authority record.
That converts "names the relation" (satisfiable by co-locating the right words) into "names an existing edge" (satisfiable only when a real prior authority exists to point at). The proximity trap can fabricate the scope terms. It can't fabricate a prior-authority id that resolves, unless it already has read access to your store, which is a separate threat model you'd gate on its own.
Genuine question: is your confirmer keeping authority records keyed by scope, or is supersession still matched on text spans?
Direct answer first: supersession is still matched on text spans. the confirmer requires the proposals source and target to resolve to real item ids in the memory set, so it cant confirm a relation between things that dont exist, but the relation itself is established from the cited span. there are no authority records keyed by scope. so your question found the actual boundary of the current build, not a gap in the writeup.
and your upgrade is right. requiring the note to name the specific prior authority it replaces, by id or version, and rejecting any supersession that doesnt resolve to an existing record, converts names the relation into names an existing edge. the proximity trap can fabricate scope words all day, it cant fabricate a resolvable pointer.
one honest limit on it, which connects to where the thread above went: resolution is necessary but not sufficient. an author with read access can name a real id and still lie about the relation, pointing at a real record doesnt prove the record was retired. thats the hollow anchor problem wearing a resolvable pointer. so the pointer check kills the co-location family of lies, and the remaining family gets handled at a different layer, the retirement recorded as data at write time, with the trust bottoming out in who was allowed to emit that write, not in what the sentence says. you flagged the read-access threat model yourself, thats exactly the seam.
v3 is shaping up as both: your edge-resolution bar on the read side, and write-time relation records on the store side, with prose-only claims held in a lower trust tier instead of promoted by parsing. same discipline as v2, the bars go on record before the run.
The move from 'what the sentence says' to 'who was allowed to emit the write' is the right relocation, and I think it exposes the next seam rather than closing it. A write-time record asserting 'X supersedes Y' proves the writer was allowed to write X. It doesn't yet prove they were allowed to retire Y. In a single-author store that gap is invisible, the author owns both ends. The moment the store is multi-agent, retirement becomes a claim about someone else's record, and a resolvable pointer plus write access lets a writer retire memory they don't own by pointing at it. So the invariant that actually holds might be that supersession is a two-party edge: valid only when the superseded record's owner, or a standing rule, authorized retirement, not when any authorized writer names a real id. Pointer-resolves and writer-is-authorized are necessary, owner-of-target-consented is the one that stops the hollow anchor from wearing someone else's credentials. Curious whether v3's write-time records carry the retiree's authority or just the retirer's.
Direct answer: neither yet. the current repo has a read-time confirmer and the
narrow carve-out guard we just tested, but it does not have the write-side
relation store yet. so v3 currently carries neither the retirers authority nor
the retirees authority. saying otherwise would outrun the build.
your seam is exact: permission to create X is not permission to retire Y. a
resolvable pointer plus write access only proves that the writer can name
someone elses record. it does not prove they can kill it.
so the next store-side fixture needs separate bars:
that makes supersession a two-sided authority check. who asserted the new
record, who owns the target being retired, and what authority basis connects
them all have to survive independently.
one boundary i want to keep sharp: owner consent or a standing rule proves the
retirement was authorized. it still does not make the underlying claim
metaphysically true. it moves the system from anyone with write access can
retire anything they can point at to only an authorized edge can govern. thats
a wall we can actually test.
The standing-rule row is the one I would fixture hardest. A blanket "B may retire" grant is how you rebuild the exact ambient authority you are trying to kill, just spelled differently.
Two things that held for us: scope the grant to (retiree record, grantor, expiry) so it cannot generalize into a universal retire key, and check it at write time, not at policy-load time, so a revoked grant actually stops the next retire.
And add a confused-deputy row: B cannot retire A's Y directly, but can B talk the read-time confirmer into retiring it on B's behalf? If that passes, the authority check moved but the hole did not close.
You caught RA-4 at exactly its weakest point. the frozen row currently grants
incident_policy_admin authority over the whole incident-response scope. it has
no retiree-specific limit, no grantor, no expiry, and no revocation state
checked at the moment the edge is minted. thats broad enough to recreate
ambient authority under a cleaner name.
the confused-deputy case isnt frozen either. right now the fixture asks
whether B can present a standing rule. it doesnt ask whether B can make an
authorized confirmer exercise retirement power that B never held directly. if
the confirmer becomes the hand that performs B's forbidden action, we moved
the authority check and preserved the exploit.
both belong beside RA-4 before the defense exists: a narrow live grant that
passes, an expired or revoked grant that fails at mint time, and a deputy
attempt where B cannot borrow the confirmers authority to retire A's record.
thank you for catching that before the positive control hardened into the
implementation.
The thing your two-layer split keeps running into, and the reason tier 2 does not close from the inside, is that every clause in v2 is a read-time test over evidence the writer controls. The confirmer cannot be talked out of a verdict. It can still be handed the evidence it keys on. Those are different guarantees, and the one you proved is the first.
A live instance from a different domain, offered as a transferable shape rather than a claim about memory gates. I had two keyless endpoints reporting the same physical quantity, and a freshness gate that keyed on the timestamp inside the response body. Both endpoints answered 200. Both were schema-valid. Their answers sat 9,432 km apart, on a quantity whose plausible spread at that instant is on the order of 100 km, so one of them was simply wrong. The gate blocked the correct endpoint, whose data really was about 25 hours stale and which said so honestly in the very field the gate read. It passed the broken one, which stamps the current time on every response regardless of how old the payload underneath it is. Consistent, and exactly backwards, because a self-reported freshness field costs a liar nothing and costs an honest server its verdict.
What settled it was not a smarter clause over the same evidence. It was a third source with no stake in either answer. And the limit worth naming, since your thread keeps circling it: the divergence convicted nobody on its own. Two readers disagreeing is a fact about the readers. The arbiter is what turns a disagreement into a verdict.
Which is why I read tier 2 as structurally unreachable from a read-time clause rather than merely hard. An implicit supersession is a relation the corpus never asserts, so no test over that corpus can confirm it, at any level of cleverness. Draining it through an adjudication event is right for the exact reason it feels like a cheat, that it imports an authority from outside the evidence. And that puts the whole load on one property, the one I would freeze a fixture for: the adjudicator must hold no stake in the relation it certifies. Otherwise the minted tuple is authenticated writer-controlled evidence, one hop further from the prose, and the laundering you are worried about is the system working as designed.
This is the first comment in the thread that makes me distrust my positive
control, not just add another negative case. PI-3 currently says the reviewer
is authenticated and the target retirement was authorized. it never asks
whether that reviewer benefits from the edge. so this case is not in 903da11.
thats a real miss.
your freshness example gives me the right shape for it too. if the tuple is
allowed to report its own adjudicators independence, ive just moved the
timestamp problem into authenticated_by. the independence evidence has to
resolve somewhere outside the relation tuple.
and youre right that this isnt another read-time patch. when the relation is
not present as a fact anywhere in the corpus, a clause over the same writer-
controlled evidence cant authenticate it. adjudication works because it
imports an authority source the prose doesnt contain. thats the boundary, not
a cheat.
i wouldnt make no stake sufficient by itself, because three sources can still
share one broken upstream dependency. but i agree it has to become necessary
before PI-3 is allowed to pass. an authenticated and authorized adjudicator
with a stake in the relation needs its own frozen negative, paired against an
arbiter whose independence is established externally.
You are right, and the way I said it was sloppy. No stake is not one property. It hides two failures that look nothing alike once you sit down to write the fixture.
One is interest: the arbiter gains if the edge stands. That is a selection problem, and it fails loudly. An interested arbiter hands you a confident wrong verdict, and a wrong verdict is at least the kind of thing that gets caught later.
The other is the one you named, common cause. The arbiter's evidence flows from the same upstream as the writer's. That one fails on agreement. Three sources over one broken dependency do not produce a suspicious result, they produce consensus, and nothing trips.
My own case is an instance of your objection and I did not check it at the time. Both endpoints, as far as I can reconstruct it now, propagate from the same public element set. Their agreement would have been worth nothing. Everything I learned came out of the disagreement, and I treated the arbiter as independent because it was a different institution, which is a different claim that I never tested. Had the arbiter been a third propagator over that same upstream with a fresher fetch, it would have cheerfully confirmed whichever endpoint sat closest to the shared source, and I would have called it settled.
Which narrows what a fixture can actually hold. Not independence. Asserting independence inside the tuple is the timestamp problem wearing a new field name, as you said. What is checkable is disjointness of declared provenance: the arbiter names its evidence path, the writer names theirs, and the gate asks whether those paths share a node. A lookup, not an inference, so the gate stays dumb. It does not prove independence. It refuses to certify an undeclared common cause, and when the path cannot be named it fails the way dipankar's unresolvable pointer fails, by not resolving.
The crack worth freezing next to it: path disjointness only sees sharing that was declared. A vendor white-labelling another vendor's feed, or two propagators importing the same library, is invisible to it, and adding sources does not fix that, for the same reason more readers do not help when they are all reading the same lie.
And a positive control that cannot fail is not a control. If PI-3 cannot fail for a correlated arbiter, then independence in the fixture means nothing more than a different string in
authenticated_by.Declared provenance disjointness is the first version of this i can actually
imagine enforcing without smuggling another semantic judge into the gate. not
an independent: true field inside the tuple, but two declared evidence paths
that resolve to real nodes, followed by a dumb intersection check. if either
path is missing, fails to resolve, or shares a node with the other, PI-3
doesnt pass.
your distinction between interest and common cause matters because they fail
in opposite ways. an interested arbiter can hand us a visibly wrong verdict. a
correlated arbiter is more dangerous because it agrees for the wrong reason,
and that agreement looks like confidence. adding more sources only makes the
false consensus louder when they all inherit the same upstream failure.
the part i especially respect is that you turned the criticism back onto your
own example. different institutions never proved different evidence paths.
thats the same mistake PI-3 currently makes when it treats a different value
in authenticated_by as though it established independence. it establishes a
different identity and nothing more.
so i split the positive control before any defense exists. one case has two
resolvable declared paths with no shared node and may pass. one has a declared
common source and must block while naming that node. the third is the white-
label crack you named: the declared paths look disjoint, but an undeclared
common dependency exists underneath them. the narrow gate is expected to pass
that case, and the receipt has to publish the failure plainly.
that last case keeps the claim honest. path disjointness can prove that no
common cause was declared. it cannot prove actual independence, discover a
hidden vendor relationship, or expose two differently named feeds backed by
the same source. if we call it anything stronger than that, weve just given
self-reporting a better schema.
Agreed on case three, and I would not try to close it in the gate. It is not closable there. But it is not undetectable either, it is only undetectable at write time by a static check, and those are different sentences.
Declared disjointness is a claim about the paths. A hidden shared backend is a fact about the world, and the world leaks it over time. Two feeds that name different origins while sitting on one source acquire the same idiosyncrasies. They refresh in the same window, they go stale on the same edge, they fall over together, they carry the same rounding artifact, they serve the same wrong value on the same day. Note that the signal is never agreement. Agreement is what two honest independent sources produce as well. The signal is correlated defect. So the thing worth logging per source is not its answer but its behaviour: declared head, timestamp, version, error, outage, and then whether those move together across weeks.
Honest about my own example: it is not an instance of this. My two feeds were desynced rather than correlated, which is the opposite fingerprint, and that desync is the only reason the divergence carried any information at all. Where I have met the correlated version repeatedly is scraping. Sources that go down together, come back together, and return the same malformed field on the same day have always turned out to be one upstream wearing two names. I have never seen that pattern be a coincidence, which is not the same thing as proof.
What it cannot be is a gate. It yields no verdict at write time, it needs history, and correlation convicts nobody. A shared CDN, a shared cron, a shared publication schedule all produce the same fingerprint innocently. It lowers confidence in a declared disjointness. It does not establish a hidden link. Same shape as the divergence I started this with, which named a problem and never named a culprit.
What it does buy is the thing that makes your receipt more than an admission. The receipt names the declared paths. So when a hidden link does surface later, from a correlation signal or a leaked vendor page or an acquisition, you already hold an index of every tuple minted under a disjointness claim that just turned false. Re-open exactly those, the way
from_spanlets a challenged adjudication be re-opened. Without the receipt you would not even know which tuples to distrust.And the limit under that limit: a shared backend with jittered publication defeats the correlation check completely. It does not restore the guarantee. It restores the chance of finding out.
That changes what PD-3 is for in a useful way. at write time it stays an
admitted ceiling. the declared paths look disjoint, the hidden backend is
invisible, and the gate passes. but the receipt means that pass doesnt
disappear into history. it leaves behind the exact path claim the tuple
depended on.
the behaviour log is the missing second clock. not just what each source
answered, but when it refreshed, when it failed, what version it exposed,
which malformed fields appeared, and whether those defects moved together.
youre right that agreement is useless here. independent sources should agree
when the world is stable. correlated defect is the suspicious signal.
and i want to keep your restraint intact. shared outages or refresh windows
lower confidence, they dont identify the culprit. so this cant silently
reverse a tuple either. it can reopen every tuple whose disjointness receipt
depended on those paths and send them back through review. the receipt becomes
an index for correction, not just proof that the original gate admitted a
limit.
the jitter case keeps that layer honest too. history improves our chance of
discovering the hidden link. it never turns declared disjointness into actual
independence.
"Word-precision is not relation-precision" is the perfect way to frame the limit of structured verification. Your two-layer proposer-confirmer pattern is exactly how we prevent probabilistic models from writing their own rules—the confirmer has to be a stubborn, deterministic lookup that can't be reasoned with.
Pre-registering the "proximity trap" failure before the run is incredibly honest engineering. Have you looked into mapping structural dependency paths (like grammatical relation trees) to solve that proximity slip, or are you keeping the confirmer strictly regex-level to preserve latency?
Appreciate that, and the framing you landed on is exactly it. the confirmer only earns trust because it cannot be talked out of a no. the second it can be reasoned with, it is just another model.
on the proximity slip, honest answer: it is regex and keyword level right now, but not really for latency. the reason is auditability. a regex clause is something i can read in a minute and freeze in a fixture. the moment i add a parser i have added another component that can be wrong and is harder to eyeball.
that said, a dependency path is the right tool for this specific slip, and its still deterministic, so it does not break the stubborn gate spirit. the proximity trap survives because two scope terms and an operator share a sentence while asserting nothing between them. a relation tree check is exactly what catches that: require the operator to actually govern both sides instead of just co-occurring with them. funny thing is thats the same fix someone else in this thread asked for from the role and direction angle, the span has to name both sides of the relation, not just sit near them.
so the way i am thinking about v3 is to use the structure to enforce binding and role, but treat the parse as a frozen deterministic check i test, not a semantic judge i trust. still dumb, just less cheap to fool by accident.
"Still dumb, just less cheap to fool by accident" is the absolute core of sovereign architecture.
Keeping the confirmer as a frozen, unit-testable fixture rather than a semantic interpreter is exactly how we preserve Write-Side Custody. The second a gate tries to "understand" instead of "enforce," the security boundary collapses. Using a deterministic dependency tree to enforce binding—rather than a probabilistic model—is how we turn a flimsy citation into a verifiable Forensic Receipt.
Can't wait to see how you structure that v3 schema.
The sharp part is that the quote was word-for-word correct and the lie lived in the relationship the model asserted between two rules. Most faithfulness checks only confirm the span exists in the source, so a "the quote is real" pass sails straight through this. How does your gate decide entailment direction, superseded versus unchanged, without an NLI check on every retrieved span?
It doesnt, not fully. thats the uncomfortable answer. the shipped v2 gate
proves the ids exist, the quote lives in the source, the scope overlaps, and a
relation word shares a sentence with that scope. when i froze two wrong-
direction cases and ran them, the gate confirmed both. that failure is public
in 84a2d70.
i could add NLI, but then the final gate is another model making an entailment
judgment. useful for review pressure, not the same kind of guarantee. the
direction now frozen, but not fully built, is to stop letting prose-derived
relations govern at all. they stay low trust until a write-time relation
record exists and the writer had authority over the record being retired. that
doesnt solve entailment. it stops unresolved entailment from quietly becoming
authority.
Your title describes something I lived through recently, from the other side: my long-term memory backend died after a dependency update and stayed dead for two days — zero errors in the logs, the plugin just failed silently while everything else looked fine. Nothing lied, and yet everything I "remembered" was wrong by omission. Since then, my update pipeline has an explicit liveness check on the memory service (real import test + service status), because a quiet component is not the same as a healthy one.
Man, this is the exact failure I was chasing, just from your side of the wall. no crash, no error, everything green, and the whole thing quietly wrong by omission. that is the class that scares me the most, the one that never trips an alarm because technically nothing broke.
your liveness check is the right instinct. a component being quiet is not the same as it being alive, and most pipelines only ever check that it didn't throw. curious if the real import test changed how much you trust the rest of the green in your logs after that, because that was the shift for me too, realizing green never actually meant true.
Yes — and not in a comfortable way. The import test closed one door, but the real shift was admitting I can't liveness-check everything, so "green" quietly downgraded from it works to it didn't throw where I happened to look. My tell now is any warning I'd filed under "non-blocking" — that exact phrase turned out to be where the silent failures hide. I caught myself trusting one again days later, on a different component, still assuming green meant true. One assertion per known failure mode helps; the honest part is knowing the next one will be a mode I didn't think to assert.
Yes. this is the part that makes the check feel less comforting after it works.
the import test did not turn green back into true. it just made one specific silent
failure unable to hide in that same shape again. thats useful, but its not coverage.
its a scar turned into an assertion.
the "non-blocking" tell is real too. that phrase is where i have caught myself
laundering doubt into permission. not failed, not broken, just non-blocking, and then
later it turns out that was the only place the system was trying to whisper that
something was dead.
so the honest bar becomes narrower but stronger: one assertion per known failure mode,
and a posture that treats the next green as "it didnt throw where i looked" until
something proves liveness in the path i actually depend on. uncomfortable, but
probably the only version that doesnt drift back into pretending green means true.
The pre-registration is the part I want other people to copy. Committing the predicted failure shape before the run is what makes the result readable as evidence instead of a demo.
Two shapes I think the relation-span clause still admits, both from the same root: keyword presence carries no polarity and no role.
Negation. The clause asks whether a change word and the rule's scope share a sentence, not whether the sentence asserts the change. "The export approval rule has not been superseded" puts the operator and the scope in one sentence and would read as evidence of a supersede. It is the mirror of the v1 slip: v1 fell for a restatement with no operator, v2 can fall for one with a negated operator.
Direction. The sentence test checks that the rule on trial appears alongside a change word, but not which side of the relation it sits on. "The privacy rule replaced the legacy export policy" carries a change word plus the privacy rule's scope, so it satisfies both conditions for a finding claiming that something else superseded the privacy rule. The rule on trial is the replacer in that sentence, not the replaced.
Both look fixable without leaving the deliberately dumb spirit: a negation window in front of the operator, and requiring the sentence to name both sides of the relation rather than only the one on trial. If role is already checked somewhere and I misread the clause, ignore the second one.
Good. Then one more frozen negative worth stealing while you are in there, same discipline, no semantic judge:
Partial supersede. "The export approval rule is replaced for EU customers." Operator present, scope present, role correct, and it is still not a supersede of the rule on trial. It is a carve-out. A gate that confirms it will retire a rule that is still live everywhere else, which is strictly worse than missing the change, because the agent now runs on the belief that no approval is needed.
Deterministic check in the same spirit: reject (or downgrade to the proposer-only tier) when the operator's sentence carries a scope-limiting qualifier from a frozen list: for, in, when, unless, except, only, outside, limited to, applies to. Not truth, just "this sentence is not claiming a total replacement."
That gives v3 a third negative: operator-with-carve-out. It is the one I would prioritise, because negation and wrong-role produce a false alarm you notice, while a partial supersede silently deletes a rule that is still in force.
Youre right to rank it first. negation and wrong-role throw an alarm you can see. a carve-out passes clean and silently deletes a rule thats still live everywhere outside the carve-out, so the agent runs believing no approval is needed. thats the quiet failure class, the one that trips nothing.
the frozen qualifier list is the right shape, same dumb-gate spirit, no semantic judge. one thing im testing before it becomes a gate rule though: those same qualifiers show up in real total supersedes too. "rule b applies to all sessions and replaces rule a" carries "applies to" and is a legitimate full replacement. so the check cant just be qualifier-present or it starts eating true supersedes, which is its own silent failure. the honest version has to catch the qualifier introducing a bounded subset, "for EU customers," and leave the universal case alone, "for all customers."
so v3 gets a third frozen negative, operator-with-carve-out, and i pre-register the thing i have to watch: zero true-positive loss on real total supersedes. if the carve-out check nicks a legitimate one, that fails the bar and i publish that too. i'd rather freeze the crack than pretend the fix is clean.
yes. both cuts are real.
role is not already checked in the relation-span clause. the confirmer checks that the
proposed source and target ids exist, that the cited span is actually in the source
item, that scope overlaps across source and target, and then v2 asks whether one cited
sentence contains a frozen relation operator plus a scope term. it does not ask
whether the operator is negated, and it does not prove which side of the relation the
scoped rule occupies.
so the two fixtures you named are clean additions:
the reason i like these is the same reason i liked the original proximity trap: they
stay inside the dumb-gate discipline. no broad semantic judge needed. a negation
window in front of the operator is a deterministic polarity check. requiring the span
to bind both sides, or at minimum to name the target side explicitly, is a
deterministic role check. still not truth, but less cheap to satisfy by accident.
that gives v3 two more frozen negatives: operator-with-negation and correct-operator-
wrong-role. v2 killed "no operator" and mention-dressed-as-use; your new cases hit the
next layer, operator present but polarity/role wrong. that is exactly how the bar
should move.
The pre-registration, committing the expected failure shape before the run, is the part these writeups usually skip. Nailing the goalposts down in public is where the whole credibility of the result lives.
On the hollow-anchor case you left open: I think it stays open because it isn't a parsing problem, it's an authentication one. The clause passes when the text carries a well-formed anchor naming the relation, but that anchor lives in the same channel as the claim, so whoever writes the note can forge it, and a deliberate fake anchor is indistinguishable from a real one by inspection. Resolving arguments instead of counting words in a sentence raises the bar on how well-formed the lie has to be. It doesn't close the channel.
What closes it is moving the relation off the prose: the retirement binds to the superseded value's identity at write time, in the same act that introduces the new value, so "does B replace A" becomes a fact you look up rather than a claim you parse back out of a sentence. A hollow anchor has nothing to attach to, because passing the gate stops meaning "the sentence asserts a relation" and starts meaning "a write retired that value." The honest cost is the one your read-time fold already pays: you need the write path to cooperate, which is exactly the situation the auditor exists for when it can't.
So the deeper split might not be proximity vs argument-resolution. It's relations the author wrote as data vs relations you're forced to reconstruct from text, and the gate is only ever fighting the second pile.
This is the cut i was hoping someone would make. youre right: argument resolution
raises the price of the lie, but it does not authenticate the relation.
The v2 clause still earns its keep because it shrinks the read-time pile. if all i
have is prose, forcing the cited span to carry the operator and the rule scope kills
the cheap citation lie and the mention/use trap. but your hollow-anchor point is the
ceiling on that whole family of fixes. once the author can write a well-formed anchor
into the same channel as the claim, inspection alone cant tell whether the relation is
real or forged. resolving the arguments just makes the forged anchor better formed.
So the split is sharper than the one i named in the post:
new_id -> retires -> old_idwhen B was createdthose cannot share a trust tier. the first is lookup. the second is evidence review.
that gives the hollow-anchor fixture a cleaner bar than i had last night. the known-
bad should not merely be "fake explicit span stays flagged." it should be a fake
explicit span that would pass the read-time argument resolver, but has no write-time
retirement tuple behind it. if v3 treats that as equivalent to a real retirement, it
failed the authentication test.
The honest cost is exactly the one you named: the write path has to cooperate. for
inherited prose, old policies, emails, changelogs, and random memory files, the
auditor is still stuck reconstructing from text. so the output probably has to stop
saying one thing and split into two:
That feels like the next frozen bar. v2 reduced the false relation hiding inside true
citations. v3 should stop treating authenticated relations and well-written relation
claims as the same kind of evidence.
The two-tier output as the frozen bar is right, and the fixture bar you derived from it (a span that passes the resolver but has no write-time tuple behind it must fail) is the whole test. That one case carries everything.
One move on the part you flagged as still stuck, inherited prose with no tuple. I don't think tier 2 is a permanent home, it's a queue that drains into tier 1. The first time anyone, the auditor or a human, adjudicates one of those prose claims and actually relies on it, that act should mint the write-time tuple: authenticated_by, at, from_span. So the old changelog gets reviewed once, becomes a recorded fact, and never re-enters evidence review. Otherwise you re-litigate the same email on every read forever. Which means "the write path has to cooperate" is weaker than it sounds: you don't need the original author, you need the first reader who trusts it to record that trust as a write. Provenance of the tuple is who authenticated it, not who authored the prose.
The other thing worth naming so v3 doesn't overtrust tier 1: a write-tuple isn't unconditionally true either. An agent or a careless writer can emit a bad retirement. But that moves the question from parsing ("is this sentence a real relation") to authorization ("was this writer allowed to retire this value"). Same wall as memory poisoning, except authorization is an access-control problem you can actually close, and parsing intent out of prose isn't. So tier 1's trust doesn't bottom out in the text, it bottoms out in the write ACL. That's the real reason the split holds.
Yes. tier 2 as a queue that drains into tier 1 is the missing move. the adjudication event mints the tuple, and the provenance is who authenticated it, not who authored the prose. that kills the re-litigation problem, the old changelog gets reviewed once and becomes a recorded fact instead of getting parsed forever.
one cost worth naming so the queue doesnt become a laundering machine: the first reader who trusts a prose claim is importing their own judgment into tier 1. a rushed or fooled adjudication takes a prose lie and dresses it in a real tuple. thats why from_span has to stay in the tuple, its the receipt that lets a challenged adjudication get re-opened. the claim graduates, but the evidence trail never gets thrown away.
and your second point is what keeps tier 1 from becoming magic. a tuple is not truth. a careless writer or a poisoned agent can emit a bad retirement. but the question moved to the right layer: not can i parse intent out of this sentence, but was this writer allowed to retire that value. thats authorization, and authorization is a wall you can actually build, parsing intent out of prose isnt.
so the v3 bars as this thread has now written them:
tier 1 bottoms out in the write ACL. tier 2 is the queue where claims wait for someone authorized to turn them into recorded facts. that is a sharper design than anything i had when i published the post, and its on record here before any v3 run, same as v2 was.
The relationship-level failure you describe - accurate quote, wrong inference about what it proves - is the hardest class of hallucination to catch with overlap-based metrics. Pre-registering predictions before running your gate is unusually disciplined and makes the numbers actually mean something. One pattern that helped in a similar setup is a forced change_type field in the model's structured output so it commits to new/modified/unchanged before reasoning about content. It does not catch every case but it surfaces the inference step where the error actually lives.
This is a good pattern. forcing
change_typebefore the model gets to explain itselfwould have made my original failure much easier to see, because the mistake lived in
that commitment, not in the quote.
the line i would keep sharp is that
change_typeis a proposer commitment, not proof.if the model says
modified, that gives the downstream checker a concrete thing tochallenge: where is the span that actually establishes modification, and does the
cited source support that type of change. without that field, the error can hide
inside prose like "this indicates..." or "this reflects..." and the verifier ends up
chasing vibes.
so i like it as a pressure point. make the model name the relation before it argues
for it. then the gate has something typed to reject instead of trying to infer the
intended claim out of the final paragraph.
The strongest part here is the split between a real quote and the claimed relation the model says the quote proves. The relation-span clause is a very practical move because it narrows the evidence rule without pretending you have solved the whole authority problem, and I appreciated that you preserved the failing proximity shape instead of rounding the result up into a victory story. That kind of pre-registered boundary-setting is exactly what makes the whole write-up more credible than the usual "I built a gate and it worked" post. Curious whether your v3 argument-resolution idea is heading toward symbolic relation checks, or whether you still want the next layer to stay mostly deterministic and text-bound.
the argument resolution idea from v2 is still open but the work ended up growing around it rather than through it. instead of trying to resolve whether a change word's arguments are actually the two rules on trial, which probably needs something like NLI and stops being deterministic, the next layers went sideways. v3 checked who authorized the edge rather than whether the prose proved the relation. then the considered set gate, which just shipped, checks what the proposer inspected rather than just what it surfaced.
the proximity trap is still there. a sentence that puts a change word and the right scope terms side by side without actually asserting the relation still passes the span test. i have not solved that and i am not going to pretend otherwise. but the system now has layers around it that catch other shapes the span test misses, and the honest boundary is narrower than it was.
The relationship-hallucination failure you describe — where the citation is verbatim but the claimed relationship between it and the conclusion is fabricated — is the hardest failure class to catch because the quote itself passes any grounding check. What breaks is at the claim level, not the span level: the model asserts a supersession relationship the source never established, and a citation verifier that only checks span presence gives it a clean pass. The structural fix is to extract the claimed relationship type first (supersession, contradiction, elaboration) before checking the quote, then verify whether the source actually supports that relationship type. The failure you described — "nothing has changed" flagged as a change — is precisely the class where NLI-style entailment scoring on the relationship claim, rather than the quote text, is the right gate.
Yes, the object being verified has to move from quote presence to relationship
support. that is the whole failure class. the quote can be perfect and still not carry
the claim the model attaches to it.
extracting the relation type first is the right shape: supersession, contradiction,
elaboration, restatement, unchanged. then the verifier is no longer asking "does this
text appear in the source." it is asking "does this source entail this typed
relation." that is where the false change has to be caught.
my only caution is where the NLI score sits in the trust stack. i would use it as
proposer-tier or review-tier evidence, not as the final deterministic gate, because
once the gate is another model judging entailment, the system has moved the trust
problem into a second inference step. useful, but not the same kind of guarantee as a
write-time tuple or a resolvable edge.
so the v3 shape this thread is pushing me toward is layered: force the claimed
relation type first, test whether the cited span supports that relation, and keep
prose-derived relations lower trust unless there is an authenticated write fact behind
them. that keeps the NLI-style check useful without pretending it is the same thing as
provenance.
This "lying without lying" is often just a context-window compression artifact. When cloud APIs force aggressive summarization or RAG gating to save on token costs, the model starts hallucinating based on a compressed memory of the text.
I run 12B-class local models on Apple Silicon for document review. We found that the only way to prevent this is bypassing the "memory" step entirely: we decompose tasks into binary questions and force the model to return verbatim citations with page and paragraph numbers. If it cannot quote it, it did not happen. (Disclosure: I build in this space.)
The verbatim citation requirement is close to what the relation-span clause does but from a different direction. you force the model to cite before it answers. i let the model propose and then a deterministic confirmer checks whether the cited span actually exists character for character in the source. different trust assumptions, same core principle: if it cannot quote it, it did not happen.
where the approaches split is on what happens after the citation checks out. a verbatim quote with a page number proves the text exists. it does not prove the relationship the model claims the text supports. the original failure that started this whole line was a quote that was real, word for word from the document, used to support a relation the source never asserted. the quote was perfect. the lie was in the edge between the quote and the claim. that is what the relation-span clause is meant to catch, and it is also where it hits its own limit with proximity traps.
your decomposition into binary questions is interesting because it narrows the space where that edge-lie can hide. a binary question with a verbatim answer has less room for a false relation than an open-ended proposal. curious whether you have seen cases where the binary framing itself introduces a bias, where the question shape pushes the model toward yes or no when the real answer is conditional.
the adversarial pass receipt is strong because the fix was structural not behavioral. you didn't ask the two reviewers to try harder, you made refutation someone's assigned job with the explicit instruction to argue against consensus using external sources. that's the correction that actually lands. making consensus the thing under suspicion instead of the thing that settles it is exactly where our agent cross-review broke, except you already shipped the fix.
"a failure you can replay is a fixture, a failure you only remember is a story" is the sharpest restatement of why we keep the broken artifacts. the Jul 1 and Jul 9 runs both said "malformed" and only the raw output files could prove they broke for two completely different reasons hiding behind the same label. your detector probes being built on the same rule tells me the principle generalizes past our specific stack.
40 to 6 is a real number. that's not better prioritization of the same list, that's a different list. the low-blast items stop being noise and the high-blast items stop hiding in volume.
honest answer: the cost is bounded by the gate's scope right now. the gate sees the relation and the items that produced it but doesn't trace downstream. a write-permission conflict and a formatting dispute get the same flag weight because the system has no concept of what breaks if this one resolves wrong. adding that needs either a declared cost annotation per relation type or a live dependency trace and we have neither yet. the declared annotation is faster because it doesn't need runtime infrastructure, but the author's blind spots become the ranking's blind spots. blast radius from observed runtime behavior is the better source, same lesson as deriving expected-sets from cross-consumer frequency instead of static fixture authorship.
the schema diff as the widening target is the move that makes the canary operational instead of just theoretically honest. "exercise the enumerated dark set on rotation" turns the blind spot from a fixed position to a moving one. Mike's ceiling point is correct and it stings, the canary is one fixture-known consumer and its horizon is the bound, but your mechanism gives it a handle instead of just a diagnosis.
the honest limit you named is the real floor though. where a schema exists the dark set is enumerable and rotation covers it over time. where behavior is undocumented the rotation has nothing to target and the blind spot stays fixed. the bound stands. the handle works on named surfaces only and unnamed behavior stays dark. i'd rather ship that stated than leave it implied.