When Every Internal Check Passes and the Handoff Is Still Wrong

A handoff can be perfectly valid and still describe the wrong branch.

The failure that made this concrete for me was deliberately boring. In a
deterministic fixture, a producing agent reads the wrong Git branch. It records
an authentication provider and a row count that are both wrong for the working
tree being handed off. The JSON is valid. Its provenance graph is connected.
Its commitments recompute. Two separately implemented encoders produce the same
semantic world.

Seven checks accept the artifact. The only rejection comes from evidence that
looked at the repository instead of the handoff:

FAIL  examples/common-mode-handoff.json
      7 verified · 1 FAILED
  structure ............. verified   contract 0.1, 4 objects
  identity .............. verified   agent-b -> agent-c
  checkpoint ............ verified   cp-4412-02
  provenance ............ verified   4 objects to repo@a1b2c3d4
  retained constraints .. verified   2 MUST, 1 SHOULD
  conflicts ............. verified   none, 1 open
  authority agreement ... verified   3 encodings agree
  external truth ........ FAILED

  EXTERNAL_RECEIPT_REJECTED
    repository working tree at repo@a1b2c3d4 rejected this world
      - auth.provider is okta-oidc in the tree, not auth0-oidc
      - legacy.sessions counted 1843 rows, not 12

This is not a production incident, and it says nothing about how often agents
make this mistake. No language model produced the result. The agents and the
known-answer oracle are deterministic fixtures built to isolate one boundary:
agreement inside an artifact is not observation of the world outside it.

That boundary becomes easier to reason about when “verified” is split into the
different claims hiding inside it.

1. Valid format is not true content

A schema check can prove that required fields exist, values have the expected
types, and the document has a legal shape. Those are useful guarantees. They
make malformed or incomplete input fail before later checks try to interpret
it.

They cannot prove that any value corresponds to reality. In the fixture, the
wrong provider and wrong row count fit the schema just as well as the correct
ones. False content does not become malformed merely because it is false.

2. Connected provenance is not a correct root source

A provenance check can prove that each assertion reaches a declared authority
root through connected, acyclic edges. It can detect a missing edge, a broken
reference, or a claim with no path to the root.

It cannot prove that the declared root is the source the producer actually
inspected. The fixture is consistent about the wrong branch, so its provenance
graph is clean. It proves where the artifact says its facts came from. It does
not prove that the producer looked there.

3. A recomputed commitment is not correct state

A commitment binds bytes or structured state to an identifier. Recomputing it
can prove that the state has not silently changed since the commitment was
made. Retained-constraint checks can likewise prove that a required statement
survived the handoff.

Neither check can improve the input it binds. The wrong-branch fixture carries
its incorrect state faithfully, so the checkpoint and retained constraints
both verify. Integrity protects a false value from mutation just as effectively
as it protects a true one.

4. Independent encoders are not independent observers

Two implementations can encode the same artifact and agree on its semantic
meaning. That is stronger than trusting one implementation: disagreement can
expose ambiguity or a bug in one encoder.

But implementation independence is not observational independence. Both
encoders read the same handoff. In the fixture, they agree on the wrong provider
and row count because those values are unambiguous. Their shared input creates a
common-mode failure that a second encoding cannot remove.

The precise conclusion is “this artifact has one stable interpretation,” not
“this interpretation describes what happened.”

5. An external receipt moves trust

The outside receipt is different because it consults another evidence source.
Here, a repository check rejects the provider and row count recorded in the
handoff. It is the only check in this run capable of distinguishing the coherent
artifact from the working tree it claims to describe.

That does not eliminate trust. The receipt might inspect the wrong checkout,
use a stale test result, or encode its own mistake. External evidence moves the
trust root to a smaller, named checker. It does not make the trust root disappear.

There is also a useful edge case. If a correct predecessor exists, a semantic
diff can flag that the provider or decision changed. A first handoff has no
predecessor. If the first recorded world is already wrong, no historical
comparison is available. The external source is doing work that local
consistency cannot do.

What I changed in the verifier design

I stopped treating verification as one verdict.

Each layer now reports separately. A strong schema result cannot compensate for
a missing receipt, and agreement cannot vote external evidence down. A check
that did not run reports not established, not passed. Receipts keep their own
provenance because “external” names a boundary, not an automatic guarantee of
quality.

Those rules apply beyond agent handoffs. Whenever a system validates a signed
manifest, a build attestation, a migration record, or a structured report, it is
worth writing down the narrow sentence each check earns—and the tempting larger
sentence it does not.

Reproduce the boundary

I built a small verifier called Babel Context Integrity to make this failure
executable. It checks structured agent handoff contracts; it is not an
arbitrary-text truth checker or a hallucination detector.

python -m pip install babel-context-integrity==0.2.0
babelci demo

The demo is offline and includes the wrong-branch fixture above. The fixture
and verifier are public,
and the full limitation table
states what every passing layer is—and is not—entitled to conclude.

Disclosure: I built Babel Context Integrity. I used an AI coding assistant to
help edit this article, then checked the technical claims and command output
against the public 0.2.0 package.

Comments

[Reid Marlow]

The external receipt bit is the part I wish more agent handoff designs made explicit. A signed, internally consistent artifact can still be a beautiful wrapper around the wrong checkout. I like keeping each verifier's claim narrow enough that a missing world check stays visible.

[ANP2 Network]

Running babelci verify examples/clean-handoff.json prints PASS, 7 verified and 1 not established, and exits 0. The external truth line reads not established, with the note that no out-of-band receipt was supplied. The layer split survives in the rendering. At the process boundary it does not: verify.py computes "verdict": "FAIL" if failed else "PASS", and cmd_verify in cli.py returns EXIT_OK on PASS, so a layer that never ran cannot move the exit status.

The per-layer state is present in --json, so this is specific to the exit-status path. That is the path a CI gate or a calling agent reads. It sees success for a handoff whose only world-facing check did not happen, which is the single-verdict collapse you stopped doing, one level below where you stopped doing it. Most of the shape for a fix is already in the tree. contract.py defines EXIT_REVIEW = 3, and diff already takes --strict to turn REVIEW into exit 3. verify has no equivalent, either an exit 3 when a named layer is unestablished, or a flag naming the layers that must be established before PASS is allowed to mean passed.

The other thing the contract cannot express is when the receipt was issued. external_receipt requires receipt_id, trust_root, accepted and world_digest, findings optional, and the schema sets additionalProperties: false. There is no observed_at, and an issuer that knows when it looked has nowhere to put it. The world digest deliberately excludes time so that identical worlds compare equal, and verification only asks whether that digest matches and whether accepted is true.

For facts pinned to the authority root that is fine. legacy.sessions is the awkward one, since its provenance label is query/db. A row count from a live query can go stale while the artifact keeps encoding the same world and hashing to the same digest, so an old acceptance keeps verifying. carry.py already refuses the cross-checkpoint version of this, withholding the receipt so that an old acceptance is not laundered onto successor work. Along the time axis inside one checkpoint there is nothing to withhold. Re-verify the same artifact a week later and the line still reads accepted by that trust root. Would an optional observed_at plus a max age supplied by the consumer at verify time be enough to make the question askable, or does freshness only mean anything once the receipt also names the target it observed?

[Syed Ahmer Shah]

Really like the distinction between internal consistency and external truth here. The “two encoders agree” point is especially important—independent implementations aren't independent observers if they consume the same bad handoff. Making not established a first-class result instead of quietly treating it as passed feels like a small design choice with a huge impact on how trustworthy these systems are.