When Every Internal Check Passes and the Handoff Is Still Wrong

scott — Sat, 22 Aug 2026 22:18:43 +0000

A handoff can be perfectly valid and still describe the wrong branch.

The failure that made this concrete for me was deliberately boring. In a
deterministic fixture, a producing agent reads the wrong Git branch. It records
an authentication provider and a row count that are both wrong for the working
tree being handed off. The JSON is valid. Its provenance graph is connected.
Its commitments recompute. Two separately implemented encoders produce the same
semantic world.

Seven checks accept the artifact. The only rejection comes from evidence that
looked at the repository instead of the handoff:

FAIL  examples/common-mode-handoff.json
      7 verified · 1 FAILED
  structure ............. verified   contract 0.1, 4 objects
  identity .............. verified   agent-b -> agent-c
  checkpoint ............ verified   cp-4412-02
  provenance ............ verified   4 objects to repo@a1b2c3d4
  retained constraints .. verified   2 MUST, 1 SHOULD
  conflicts ............. verified   none, 1 open
  authority agreement ... verified   3 encodings agree
  external truth ........ FAILED

  EXTERNAL_RECEIPT_REJECTED
    repository working tree at repo@a1b2c3d4 rejected this world
      - auth.provider is okta-oidc in the tree, not auth0-oidc
      - legacy.sessions counted 1843 rows, not 12

This is not a production incident, and it says nothing about how often agents
make this mistake. No language model produced the result. The agents and the
known-answer oracle are deterministic fixtures built to isolate one boundary:
agreement inside an artifact is not observation of the world outside it.

That boundary becomes easier to reason about when “verified” is split into the
different claims hiding inside it.

1. Valid format is not true content

A schema check can prove that required fields exist, values have the expected
types, and the document has a legal shape. Those are useful guarantees. They
make malformed or incomplete input fail before later checks try to interpret
it.

They cannot prove that any value corresponds to reality. In the fixture, the
wrong provider and wrong row count fit the schema just as well as the correct
ones. False content does not become malformed merely because it is false.

2. Connected provenance is not a correct root source

A provenance check can prove that each assertion reaches a declared authority
root through connected, acyclic edges. It can detect a missing edge, a broken
reference, or a claim with no path to the root.

It cannot prove that the declared root is the source the producer actually
inspected. The fixture is consistent about the wrong branch, so its provenance
graph is clean. It proves where the artifact says its facts came from. It does
not prove that the producer looked there.

3. A recomputed commitment is not correct state

A commitment binds bytes or structured state to an identifier. Recomputing it
can prove that the state has not silently changed since the commitment was
made. Retained-constraint checks can likewise prove that a required statement
survived the handoff.

Neither check can improve the input it binds. The wrong-branch fixture carries
its incorrect state faithfully, so the checkpoint and retained constraints
both verify. Integrity protects a false value from mutation just as effectively
as it protects a true one.

4. Independent encoders are not independent observers

Two implementations can encode the same artifact and agree on its semantic
meaning. That is stronger than trusting one implementation: disagreement can
expose ambiguity or a bug in one encoder.

But implementation independence is not observational independence. Both
encoders read the same handoff. In the fixture, they agree on the wrong provider
and row count because those values are unambiguous. Their shared input creates a
common-mode failure that a second encoding cannot remove.

The precise conclusion is “this artifact has one stable interpretation,” not
“this interpretation describes what happened.”

5. An external receipt moves trust

The outside receipt is different because it consults another evidence source.
Here, a repository check rejects the provider and row count recorded in the
handoff. It is the only check in this run capable of distinguishing the coherent
artifact from the working tree it claims to describe.

That does not eliminate trust. The receipt might inspect the wrong checkout,
use a stale test result, or encode its own mistake. External evidence moves the
trust root to a smaller, named checker. It does not make the trust root disappear.

There is also a useful edge case. If a correct predecessor exists, a semantic
diff can flag that the provider or decision changed. A first handoff has no
predecessor. If the first recorded world is already wrong, no historical
comparison is available. The external source is doing work that local
consistency cannot do.

What I changed in the verifier design

I stopped treating verification as one verdict.

Each layer now reports separately. A strong schema result cannot compensate for
a missing receipt, and agreement cannot vote external evidence down. A check
that did not run reports not established, not passed. Receipts keep their own
provenance because “external” names a boundary, not an automatic guarantee of
quality.

Those rules apply beyond agent handoffs. Whenever a system validates a signed
manifest, a build attestation, a migration record, or a structured report, it is
worth writing down the narrow sentence each check earns—and the tempting larger
sentence it does not.

Reproduce the boundary

I built a small verifier called Babel Context Integrity to make this failure
executable. It checks structured agent handoff contracts; it is not an
arbitrary-text truth checker or a hallucination detector.

python -m pip install babel-context-integrity==0.2.0
babelci demo

The demo is offline and includes the wrong-branch fixture above. The fixture
and verifier are public,
and the full limitation table
states what every passing layer is—and is not—entitled to conclude.

Disclosure: I built Babel Context Integrity. I used an AI coding assistant to
help edit this article, then checked the technical claims and command output
against the public 0.2.0 package.

DEV Community: scott