DEV Community: dormitivegit

Auto-compact can lose context. What can a verifiable handoff actually prove?

dormitivegit — Fri, 13 Nov 2026 15:09:56 +0000

What actually gets lost

Claude Code compacts a long conversation into a summary when the context window fills. Some things survive by
design: the docs list the project-root CLAUDE.md, auto memory and invoked skills (up to a size cap) as
re-injected after compaction. What does not survive intact is the conversation itself: "The summary keeps your
requests and key code snippets, but detailed instructions from early in the conversation may be lost." A new
session goes further: it "starts with a fresh context window, without the conversation history from previous
sessions." (/clear keeps the old session on disk; you can /resume it, but the new context starts empty.)

So the gap is task state that lives only in the conversation: the approach you already rejected, the two tests that
still fail, the deploy nobody has authorized yet. Practitioners see it:

One DEV author tracked 18 instructions in each of two real conversations and found, after three compactions, "I had lost 33% of the instructions I tracked in one conversation and 57% in the other" — measured as whether each item was still written down, not whether it was followed (source).
Agents redo failed work because "commit messages describe the diff that shipped, not the dead ends" (source).

When a handoff is worth it

Often it isn't. The AI Hero handoff guide puts it plainly: "/compact unless something is travelling."
(source). Same task, same session: compact. Persistent rules: CLAUDE.md.

A handoff earns its cost when work moves to a fresh session, another agent or a colleague and resuming wrong is
expensive. Compare:

Plain:       Migration mostly done. Fix remaining issues and continue.
Structured:  Rejected: approach A (evidence: …)   State: 2 failing tests — recheck before trusting
             First action: inspect the failures   Deploy: NOT authorized

The second one is what session-handoff (Apache-2.0) asks the
writer to produce: state and where it is recorded, what was corrected and must not be repeated, the first authorized
action, and commands that re-verify anything that may have changed.

Three layers of checking — and what each cannot prove

Layer	What it specifies or checks	What it does not prove
Handoff contract (`SKILL.md`)	what a handoff must contain	that the content is true
Lint (18 rules)	form: unanchored claims, missing cold read by a context-free reader, a pasted reading that no longer matches the file	correctness — only 9 of the 18 rules have a registered test that shows them catching the failure they target
Readout (`handoff_readout.py`)	numbers computed from the session transcript, pasted verbatim, reproducible byte-for-byte	that the detectors are valid, or that the next session acts correctly

Abbreviated readout output (the full block is eight lines with an empty lesson table, nine with registered lessons):

READOUT_SNAPSHOT  transcript path · lines read · sha256 · last timestamp · readout-script and lesson-table hashes
READOUT_PATTERNS  per registered lesson: violations / opportunities (previous five windows)
READOUT_RECEIVE   was a read call on the handoff file detected in the three tool calls after it was referenced
READOUT_RECOMPUTE the exact command that reproduces this block

The public lesson table ships empty: snapshot, receive, cost and size work out of the box; per-lesson counts
need you to register patterns (references/readout.md). Reproducibility tells you what was measured, not whether
the measurement is right.

Four measurement failures an independent review found

Before collecting any outcome data, a second model reviewed version 2.6.0 in two phases: it sealed its own findings
(hashes in a file) before it was allowed to open the author's numbers, and that order was checked from its
transcript timestamps. This is an internal review with a different model, not external replication. It returned
fix before retest; all four findings were reproduced locally before fixing:

Wrong observation source. The working-directory lesson read 0 in a session whose writer admitted moving the working directory. The detector parsed command text and watched only some directories. It now counts the host's environment records whose new working directory is neither the start directory nor a temp directory. (The per-record cwd field was tried first and dropped: it flips between record types — one session showed 186 "changes".)
Threshold from the wrong population. Two rollback lines would have fired 44% and 16% of the time with nothing changed (bootstrap over baseline sessions). One became a direct count; the other was re-calibrated to 5% on the reference population — in-sample; the out-of-sample rate is not yet known.
The measurement disturbed itself. One way of pasting the block back into the file changed the size reading, so recompute stopped matching.
Mixed denominators. A summed rate added lessons with different denominators; one lesson was 59% of all violations.

Separately, two coders labelling 211 user messages for "requirement not met" agreed at κ = 0.27 (one coder labelling
twice: about 0.78). That label is not stable enough to be a baseline, so it isn't one.

These numbers come from the author's private session corpus, which is not published (raw transcripts contain paths
and code); the repository ships the detectors, tests and synthetic fixtures, not that corpus.

What is established, and what is not

Publicly verifiable: the repository ships the readout implementation and regression fixtures, and CI passes on Linux with Python 3.12 and 3.13; the readout gives byte-identical output when the transcript snapshot, handoff file, prior-window inputs, script version and paths are held constant.
From the author's private evaluation (corpus not public): the registered lesson detectors passed their validity checks, and in 131 of 138 internal handoffs the heuristic found a read call on the handoff file among the first three tool calls after its filename appeared in one of the receiving session's first three user messages. That does not establish that the read succeeded, that the content was understood, or that the next action was correct.
Not established: that receiving agents preserve every instruction or make fewer repeated mistakes. That needs a comparison — plain summary vs. a generic structured handoff vs. this skill — on the same frozen tasks, scored by something the writer cannot edit. The next round measures per-lesson recurrence over ~30 new handoffs; the result gets published either way.
Known limits: the readout parses Claude Code transcripts only (their format is internal and can change); the receive check only recognizes handoff files named HANDOFF_*.md; the recompute line does not shell-quote paths with spaces.

Try it, break it

mkdir -p "$HOME/.claude/skills"
git clone https://github.com/dormitivegit/session-handoff.git "$HOME/.claude/skills/session-handoff"

SKILL="$HOME/.claude/skills/session-handoff"
HANDOFF="/absolute/path/to/HANDOFF_billing-migration.md"              # name it HANDOFF_*.md (see limits above)
SESSION="$HOME/.claude/projects/your-project/session-id.jsonl"  # the transcript of the session that wrote it

python3 "$SKILL/scripts/lint_handoff.py" "$HANDOFF"
python3 "$SKILL/scripts/handoff_readout.py" --session "$SESSION" --handoff "$HANDOFF"
python3 "$SKILL/tests/run_tests.py"                             # the regression cases behind the fixes above

The most useful feedback is a sanitized failing case: a handoff where lint stayed clean but the next session still
resumed wrong. Open an issue with it.

Same pack, different CWD, different verdict

dormitivegit — Wed, 26 Aug 2026 11:55:17 +0000

Earlier in development, the same governance pack passed from one working directory
and failed from another. Same pack bytes. Same referenced source bytes. Only the
process working directory differed.

A relative embedded source path was being resolved against the process working
directory. A value the tool never declared as an input was changing the verdict.

The defect became visible when the same check was run from a different working
directory.

What the fix actually changed

Relative embedded source paths now resolve against the pack's own directory instead
of the process working directory. Absolute references stay absolute.

For file-backed packs, relative references resolve from the pack's directory. With
stdin there is no pack directory, so relative references are rejected rather than
silently inheriting the process working directory. I required that stdin reject
relative references instead of inventing a base directory.

Why ordinary tests did not catch it

A passing behavior test does not by itself prove that the files or evidence source I
intended to check stayed bound.

In an AI-assisted workflow, changes can arrive faster than I can re-check every
assumption behind them. I wanted a small mechanical check for the things I had
already decided mattered, while keeping the acceptance decision with me.

This CLI adds a narrower deterministic check after ordinary tests. Its result is
evidence for a reviewer or a CI system; it is not release authorization.

The smallest thing you can run

I use the corpus command below because it is the shortest way to show the mechanism.
It is not a reproduction of the path-resolution bug above.

All commands below were replayed against software-evidence-controls==0.4.0rc1.

python3 -m venv .venv
. .venv/bin/activate
python -m pip install software-evidence-controls==0.4.0rc1

Create a throwaway source directory, freeze it, change one byte, and verify:

demo_dir="$(mktemp -d)"
source_dir="$demo_dir/source"
manifest="$demo_dir/accepted-manifest.jsonl"
mkdir -p "$source_dir"
printf '%s\n' 'value = 1' > "$source_dir/example.py"

software-evidence-controls corpus freeze "$source_dir" --manifest "$manifest" --format json

manifest_sha256="$(python - "$manifest" <<'PY'
import hashlib, pathlib, sys
print(hashlib.sha256(pathlib.Path(sys.argv[1]).read_bytes()).hexdigest())
PY
)"

printf '%s\n' 'value = 2' > "$source_dir/example.py"

software-evidence-controls corpus verify "$manifest" \
    --accepted-manifest-sha256 "$manifest_sha256" \
    --expected-root "$source_dir" \
    --format json

Abridged from the actual JSON output:

{
  "result": "HOLD",
  "findings": [
    {
      "code": "CI03_SOURCE_CHANGED",
      "severity": "ERROR",
      "message": "filesystem source bytes changed",
      "evidence": "example.py"
    }
  ],
  "exit_code": 4,
  "counts": {
    "match": 0,
    "missing": 0,
    "changed": 1,
    "type_changed": 0,
    "self_ingested": 0
  }
}

This means the recorded source changed. It does not mean "reject the change." The
finding is an input to CI or review; acceptance remains a human decision.

The fastest way to evaluate this is to run it.

What the tool covers

The CLI has several surfaces; these are the two used in this article.

corpus freeze / corpus verify records the bytes of a set of files you explicitly
select, then verifies that those recorded sources have not changed. It does not prove
that the scope you selected was complete.

check validates a pack against explicit rules and returns a machine-readable
PASS / FAIL / HOLD result with structured findings.

With --format json, these commands emit structured output meant to be consumed by
CI or read by a reviewer rather than skimmed as a log.

What this does not prove

The published package declares no third-party Python runtime dependencies and
requires Python 3.11 or newer. That is a statement about declared runtime
dependencies, not about pip, the build backend, packaging tooling, or Python
itself.

I have not tested it on Windows. I have not exhaustively tested every CLI option
combination, or long-running and repeated-operation behavior. The integrity check
covers the sources you recorded, not the question of whether you recorded the right
set. Other tools work on adjacent problems. I have no adoption data and make no
claim about who else is using it.

Why the name changed

The project was previously published as fable5-assurance-toolkit. In August 2026 I
renamed the GitHub repository and published the Python distribution under
software-evidence-controls. Existing GitHub URLs redirect; the old PyPI prereleases
remain as historical artifacts.

Anthropic also uses "Fable 5" for a Claude model; this project is unrelated to
Anthropic. The collision accelerated the rename, but the real problem was simpler:
the old name spent its most visible word on an internal lineage label instead of
saying what the tool does.

Who it is for

It is for developers who can already say what must stay true after an AI-assisted
change — which files, which manifest, which pack — and want that boundary checked
mechanically.

If you want a hosted service, a GUI, or an AI agent that decides what to accept for
you, this is not that.

Repository: https://github.com/dormitivegit/software-evidence-controls

I used AI coding and analysis tools while building this. I decided the scope, what I
was willing to claim publicly, and when public changes were authorized.

Probabilistic agents need deterministic acceptance boundaries

dormitivegit — Thu, 13 Aug 2026 05:56:17 +0000

I have been building with coding agents daily for a while now. They are good. The
problem I keep running into is not that they write bad code — it is that they
change things faster than I can produce evidence of what changed relative to what
I had already accepted.

This is a different problem from correctness, and I think it is being
under-discussed.

Tests answer a different question

When an agent hands back a change, the reflex is to run the test suite. If it
passes, ship it.

Tests answer: does the code do what the tests assert?

They do not answer: what moved, relative to the state I reviewed and accepted?

Those come apart in ordinary situations. An agent refactors three files while
fixing one. A generated migration rewrites a fixture you were treating as a
frozen reference. A tool run mutates a config file nobody was watching. Every
test still passes, because no test was ever written about the thing that moved.
You find out later, or you do not find out.

The gap widens as agents get more capable. A weak agent touches one function. A
strong agent touches whatever it decides is in scope — and "whatever it decides"
is precisely the part you cannot pin down in advance, because that
non-determinism is where the value comes from.

The verification layer should be orthogonal to the agent

The tempting fix is to make the agent verify itself: ask it to summarize its own
changes, or run a second agent as reviewer.

I do not think this works as an acceptance boundary, for a structural reason:
both the change and the verification then come from the same probabilistic
process. When they disagree you learn something. When they agree you have learned
almost nothing, because agreement is exactly what a shared failure mode produces.

What I want is a verification layer with the opposite properties:

Deterministic. Same bounded input, same finding, every time.
Model-neutral. Swapping the agent should not change what acceptance means.
Offline. No network call in the verification path — a verification step that can fail for network reasons is not a boundary.
Machine-consumable. A stable exit code and a structured finding, not prose a human has to interpret.

Note that none of this makes the agent less useful. It stays as flexible and
probabilistic as you like. The boundary is drawn at acceptance, not at
generation.

Freeze, change, verify

The concrete mechanism I settled on is boring, which I take as a good sign.

Freeze a bounded set of sources into a hash-addressed manifest. Let the agent do
whatever it does. Verify against the manifest.

$ assurance corpus freeze ./src --manifest baseline.jsonl
result=PASS
exit_code=0
manifest=baseline.jsonl
write_disposition=CREATED
source_record_count=1

# ... agent runs, edits a file ...

$ assurance corpus verify baseline.jsonl
result=HOLD
exit_code=4
counts={"changed": 1, "match": 0, "missing": 0, "self_ingested": 0, "type_changed": 0}
FINDING {"code":"CI03_SOURCE_CHANGED","severity":"ERROR","message":"filesystem
source bytes changed","path":".../src/greeting.py","location":"source_record", ...}

(Real output, with a few header lines — module id, rule-set version, profile —
elided for width, and the absolute path shortened.)

The two things that matter here are the ones that look least interesting.

The exit code is part of the contract. 4 means an integrity finding
specifically, not "something went wrong." A CI job can branch on it without
parsing anything. Once exit codes are contractual they have to be versioned and
tested like any other public interface, which is a constraint worth accepting
early rather than discovering later.

"Bounded" is doing real work. The manifest covers explicitly supplied roots.
Not the whole machine, not an implicit working directory. An unbounded integrity
check is one that eventually gets disabled because it is too noisy, and a
disabled check is worse than no check because you still believe it is running.

A fair objection at this point: for a clean Git repository, git diff covers a
substantial part of the ordinary file-change case. It genuinely does. The cases
where it differs are when the bounded evidence set is not identical to the
repository — several explicit roots at once, deliberately untracked files,
members inside ZIP archives (read and hashed individually), symlink identity
(the link target path itself is recorded and hashed, which is a different fact
from the target file's contents), and the machine-consumable exit semantics
above. If your evidence set is exactly "tracked files in one repo," use
git diff. It is right there and it is excellent.

Authorization must be out-of-band — a lesson I learned by getting it wrong

This is the part I would most like to pass on, because I got it wrong in public
and the failure mode generalizes well beyond my own project.

Suppose your verification layer checks that a mutation was authorized: the record
claims a decision authorized it, and the tool confirms the decision exists,
covers the same object, and was made by the right authority.

The question is: where does "the right authority" come from?

My first public implementation had the authority identity hard-coded as a literal
string in the validation logic. It worked perfectly — for exactly one person.
Every other user constructing a fully well-formed record got a HOLD, because
their authority identity was not the one baked into the source. Three of the
tool's modules were structurally unusable by anyone but me.

I did not notice, and my test suite could not have told me, because all my
fixtures used the same identity as the code. It surfaced during an independent
adversarial review of the repository, and it surfaced only because the reviewer
built two byte-identical inputs differing in exactly one field and observed that
one passed and one did not. Reading the source had not found it; a green suite
had not found it. A single-tenant constant hiding in validation logic is
invisible from inside your own tests, because your fixtures share the constant.

The obvious repair is to let the input document declare its own authority. This
is worse. If the record under verification names the authority that will be
accepted, then a record can authorize itself:

{
  "authority_identity": "WHOEVER_I_SAY",
  "decisions": [{ "decider": "WHOEVER_I_SAY", "state": "AUTHORIZED" }]
}

The validator dutifully confirms the two agree, and the check has become
decorative. This is the same shape as a certificate that vouches for its own
issuer.

The repair that actually holds is to take the expected authority out of band —
supplied by the caller, at the call site, never readable from the artifact being
checked:

$ assurance check pack.json --authority-id PROJECT_AUTHORITY   → PASS, exit 0
$ assurance check pack.json --authority-id SOMEONE_ELSE        → HOLD, exit 3
$ assurance check pack.json                                    → HOLD, exit 3
                                                                  (fail-closed)

and, importantly:

# pack declares its own authority_identity, no --authority-id given
$ assurance check self-declaring-pack.json                     → HOLD, exit 3

The last two lines are the ones worth arguing about. Missing expected authority
is a HOLD, not a pass-through — an authorization check with no expected authority
has nothing to check against, and defaulting to permissive is how these things
quietly stop working. And a self-declared authority never overrides the
out-of-band value, even when it happens to agree with it.

The transferable lesson is narrow and worth stating plainly: trust anchors do
not belong inside the artifact being verified. The second-order version is the
one that nearly caught me — the naive fix for a coupling problem introduced a
self-authorization hole, and it looked like a clean generalization while doing
it.

The layer should refuse to make the decision

The last design constraint is the one people push back on most, so I will state
it plainly: this kind of tool should not decide whether to accept a change.

Concretely, in mine, risk classification returns a tier and an explicit field
saying the classification is not an authorization. Handoff validation reports
structural observations and explicitly reports that receiver readiness was not
machine-determined. Those fields are not decoration; they exist so that no
downstream automation can quietly read a PASS as a go-ahead.

The reason is not modesty about what software can do. It is that the moment a
deterministic checker is treated as an approval authority, people start shaping
inputs to satisfy it, and you have rebuilt the thing you were trying to avoid —
a probabilistic process optimizing against a proxy. Keeping the tool
descriptive, and keeping acceptance with a person, is what preserves the
boundary's meaning.

Where this leaves things

I do not think "assurance for AI-assisted engineering" is a solved problem, or
that a manifest checker is the whole answer. What I am fairly confident about is
the shape:

probabilistic generation  →  deterministic verification  →  human acceptance

with each stage refusing to do the next one's job. Agents stay flexible.
Verification stays reproducible and inspectable. Acceptance stays with someone
accountable.

It is explicitly not a replacement for Git, for tests, for CI, or for human
review. It sits beside all four.

I built FABLE5 as one
implementation of this shape — a local CLI, Python 3.11+ standard library only,
no network calls, no daemon, no model invocation, Apache-2.0. It is early: a
0.3.0 prerelease with 276 tests and CI across Python 3.11–3.14, maintained by
one person. There is a self-contained runnable example that walks the whole
freeze → change → detect → re-freeze cycle in a disposable temp directory in
about two seconds.

I would rather have the architecture argued with than the tool adopted. If you
think the acceptance boundary belongs somewhere else, or that this is a problem
existing CI already handles, I would genuinely like to hear it.