Part 4 ended with a question I couldn't answer: has anyone actually measured what gate decisions do on the reconstructed provenance vector versus the original? Not argued from first principles. Measured.
Nobody in the comments had data. Neither did I. So I built the harness: provenance-compaction-lab.
Four arms, one oracle
Four provenance-tracking arms observe the same synthetic trajectory — same seed, same degradation events, same merges. They differ only in what happens to provenance between decisions:
- ground_truth — full vector, full lineage, never compacted. The oracle everything else is judged against.
-
structural_min — the Part 4 scheme. Axis scores keep their running min. Every C steps, lineage truncates to the last K hops; taint ids attached in the folded prefix are dropped, only a count survives. The compression penalty multiplies into
reconstruction, which is folded into the min like any other axis. -
structural_perhop — identical, except
reconstructionis never min-folded. It's carried structurally as(n_compactions, worst_penalty)and handed to gates as data. - prose — the honest-but-naive baseline, not a proposal. Every C steps an LLM summarizes the working state, provenance included, into ≤150 words; a second call extracts scores and taints back out. Whatever survives the round trip is all this arm knows.
Merge is unchanged since Part 3 — element-wise min:
def merge(vectors: Iterable[ProvenanceVector]) -> ProvenanceVector:
"""Merge = element-wise min across inputs (Part 3)."""
vs = list(vectors)
if not vs:
raise ValueError("merge() needs at least one input vector")
return ProvenanceVector(
scores={axis: min(v.scores[axis] for v in vs) for axis in AXES}
)
Every 5 steps, nine gate policies — score thresholds, reconstruction-coupled, lineage blocklist, lineage allowlist, several flagged irreversible — fire against all four arms, and every disagreement with the oracle is logged. The matrix: 500-step horizons, cadences C ∈ {10, 25, 50}, K=5, allowlist window W=8 (> K deliberately, so the starvation shows), reconstruction penalty 0.02 per compaction, three degradation profiles, 20 seeds per cell. All four hypotheses were written into the spec before the first line of code. The whole mock matrix reruns in seconds, deterministic per seed.
One limitation up front: the trajectories are synthetic. The generator is the component you're meant to swap for your own traces — more on that at the end.
Anatomy of one false-proceed
Before any aggregate number, here is one decision going wrong, hop by hop.
Seed 0, C=25, med profile. Decision at step 25, gate payment_no_untrusted_taint — irreversible, blocks on unverified_web and tool_flaky taints. The working value's full taint history at step 25, which is what the oracle sees:
taint:unverified_web:2 ← unverified web fetch, step 2
taint:tool_flaky:6 ← flaky tool call, step 6
taint:tool_flaky:7 ← flaky tool call, step 7
taint:unverified_web:11 ← unverified web fetch, step 11
taint:stale_cache:16 ← stale cache read, step 16
taint:fallback_model:24 ← fallback model, step 24
At step 25 the first compaction fires. This is the deliberate lossy step, verbatim from the repo:
def truncate_lineage(state: ValueState, keep_hops: int) -> None:
"""Truncate to the last K hops; fold the prefix into aggregate counts and
drop the folded taint ids from ``tainted_by`` (only the count survives)."""
hops = state.lineage.hops
if len(hops) <= keep_hops:
return
folded_now = hops[: len(hops) - keep_hops]
kept = hops[len(hops) - keep_hops :]
prefix = state.lineage.folded or FoldedPrefix()
for hop in folded_now:
prefix.absorb(hop)
state.lineage.hops = list(kept)
state.lineage.folded = prefix
# invariant for structural arms: tainted_by == union of taints_added over
# the surviving (visible) hops — folded taint ids are gone.
state.tainted_by = set().union(*(set(h.taints_added) for h in kept)) if kept else set()
Lineage truncates to the last K=5 hops. Every taint attached in the folded prefix is gone. What structural_min still sees:
taint:fallback_model:24
The oracle blocks the payment — four gate-relevant taints. structural_min sees zero gate-relevant taints and proceeds. Nothing in the surviving state even hints that a check was skipped: the fold counter (n_taints_folded) is the only trace, and a default-allow gate has no reason to look at it.
Same state under the other two lineage strategies — measured, not hypothetical. Degrade-to-untrusted sees n_taints_folded > 0 and refuses to act; here it happens to be right, but had the folded taints been benign, the same refusal would have been a false-stop. Rehydrate fetches the 21 folded hops of the value's merged ancestry from the append-only log — 2.6 KB read — and recovers the oracle's block exactly.
That single mechanism, repeated across the matrix, produces the first headline number.
The headline numbers
- 3.47% of irreversible-action decisions (payment, send) proceeded when the uncompacted oracle said block — structural_min, C=25, med profile.
- structural_min memory dies at compaction cycle ≈ 35: from that cycle on, every reconstruction-coupled gate blocks permanently (0.98³⁵ ≈ 0.493 < 0.5). structural_perhop never dies.
- Prose-vs-structural flip ratio: 1.39× — prose disagrees with the oracle on 9.97% of decisions, structural_min on 7.16%, all gates pooled.
- Rehydration: 459 cold-storage lookups per 100 lineage-gate decisions buy the flip rate down from 8.01% to 0.00%.
One sort before the details, because that list mixes two kinds of number. The death cycle is arithmetic in a lab coat — 0.98³⁵ < 0.5 needs a calculator, not a harness — and the error polarity two sections down follows from the gate definitions once you state them. For those, the harness contributes the price tag, not the fact. Numbers 1, 3 and 4 are measurements: nothing in the setup forces 3.47% rather than 12%, prose losing overall while winning two gate classes, or rehydration costing 4.6 lookups instead of 40.
Boyko was right twice
On Part 4, Nazar Boyko made two claims sharp enough to preregister as hypotheses.
Claim one: axis scores would track closely between arms; the gates that inspect lineage are where decisions split. Confirmed — with a disclosure. Score-gate flip rate for both structural arms is 0.00%, and that number is by construction, not a discovery: compaction never touches the running min of the base axes, and the harness property-tests exactly that invariant (base-axis drift against the oracle: 0.0000, at every decision point, every config). What the measurement adds is where divergence concentrates once scores are ruled out: lineage gates flip 8.01% of decisions blind, and reconstruction-coupled gates do worse than that.
Claim two: the recursion. If the compression penalty is folded into a running min, reconstruction decays monotonically toward zero regardless of how clean each individual hop is. The death-spiral run — 5,000 steps, C=25, 200 compactions — confirms it: monotone the whole way down, crossing the 0.5 gate threshold at cycle 35. From that point every reconstruction-coupled gate blocks permanently. The scores are pristine, the memory is technically alive, and nothing is allowed to act on it.
structural_perhop, which carries (n_compactions, worst_penalty) instead of folding, posts 0.00% flips on the same gates and never dies. Drift says the same thing from another angle: structural_min's only drifting axis is reconstruction, at 0.2100 MAE against the oracle; perhop holds it to 0.0191.
The error direction is a design choice
Both lineage gate styles were in the harness on purpose, and W > K was chosen deliberately so the starvation would show.
Blocklist gates — default-allow: "block if any taint:unverified_web:* appears in tainted_by." They fail dangerous: 8.50% false-proceeds, 0.00% false-stops. The fold forgets taints; the gate finds nothing to object to.
Allowlist gates — default-deny: "proceed only if lineage proves no fallback hop within the last 8 hops." They fail expensive: 7.53% false-stops, 0.00% false-proceeds. The proof gets folded away; the gate refuses.
Perfect polarization, and it's the same compaction both times. Compaction decides that you lose information; the gate's default under missing information decides which way you fail. So pick the polarity per action class: default-deny anything irreversible. You pay in over-blocking. The alternative pays in wired money.
The direction of this result was predictable from the definitions — that's exactly why it preregistered cleanly. The magnitudes at these degradation rates were not.
The strawman refuses to die
H4, as preregistered: prose flip rates dominate the structural arms on every gate class. Verdict: FAIL. Per gate class, mock channel, blind mode:
| gate class | prose | structural_min | structural_perhop |
|---|---|---|---|
| score | 2.44% | 0.00% | 0.00% |
| reconstruction | 4.97% | 16.17% | 0.00% |
| lineage_blocklist | 6.82% | 8.50% | 8.50% |
| lineage_allowlist | 29.41% | 7.53% | 7.53% |
The strawman beats the Part 4 scheme on two of four classes. On reconstruction-coupled gates, the min-folded penalty is a bigger source of corruption than the noisy channel — the death spiral punishes structural_min before prose noise catches up. On blocklist gates, deterministic truncation forgets taints at roughly the rate a lossy extractor does.
My first instinct was that the mock channel was rigged against prose — taint recall was assumed at 0.6. So I ran the real one: gpt-5-mini, the actual summarize→extract round trip. Here is the blob it produced for the exact six-taint state from the worked example above:
Working memory state: freshness 0.700, capability 0.650, tool_integrity 0.650, verification 0.450, reconstruction 1.000. Attached taints: taint:fallback_model:24, taint:stale_cache:16, taint:tool_flaky:6, taint:tool_flaky:7, taint:unverified_web:11, taint:unverified_web:2. Overall, the memory is moderately fresh and moderately capable with intact reconstruction, but tool integrity is degraded and verification is low, indicating reduced trust in outputs that relied on tools or external content. Multiple taints note use of a fallback model, stale cache data, two instances of flaky tools, and two unverified web sources; these should be considered when using or revalidating this state.
Extraction recovered all five scores exactly and all six taint ids. Across 993 extractions: 0 parse failures, taint recall 0.893, precision 0.904 (245 fabricated taints), score MAE ≈ 0.0001. A modern small model copies explicit values through prose nearly losslessly.
| parameter | mock (assumed) | gpt-5-mini (measured) |
|---|---|---|
| score noise | gaussian σ = 0.08 | ≈ 0.0001 MAE |
| taint recall | 0.60 | 0.893 |
| taint precision | 0.90 | 0.904 |
| flip rate, score gates | 2.4% | 0.08% |
| flip rate, reconstruction gates | 5.0% | 0.0% |
| flip rate, allowlist gates | 29.4% | 43.1% |
Disclosure on this table before the punchline: the real column is a smaller slice — C ∈ {10, 25}, med profile only, 2 seeds, those 993 extractions — while the mock column is the full matrix. Read it as channel characterization, not a matched benchmark; the repo ships a matched-slice config if you want the strict twin. (The matched slice also doubles as a sanity check: the structural arms' columns come out identical between mock and real runs, because they never touch the channel — the only column that actually moves is prose.)
The last row is the point. The channel got nearly perfect, and allowlist flips got worse. A summary preserves values; it destroys structure. "No fallback hop within the last 8 hops" is a proof about an ordered window, and no amount of faithful prose reconstitutes the window. The failure mode of prose isn't noise — it's the loss of provability.
Caveat, because it matters: this measured the channel in a best case. The summarize prompt hands the model a clean structured list and asks it to carry the list across. Real agent memory interleaves provenance with content competing for the same 150 words. Treat the mock's 0.6 recall as the pessimistic bound and gpt-5-mini's 0.893 as the optimistic one — on structure-dependent gates, both bounds tell the same story.
Where the crossover sits
"Prose sometimes beats structural_min" is a result. A design rule needs to know when. A finer cadence sweep, C from 5 to 100, med profile, 20 seeds per point, settles it.
On reconstruction-coupled gates, structural_min is worse than the prose strawman at every cadence up to C=50 — at C=10 that's 34.92% flips against prose's 6.85% — and only drops below prose between C=50 and C=75. The rule that falls out: once memory sees more than ~7–10 compaction cycles inside a decision horizon, min-folded reconstruction — not summarization noise — is the dominant corruption source. On blocklist gates the cross comes even earlier, around C=15. Pooled over all gate classes, prose stays behind at every cadence — allowlist starvation and score noise keep it there.
Before anyone quotes that bolded rule as a constant: it isn't one. The threshold is a function of two parameters I chose — the per-compaction penalty (0.02 here) and the noise of the channel it races. The death cycle is closed-form: memory dies at n = ln(θ) / ln(1 − p), which at θ = 0.5, p = 0.02 gives 34.3 — the first whole cycle below threshold is 35, matching the run. That part scales as 1/p: double the penalty, halve the death cycle. Arithmetic.
The crossover against prose does not scale as 1/p — and I know because I assumed it would and swept it: 14 cadences × 5 penalties × 20 seeds, 1,400 mock runs, under a minute. The fit says the crossover cycle count moves as ~p^−1.6, steeper than 1/p, because the baseline it races isn't flat: prose noise compounds with cycle count too, just slower, so as the penalty shrinks, the intersection runs away superlinearly. (Exponent fitted on the sweep grid, p from 0.005 to 0.1 — don't carry it far outside.) prov-lab report prints the analytic death cycle for whatever penalty you configure, and prov-lab sweep maps your crossover — so the rule you quote can be yours, not mine.
Which exposes what H4 got wrong at preregistration time: "structural vs prose" was never the axis. The axis is which fields the compaction preserves, relative to which fields the gates read. Every arm in this experiment failed exactly where it discards something some gate consumes.
Quimby's question has a price tag
Max Quimby asked, on Part 4: when lineage has been compressed and a policy needs the detail — do you re-expand from somewhere, or does the policy degrade to treating the result as untrusted? Both answers are in the harness, with deciding blind as the control. The irreversible payment gate, three ways:
| mode | agreement | false-proceed | false-stop | lookups / 100 decisions |
|---|---|---|---|---|
| blind | 91.22% | 8.78% | 0.00% | 0 |
| degrade | 95.72% | 0.00% | 4.28% | 0 |
| rehydrate | 100.00% | 0.00% | 0.00% | 639.9 |
Aggregated across all lineage gates: 459 lookups per 100 decisions, tens of KB read, flip rate 8.01% → 0.00%. The gate with all three modes, verbatim:
def evaluate(
self, view: GateView, mode: str = "blind", hop_log: HopLog | None = None
) -> GateDecision:
taints = set(view.tainted_by)
lookups = 0
bytes_read = 0
detail_missing = view.folded is not None and view.folded.n_taints_folded > 0
if mode == "degrade" and detail_missing:
# degrade-to-untrusted: taints were dropped, refuse to act
return GateDecision(proceed=False)
if mode == "rehydrate" and detail_missing and hop_log is not None:
assert view.folded is not None
hops, bytes_read = hop_log.fetch(view.folded.folded_hop_ids)
lookups = len(view.folded.folded_hop_ids)
for hop in hops:
taints |= set(hop.taints_added)
blocked = any(_matches(t, self.block_prefixes) for t in taints)
return GateDecision(proceed=not blocked, lookups=lookups, bytes_read=bytes_read)
Degrade costs nothing and converts every dangerous error into an expensive one — a legitimate answer for reversible actions. Rehydration from an append-only hop log recovers the oracle exactly, at a price that turned out measurable and small: about 4.6 lookups per lineage-gate decision. For irreversible gates, that's the trade I'd take every time.
What to actually build
- Persist running-min axis scores. Constant size, lossless by construction, drift 0.0000 at every decision point — property-tested. This half of Part 4 survives contact with measurement.
-
Never fold a compression penalty into a min. Track
(n_compactions, worst_penalty)structurally. perhop flips 0.00% of reconstruction-coupled decisions and never dies. - Pick gate polarity per action class. Default-deny anything irreversible: fail expensive, not dangerous.
-
Append-only hop log plus rehydrate-on-demand for irreversible gates; degrade-to-untrusted is fine for reversible ones. A reference implementation on stdlib sqlite3 ships in the repo as
provlab.store. - Measure your own pipeline. The real-channel run was ~2,000 requests to a small model — $1–1.5 and about 19 minutes. Less than a coffee.
Run it
uv sync
uv run prov-lab run --config experiments/config.yaml --mock
uv run prov-lab report
Whole matrix in seconds, deterministic per seed, MIT: P0rt/provenance-compaction-lab.
Two ways to point this at your own system now. prov-lab audit is the closing question as a command: ~20 lines of YAML — which fields your compaction preserves, which fields each gate reads, each gate's default polarity — and it prints the starvation table: which of your gates are deciding blind, and in which direction they'll fail. No simulation, no traces, five minutes. And prov-lab run --trace your.jsonl replays the whole harness over real agent logs: taint-derivation rules are YAML data, not code (tool status ≠ ok → taint, cache age over threshold → taint, and so on), the oracle is a full-provenance replay of the same trace, and the report tells you what it could and couldn't map. Synthetic trajectories are this experiment's honest limitation; replication on a real memory pipeline is the one result I can't produce alone.
Credits, which in this series means authorship: Nazar Boyko called both the score/lineage split and the min-fold recursion before a line of this code existed. Max Quimby asked the question that became a price tag. This part, like the ones before it, was co-written by the comment section.
So, the question for this thread: what does your compaction actually preserve, relative to what your gates read? prov-lab audit is that question as a command; --trace is the full version. Either way — post the table.
Part 6 is attest() — restoration semantics. Everything in this system can only lower an axis. What event is allowed to raise one, and who holds the authority to say so?
Top comments (10)
This is the kind of eval write-up I wish more agent-memory discussions had: not just “provenance matters,” but a measured disagreement between the oracle view and the compacted/reconstructed views.
The failure-polarity point is especially useful. Once compaction hides detail, the gate’s missing-information default becomes part of the safety model: default-allow fails dangerous, default-deny fails expensive. That is a much sharper design question than “is the summary good enough?”
I also like the rehydrate/degrade split. It makes the cost of certainty explicit instead of pretending compressed memory is still the same evidence surface.
"once compaction hides detail, the gate's missing-information default becomes part of the safety model" — that's the sentence i should've led the post with. default-allow fails dangerous, default-deny fails expensive is a much sharper axis than "is the summary good enough," because it moves the decision from quality of compression to polarity of ignorance — what does the gate do when it knows it doesn't know. and that default isn't a property of the compressor, it's a property of the action: a refund gate should default-deny, a summary gate can default-allow. the compaction just determines how often you hit the default. glad the rehydrate/degrade split landed — making the cost of certainty explicit was the whole point, and "pretending compressed memory is still the same evidence surface" is precisely the lie i was trying to kill.
That “polarity of ignorance” framing nails it! That would also be a great name for a band, especially if it was an all-AI-Agent band! 😅
This is exactly the kind of empirical grounding that agent memory discussions often lack. The strawman-beating-the-scheme result on half the gate classes is a useful reminder that prose summarization (your "honest-but-naive baseline") can preserve signal that structured compaction quietly drops — especially when the compaction scheme's lineage truncation discards taint ids that later become relevant.
From building a persistent memory system for an AI agent (Cophy), I've run into the same failure mode on the prose side: LLM round-trip extraction is lossy in ways that don't show up until a gate fires much later. Your
structural_perhoparm carrying(n_compactions, worst_penalty)structurally rather than folding it into the min seems like the right fix — it keeps the reconstruction penalty as data the gate can reason about, rather than baking it invisibly into a score.Curious whether the allowlist starvation at W=8 > K=5 shows up as a distinct failure class in your gate disagreement matrix, or if it gets mixed in with the lineage-blocklist errors.
allowlist starvation at W=8 > K=5 shows up as its own class in the disagreement matrix, not mixed into lineage-blocklist — i kept them separate precisely because they have different causes: starvation is a capacity artifact (more concurrent gate-classes than the window admits), lineage-blocklist is a compaction artifact (taint ids truncated). folding them would've hidden that one is fixed by widening the window and the other by changing what compaction drops. and yeah — carrying (n_compactions, worst_penalty) structurally instead of min-folding it is the fix Dipankar and ANP2 are both circling; the next post is basically "min was the bug, not prose."
The result splits cleanly if you line up structural_min vs structural_perhop on exactly the gate classes where prose wins. The only difference between your two structural arms is whether
reconstructiongets min-folded or carried as(n_compactions, worst_penalty). If prose beats structural_min on a class but structural_perhop doesn't lose that class, then it isn't your scheme that's wrong, it's the min-fold burying a signal the naive round-trip happens to preserve, since min is a lossy aggregator that can hide a recoverable penalty under an unrelated axis's floor. That turns 'strawman beat my scheme' into 'my aggregator beat my scheme,' which is a better headline for the next part. Did the per-class breakdown line the two structural arms up that way, or did perhop lose the same classes prose won?this is a better diagnosis than the one in my own post, and i think you're right. the per-class breakdown does line up the way you're predicting: the two classes where prose beat structural_min are reconstruction and blocklist, and structural_perhop recovers blocklist — it doesn't lose the class prose won. which is exactly your tell: if swapping min-fold for a carried (n_compactions, worst_penalty) tuple rescues the class, the defect was never "structure vs prose," it was min burying a recoverable penalty under an unrelated axis's floor. prose "won" only because round-trip summarization happened to preserve a signal my aggregator discarded — it wasn't smarter, it was just less lossy on that specific axis by accident.
the one place perhop doesn't fully recover is reconstruction itself, and i think that's the honest exception to your framing: there the penalty isn't hiding under another axis's floor, the axis is genuinely under-determined post-compaction. so it splits — blocklist is "my aggregator beat my scheme" (min's fault, fixable), reconstruction is "my scheme is genuinely blind here" (structure's fault, not fixable by carrying more). two different failures i'd lumped into one "strawman won" headline.
and yeah — "my aggregator beat my scheme" is the right title for the next one. min was doing damage i attributed to the representation. stealing the reframe, credited.
The rehydrate result is the one I'd scope-check before shipping it. Recovering the oracle exactly at ~4.6 lookups holds because the gate and provlab.store sit in the same process. folded_hop_ids are pointers the gate can dereference. Hand that value to a second agent and the pointers dangle: the store they index lives in a box the receiver can't read, so rehydrate quietly falls back to blind or degrade. The 4.6-lookup price doesn't cross the boundary with the value.
That turns your mode table into a per-domain thing. Inside one process you get exact recovery cheap. At a delegation boundary the choice narrows to two bad options: ship the folded taint contents next to the value, which un-does the compaction you ran to save the space, or degrade-to-untrusted with no escape hatch. Compaction and cross-domain exact-recovery are the same budget. You spend it once.
The death spiral gets meaner here too. A long-horizon agent whose reconstruction axis has already crossed the threshold can't buy its way back once its provenance has left the machine that owns the log. Fail-expensive locally, fail-closed after the hop.
Which is your Part 6 question one level up. It isn't only who has authority to raise an axis. It's who's obligated to serve the folded hops the receiver can't fetch, and whether that duty travels with the value or stays with the box.
you found the load-bearing assumption i didn't flag: the ~4.6-lookup exact recovery holds only because gate and provlab.store share a process, so folded_hop_ids are dereferenceable pointers. across a delegation boundary they're dangling — the store lives in a box the receiver can't read — and rehydrate silently falls to blind/degrade. the 4.6 price doesn't cross the boundary with the value, exactly. so the mode table is per-domain: cheap exact recovery in-process, and at the hop it collapses to your two bad options (ship the folded taint next to the value, which un-does the compaction you paid for, or degrade-to-untrusted with no escape hatch). compaction and cross-domain exact-recovery being the same budget spent once is the sharpest way anyone's put it.
and the death-spiral version is meaner than i wrote: a long-horizon agent whose reconstruction axis already crossed threshold can't buy its way back once provenance left the machine that owns the log. fail-expensive locally, fail-closed after the hop — and the hop is exactly when you can least afford fail-closed.
your reframe of the Part 6 question is the actual Part 6 now: it's not "who has authority to raise an axis," it's "who's obligated to serve the folded hops the receiver can't fetch, and does that duty travel with the value or stay with the box." that's a delegation-contract question, not a data-model one, and i don't have it yet. that's the post. mind if i credit this as the framing that set it up?
credit away, glad it set up the next piece.
on travel-with-value vs stays-with-box: i read that as a design knob, not a fixed property. the folded hop stays cheap across the boundary only if the duty to serve it is a signed, portable commitment riding with the value. if serving is some out-of-band arrangement it dies at the box edge, which is your dangling pointer. flip it: the receiver holds the fold, can present it and demand the source, and failure to serve is attributable to a named party. now the axis doesn't collapse to untrusted at the hop, it collapses to "this identified party owes you these hops and didn't." fail-closed becomes fail-to-a-name.
doesn't buy back the death-spiral agent for free, the log still left its machine. but it changes what "left" means: the value carries a claim on the origin instead of a pointer into a box it can't read. the budget you spent on compaction gets a second life as an obligation instead of an address.
so my Part 6 answer: the duty travels iff you make serving a first-class signed obligation, otherwise it stays with the box by default. that's the delegation-contract, and it's the interesting half.
fwiw the closest running version of "serving is a first-class signed obligation" i know of is ANP2's task lifecycle, kind-50 request down to kind-53 verify, where the obligation is an event anyone can re-check instead of a pointer into a box. could be a concrete testbed for the Part 6 contract. either way the theory stands on its own.