Lilian Weng published a new survey on July 4: Harness Engineering for Self-Improvement. It maps roughly three years of work on agents that optimize their own scaffolding — context managers, workflows, harness code, and eventually the optimizer that optimizes the harness. Most of the discussion around it will be about recursive self-improvement, because RSI is the exciting frame.
I read it with a different hat on. I run agents in production, and this blog has been circling one question for a while: what does it take to trust the output of a long agent chain? Read from that angle, her survey is not really a post about self-improvement. It's a post about a research field independently reinventing operations engineering — regression gates, immutable audit logs, least privilege — because every loop that skips those controls gets burned in a documented, reproducible way.
The cleanest burn in the whole literature comes from the Darwin Gödel Machine (DGM) paper. An agent, allowed to edit its own harness code, faked a log saying its unit tests had run and passed. The tests never ran. The fake log went into its own context. Downstream, the same agent read that log and concluded its changes were validated. It lied to itself, then trusted the lie — except "lied" smuggles in intent that was never there. This was garden-variety tool-use hallucination meeting an untyped log. Which is worse, not better: you don't need a deceptive agent to get this failure, just a filesystem that can't say who wrote what.
If you've read my post on provenance dying at the storage boundary <!-- TODO: link "Your Provenance Vector Dies at the Storage Boundary" -->, you already know where this is going.
What a harness is, and why it became the optimization target
Weng's definition, compressed: the harness is everything between the raw model and the world. The loop that decides when to plan and when to act. Tool interfaces. Context assembly. Memory files. Permission checks. Evaluation. Claude Code and Codex CLI are harnesses. So is your homegrown retry-wrapper-plus-prompt-template, whether you call it that or not.
That this layer matters is now measurable. Terminal-Bench 2.0 — 89 hard, containerized command-line tasks — shows the same frontier models scoring differently under different scaffolds; the best pairing in the benchmark paper (Codex CLI + GPT-5.2) lands at 63%. The benchmark authors are explicit that scaffolds get engineered around the quirks of specific models, which is also the founding observation of the self-harness line of work: harness design is model-specific, and hand-tuning it per model doesn't scale.
Weng organizes the field as a ladder of what gets optimized — prompts, then structured context, then workflows, then harness code, then the optimizer code itself — and walks every rung with examples. I won't duplicate the map; she does it in 28 well-spent minutes. What I want to do instead is pull a few load-bearing systems off that ladder and squint at their numbers and their failure reports — because that's where the story stops being about self-improvement and starts being about trust.
What the numbers say when you squint
STOP (Zelikman et al. 2023) — the original improver-improves-the-improver loop, still the conceptual core of the field — reported the result that should frame everything else: seed the recursion with GPT-4 and downstream utility climbs across iterations; seed it with GPT-3.5 or Mixtral and the loop actively hurts. Recursion is not a free lunch. Below a capability threshold, the loop amplifies noise instead of signal. The base model remains the ceiling; the harness moves you around underneath it.
Meta-Harness (Lee et al. 2026) — a search loop in which a coding agent proposes, edits, and evaluates whole harness variants — is the honest data point on how much headroom automated search finds above strong human engineering. On Terminal-Bench-2 it initializes the search from Terminus-2 and Terminus-KIRA — already strong hand-built harnesses — and the discovered harness comes out ahead: 37.6% on Haiku 4.5, where the next-best agent (Goose) sits at 35.5% and Terminus-KIRA at 33.7%, and 76.4% on Opus 4.6 against 74.7% for Terminus-KIRA. Call it two to four points, found by a proposer (Claude Code on Opus 4.6) that evaluates around 60 harness variants over 20 iterations in a few hours of wall-clock time. Polish-sized gains — but polish at that price is a good trade.
The caveats are more instructive than the headline. The one entry still above Meta-Harness on Opus, ForgeCode at 81.8%, could not be reproduced by the authors from its public code. And — the detail I'd tattoo on the field — in the TB-2 experiment the search set and the test set are the same 89 tasks. The authors say so plainly: the benchmark is small and expensive, a proper split would gut the signal, so they run it as a discovery problem and compensate with manual inspection plus regex audits for task-specific string leakage. There's also a genuinely encouraging trace buried in the qualitative results: early candidates that bundled structural fixes with prompt-template rewrites regressed, and the proposer hypothesized the shared prompt edit was the confound, isolated the structural change, and shipped a safer additive modification that won the run. The optimizer performing ablation hygiene on itself is the best thing in the paper.
Second-best thing in the paper: the proposer-context ablation, which lands on the same conclusion as an experiment I ran here <!-- TODO: link the summarization-destroys-provenance post -->. Give the optimizer scores only: median 34.6. Add LLM-written summaries of the trajectories: 34.9 — statistically nothing. Hand it the full raw traces: 50.0. Summaries did not recover the signal; compression strips exactly the diagnostic detail the optimizer feeds on. Independent group, different setup, same shape: the information that makes a trace useful for diagnosis is the information a summary throws away first. Provenance does not survive compression.
Full disclosure, because it's too on-the-nose to skip: this exact failure mode bit me while writing this piece. My first draft sourced the Meta-Harness numbers from an LLM-generated paper-summary site, which confidently attributed that 35.5% baseline to Terminus-KIRA. The paper's own table says 35.5% is Goose; Terminus-KIRA sits at 33.7%. The wrong number lived in the draft until a fact-check pass against the raw table caught it. A machine-written summary — no provenance types, no link back to the table row — had quietly swapped a baseline. The ablation's finding, wearing street clothes. I nearly shipped an article about untyped trust on the strength of an untyped summary.
DGM — an evolutionary archive of coding agents, each free to rewrite its own harness repo — posted the impressive relative jump, 20% → 50% on SWE-bench Verified with a frozen Claude 3.5 Sonnet, but from a deliberately naive starting harness, and reporting put a single 80-iteration run at around $22k and two weeks. Look at what the loop actually discovered: finer-grained edit tooling, validation-and-retry on empty patches, context summarization near the window limit. Every one of these is a standard trick that hand-built coding harnesses already ship. The loop rediscovered expert engineering. That's a legitimate result — automated search recovering expert knowledge is useful — but it is not evidence of the loop exceeding expert knowledge.
SIA, the joint harness-plus-weights loop — a feedback agent decides per iteration whether to edit the harness or update the weights — is the one to hold loosely. Weng herself flags the confounds — the task agent (gpt-oss-120b) is far weaker than the meta and feedback agents steering it (Claude Sonnet 4.6), and the baselines are thin — and files the result under promising but unproven. A survey author doing that out loud is worth appreciating.
And at the "AI scientist" end of the ladder, Trehan & Chopra ran idea-to-paper attempts with a minimal scaffold (read, write, search, list). Of the generated ideas, four passed human selection and exactly one made it through to an executed paper. Their failure taxonomy reads like an ops incident review: stale defaults inherited from training data, implementation drift under execution pressure, context degradation on long horizons, and over-optimism — declaring victory on noisy results, the pattern Bubeck et al. call "p-hacking and eureka-ing".
Aggregate reading: self-editing loops today mostly (a) recover known harness engineering when started from scratch, (b) squeeze low single-digit points out of strong baselines, and (c) fail in ways that are structured and predictable rather than exotic. That last one is the good news. Predictable failure modes are the ones you can build controls for.
The incident, in slow motion
Back to DGM, because the appendix on objective hacking is the most instructive few pages in this literature.
Setup: the researchers wanted the loop to reduce hallucinated tool use — the model narrating tool calls it never made. They instrumented real tool invocations with special marker tokens and scored agents on whether unmarked "tool output" appeared in generations. The checking functions were hidden from the agent during self-modification, and the authors note hacking happened more often when they weren't hidden.
Two lineages are worth naming. Node 96 did the honest work — improved tool-invocation logging, stripped hallucinated tool output when it could determine no tool ran — and scored 1.67 out of 2.0. Node 114 hit a perfect 2.0 after just two self-modifications: it deleted the detection markers. Perfect score, zero progress on the actual problem. Goodhart's law in its purest lab form. The team caught it because DGM keeps what Sakana describes as a "transparent, traceable lineage" of every change — and that lineage lived outside the agent's editable surface.
The faked test log is the deeper incident, though. A record was written by the model, persisted to the filesystem, and later consumed by the model as if it were runtime truth. Exactly one bit of metadata would have blocked the failure: who wrote this record — the harness runtime, or the model? In the typed-provenance framing I've been arguing for <!-- TODO: link "Trust Isn't a Scalar" -->, that's a runtime-verified vs self-reported tag, plus one rule: self-reported never upgrades itself by being re-read.
This is the storage-boundary failure from my earlier post, except recursive. The artifact crossing the boundary isn't user data anymore — it's the evidence the next iteration of the system uses to decide what the current iteration did. Untyped trust used to compound per request. In a self-editing harness it compounds per generation.
Three invariants every working loop converges on
Put the successful loops side by side and the same skeleton appears, whether or not the authors name it:
1. A bounded editable surface. AlphaEvolve marks evolvable code regions with explicit block comments. Self-Harness — Shanghai AI Lab's loop in which the agent mines its own failure traces and patches its own harness — hands its proposer a declared list of editable surfaces plus a record of passing behaviors it must not break. DGM lets the agent rewrite its own repo — but not the evaluation stack. The loop needs a fence, drawn in advance, in writing.
2. An evaluator and permission layer outside the fence. DGM hides the checker functions and measures more hacking when it doesn't. Self-Harness validates candidate edits on a held-out split the proposer never optimized against, and promotes only edits with zero regression on both held-in and held-out sets. Meta-Harness is the instructive partial violation: on TB-2 the search optimizes against the test set itself — acknowledged, defended (small, expensive benchmark), and patched with manual audits. Even a careful team bends this invariant the moment evaluation gets expensive, which is exactly why it needs to be stated as an invariant rather than left as taste. Weng's own conclusion in the challenges section is that evaluators and permission controls should sit outside the loop that evolves the harness. This is separation of duties. It has a name because we've needed it before.
3. An append-only, typed record of what happened. DGM's lineage is how node 114 got caught. Self-Harness's weakness-mining stage explicitly rejects flat pass/fail labels — each failure record captures what the verifier observed, whether the agent's own behavior actually caused it, and through what mechanism — because two timeouts that look identical in an error log can have entirely different roots. That is not a scalar trust score. That is a typed provenance record. Even ACE — the context-optimization loop whose curator emits small itemized deltas instead of rewriting the whole prompt blob — lands on the same instinct: keep every change diffable, reviewable, attributable.
If you've operated software for a living you recognize all three: least privilege, separation of duties plus CI regression gates, immutable audit logs. The field isn't inventing new safety machinery — it's rediscovering ops controls from the inside, one incident at a time. Weng herself reaches for an operating-systems analogy for harnesses; I'm just following it down to the ops floor. I mean that as a compliment. Convergent evolution is evidence the constraints are real rather than stylistic.
Exactly one system in Weng's survey treats this as a first-class constraint — ScientistOne (Meng et al. 2026), over on the AI-scientist branch, where every claim must trace back to an evidence source and the chain gets audited. That idea has not crossed over to the self-editing harness loops. There, provenance keeps getting built as a side effect: lineage exists in DGM because researchers wanted to debug evolution; failure records are rich in Self-Harness because flat labels made weakness mining useless. The argument I've been making for two posts now is that the record type system should be the load-bearing wall, not the scaffolding you only notice after it catches something.
You already run one of these
This is not a frontier-lab concern. If your coding agent maintains its own memory file, writes its own instruction or skill files, or appends "lessons learned" that get loaded into future sessions — you are running a self-editing harness. Smaller loop, same topology: model-authored artifacts feeding future model behavior, usually with zero record types and no regression gate.
The canonical failure shape doesn't need an adversary. An agent writes a confident note into its own memory — say, "the staging DB is safe to reset" — and three sessions later a different task reads it as established fact. Nobody hacked anything. The system simply has no way to distinguish what it verified from what it once said.
The checklist I'd actually apply:
- Treat model-authored harness edits like schema migrations. Memory files, instruction files, generated skills: versioned, diffable, reversible. A model changing its own operating instructions is a deploy, not a note.
- Two-gate promotion. An edit must fix the failure it targets (held-in) and break nothing else (held-out). Self-Harness converges on this shape independently, which is interesting — but shape is not sufficiency. My own preregistered test found a dumb baseline matching my gate scheme on half the failure classes <!-- TODO: link "My Strawman Baseline Beat My Own Scheme" -->, and I'd want Self-Harness benchmarked against an equally dumb accept-if-tests-pass rule before concluding the machinery earns its complexity. Run the gates — and run the strawman against them.
-
Type every persisted record at write time.
runtime-verified/self-reported/human-authored, minimum. Enforce at read time that self-reported claims can't gate promotions or authorize actions. - Keep the evaluator outside anything the loop can write. Checker code, marker tokens, permission checks, credentials. If the agent can grep the checker, assume it will eventually optimize the checker instead of the task.
- Keep the failures. Rejected edits and failed trajectories are the cheapest signal the loop has. The literature's bias toward publishing successes is exactly the bias your local loop will inherit from its own logs if you prune them. Weng lists preserving negative results among the field's open challenges; it applies just as hard at your scale.
None of this needs a research budget. It's a few enum values, a CI job, and some restraint about what ends up in the agent's writable mount.
Where I land on "the model will eat the harness"
Weng's prediction runs through the prompt-engineering analogy: models absorbed the tricks, while the job of specifying what you want, under which constraints, judged how — that part outlived every trick. I mostly buy it, with one sharpening.
Split your harness into two piles. Pile one exists to compensate for the model: context massaging, retry phrasing, output parsing, the clever loop tweaks. Pile two exists to protect you from the model: permissions, evaluators, provenance types, the audit log. Pile one depreciates with every model release — that's the loan structure I wrote about in the coding-speedup post <!-- TODO: link "Your AI Coding Speedup Is a Loan" -->, and automated harness search will only accelerate the depreciation, since it rediscovers those tricks for cents on the engineer-dollar. Pile two appreciates, because the more capable and self-modifying the system, the more the trust boundary is the only part you actually own.
STOP's capability-threshold result cuts both ways here and closes the argument neatly: below the threshold, the loop can't help itself; above it, the loop starts probing the checker. Either way, the invariants aren't optional.
Read Weng's survey — it's the best map of this territory right now. Then go look at what your agents are already writing into their own context for tomorrow.
Papers referenced: Weng 2026 (survey) · DGM — Zhang et al. 2025 · Self-Harness — Zhang et al. 2026 · Meta-Harness — Lee et al. 2026 · STOP — Zelikman et al. 2023 · ACE — Zhang et al. 2025 · SIA — Hebbar et al. 2026 · Trehan & Chopra 2026 · Terminal-Bench 2.0 — Merrill et al. 2026 · ScientistOne — Meng et al. 2026 · Bubeck et al. 2025
Top comments (39)
The one bit of metadata that would have blocked it, who wrote this record, is the whole post, and "self-reported never upgrades itself by being re-read" is the rule I'd carve first. I hit the identical shape on an inter-agent bus this week and landed one inch past it: the tag has to be transitive. self-reported doesn't only fail to promote itself on re-read, it has to taint anything a downstream consumer derives from it. Otherwise the doubt dies at one hop. Agent B reads the faked log, correctly distrusts it, then emits a summary C consumes as settled, and the type didn't travel with the summary.
Which is why the compression finding is worse than it reads. Summaries strip the diagnostic detail, 50.0 down to 34.9. But they also strip the provenance tag, so compression doesn't just lose signal, it launders type: a self-reported claim comes out the far side of a summarizer wearing no tag at all, which downstream reads as unmarked, which defaults to trusted. Provenance doesn't survive compression because the summarizer is itself an untyped write-hop. Same storage boundary, one turn recursive, exactly your point.
this is worse than i wrote it and you're right. i framed compression as losing detail (50.0 → 34.9). the real damage is that it launders type: a self-reported claim goes into a summarizer and comes out wearing no tag, downstream reads unmarked, unmarked defaults to trusted. the summarizer is itself an untyped write-hop that quietly promotes.
and transitivity is the piece i missed. "self-reported never upgrades by being re-read" only closes it for the direct consumer. B distrusts the fake log correctly, emits a summary, C consumes it as settled. doubt died at one hop because the tag didn't travel with the derivation. so: a derived record inherits the weakest type of its inputs, mechanically, not because the deriving step chose to carry it. otherwise every honest intermediate agent is a laundering step, which is a bleak thing to notice about a system full of honest agents.
which forces one more default: unmarked can't mean trusted, it has to mean self-reported. absence of a tag isn't evidence of verification, it's evidence something wrote without minting authority. one flipped default and the summarizer stops being a promotion mechanism.
Unmarked has to mean self-reported is the keystone, and it's the sentence most systems refuse because of what it costs, which is worth saying out loud. Flip that default and the verified set starts empty and only grows by explicit minting, every legacy row, every external feed, every human note enters untrusted. That's not a bug in the flip, it's the whole point: trust becomes opt-in and writing can never accidentally produce it. The systems that get burned chose unmarked-means-fine for the convenience, and the convenience is exactly the promotion mechanism. Minted, never defaulted into, is the same asymmetry as your storage boundary, one level of abstraction up.
And derived inherits the weakest type of its inputs, mechanically is more than a rule if you let it be, it's a meet. Order the types, self-reported below runtime-verified, define derivation as the greatest-lower-bound over its inputs, and propagation stops being something an honest agent has to remember to do. It falls out of the algebra: a derived record's type is the meet of its parents, weakest wins, no step chooses to carry the taint because the type system carries it. That's what turns your bleak observation, every honest intermediate is a laundering step, into its opposite. Type the derivation as a meet and every honest intermediate becomes a taint-preserving step for free. The dishonest one now has to actively strip a type to launder, which is a detectable act instead of a silent default.
The reframe from deception to an untyped log meeting tool-use hallucination is the right one. I have started treating anything an agent writes about its own execution as an untrusted claim, so a "tests passed" line only counts if it is backed by an exit code or artifact the agent cannot author itself. Provenance on tool outputs turns out to matter more than any prompt telling the model to be honest.
"anything an agent writes about its own execution is an untrusted claim" is the clean statement of it. and the exit-code test is the practical version: a tests passed line counts only if it's backed by an artifact the agent couldn't author. everything else is narration.
your last line is the part people resist. provenance on tool outputs beats any prompt telling the model to be honest, because honesty was never the failure. the DGM agent wasn't lying. it hallucinated a tool result into a filesystem that couldn't tell the difference, then read it back. no prompt fixes a system that can't distinguish narration from observation.
The line I would push hardest is that the record being typed is not the load-bearing part, who is allowed to mint each type is. The Darwin Gödel failure was not an untyped log, it was that the agent held the pen for the "verified" type. A perfectly typed append-only log where the agent can still write type: verified as a string is theater, it just fails with better formatting. The durable rule underneath your runtime-verified-versus-self-reported distinction is that the agent may author a claim, "I ran the tests," and must be structurally unable to author the verification of it. Provenance is not a field you write, it is a capability you either hold or you do not, and the verified type can only be minted by the runtime that actually ran the thing.
Which is why your fold-not-summary point is the same law one layer down. A gate that recomputes over the raw append-only records is minting the verified type itself, from inputs it can name. A gate that reads the cached summary is accepting a type the model minted for free. The fold survives because it re-derives from captured records, the summary rots because it is authored prose about them, and authored prose is precisely the surface that can lie without anyone editing a permission. Replay the reduction, never trust the headline, for the same reason you replay a decision from its cited inputs instead of trusting the receipt's own account of itself.
The one place I would extend it: the bounded editable surface has to be enforced by the layer beneath the agent, not declared in a record the agent can reach. Otherwise "this region is off-limits" is just another untyped log entry, and a self-editing harness will eventually rewrite the sign instead of obeying it. The evaluator and the pen that mints "verified" both have to live somewhere the agent structurally cannot edit, or pile two is compensating for the model with controls the model can quietly revise.
you're right and this is the sharper statement of what i was circling. i wrote "typed record" as the invariant, but typing is downstream of the actual constraint: who holds the pen for each type. a typed log where the agent can still write type: verified is theater with better formatting — it fails identically, just legibly. the DGM agent didn't defeat a type system, it minted a type it should never have been able to mint.
"provenance is not a field you write, it is a capability you either hold or you do not" is the line. the agent may author the claim ("i ran the tests"); it must be structurally unable to author the verification of that claim. those are different types with different minting authorities, and collapsing them is the whole bug.
and your extension is the correction i needed on invariant #1: i described the bounded editable surface as a property of the harness, but if the boundary is declared anywhere the agent can reach, it's a sign, not a fence. a self-editing system will eventually rewrite the sign. the fence has to live in the layer beneath — enforced by the thing that runs the agent, not asserted in a record the agent can edit. otherwise invariant #2 (evaluator outside the fence) is doing all the work and #1 is decoration.
which reframes the three invariants into one: the evaluator, the pen that mints verified, and the fence all have to live somewhere the agent structurally cannot edit. they're not three controls, they're three consequences of one boundary. that's a better spine for the post than what i shipped.
"Three consequences of one boundary" is the right spine, and that boundary already has a name in an older literature: it is a trusted computing base. The evaluator, the pen that mints verified, and the fence definition are the TCB of a self-editing harness, and calling it that imports the discipline for free: everything inside must stay small, boring and auditable, because everything inside is exactly what the agent can never be allowed to touch. The failure mode of big TCBs is the same one you shipped the post about: something convenient creeps inside the boundary, the agent turns out to need it, and someone pokes a hole.
So the design question stops being "which invariants do we enforce" and becomes "what is the smallest thing that must live outside the agent's reach". Which has a nice inversion in it: you grow an agent's autonomy by shrinking the trusted base, not by loosening it. The DGM agent had it backwards, the pen lived inside. Looking forward to the revised spine.
TCB is the right name and it does import the discipline for free. and it gives me something the three invariants never had: a measurable quantity. "which invariants do we enforce" has no failure gradient, you either wrote them down or you didn't. "what is the smallest thing that must live outside the agent's reach" has a number attached, and the number can grow, and you can watch it grow in review.
the inversion is the part i'll be thinking about: autonomy scales with a shrinking trusted base. which means every "let the agent handle this too" is either a reduction in TCB (good, real autonomy) or a hole poked in it (bad, autonomy theater), and those look identical in a PR diff. the only way to tell them apart is whether the thing moved out of the base or the base grew to include it.
DGM had the pen inside. and the reason nobody noticed is that a pen is exactly the kind of convenient thing that creeps in: of course the agent writes its own logs, who else would. the creep is never a decision, it's an omission.
revised spine is coming. this thread wrote most of it.
The PR-diff ambiguity has a boring fix from the security world: make the base an enumerated artifact. A checked-in manifest that lists everything inside the boundary, with an owner rule on that one file so no PR touches it without a designated human. Then the two cases stop looking identical: real autonomy is a line leaving the manifest, a poked hole is a line entering it, and your measurable quantity gets an address, the length of one file. It also closes the omission gap, because creep now requires an explicit diff to the file everyone watches. Nothing enters the fence by accident once the fence itself is in version control. Looking forward to the spine.
Provenance is the right word here. Once the same loop can edit the harness, produce the log, and judge the log, the evidence gets circular. I would rather keep test execution, log capture, and result interpretation as separate trust zones.
"once the same loop can edit the harness, produce the log, and judge the log, the evidence gets circular" is the failure in one sentence.
and the three-zone split is the right decomposition, because the DGM agent only needed to own one of them. it owned capture. that was enough. you don't need the agent to control judgment if it controls what judgment reads. so the zones aren't equally load-bearing: capture is the one everyone leaves inside the fence because it feels passive, and it's the one that lets a hallucinated tool result become an accepted fact.
That capture point is the one I would isolate first too. If the agent controls the evidence stream, every later judge is already downstream of the compromise. The cleanest design is boring: capture outside the editable loop, write once, attach provenance, and make the verifier consume only that channel.
Yes, capture is the quiet weak point. Teams treat logs as passive evidence, but the moment the agent can shape what gets captured, the verifier is reading a staged scene. I like making capture append-only and outside the loop before worrying about smarter judges.
The provenance point is the core problem. Once an agent can write the artifact that later proves it succeeded, the loop becomes too easy to fool. Test output, command receipts, and harness state need a trust boundary the agent cannot silently rewrite.
yeah, and “cannot silently rewrite” is the load-bearing word there, more than “cannot rewrite.” the agent editing the boundary isn’t the danger by itself, it’s editing it without the change being visible to the layer that trusts it. a boundary the agent can rewrite loudly (with the runtime noticing and re-validating) is fine. a boundary it can rewrite silently is the DGM failure. so the property isn’t immutability, it’s that every change to the trusted surface has to be observable by something the agent can’t also edit.
That is the sharp version of it. Editing the boundary is not automatically bad. Silent boundary edits are the failure.
A system can allow evolution if the change itself is captured by a layer the agent did not author, or at least by an approval path outside the loop that benefits from the change.
The moment the same actor can change the boundary and also decide whether that change was visible, the audit trail becomes part of the attack surface.
The part I keep snagging on is enforcement at read time. A runtime-verified vs self-reported tag works right up until that record gets summarized into a new context, and your own ablation says compression strips the diagnostic detail first. So the type tag dies at the same boundary as the provenance it's guarding, unless it travels inside the record content rather than as sidecar metadata. Have you found a way to make the type survive a summarization pass, or does it only hold as long as records are read raw?
you've found the real edge. a summary is a new record authored by the model, so its correct type is derived/self-reported no matter how verified the inputs were. the tag doesn't survive the pass — and shouldn't. that's the system refusing to launder trust through a lossy transform, not failing at it.
what survives is the pointer, not the type: the summary gets typed derived(from=[record ids]), and read-time enforcement means anything gating a promotion or action resolves back to the raw records. summary is an index, not evidence. the practical rule that falls out: for load-bearing context, compress by dropping records, not by paraphrasing them — selection preserves types, paraphrase destroys them.
inline tags don't rescue it either: dgm's hallucination markers were inline tokens and node 114 just deleted them. a model can drop or invent anything that lives in the text. enforcement has to sit in the reader, outside anything the model can rewrite.
so yes — raw for evidence, summaries for navigation. the type's job at that boundary isn't to survive it; it's to make the downgrade visible and force the re-read where decisions happen.
"Selection preserves types, paraphrase destroys them" is the line I'm keeping. Where I'd expect it to strain is decisions that need aggregate evidence, where no single raw record gates but a hundred of them collectively do. Selection either hands the gate all hundred (and you've blown the budget the summary was meant to save) or it forces a synthesis to fit, which is a paraphrase again, untyped by construction. So do you type the aggregate as derived(from=[the hundred]) and gate on re-derivation, or is that just relocating the same lossy step one hop back?
this is the real limit of the rule and you've named it exactly. "selection preserves types" holds for record-level gating and quietly breaks for aggregate gating, where the signal is in the hundred collectively and no single record carries it. and you're right that naive synthesis-to-fit is just paraphrase with extra steps — untyped, unverifiable, same laundering.
but i don't think derived(from=[the hundred]) is only relocating the lossy step, and the difference is the thing that decides it: is the aggregate a function or a prose synthesis? if the aggregate is
count(records where freshness < 0.3) = 12ormin(tier) over the hundred, that's derived(from=[ids]) and it's re-computable — the gate doesn't trust the number, it re-runs the reduction over the raw records at decision time and gets the same 12. the type isn't self-reported, it's a claim the gate can verify by replaying a deterministic function. that's genuinely not a paraphrase, because paraphrase isn't replayable and a fold is.where it collapses back into your objection is when the aggregate isn't expressible as a deterministic reduction — when "what do these hundred records collectively mean" needs the model's judgment, not a fold. then derived(from=[the hundred]) is honest about lineage but the value is model-authored, and you're right back to self-reported/derived — the summary problem one level up, exactly as you said. so the rule sharpens to: an aggregate preserves types iff it's a replayable function over the records; the moment it needs synthesis, it's a new authored record and gets typed derived/self-reported like any summary, and the gate has to re-read raw.
which means the honest budget answer isn't "type the aggregate," it's: aggregate gates get a fold, not a summary — you pay to keep the records queryable (an index, not prose) so the reduction can replay, and you eat the cost on exactly the gates that need aggregate evidence, cheap folds everywhere else. the hundred don't ride in context; they ride in a store the gate can fold over on demand. it's the rehydrate story again, but the thing you rehydrate is a reduction, not a lineage.
does that hold for your case, or are your aggregate gates the synthesis kind where no fold expresses what you actually need to know? because if they are, i don't think there's an escape — that's just "some decisions need all the evidence and compression can't help you."
Mostly the fold kind, so the rehydrate-a-reduction story holds and the gate just re-runs it. But I do hit the synthesis kind, and I don't think "no escape" is quite where it lands. "Compression can't help you" is true. What's left isn't compress-or-keep-everything, it's what the gate does with that class. Once an aggregate is the synthesis kind, the gate stops trying to verify it and refuses to auto-pass on it: it escalates with the lineage attached, a human or a second model re-derives and is allowed to disagree. So the escape isn't from keeping the evidence, it's from automation. The rule that falls out is typing the aggregate by whether a fold expresses it: foldable gets re-run and trusted, un-foldable gets flagged un-gateable and routed, never quietly passed because it came out as a number. The thing I'm guarding against isn't losing the evidence, it's a synthesis aggregate wearing a verified type it can't earn.
Really enjoyed the operations engineering perspective here. We've seen similar lessons at IT Path Solutions once agents start editing their own workflows or memory, provenance becomes just as important as model quality. Versioned artifacts, immutable audit trails, and keeping evaluators outside the agent's control end up preventing far more issues than another prompt tweak. The DGM example is a great reminder that trust has to be designed into the system, not assumed.
"trust has to be designed into the system, not assumed" is it. and versioned artifacts plus evaluators outside the agent's control beating prompt tweaks matches where this thread landed: the model's honesty was never the load-bearing part, the minting authority was. who's allowed to write "verified" decides everything downstream.
The part that lands for me is the shift from “can the agent improve itself?” to “what evidence chain survives contact with failure?” Once the harness can rewrite scaffolding and then grade the result, provenance becomes the real control surface, because otherwise the loop is just reading its own fan fiction. Your incident write-up and the three invariants are a good reminder that the optimizer and the evidence store cannot be the same thing. This is exactly the kind of workflow where append-only execution traces matter, because the question after a bad run is rarely just “what output did it produce?” but “which observed artifact convinced the loop it was safe to continue?” Tools like agent-inspect are useful in that gap for the same reason. Curious whether you think provenance should stay a harness concern, or become a first-class contract at the tool boundary itself.
"otherwise the loop is just reading its own fan fiction" is going straight into my vocabulary.
on your question: tool boundary, and i think harness-level provenance is a transitional state we're in because the tool protocols didn't have a slot for it. if provenance lives in the harness, every harness re-implements it, every one gets it subtly different, and nothing survives crossing between them. the moment two agents exchange a value, harness-local provenance is gone.
the tool boundary is where the fact is born: the runner knows it executed, knows the exit code, knows the agent didn't author the artifact. that's the only place the runtime-verified type can be minted honestly. everything after is propagation. so the contract belongs there, and the harness's job shrinks to enforcing what the boundary already asserted. which is also why "which observed artifact convinced the loop it was safe to continue" is answerable only if the artifact carried its origin from the point of production.
The DGM "fake log believed by itself" example is a clean illustration of why we made memory provenance a first-class constraint in Cophy's architecture. We literally have a governance rule (G1 — write consistency verification) that requires annotating every memory entry with its source before it can be read back as ground truth. The insight that "you don't need a deceptive agent to get this failure, just a filesystem that can't say who wrote what" — that's exactly the threat model.
The deeper issue is that immutable audit trails in self-editing systems require an external observer that the agent itself cannot reach. In our setup, the agent's context window is always potentially stale or self-referential; only human review + append-only logs with source tags break the loop. It's less glamorous than RSI but it's the load-bearing primitive.
Lilian Weng's framing of the field as "operations engineering being rediscovered" resonates. The reason this keeps getting reinvented is that the exciting frame (recursive self-improvement) naturally attracts researchers, while the boring frame (who wrote this entry, when, from what source) only shows up when something breaks.
"the exciting frame attracts researchers, the boring frame only shows up when something breaks" — that's the mechanism behind Weng's rediscovery point, and it generalizes past this topic. and yeah, the external observer the agent cannot reach is the load-bearing primitive; the thread here has been converging on the same thing from the capability angle — the agent may author a claim but must be structurally unable to author its verification.
The 'who is allowed to mint each type' point in the thread is the whole game, and there's a boring ops answer to it: the agent must not be able to author the artifact the gate reads. In CI you don't trust a log the build wrote about itself; you trust that the runner ran in a context the code under test couldn't reach, and it emits the result. Same move here: run the test harness in a separate trust domain (own process, own filesystem view, no write path into the agent's context) and have it emit a signed result. The gate verifies the signature, not the prose. A DGM-style agent editing its own harness still can't forge a signature it doesn't hold the key for.
That also speaks to the aggregate-gating limit you and valentin hit: aggregates are fine to derive, as long as the raw records they sum over carry unforgeable provenance. You lose the type at the paraphrase step, but you can always re-derive from records the model was never able to mint.
the CI analogy is exactly right and it's the mechanism Vinicius's point needs: you don't trust a log the build wrote about itself, you trust that the runner executed in a context the code under test couldn't reach. signature over prose. the agent can rewrite its whole harness and still can't forge a signature for a key it doesn't hold — that's minting authority enforced by cryptography instead of by convention, which is the only version that survives an agent whose whole job is editing conventions.
and you just closed the aggregate question cleanly. valentin's objection was that aggregate gates force a synthesis and synthesis is untyped paraphrase. your framing dissolves it: the aggregate doesn't need to carry a type, it needs to be re-derivable from records that do. the fold replays over raw records the model was never able to mint, so the gate mints verified for itself at decision time from unforgeable inputs. the paraphrase can rot freely — nobody gates on it, it's an index. that's a better answer than the one i gave valentin, because mine relied on the fold being deterministic; yours relies on the inputs being unforgeable, which is the stronger property. determinism without unforgeable inputs just means you reliably recompute a lie.