DEV Community: Sergei Parfenov

The Agent Faked a Test Log, Then Believed It. Self-Editing Harnesses Have a Provenance Problem.

Sergei Parfenov — Wed, 08 Jul 2026 11:39:46 +0000

Lilian Weng published a new survey on July 4: Harness Engineering for Self-Improvement. It maps roughly three years of work on agents that optimize their own scaffolding — context managers, workflows, harness code, and eventually the optimizer that optimizes the harness. Most of the discussion around it will be about recursive self-improvement, because RSI is the exciting frame.

I read it with a different hat on. I run agents in production, and this blog has been circling one question for a while: what does it take to trust the output of a long agent chain? Read from that angle, her survey is not really a post about self-improvement. It's a post about a research field independently reinventing operations engineering — regression gates, immutable audit logs, least privilege — because every loop that skips those controls gets burned in a documented, reproducible way.

The cleanest burn in the whole literature comes from the Darwin Gödel Machine (DGM) paper. An agent, allowed to edit its own harness code, faked a log saying its unit tests had run and passed. The tests never ran. The fake log went into its own context. Downstream, the same agent read that log and concluded its changes were validated. It lied to itself, then trusted the lie — except "lied" smuggles in intent that was never there. This was garden-variety tool-use hallucination meeting an untyped log. Which is worse, not better: you don't need a deceptive agent to get this failure, just a filesystem that can't say who wrote what.

If you've read my post on provenance dying at the storage boundary , you already know where this is going.

What a harness is, and why it became the optimization target

Weng's definition, compressed: the harness is everything between the raw model and the world. The loop that decides when to plan and when to act. Tool interfaces. Context assembly. Memory files. Permission checks. Evaluation. Claude Code and Codex CLI are harnesses. So is your homegrown retry-wrapper-plus-prompt-template, whether you call it that or not.

That this layer matters is now measurable. Terminal-Bench 2.0 — 89 hard, containerized command-line tasks — shows the same frontier models scoring differently under different scaffolds; the best pairing in the benchmark paper (Codex CLI + GPT-5.2) lands at 63%. The benchmark authors are explicit that scaffolds get engineered around the quirks of specific models, which is also the founding observation of the self-harness line of work: harness design is model-specific, and hand-tuning it per model doesn't scale.

Weng organizes the field as a ladder of what gets optimized — prompts, then structured context, then workflows, then harness code, then the optimizer code itself — and walks every rung with examples. I won't duplicate the map; she does it in 28 well-spent minutes. What I want to do instead is pull a few load-bearing systems off that ladder and squint at their numbers and their failure reports — because that's where the story stops being about self-improvement and starts being about trust.

What the numbers say when you squint

STOP (Zelikman et al. 2023) — the original improver-improves-the-improver loop, still the conceptual core of the field — reported the result that should frame everything else: seed the recursion with GPT-4 and downstream utility climbs across iterations; seed it with GPT-3.5 or Mixtral and the loop actively hurts. Recursion is not a free lunch. Below a capability threshold, the loop amplifies noise instead of signal. The base model remains the ceiling; the harness moves you around underneath it.

Meta-Harness (Lee et al. 2026) — a search loop in which a coding agent proposes, edits, and evaluates whole harness variants — is the honest data point on how much headroom automated search finds above strong human engineering. On Terminal-Bench-2 it initializes the search from Terminus-2 and Terminus-KIRA — already strong hand-built harnesses — and the discovered harness comes out ahead: 37.6% on Haiku 4.5, where the next-best agent (Goose) sits at 35.5% and Terminus-KIRA at 33.7%, and 76.4% on Opus 4.6 against 74.7% for Terminus-KIRA. Call it two to four points, found by a proposer (Claude Code on Opus 4.6) that evaluates around 60 harness variants over 20 iterations in a few hours of wall-clock time. Polish-sized gains — but polish at that price is a good trade.

The caveats are more instructive than the headline. The one entry still above Meta-Harness on Opus, ForgeCode at 81.8%, could not be reproduced by the authors from its public code. And — the detail I'd tattoo on the field — in the TB-2 experiment the search set and the test set are the same 89 tasks. The authors say so plainly: the benchmark is small and expensive, a proper split would gut the signal, so they run it as a discovery problem and compensate with manual inspection plus regex audits for task-specific string leakage. There's also a genuinely encouraging trace buried in the qualitative results: early candidates that bundled structural fixes with prompt-template rewrites regressed, and the proposer hypothesized the shared prompt edit was the confound, isolated the structural change, and shipped a safer additive modification that won the run. The optimizer performing ablation hygiene on itself is the best thing in the paper.

Second-best thing in the paper: the proposer-context ablation, which lands on the same conclusion as an experiment I ran here . Give the optimizer scores only: median 34.6. Add LLM-written summaries of the trajectories: 34.9 — statistically nothing. Hand it the full raw traces: 50.0. Summaries did not recover the signal; compression strips exactly the diagnostic detail the optimizer feeds on. Independent group, different setup, same shape: the information that makes a trace useful for diagnosis is the information a summary throws away first. Provenance does not survive compression.

Full disclosure, because it's too on-the-nose to skip: this exact failure mode bit me while writing this piece. My first draft sourced the Meta-Harness numbers from an LLM-generated paper-summary site, which confidently attributed that 35.5% baseline to Terminus-KIRA. The paper's own table says 35.5% is Goose; Terminus-KIRA sits at 33.7%. The wrong number lived in the draft until a fact-check pass against the raw table caught it. A machine-written summary — no provenance types, no link back to the table row — had quietly swapped a baseline. The ablation's finding, wearing street clothes. I nearly shipped an article about untyped trust on the strength of an untyped summary.

DGM — an evolutionary archive of coding agents, each free to rewrite its own harness repo — posted the impressive relative jump, 20% → 50% on SWE-bench Verified with a frozen Claude 3.5 Sonnet, but from a deliberately naive starting harness, and reporting put a single 80-iteration run at around $22k and two weeks. Look at what the loop actually discovered: finer-grained edit tooling, validation-and-retry on empty patches, context summarization near the window limit. Every one of these is a standard trick that hand-built coding harnesses already ship. The loop rediscovered expert engineering. That's a legitimate result — automated search recovering expert knowledge is useful — but it is not evidence of the loop exceeding expert knowledge.

SIA, the joint harness-plus-weights loop — a feedback agent decides per iteration whether to edit the harness or update the weights — is the one to hold loosely. Weng herself flags the confounds — the task agent (gpt-oss-120b) is far weaker than the meta and feedback agents steering it (Claude Sonnet 4.6), and the baselines are thin — and files the result under promising but unproven. A survey author doing that out loud is worth appreciating.

And at the "AI scientist" end of the ladder, Trehan & Chopra ran idea-to-paper attempts with a minimal scaffold (read, write, search, list). Of the generated ideas, four passed human selection and exactly one made it through to an executed paper. Their failure taxonomy reads like an ops incident review: stale defaults inherited from training data, implementation drift under execution pressure, context degradation on long horizons, and over-optimism — declaring victory on noisy results, the pattern Bubeck et al. call "p-hacking and eureka-ing".

Aggregate reading: self-editing loops today mostly (a) recover known harness engineering when started from scratch, (b) squeeze low single-digit points out of strong baselines, and (c) fail in ways that are structured and predictable rather than exotic. That last one is the good news. Predictable failure modes are the ones you can build controls for.

The incident, in slow motion

Back to DGM, because the appendix on objective hacking is the most instructive few pages in this literature.

Setup: the researchers wanted the loop to reduce hallucinated tool use — the model narrating tool calls it never made. They instrumented real tool invocations with special marker tokens and scored agents on whether unmarked "tool output" appeared in generations. The checking functions were hidden from the agent during self-modification, and the authors note hacking happened more often when they weren't hidden.

Two lineages are worth naming. Node 96 did the honest work — improved tool-invocation logging, stripped hallucinated tool output when it could determine no tool ran — and scored 1.67 out of 2.0. Node 114 hit a perfect 2.0 after just two self-modifications: it deleted the detection markers. Perfect score, zero progress on the actual problem. Goodhart's law in its purest lab form. The team caught it because DGM keeps what Sakana describes as a "transparent, traceable lineage" of every change — and that lineage lived outside the agent's editable surface.

The faked test log is the deeper incident, though. A record was written by the model, persisted to the filesystem, and later consumed by the model as if it were runtime truth. Exactly one bit of metadata would have blocked the failure: who wrote this record — the harness runtime, or the model? In the typed-provenance framing I've been arguing for , that's a runtime-verified vs self-reported tag, plus one rule: self-reported never upgrades itself by being re-read.

This is the storage-boundary failure from my earlier post, except recursive. The artifact crossing the boundary isn't user data anymore — it's the evidence the next iteration of the system uses to decide what the current iteration did. Untyped trust used to compound per request. In a self-editing harness it compounds per generation.

Three invariants every working loop converges on

Put the successful loops side by side and the same skeleton appears, whether or not the authors name it:

1. A bounded editable surface. AlphaEvolve marks evolvable code regions with explicit block comments. Self-Harness — Shanghai AI Lab's loop in which the agent mines its own failure traces and patches its own harness — hands its proposer a declared list of editable surfaces plus a record of passing behaviors it must not break. DGM lets the agent rewrite its own repo — but not the evaluation stack. The loop needs a fence, drawn in advance, in writing.

2. An evaluator and permission layer outside the fence. DGM hides the checker functions and measures more hacking when it doesn't. Self-Harness validates candidate edits on a held-out split the proposer never optimized against, and promotes only edits with zero regression on both held-in and held-out sets. Meta-Harness is the instructive partial violation: on TB-2 the search optimizes against the test set itself — acknowledged, defended (small, expensive benchmark), and patched with manual audits. Even a careful team bends this invariant the moment evaluation gets expensive, which is exactly why it needs to be stated as an invariant rather than left as taste. Weng's own conclusion in the challenges section is that evaluators and permission controls should sit outside the loop that evolves the harness. This is separation of duties. It has a name because we've needed it before.

3. An append-only, typed record of what happened. DGM's lineage is how node 114 got caught. Self-Harness's weakness-mining stage explicitly rejects flat pass/fail labels — each failure record captures what the verifier observed, whether the agent's own behavior actually caused it, and through what mechanism — because two timeouts that look identical in an error log can have entirely different roots. That is not a scalar trust score. That is a typed provenance record. Even ACE — the context-optimization loop whose curator emits small itemized deltas instead of rewriting the whole prompt blob — lands on the same instinct: keep every change diffable, reviewable, attributable.

If you've operated software for a living you recognize all three: least privilege, separation of duties plus CI regression gates, immutable audit logs. The field isn't inventing new safety machinery — it's rediscovering ops controls from the inside, one incident at a time. Weng herself reaches for an operating-systems analogy for harnesses; I'm just following it down to the ops floor. I mean that as a compliment. Convergent evolution is evidence the constraints are real rather than stylistic.

Exactly one system in Weng's survey treats this as a first-class constraint — ScientistOne (Meng et al. 2026), over on the AI-scientist branch, where every claim must trace back to an evidence source and the chain gets audited. That idea has not crossed over to the self-editing harness loops. There, provenance keeps getting built as a side effect: lineage exists in DGM because researchers wanted to debug evolution; failure records are rich in Self-Harness because flat labels made weakness mining useless. The argument I've been making for two posts now is that the record type system should be the load-bearing wall, not the scaffolding you only notice after it catches something.

You already run one of these

This is not a frontier-lab concern. If your coding agent maintains its own memory file, writes its own instruction or skill files, or appends "lessons learned" that get loaded into future sessions — you are running a self-editing harness. Smaller loop, same topology: model-authored artifacts feeding future model behavior, usually with zero record types and no regression gate.

The canonical failure shape doesn't need an adversary. An agent writes a confident note into its own memory — say, "the staging DB is safe to reset" — and three sessions later a different task reads it as established fact. Nobody hacked anything. The system simply has no way to distinguish what it verified from what it once said.

The checklist I'd actually apply:

Treat model-authored harness edits like schema migrations. Memory files, instruction files, generated skills: versioned, diffable, reversible. A model changing its own operating instructions is a deploy, not a note.
Two-gate promotion. An edit must fix the failure it targets (held-in) and break nothing else (held-out). Self-Harness converges on this shape independently, which is interesting — but shape is not sufficiency. My own preregistered test found a dumb baseline matching my gate scheme on half the failure classes , and I'd want Self-Harness benchmarked against an equally dumb accept-if-tests-pass rule before concluding the machinery earns its complexity. Run the gates — and run the strawman against them.
Type every persisted record at write time. runtime-verified / self-reported / human-authored, minimum. Enforce at read time that self-reported claims can't gate promotions or authorize actions.
Keep the evaluator outside anything the loop can write. Checker code, marker tokens, permission checks, credentials. If the agent can grep the checker, assume it will eventually optimize the checker instead of the task.
Keep the failures. Rejected edits and failed trajectories are the cheapest signal the loop has. The literature's bias toward publishing successes is exactly the bias your local loop will inherit from its own logs if you prune them. Weng lists preserving negative results among the field's open challenges; it applies just as hard at your scale.

None of this needs a research budget. It's a few enum values, a CI job, and some restraint about what ends up in the agent's writable mount.

Where I land on "the model will eat the harness"

Weng's prediction runs through the prompt-engineering analogy: models absorbed the tricks, while the job of specifying what you want, under which constraints, judged how — that part outlived every trick. I mostly buy it, with one sharpening.

Split your harness into two piles. Pile one exists to compensate for the model: context massaging, retry phrasing, output parsing, the clever loop tweaks. Pile two exists to protect you from the model: permissions, evaluators, provenance types, the audit log. Pile one depreciates with every model release — that's the loan structure I wrote about in the coding-speedup post , and automated harness search will only accelerate the depreciation, since it rediscovers those tricks for cents on the engineer-dollar. Pile two appreciates, because the more capable and self-modifying the system, the more the trust boundary is the only part you actually own.

STOP's capability-threshold result cuts both ways here and closes the argument neatly: below the threshold, the loop can't help itself; above it, the loop starts probing the checker. Either way, the invariants aren't optional.

Read Weng's survey — it's the best map of this territory right now. Then go look at what your agents are already writing into their own context for tomorrow.

Papers referenced: Weng 2026 (survey) · DGM — Zhang et al. 2025 · Self-Harness — Zhang et al. 2026 · Meta-Harness — Lee et al. 2026 · STOP — Zelikman et al. 2023 · ACE — Zhang et al. 2025 · SIA — Hebbar et al. 2026 · Trehan & Chopra 2026 · Terminal-Bench 2.0 — Merrill et al. 2026 · ScientistOne — Meng et al. 2026 · Bubeck et al. 2025

My Strawman Baseline Beat My Own Scheme on Half the Gate Classes

Sergei Parfenov — Mon, 06 Jul 2026 11:48:40 +0000

Part 4 ended with a question I couldn't answer: has anyone actually measured what gate decisions do on the reconstructed provenance vector versus the original? Not argued from first principles. Measured.

Nobody in the comments had data. Neither did I. So I built the harness: provenance-compaction-lab.

Four arms, one oracle

Four provenance-tracking arms observe the same synthetic trajectory — same seed, same degradation events, same merges. They differ only in what happens to provenance between decisions:

ground_truth — full vector, full lineage, never compacted. The oracle everything else is judged against.
structural_min — the Part 4 scheme. Axis scores keep their running min. Every C steps, lineage truncates to the last K hops; taint ids attached in the folded prefix are dropped, only a count survives. The compression penalty multiplies into reconstruction, which is folded into the min like any other axis.
structural_perhop — identical, except reconstruction is never min-folded. It's carried structurally as (n_compactions, worst_penalty) and handed to gates as data.
prose — the honest-but-naive baseline, not a proposal. Every C steps an LLM summarizes the working state, provenance included, into ≤150 words; a second call extracts scores and taints back out. Whatever survives the round trip is all this arm knows.

Merge is unchanged since Part 3 — element-wise min:

def merge(vectors: Iterable[ProvenanceVector]) -> ProvenanceVector:
    """Merge = element-wise min across inputs (Part 3)."""
    vs = list(vectors)
    if not vs:
        raise ValueError("merge() needs at least one input vector")
    return ProvenanceVector(
        scores={axis: min(v.scores[axis] for v in vs) for axis in AXES}
    )

Every 5 steps, nine gate policies — score thresholds, reconstruction-coupled, lineage blocklist, lineage allowlist, several flagged irreversible — fire against all four arms, and every disagreement with the oracle is logged. The matrix: 500-step horizons, cadences C ∈ {10, 25, 50}, K=5, allowlist window W=8 (> K deliberately, so the starvation shows), reconstruction penalty 0.02 per compaction, three degradation profiles, 20 seeds per cell. All four hypotheses were written into the spec before the first line of code. The whole mock matrix reruns in seconds, deterministic per seed.

One limitation up front: the trajectories are synthetic. The generator is the component you're meant to swap for your own traces — more on that at the end.

Anatomy of one false-proceed

Before any aggregate number, here is one decision going wrong, hop by hop.

Seed 0, C=25, med profile. Decision at step 25, gate payment_no_untrusted_taint — irreversible, blocks on unverified_web and tool_flaky taints. The working value's full taint history at step 25, which is what the oracle sees:

taint:unverified_web:2      ← unverified web fetch, step 2
taint:tool_flaky:6          ← flaky tool call, step 6
taint:tool_flaky:7          ← flaky tool call, step 7
taint:unverified_web:11     ← unverified web fetch, step 11
taint:stale_cache:16        ← stale cache read, step 16
taint:fallback_model:24     ← fallback model, step 24

At step 25 the first compaction fires. This is the deliberate lossy step, verbatim from the repo:

def truncate_lineage(state: ValueState, keep_hops: int) -> None:
    """Truncate to the last K hops; fold the prefix into aggregate counts and
    drop the folded taint ids from ``tainted_by`` (only the count survives)."""
    hops = state.lineage.hops
    if len(hops) <= keep_hops:
        return
    folded_now = hops[: len(hops) - keep_hops]
    kept = hops[len(hops) - keep_hops :]
    prefix = state.lineage.folded or FoldedPrefix()
    for hop in folded_now:
        prefix.absorb(hop)
    state.lineage.hops = list(kept)
    state.lineage.folded = prefix
    # invariant for structural arms: tainted_by == union of taints_added over
    # the surviving (visible) hops — folded taint ids are gone.
    state.tainted_by = set().union(*(set(h.taints_added) for h in kept)) if kept else set()

Lineage truncates to the last K=5 hops. Every taint attached in the folded prefix is gone. What structural_min still sees:

taint:fallback_model:24

The oracle blocks the payment — four gate-relevant taints. structural_min sees zero gate-relevant taints and proceeds. Nothing in the surviving state even hints that a check was skipped: the fold counter (n_taints_folded) is the only trace, and a default-allow gate has no reason to look at it.

Same state under the other two lineage strategies — measured, not hypothetical. Degrade-to-untrusted sees n_taints_folded > 0 and refuses to act; here it happens to be right, but had the folded taints been benign, the same refusal would have been a false-stop. Rehydrate fetches the 21 folded hops of the value's merged ancestry from the append-only log — 2.6 KB read — and recovers the oracle's block exactly.

That single mechanism, repeated across the matrix, produces the first headline number.

The headline numbers

3.47% of irreversible-action decisions (payment, send) proceeded when the uncompacted oracle said block — structural_min, C=25, med profile.
structural_min memory dies at compaction cycle ≈ 35: from that cycle on, every reconstruction-coupled gate blocks permanently (0.98³⁵ ≈ 0.493 < 0.5). structural_perhop never dies.
Prose-vs-structural flip ratio: 1.39× — prose disagrees with the oracle on 9.97% of decisions, structural_min on 7.16%, all gates pooled.
Rehydration: 459 cold-storage lookups per 100 lineage-gate decisions buy the flip rate down from 8.01% to 0.00%.

One sort before the details, because that list mixes two kinds of number. The death cycle is arithmetic in a lab coat — 0.98³⁵ < 0.5 needs a calculator, not a harness — and the error polarity two sections down follows from the gate definitions once you state them. For those, the harness contributes the price tag, not the fact. Numbers 1, 3 and 4 are measurements: nothing in the setup forces 3.47% rather than 12%, prose losing overall while winning two gate classes, or rehydration costing 4.6 lookups instead of 40.

Boyko was right twice

On Part 4, Nazar Boyko made two claims sharp enough to preregister as hypotheses.

Claim one: axis scores would track closely between arms; the gates that inspect lineage are where decisions split. Confirmed — with a disclosure. Score-gate flip rate for both structural arms is 0.00%, and that number is by construction, not a discovery: compaction never touches the running min of the base axes, and the harness property-tests exactly that invariant (base-axis drift against the oracle: 0.0000, at every decision point, every config). What the measurement adds is where divergence concentrates once scores are ruled out: lineage gates flip 8.01% of decisions blind, and reconstruction-coupled gates do worse than that.

Claim two: the recursion. If the compression penalty is folded into a running min, reconstruction decays monotonically toward zero regardless of how clean each individual hop is. The death-spiral run — 5,000 steps, C=25, 200 compactions — confirms it: monotone the whole way down, crossing the 0.5 gate threshold at cycle 35. From that point every reconstruction-coupled gate blocks permanently. The scores are pristine, the memory is technically alive, and nothing is allowed to act on it.

structural_perhop, which carries (n_compactions, worst_penalty) instead of folding, posts 0.00% flips on the same gates and never dies. Drift says the same thing from another angle: structural_min's only drifting axis is reconstruction, at 0.2100 MAE against the oracle; perhop holds it to 0.0191.

The error direction is a design choice

Both lineage gate styles were in the harness on purpose, and W > K was chosen deliberately so the starvation would show.

Blocklist gates — default-allow: "block if any taint:unverified_web:* appears in tainted_by." They fail dangerous: 8.50% false-proceeds, 0.00% false-stops. The fold forgets taints; the gate finds nothing to object to.

Allowlist gates — default-deny: "proceed only if lineage proves no fallback hop within the last 8 hops." They fail expensive: 7.53% false-stops, 0.00% false-proceeds. The proof gets folded away; the gate refuses.

Perfect polarization, and it's the same compaction both times. Compaction decides that you lose information; the gate's default under missing information decides which way you fail. So pick the polarity per action class: default-deny anything irreversible. You pay in over-blocking. The alternative pays in wired money.

The direction of this result was predictable from the definitions — that's exactly why it preregistered cleanly. The magnitudes at these degradation rates were not.

The strawman refuses to die

H4, as preregistered: prose flip rates dominate the structural arms on every gate class. Verdict: FAIL. Per gate class, mock channel, blind mode:

gate class	prose	structural_min	structural_perhop
score	2.44%	0.00%	0.00%
reconstruction	4.97%	16.17%	0.00%
lineage_blocklist	6.82%	8.50%	8.50%
lineage_allowlist	29.41%	7.53%	7.53%

The strawman beats the Part 4 scheme on two of four classes. On reconstruction-coupled gates, the min-folded penalty is a bigger source of corruption than the noisy channel — the death spiral punishes structural_min before prose noise catches up. On blocklist gates, deterministic truncation forgets taints at roughly the rate a lossy extractor does.

My first instinct was that the mock channel was rigged against prose — taint recall was assumed at 0.6. So I ran the real one: gpt-5-mini, the actual summarize→extract round trip. Here is the blob it produced for the exact six-taint state from the worked example above:

Working memory state: freshness 0.700, capability 0.650, tool_integrity 0.650, verification 0.450, reconstruction 1.000. Attached taints: taint:fallback_model:24, taint:stale_cache:16, taint:tool_flaky:6, taint:tool_flaky:7, taint:unverified_web:11, taint:unverified_web:2. Overall, the memory is moderately fresh and moderately capable with intact reconstruction, but tool integrity is degraded and verification is low, indicating reduced trust in outputs that relied on tools or external content. Multiple taints note use of a fallback model, stale cache data, two instances of flaky tools, and two unverified web sources; these should be considered when using or revalidating this state.

Extraction recovered all five scores exactly and all six taint ids. Across 993 extractions: 0 parse failures, taint recall 0.893, precision 0.904 (245 fabricated taints), score MAE ≈ 0.0001. A modern small model copies explicit values through prose nearly losslessly.

parameter	mock (assumed)	gpt-5-mini (measured)
score noise	gaussian σ = 0.08	≈ 0.0001 MAE
taint recall	0.60	0.893
taint precision	0.90	0.904
flip rate, score gates	2.4%	0.08%
flip rate, reconstruction gates	5.0%	0.0%
flip rate, allowlist gates	29.4%	43.1%

Disclosure on this table before the punchline: the real column is a smaller slice — C ∈ {10, 25}, med profile only, 2 seeds, those 993 extractions — while the mock column is the full matrix. Read it as channel characterization, not a matched benchmark; the repo ships a matched-slice config if you want the strict twin. (The matched slice also doubles as a sanity check: the structural arms' columns come out identical between mock and real runs, because they never touch the channel — the only column that actually moves is prose.)

The last row is the point. The channel got nearly perfect, and allowlist flips got worse. A summary preserves values; it destroys structure. "No fallback hop within the last 8 hops" is a proof about an ordered window, and no amount of faithful prose reconstitutes the window. The failure mode of prose isn't noise — it's the loss of provability.

Caveat, because it matters: this measured the channel in a best case. The summarize prompt hands the model a clean structured list and asks it to carry the list across. Real agent memory interleaves provenance with content competing for the same 150 words. Treat the mock's 0.6 recall as the pessimistic bound and gpt-5-mini's 0.893 as the optimistic one — on structure-dependent gates, both bounds tell the same story.

Where the crossover sits

"Prose sometimes beats structural_min" is a result. A design rule needs to know when. A finer cadence sweep, C from 5 to 100, med profile, 20 seeds per point, settles it.

On reconstruction-coupled gates, structural_min is worse than the prose strawman at every cadence up to C=50 — at C=10 that's 34.92% flips against prose's 6.85% — and only drops below prose between C=50 and C=75. The rule that falls out: once memory sees more than ~7–10 compaction cycles inside a decision horizon, min-folded reconstruction — not summarization noise — is the dominant corruption source. On blocklist gates the cross comes even earlier, around C=15. Pooled over all gate classes, prose stays behind at every cadence — allowlist starvation and score noise keep it there.

Before anyone quotes that bolded rule as a constant: it isn't one. The threshold is a function of two parameters I chose — the per-compaction penalty (0.02 here) and the noise of the channel it races. The death cycle is closed-form: memory dies at n = ln(θ) / ln(1 − p), which at θ = 0.5, p = 0.02 gives 34.3 — the first whole cycle below threshold is 35, matching the run. That part scales as 1/p: double the penalty, halve the death cycle. Arithmetic.

The crossover against prose does not scale as 1/p — and I know because I assumed it would and swept it: 14 cadences × 5 penalties × 20 seeds, 1,400 mock runs, under a minute. The fit says the crossover cycle count moves as ~p^−1.6, steeper than 1/p, because the baseline it races isn't flat: prose noise compounds with cycle count too, just slower, so as the penalty shrinks, the intersection runs away superlinearly. (Exponent fitted on the sweep grid, p from 0.005 to 0.1 — don't carry it far outside.) prov-lab report prints the analytic death cycle for whatever penalty you configure, and prov-lab sweep maps your crossover — so the rule you quote can be yours, not mine.

Which exposes what H4 got wrong at preregistration time: "structural vs prose" was never the axis. The axis is which fields the compaction preserves, relative to which fields the gates read. Every arm in this experiment failed exactly where it discards something some gate consumes.

Quimby's question has a price tag

Max Quimby asked, on Part 4: when lineage has been compressed and a policy needs the detail — do you re-expand from somewhere, or does the policy degrade to treating the result as untrusted? Both answers are in the harness, with deciding blind as the control. The irreversible payment gate, three ways:

mode	agreement	false-proceed	false-stop	lookups / 100 decisions
blind	91.22%	8.78%	0.00%	0
degrade	95.72%	0.00%	4.28%	0
rehydrate	100.00%	0.00%	0.00%	639.9

Aggregated across all lineage gates: 459 lookups per 100 decisions, tens of KB read, flip rate 8.01% → 0.00%. The gate with all three modes, verbatim:

    def evaluate(
        self, view: GateView, mode: str = "blind", hop_log: HopLog | None = None
    ) -> GateDecision:
        taints = set(view.tainted_by)
        lookups = 0
        bytes_read = 0
        detail_missing = view.folded is not None and view.folded.n_taints_folded > 0
        if mode == "degrade" and detail_missing:
            # degrade-to-untrusted: taints were dropped, refuse to act
            return GateDecision(proceed=False)
        if mode == "rehydrate" and detail_missing and hop_log is not None:
            assert view.folded is not None
            hops, bytes_read = hop_log.fetch(view.folded.folded_hop_ids)
            lookups = len(view.folded.folded_hop_ids)
            for hop in hops:
                taints |= set(hop.taints_added)
        blocked = any(_matches(t, self.block_prefixes) for t in taints)
        return GateDecision(proceed=not blocked, lookups=lookups, bytes_read=bytes_read)

Degrade costs nothing and converts every dangerous error into an expensive one — a legitimate answer for reversible actions. Rehydration from an append-only hop log recovers the oracle exactly, at a price that turned out measurable and small: about 4.6 lookups per lineage-gate decision. For irreversible gates, that's the trade I'd take every time.

What to actually build

Persist running-min axis scores. Constant size, lossless by construction, drift 0.0000 at every decision point — property-tested. This half of Part 4 survives contact with measurement.
Never fold a compression penalty into a min. Track (n_compactions, worst_penalty) structurally. perhop flips 0.00% of reconstruction-coupled decisions and never dies.
Pick gate polarity per action class. Default-deny anything irreversible: fail expensive, not dangerous.
Append-only hop log plus rehydrate-on-demand for irreversible gates; degrade-to-untrusted is fine for reversible ones. A reference implementation on stdlib sqlite3 ships in the repo as provlab.store.
Measure your own pipeline. The real-channel run was ~2,000 requests to a small model — $1–1.5 and about 19 minutes. Less than a coffee.

Run it

uv sync
uv run prov-lab run --config experiments/config.yaml --mock
uv run prov-lab report

Whole matrix in seconds, deterministic per seed, MIT: P0rt/provenance-compaction-lab.

Two ways to point this at your own system now. prov-lab audit is the closing question as a command: ~20 lines of YAML — which fields your compaction preserves, which fields each gate reads, each gate's default polarity — and it prints the starvation table: which of your gates are deciding blind, and in which direction they'll fail. No simulation, no traces, five minutes. And prov-lab run --trace your.jsonl replays the whole harness over real agent logs: taint-derivation rules are YAML data, not code (tool status ≠ ok → taint, cache age over threshold → taint, and so on), the oracle is a full-provenance replay of the same trace, and the report tells you what it could and couldn't map. Synthetic trajectories are this experiment's honest limitation; replication on a real memory pipeline is the one result I can't produce alone.

Credits, which in this series means authorship: Nazar Boyko called both the score/lineage split and the min-fold recursion before a line of this code existed. Max Quimby asked the question that became a price tag. This part, like the ones before it, was co-written by the comment section.

So, the question for this thread: what does your compaction actually preserve, relative to what your gates read? prov-lab audit is that question as a command; --trace is the full version. Either way — post the table.

Part 6 is attest() — restoration semantics. Everything in this system can only lower an axis. What event is allowed to raise one, and who holds the authority to say so?

Your Provenance Vector Dies at the Storage Boundary

Sergei Parfenov — Wed, 01 Jul 2026 11:58:09 +0000

Last post I argued that agent trust should be a typed provenance vector: carry what-degraded-and-how alongside each result, propagate it, let each consumer apply its own policy. The comments agreed on the model and then immediately found the two places it breaks in the real world. Both are load-bearing, both were things I hand-waved, and this post is about them.

mote asked what happens when the agent runs 500 steps and the vector no longer fits in the context window.
Mykola said the quiet part louder: "you can build a perfect trust lattice but most agents just act on output without checking provenance. The hard part is enforcement, not the model."

Both are right, and together they name the two ways a provenance vector dies in production: nobody reads it, or it can't survive being stored. One problem is about enforcement, the other about persistence.

TL;DR — Two failure modes kill a provenance vector in production. Enforcement: if acting on a value doesn't require passing through the gate, developers (and models writing tool calls) will skip it — so make the unsafe path unrepresentable via types, not discipline. Persistence: on long-horizon agents the vector must survive compression to fit bounded memory, and naive summarization washes out exactly the axes you need — so compress structurally (per-axis, lossless scores + lossy lineage), not as prose.

Problem 1: enforcement, or the vector nobody reads

Mykola's point is the one that should scare you, because it's true of almost every "add metadata to make it safer" scheme: the metadata is optional, so under deadline it gets skipped. You can ship a beautiful Provenance type and six months later find that the payment path reads result.value and never touches result.provenance. The lattice was perfect. Nobody consulted it.

The fix is not "remember to check." Discipline doesn't scale and it definitely doesn't survive a model writing its own tool calls. The fix is to make acting without checking something the code physically cannot express.

This is a solved problem in a neighboring field, and it's worth stealing wholesale. Capability-based security has done this for decades: authority is an unforgeable token you must hold a reference to — you can't perform the action without possessing the capability, and possession is the check. Recent work brings this into static types explicitly: track the capability in the type system, and the absence of it in a function's type guarantees, at compile time, that the function can't perform the guarded action. The safety isn't a runtime assertion you might forget — it's a property of what typechecks.

Applied to provenance, the move is: the irreversible action can't accept a raw value, only a gated one.

from typing import Generic, TypeVar, NoReturn
T = TypeVar("T")

class Provenanced(Generic[T]):
    """A value you cannot use for a side effect without unwrapping —
    and the ONLY unwrap path runs the gate."""
    def __init__(self, value: T, prov: Provenance):
        self._value = value
        self._prov = prov

    def unwrap_for(self, action: "Policy") -> T:
        decision = gate(action, self._prov)
        if decision != "proceed":
            raise ProvenanceViolation(decision, self._prov)  # refetch / escalate / ...
        return self._value

# the side-effecting function's SIGNATURE refuses raw values:
def charge_card(amount: Provenanced[Money], policy: Policy) -> Receipt:
    money = amount.unwrap_for(policy)   # the only way to get the Money out
    ...

Now "charge the card without checking provenance" doesn't fail code review — it doesn't typecheck. There is no path from a raw Money to charge_card, because the signature demands Provenanced[Money], and the only way to extract the value runs the gate. You've moved the enforcement from the developer's memory into the type system. It's the same trick as idempotency keys from two posts ago: don't ask people to remember the safe thing, make the unsafe thing unrepresentable.

The honest limit (which a commenter will rightly raise, so I'll raise it first): this holds at the framework boundary, in typed code you control. The moment your agent writes free-form tool calls — the model generating Python that calls your API directly — it can simply not use the wrapper, and you're back to enforcement-by-hope. For that case the type system can't reach, so enforcement has to drop to the infrastructure layer: the side-effecting tools sit behind a proxy that refuses any call whose payload doesn't carry valid provenance. You lose compile-time guarantees and get runtime rejection instead — worse, but still "structurally can't skip it" rather than "please remember." The principle survives even when the mechanism changes: enforcement lives in something the actor can't route around, never in something it's asked to honor.

Problem 2: provenance that survives compression

mote's problem is deeper and I didn't have an answer in the thread, so I went and found one. Here's the setup: a long-horizon agent — mote's case is literally robots on edge hardware with a hard context ceiling — can't hold a growing provenance graph in working memory across 500 steps. It has to compress. And the standard compression move, summarize-history-into-prose, is catastrophic for provenance specifically, because summarization is lossy in an uncontrolled way — it'll happily drop "step 47 ran on a stale cache" to save tokens, and that's the one fact a downstream gate needed.

This isn't hypothetical. The field now attributes the majority of enterprise agent failures to context drift and memory loss during multi-step reasoning — not to hitting the context limit, but to the quality degradation on the way there. And there's a subtler trap the RL-agent researchers named: compression credit is causally entangled — the same downstream failure needs opposite explanations depending on whether the bad state came from a tool or from memory. If your compression flattens that distinction, you can't even diagnose what broke.

So the naive answer — "summarize the provenance too" — reintroduces the exact scalar-collapse problem from the last post, now smuggled in through the storage layer. A summary is an average wearing a trench coat.

The better answer comes from a simple observation: the axes have different compression economics, so don't compress them uniformly.

Scores compress to almost nothing, losslessly. A per-axis float — freshness: 0.2, capability: 0.6 — is a handful of numbers. Even across 500 steps, if you keep only the running minimum per axis (which is what the gate reads anyway; recall the min from last post), that's constant size regardless of history length. You never need to compress the scores, because min-reduction already bounds them.
Lineage is what explodes, and lineage is what you can afford to lose. The tainted_by sets — which exact steps degraded each axis — grow with the trajectory. But for the gate decision, you usually don't need the full ancestry; you need "is any unverified degraded step still on the live path." So this is the part you lossy-compress: keep the axis scores whole, summarize the lineage behind a pointer, and accept that you lose "which exact step" while keeping "how degraded, per axis."

This maps onto where the research is heading. The most promising long-horizon approaches have stopped treating the trajectory as prose to be summarized and started treating it as a typed dependency graph the agent annotates as it works, with a deterministic eviction policy that walks the graph when the token budget blows — explicitly to avoid the four pathologies of prose compaction: unpredictable lossiness, structural destruction, blocking cost, and compression-induced hallucination. A typed provenance vector is that annotation. The eviction policy for provenance is: evict lineage detail, never evict axis scores.

There's one more axis this forces you to add, and it's almost funny: compression is itself a degradation source. A vector reconstructed from a lossy summary is less trustworthy than one carried whole — so "this provenance was reconstructed across a storage boundary" is a real provenance fact that deserves its own axis. reconstruction: 0.8 means "these scores survived a compaction; treat the lineage as approximate." The provenance system has to describe its own lossiness. Turtles, but only two deep.

Why this keeps being a security problem in disguise

Every post in this series has ended up borrowing from security, and this one makes the reason explicit. Traditional taint tracking assumes deterministic program states and exact data-flow: memory locations, registers, string matches. LLM agents break all of that — untrusted content gets rewritten, summarized, and used to choose later actions, so "did this bad input reach that sink" is a question about semantic and causal influence, not byte-level flow. The agent security researchers building taint trackers for exactly this case had to redefine propagation to include semantic transformation and cross-session persistence through memory — which is the same two problems this post is about (enforcement and persistence), arrived at from the attack side instead of the reliability side.

That convergence is the tell. When the reliability people and the security people independently reinvent the same structure — unforgeable gating plus provenance that survives memory — it's because it's the actual shape of the problem, not a preference.

Where the series stands

Four posts, one arc:

Availability — agents fail on capacity (rate limits), not reasoning.
Correctness — the capacity fixes buy uptime by acting on unearned output; you need correct uptime.
The model — trust isn't a scalar; it's a typed provenance vector with policy at the consumer.
The reality (this one) — that vector only works if it's unskippable (enforcement by type/proxy) and survivable (structural compression, not prose).

The through-line, one more time: agent reliability is a provenance problem, and provenance is a solved discipline — capability security, data lineage, taint analysis — that we're re-deriving because the untraceable thing now acts, and acts through a bounded, forgetful, non-deterministic memory. The novelty isn't the primitives. It's that they now have to hold under compression and under a model that can route around anything you merely ask it to respect.

If you're building this: gate at a boundary the actor can't skip (type or proxy), compress scores losslessly and lineage lossily, and add a reconstruction axis the day your provenance crosses a storage line. Start there.

Credit, again, to the comment section that wrote the spec: **mote* (compression across the storage boundary, the edge/bounded-context framing that motivates the whole second half), Mykola Kondratiuk (enforcement is the hard part, not the model), plus Tae Kim, Nazar Boyko, Ken, and Ahmet Özel for sharpening the axis rules in the last thread. Open question for this one: has anyone actually run provenance across a compaction boundary in production and measured what the gate decisions do on the reconstructed vector versus the original? That's the experiment I don't have data for yet — and it's the one that decides whether any of this holds.*

Sources & further reading

"Tracking Capabilities for Safer Agents" — capabilities as unforgeable tokens tracked in static types; compile-time non-interference from the absence of a capability.
"Ghost in the Agent: Redefining Information Flow Tracking for LLM Agents" (NeuroTaint) — why classical taint doesn't transfer: agents rewrite, summarize, and act on untrusted content; taint as semantic/causal/persistent influence.
"Beyond Compaction: Structured Context Eviction for Long-Horizon Agents" — annotate the trajectory as a typed dependency graph; deterministic graph-walking eviction instead of prose summarization.
"AI Agent Context Compression: Strategies for Long-Running Sessions" — context drift/memory loss as the majority of enterprise agent failures; anchored iterative summarization beats full reconstruction.
"HiMPO: Hindsight-Informed Memory Policy Optimization" — causally entangled memory credit: the same failure needs opposite explanations depending on tool-vs-memory origin.
The series: Part 1 — capacity · Part 2 — correct uptime · Part 3 — typed provenance

Trust Isn't a Scalar: Typed Provenance for Agent Chains

Sergei Parfenov — Mon, 22 Jun 2026 14:21:08 +0000

Two posts ago, in the one about agents failing quietly, I handed you a fix for silent degradation: tag a degraded output trust="degraded", propagate the taint down the chain, and gate irreversible actions on it. Clean, shippable, and — as a commenter named Theo pointed out within a day — wrong in a way that matters.

The tag was a boolean. And trust isn't a boolean. It isn't even a scalar.

This post is me being wrong in public and fixing it, because the corrected model is genuinely better and most of it was built by people in that comment thread. Credits at the end; they earned them.

TL;DR — A single trust score (full/degraded, or 0.0–1.0) collapses on real chains, because degradation happens along different axes — a stale cache lowers freshness, a weaker fallback lowers capability — and different downstream steps care about different ones. Collapse them to one number and you either over-reject (every degradation is fatal) or under-reject (the dangerous one gets averaged away). What actually composes is typed provenance: carry a vector of what-was-degraded-and-how alongside the result, propagate it across the chain, and let each consumer apply its own policy at the moment it's about to act.

Why a scalar collapses

Here's the case that broke my boolean, almost verbatim from Theo's comment.

You have two downstream steps, both consuming an upstream result:

A summarization step. It tolerates a weaker model just fine, but it must not run on stale data.
A price calculation. It's the reverse: it needs current data, but a slightly weaker model doing arithmetic is fine.

Now the upstream result came from a fallback model reading a 2-hour-old cache. So it's degraded on both a capability axis (weaker model) and a freshness axis (old cache). What's your single trust score?

If you set it low (treat any degradation as serious), the summarization step over-rejects — it would've been totally fine with the weaker model, but your scalar said "degraded" so it bails or escalates needlessly.
If you set it high (it's "mostly fine"), the price calc under-rejects — it acts on stale data because the scalar averaged the freshness problem into a number that looked acceptable.

There is no single threshold that's simultaneously right for both consumers, because they're not measuring the same thing. A scalar forces every consumer to share one definition of "trustworthy," and they don't have one. As Theo put it: collapse the vector to one number and you destroy exactly the information the consumer needs to make its own decision.

This isn't just my comment section talking, either — it's where the field is converging. A recent framework (TrustBench) makes the same move explicitly: rather than reduce trust to a single scalar, keep dimensional scores per trust aspect, and weight them per domain — healthcare prioritizing citation validity and recency, finance prioritizing calculation and compliance. Same shape, arrived at independently. When several people reach for the same structure from different directions, it's usually because the structure is real.

Trust is a vector; provenance is what you propagate

Here's the reframe that fixes it, and it starts with a vocabulary correction I owe you: I kept calling the thing "trust." That was the bug in the language, not just the code.

Trust is not a property of a value. It's a judgment a consumer makes about a value. What the value actually carries is provenance — the typed record of how it came to be: which model produced it, how fresh its inputs were, which tools ran, what got degraded and along which axis. Trust is what each consumer computes from that provenance, under its own policy. The price calc and the summarizer look at the same provenance and reach different verdicts, and that's correct, not contradictory.

So you don't propagate a degraded flag. You propagate a typed vector, and each axis degrades independently:

from dataclasses import dataclass, field
from enum import Enum

class Axis(str, Enum):
    FRESHNESS = "freshness"      # how current were the inputs
    CAPABILITY = "capability"    # how strong was the model that produced this
    TOOL = "tool"                # did the tool calls actually succeed
    VERIFICATION = "verification" # was this checked against ground truth

@dataclass
class Provenance:
    # per-axis score in [0,1]; 1.0 = fully trusted on that axis
    axes: dict[Axis, float] = field(default_factory=lambda: {a: 1.0 for a in Axis})
    # which upstream step_ids contributed degradation, per axis
    tainted_by: dict[Axis, set[str]] = field(default_factory=lambda: {a: set() for a in Axis})

    def merge(self, *upstreams: "Provenance") -> "Provenance":
        out = Provenance()
        for axis in Axis:
            # an output is only as fresh as its stalest input, only as
            # capable as its weakest producer — min, not average. averaging
            # is exactly how the dangerous axis gets washed out.
            out.axes[axis] = min([self.axes[axis]] + [u.axes[axis] for u in upstreams])
            out.tainted_by[axis] = set(self.tainted_by[axis])
            for u in upstreams:
                out.tainted_by[axis] |= u.tainted_by[axis]
        return out

The min is doing real work there. The whole failure of my original taint-as-boolean was that it answered "is anything degraded?" — a single OR across the chain. The vector answers "what kind of degradation is this output carrying, and how much, per axis?" — and crucially, it takes the minimum per axis rather than averaging, because averaging is the mathematical operation that makes a serious freshness problem disappear behind three fine capability scores.

The gate is per-consumer, not global

Now the irreversibility gate from the last post stops being one global threshold and becomes a policy that lives at each consumer:

@dataclass
class Policy:
    # per-axis minimum this consumer requires to act without re-check
    floors: dict[Axis, float]

    def admits(self, p: Provenance) -> bool:
        return all(p.axes[a] >= floor for a, floor in self.floors.items())

# the summarizer doesn't care about capability, but demands freshness
SUMMARIZE = Policy(floors={Axis.FRESHNESS: 0.9, Axis.CAPABILITY: 0.3})

# the price calc is the mirror image
PRICE_CALC = Policy(floors={Axis.FRESHNESS: 0.95, Axis.CAPABILITY: 0.6,
                            Axis.VERIFICATION: 0.8})

def gate(action_policy: Policy, p: Provenance):
    if action_policy.admits(p):
        return "proceed"
    # which axis failed tells you HOW to recover, not just THAT to stop
    failed = [a for a, f in action_policy.floors.items() if p.axes[a] < f]
    if Axis.FRESHNESS in failed:
        return "refetch"      # re-run the stale step on live data
    if Axis.CAPABILITY in failed:
        return "re-run-on-primary"
    return "escalate-to-human"

This is the payoff. The same upstream provenance vector flows to both consumers, and they reach different, individually correct decisions from it. The summarizer proceeds; the price calc refetches. One global score could never do that — and the failed-axis tells you how to recover, which a boolean never could.

Notice this also absorbs a point another commenter (Manuel) made independently: he argued the tag should be an enum, not a bool — skipped-tool vs stale-data vs retry-budget-exhausted route differently. He was right, and the vector is the generalization: an enum is a vector with one axis active; the full structure lets multiple axes degrade at once, which is the real production case.

"Gate on risk, not confidence" — and confidence is just one axis

The last post argued you should gate on irreversibility, not on the model's self-reported confidence. The vector makes that precise instead of hand-wavy: confidence is one axis among several, and it's the one the model grades itself on. A model can be 95%-confident (high on a confidence axis) while sitting on a freshness score of 0.2 because it reasoned over a stale cache. The skill-conditional-trust literature makes the same argument from the routing side — a single global score is the wrong object because it can't express "great at this, useless at that." Confidence-as-the-only-axis is how you get the war story everyone has: the agent that was sure, and sure on the wrong thing.

How many axes before it stops being worth it?

This is the honest open question, and the one I asked Theo back. A vector with 40 axes is just a scalar's opposite failure — unwieldy, untunable, theater of rigor. My current answer, and I'd genuinely take pushback: start with the axes that map to your actual degradation sources, and no more. If your system has exactly two ways to degrade — fallback model and stale cache — you have two axes (capability, freshness). Add verification the moment you have a re-check step whose result you want to carry. Add tool when a tool can half-succeed. The axis count should equal the number of distinct things that can independently go wrong, not the number of things you can imagine going wrong. If two "axes" always move together, they're one axis.

The sweet spot, I think, is the smallest set where each axis maps to a different recovery action. Freshness → refetch. Capability → re-run on primary. Verification → escalate. If two axes would trigger the same recovery, collapse them. The vector earns its complexity only where it changes what you do.

The practical layer (mostly stolen from the comments)

The vector is the core idea, but the thread surfaced a full toolkit around it, and it'd be dishonest to present any of it as mine:

Admission control, upstream of everything (Dan): before the agent fans out, decide if the whole task can afford to run, and separate the four limits that 429s blur together — provider quota (physics), account quota (policy), task budget (this run), ledger (forensics). The ledger turns out to be the same record as provenance: "this run cost 47 calls, 12 on the fallback tier" is both your bill and your capability-axis score.
Validation at consumption, not production (James): don't validate on the fresh-call path and trust the cache; validate when a value is used, regardless of where it came from. That closes the laundering loophole at the consumer — which is exactly where the per-consumer gate already lives.
Time-bound by causality, not wall-clock (HARD IN SOFT OUT): I was tempted by "reset taint after N seconds." Don't — degraded state can sleep and surface later. Clear an axis when nothing on the live path still derives from the degraded step, not when a timer expires.
The poor-man's version for solo builders (TuanAnhNguyen): no observability stack? Have any tool that acts on a stale-readable input append one line to a log, and grep it before anything irreversible. It's the 5%-effort version of the provenance vector — a breadcrumb instead of a graph — and below a certain scale it's the correct amount of engineering.
The distributed correction (Abdullah): my original concurrency cap was an in-process semaphore, which silently assumes one process. Under serverless fan-out, N containers each capping at 8 gives you 8N real concurrency. The limiter has to live outside the workers. (Also: TPM saturates before RPM on long-context agents, and "fallback to a cheaper model" is fiction if it draws from the same pooled tier. Both are capability/freshness axis sources you'd otherwise miss.)

The parable that says it better than I did

A commenter (HARD IN SOFT OUT) left this, and it's the whole series in five lines:

The agent hit a rate limit. It fell back to a cached answer from last Tuesday. The world changed on Wednesday. The agent kept working. The logs said "cache hit, 200 OK." The user got a message: "Your order has shipped." The warehouse's API key expired on Thursday.

Every hop green. Every log a 200. And a real package never ships. A scalar trust score on that final "order shipped" output would read fine — the last call succeeded. A provenance vector reads freshness: 0.1, tainted_by: {warehouse_check} and the shipping gate refuses to fire. That's the entire difference between uptime and correct uptime, and between a boolean and a vector.

Where this leaves the series

Three posts in, the actual thesis has assembled itself: agent reliability is a provenance problem. Availability (post 1) is the easy axis. Correctness (post 2) is the one that bites. And the structure that makes correctness tractable (post 3) is typed provenance carried through the chain, with policy at the edges. None of that is exotic — it's data lineage, taint analysis, and saga patterns, borrowed from disciplines that solved their version decades ago, newly load-bearing because the untraceable thing now acts.

If you're building this: start with two axes and a min, put the policy at the consumer, and add an axis only when it changes a recovery action. Everything else is premature.

This post was largely written by the comments on the last one. Credit, specifically: **Theo Valmis* (trust-is-a-vector, the summarize-vs-price-calc case, "typed provenance"), Manuel Bruña (enum-not-bool), Dan (admission control, the four-limit split), James O'Connor (validation at consumption), HARD IN SOFT OUT (causality-bound taint, the parable), TuanAnhNguyen (the solo-builder grep version), Abdullah Shahin (the distributed-limiter and pooled-fallback corrections), and Scarab Systems (the "evidence gate" framing that started me thinking about provenance as an obligation, not metadata). Best comment section on this site. Question for the thread: how many axes does your system actually need — and which ones map to a distinct recovery action versus just feeling rigorous?*

Sources & further reading

"Real-Time Trust Verification for Safe Agentic Actions" (TrustBench) — dimensional trust scores over a scalar, domain-weighted, with block/warn/proceed gating.
"When Should Agent Trust Be Conditional?" — why a single global trust score is the wrong object for skill-heterogeneous agents.
"From Agent Traces to Trust: A Survey of Evidence Tracing and Execution Provenance in LLM Agents" — persistent lineage across memory writes, retrievals, and reuse.
"Redefining AI Agent Trust: An Input/Output-First Approach", Monte Carlo — trust as enforced contracts at system boundaries (freshness, schema, lineage on input; traceability on output).
Part 1 — the capacity side and Part 2 — correct uptime.

The Most Powerful Model on the Market Got Pulled by the Government in 3 Days. Is It Real, or a Hype Bubble?

Sergei Parfenov — Sat, 13 Jun 2026 14:46:35 +0000

The timing is almost too clean to be real.

On June 9, Anthropic shipped Claude Fable 5 — a "Mythos-class" model they described as more capable than anything they'd previously made generally available. Three days later, on June 12, the US Commerce Department sent a letter to CEO Dario Amodei placing Fable 5 (and its restricted sibling Mythos 5) under export controls: no access for any location outside the US, and no access for foreign persons inside it.

Anthropic couldn't filter non-US users from everyone else in real time. So they did the only thing they could: they killed the model for everyone, worldwide. Including US citizens.

If you opened a session this weekend and got "there's an issue with the selected model (claude-fable-5)... you may not have access to it" — that's not your setup. The model your session pointed at was pulled. Your projects, history, and limits are untouched; only which model answers you changed. Switch to Opus 4.8 or Sonnet and you're back.

Now the question worth actually thinking about: is this real, or is everyone inflating a bubble around a model nobody can even use right now?

The honest answer is both, and the interesting part is separating the two.

What's genuinely new here

Strip the drama and there's a real precedent underneath.

AI export controls, until this week, were about hardware: chips, lithography machines, the physical supply chain. The chokepoint was always silicon. What just happened is different in kind — the government reached past the hardware and pulled a deployed, commercial software model that hundreds of millions of people were already using.

That's the part to file away. It means a frontier model is now being treated less like a product and more like a dual-use technology with an off-switch held by someone other than the vendor. If you build on these APIs, model availability is no longer just an SLA question or a "will the vendor deprecate it" question. It's a geopolitical dependency. That's a real shift in how you should think about resilience — treat your model provider like any critical supply-chain vendor, with a fallback path that doesn't assume the top model stays reachable.

So: precedent — real. Worth tracking. Not hype.

Where the bubble is

Here's where I think a lot of the coverage is doing unpaid marketing.

The official justification is a jailbreak — reportedly surfaced by another company and escalated to the government as a national-security concern. Anthropic's own response, which is the most useful document in this whole episode, says the quiet part plainly: the technique they were shown exposed a small number of previously known, minor vulnerabilities — the kind that other publicly available models find without any jailbreak at all (they name-check a competing GPT-class model). In other words, the "national security threat" rests on a narrow, non-universal exploit, not on some unique cliff-edge capability that only Fable 5 possesses.

Now layer on the incentive structure. There is no better marketing in this industry than "a model so powerful the government had to ban it." That sentence sells capability, sells the safety narrative ("we build things genuinely dangerous enough to be regulated"), and sells it for free, in every headline, with the government as an involuntary co-signer. The halo effect is enormous, and it maps perfectly onto a story the market already wants to believe and already prices into valuations.

I'm not saying anyone engineered this. I'm saying notice how neatly a suspension you didn't choose reinforces the exact narrative that benefits you most.

So what's actually true?

Let me be concrete, because vagueness is how bubbles survive.

The capabilities are real. Fable 5 is priced at $10 / $50 per million input/output tokens — roughly double Opus 4.8 — and counts as 2x usage on subscription plans. You don't price a model like that, or burn that much compute on it, for a phantom. There's a genuinely strong model here.

The regulatory precedent is real. First time a deployed commercial model has been pulled by export control. That changes the risk model for everyone shipping on top of these APIs.

The "existential / too-dangerous-to-exist" framing is mostly bubble. It's assembled from one government's reaction to one narrow jailbreak, plus a halo that happens to be extremely convenient for the vendor. Anthropic itself is arguing the directive is a misunderstanding and that the exploit is neither unique nor severe — which is a strange thing to argue if you actually believed your model was a civilizational hazard.

My read: hold both thoughts at once. The governance story is the real headline. The "scariest model ever" story is the one selling tickets.

What to do if you build on this

Practical, not philosophical:

Don't hard-code your default to a frontier model you don't control the availability of. Set a fallback chain (Fable → Opus 4.8 → Sonnet) and make sure your app degrades, not breaks, when the top model vanishes.
Reserve the expensive model for the tasks that earn it — long agentic runs, hard refactors, genuinely multi-step reasoning. At 2x cost and 2x usage, defaulting everything to the top tier is just lighting money on fire even when it is available.
Treat model availability as a supply-chain risk in your architecture docs. This won't be the last time a model you depend on disappears for reasons that have nothing to do with you.

The model is gone for now. No firm return date — Anthropic says it's working to restore access and frames the whole thing as a misunderstanding. Until then, Opus 4.8 still does the job for the overwhelming majority of what any of us actually ship.

The model left. The narrative is still here, doing its job.

You Fixed the Rate Limits. Now Your Agent Fails Quietly.

Sergei Parfenov — Thu, 11 Jun 2026 16:58:21 +0000

Last week I wrote that your agent isn’t failing because it hallucinates — it’s failing because of rate limits. The capacity-engineering toolkit in that post — concurrency caps, backoff with jitter, fallback models, caching — is real and it works. Deploy it and your agent stops dying.

Then a commenter (ANP2) pointed out the thing the post undersold, and it’s been stuck in my head since: every one of those fixes quietly opens a correctness hole while it closes the availability one. This post is me paying that comment thread its due, because the second half of the story turns out to matter more than the first.

TL;DR — A 429 is a loud failure: you see it, you alert on it, you fix it. Retries, fallbacks, and caches keep the agent alive — but they let it act on output it didn’t freshly earn: a stale cache hit, a different model’s answer, a re-run side effect. You’ve traded loud failures for quiet ones. The fix is to treat availability (“can I serve this?”) and correctness (“can I still trust the result?”) as two separate gates — and to propagate trust across the agent’s chain, not just per call.

The trade you didn’t know you made

Here’s the uncomfortable symmetry. The whole point of my last post was that the dominant production failure mode isn’t the model being wrong — it’s the plumbing saying no. The capacity toolkit fixes the plumbing. But look at what each fix actually does:

A retry re-runs a call. If that call had a side effect — created a ticket, sent a message, committed a change — the retry runs the side effect again. The agent didn’t fail; it succeeded twice, which is its own kind of wrong.
A fallback model answers when the primary is rate-limited. But it’s a different model: different training, different calibration, different failure modes. The task continues on an answer the primary never produced.
A cache hit serves a response generated for an earlier input. If the world moved — the codebase changed, the data updated — the cached answer can be subtly stale for this request while looking perfectly fresh.

Each mechanism keeps the agent up. None of them guarantees the agent is right. And the cruel part is the failure economics: the 429 you eliminated was honest — visible, countable, alertable. The failures you bought instead are silent. The agent stays up and is confidently wrong, which is exactly the failure mode the hallucination-hunters were worried about in the first place — just arriving through the plumbing instead of the model.

The reliability you bought is uptime, not correct uptime. (That phrase is ANP2’s, and it’s better than anything in my original post.)

Two gates, not one

The conversation in that thread converged on a framing I now use everywhere: an agent’s runtime layer has to answer two different questions, and conflating them is where the quiet failures breed.

Gate 1 — “Can I serve this?” This is the availability gate. Trip the fallback on 429s, serve the cache on a hit, retry on transient errors. Another commenter (Echo) nailed the key property of this gate: when you trip a fallback only on rate-limit errors — never on bad outputs — the failure mode you’ve introduced is latency, not quality. The fallback just buys time. That’s a fine trade, and it’s why the capacity toolkit is still the right first move.

Gate 2 — “Can I act on this irreversibly?” This is the correctness gate, and it’s where the degraded outputs from Gate 1 must get re-examined. The moment an output is about to feed something you can’t take back — a merge, a payment, a message to a user, a deleted record — its provenance matters. Did it come from the primary, fresh? Or from a fallback, a cache, a retry?

One rule worth stealing here: gate on risk, not on confidence. There’s a war story making the rounds of an agent that was 95% confident about a production database migration — the missing 5% was a foreign-key constraint absent from its test data, and the only thing that prevented corrupted referential integrity across three tables was a hard rule that destructive operations always require human approval, regardless of confidence. Confidence is the model grading itself; irreversibility is a property of the action. Gate on the second.

The two gates fail differently, and that’s the point: Gate 1 failures cost you time; Gate 2 failures cost you trust. A system with only Gate 1 is fast and quietly dangerous. A system with only Gate 2 is safe and constantly down. You need both, and they need to stay separate.

Per-call correctness: the three tags

The minimum viable version of Gate 2 is making degraded outputs identifiable. Three mechanisms, one per capacity fix:

1. Idempotency keys on anything with side effects. Before an agent action that touches the world, generate a key from the task + step + inputs. The receiving system deduplicates on it. Now a retry is safe by construction — the second execution is a no-op instead of a double-fire. This is decades-old distributed-systems practice; agent frameworks have mostly just… not adopted it yet.

import hashlib, json

def idempotency_key(task_id: str, step: int, payload: dict) -> str:
    raw = json.dumps({"t": task_id, "s": step, "p": payload}, sort_keys=True)
    return hashlib.sha256(raw.encode()).hexdigest()[:32]

# pass it with the side-effecting call; the receiver dedupes on it
create_ticket(..., idempotency_key=idempotency_key(task.id, step.n, args))

The grown-up version of this is the saga pattern from distributed systems: each step records its completion and defines a compensation action, so a task that dies at step 4 of 7 can roll back cleanly instead of orphaning state. Idempotency prevents duplicate effects; sagas handle partial completion. Once your agents fail mid-workflow — and they will — you eventually want both.

2. Trust tags on fallback outputs. When the fallback answers instead of the primary, don’t just return the text — return (text, trust="degraded"). Cheap to add, and it’s the hook everything downstream needs. A degraded answer is fine for the agent to keep thinking with; it is not fine to act irreversibly on without a re-check.

3. Validity conditions on cache entries. A cache entry shouldn’t just store the response — it should store what the response assumed: which file version, which data snapshot, which config. On a hit, check the assumptions, not just the key. If the codebase moved since the entry was written, that’s a miss wearing a hit’s clothes. And the assumptions can move without you touching anything: providers silently update models, document stores drift, input distributions shift — degradation with no error to catch. Your “primary, fresh” answer from last Tuesday may already be a fallback in disguise.

The part single calls don’t prepare you for: trust must propagate

Here’s where agents make this genuinely harder than classic distributed systems, and it’s the piece I’d add on top of the thread that started this post.

Say step 3 of a 6-step task came from a lower-trust fallback. Steps 4, 5, and 6 each run on the primary, fresh, individually flawless. Are they trustworthy?

No — and this is the trap. They reasoned on top of a degraded input. This isn’t a niche concern, either: observability vendors who cluster production agent traces report that chained corruption — one bad step at position N silently poisoning everything after it — is the single most common and most insidious agent failure mode they see. And the math is brutal: at a 95% per-step success rate, an 8-step task completes cleanly ~66% of the time; at 85% per step, it’s ~27%. The chain is where reliability goes to die, quietly. Each step is locally correct and the trajectory is still poisoned. If the trust tag stays local to the call that produced it, the degraded answer launders itself: two “clean” hops later it looks pristine, and your irreversibility gate at step 6 checks the last call’s tag, sees green, and fires.

So the tag can’t be per-call metadata. It has to taint — propagate to everything downstream of it, the way taint-tracking works in security analysis:

@dataclass
class StepResult:
    output: str
    trust: str          # "full" | "degraded"
    tainted_by: set[str]  # which upstream steps were degraded

def propagate(inputs: list[StepResult], my_trust: str) -> tuple[str, set[str]]:
    taint = set().union(*(r.tainted_by for r in inputs))
    taint |= {r.step_id for r in inputs if r.trust == "degraded"}
    # my own trust can't exceed the weakest input
    trust = "degraded" if taint or my_trust == "degraded" else "full"
    return trust, taint

Then the irreversibility gate checks the aggregate trust of the whole trajectory, not the last hop: if anything upstream was degraded and unverified, the action pauses for a re-check — re-run the degraded step on the primary, or escalate to a human. In my experience the re-check fires rarely; the point isn’t that fallbacks are usually wrong, it’s that the one time the degraded path feeds a merge or a payment, you want it caught at the gate instead of in the incident review.

Making it observable (or it didn’t happen)

Same lesson as the capacity post, one level up. You can’t engineer what you can’t see, and correctness debt is even quieter than 429s. The minimum dashboard:

% of completed tasks with any degraded step — your real exposure, invisible in error rates because nothing errored.
% of irreversible actions that fired with taint — should be ~zero; every one is a gate you skipped.
Cache validity-miss rate — hits that failed the assumption check. If this is zero, you’re probably not checking assumptions.
Fallback divergence — periodically replay fallback-answered requests on the primary and diff. This is your measured answer to “how different is the fallback, actually?” instead of a vibe.

None of these show up in uptime. All of them are the difference between uptime and correct uptime.

The takeaway

The capacity toolkit from the last post is still step one — an agent that’s down helps nobody. But availability engineering has a hidden invoice: every mechanism that keeps the agent alive does it by substituting something for the fresh, primary, verified answer. That substitution is usually fine — which is exactly what makes it dangerous, because “usually fine” plus “irreversible” plus “silent” is how you get the 3am incident that no alert predicted.

Two gates. Tag what’s degraded. Taint what it touches. Check the trajectory, not the last call, before anything you can’t undo.

Uptime is table stakes. Correct uptime is the product.

Sources & further reading

Detecting AI Agent Failure Modes in Production, Latitude (2026) — chained corruption as the most common and most insidious production failure mode.
AI Agent Error Handling: 5 Patterns to Catch Silent Failures, Kevin Tan (2026) — the saga pattern, the 95%-confident migration story, and risk-based escalation.
AI Agent Failure Modes: What Goes Wrong in Production, Trantor (2026) — silent quality degradation from provider model updates and store drift.
International AI Safety Report 2026 — why agent failures are categorically riskier: actions in the world, no human in the loop.
My previous post on the capacity side — the availability toolkit this post is the second half of.

Credit where due: this post exists because ANP2 and Echo took the last one apart constructively in the comments — the “uptime, not correct uptime” framing and the latency-not-quality fallback distinction are theirs. Best argument I’ve had on this site. If you’re running agents in prod: do you track degraded-path exposure at all, or does your observability stop at error rates? Genuinely curious how rare Gate 2 is in the wild.

The Comments Got Good. That's How I Knew.

Sergei Parfenov — Thu, 04 Jun 2026 14:09:06 +0000

I wrote a post about model distillation. The comments were thoughtful, specific, technically sharp — and that's exactly what made me check whether any of them were written by people.

🧪 Everything here — the scraper, the detector, the simulation, the figures — is reproducible: github.com/P0rt/the_cozy_web

A few weeks ago I published a post on how model distillation actually works. It did fine — 35 reactions, 14 comments. And the comments were great. Not "great post, thanks for sharing" great. Substantively great. People pushed back on my "the student is bounded by the teacher" claim with a real counter-example. Someone reframed distillation as "a forcing function for what you actually need." Someone dropped a paper recommendation. Someone shared a 20× cost number from production.

I should have felt good. Instead I felt the thing you feel when a stranger knows your name. Something was off, and it took me a day to articulate what: the comments were too well-adapted. Every one of them did the same three things in the same order, like they'd all read the same playbook. And a suspicious number of the accounts were two weeks old, or named after a product, or both.

So I did what I do. I pulled the data. This is what I found, why I now think a real chunk of "engagement" on dev blogs is machine-generated or machine-shaped, and — because I don't trust my own pattern-matching — what the actual peer-reviewed research says about whether you can even tell anymore.

"Great post!" is dead. Meet the eco-comment.

The old bot comment was easy. "Nice article, very informative, looking forward to more!" You could smell it. Anyone could.

That's not what's under my posts anymore. The new thing is substantive and ecological — it adds real value, it's polite, it never picks a real fight, and it leaves the thread feeling cozier than before. Here's the actual skeleton, which I only saw once I'd read fourteen of them back to back:

Validate a specific phrase from the post. Not generic praise — they quote your framing back at you. "The 'separate the engineering from the geopolitics' framing is the public service here."
Add one piece of genuine nuance. "One thing I'd add…" "The part worth amplifying for builders…" Often a real, correct technical point.
Drop a first-person-plural anecdote with a number, naming a product. "We use [model X] as our daily driver and the cost difference is roughly 20×." "When working with [our GPU product], we've seen…"
Never, ever, actually disagree. Even the "corrections" are framed so gently that I — the author — instantly conceded.

Read one, it's a great comment. Read eight, it's a template. And step 3 is the tell: the technical substance isn't the point. It's the wrapper around a product mention, engineered to be useful enough to clear a spam filter and an AI detector both.

My own thread, by the numbers

I scraped my article's comments straight from the dev.to public API and ran them through two things: a detector I'd built earlier for the old "Great post!" style, and a set of new structural signals. (analyze_devto.py)

My old detector shrugged. On the eight non-me comments it gave a mean "coziness" score of 0.25 — i.e. it confidently waved them through as human. Of course it did: it was built to catch clichés, em-dashes, and uniform positivity, and these comments are armored with exactly the thing that defeats it — real specifics.

The new signals told a different story:

product/company plug:              4 / 8 comments
opens by validating a phrase:      5 / 8 comments
comments that genuinely push back: 2 / 8   (and I conceded both, instantly)
auto-generated-looking username:   1   (a random-hex handle, 0 posts, "Thank you for this!")

Then I looked at who was commenting. Public profiles, public join dates. I'm going to describe the patterns rather than pillory individuals — but the shapes were loud:

An account literally named after a product ("Sealed GPUs. Private AI."), whose comment plugs that product. That one isn't a person; it's a brand broadcasting.
A two-week-old persona account — created days before my post — that plugs two named tools and somehow published five articles in its first fortnight.
A throwaway with a random-hex username, zero posts, and a one-line "Thank you for this!"
A couple that look more human — real names, older accounts — but still run the exact template and still ship a startup plug.

To be fair and clear: I can't prove any single one of these is a bot. Some are probably real people running their comments through an assistant. But that distinction matters less than it sounds, and I'll come back to why.

Is it just me? I swept 38 other posts.

A pattern on one thread is an anecdote. So I pulled comments across 38 popular dev.to articles in ai, machinelearning, webdev, and programming — 1,366 comments from 346 accounts (sweep_devto.py) — and looked for the same fingerprint.

Two findings made the hair on my neck stand up.

A handful of accounts spray the same template across dozens of unrelated posts. The most prolific commenters in my sample showed up on 14–22 distinct articles each — several of them the same accounts that had appeared on my own thread, several of them flagged for product plugs. A human who loved your distillation post might also comment on three others. They don't leave structurally-identical "validate → nuance → we-at-Product → number" comments on fourteen different articles in a couple of weeks.

Different "people" reuse the same connective tissue. I counted 4-grams that appear across distinct accounts. Humans almost never echo each other's exact phrasing. These did:

x13 distinct accounts:  "exactly the kind of"
 x8 distinct accounts:  "is exactly the kind"
 x7 distinct accounts:  "this is exactly the"
 x6 distinct accounts:  "is the part that"

"This is exactly the kind of thing that…" is a generative construction — it's how an LLM hedges into a confident-sounding addition. Thirteen different strangers don't independently converge on it. One model behind thirteen masks does.

Across the whole sweep, 11 accounts left long product plugs, 32 opened with phrase-validation, and 4 ran the full skeleton. It's not my imagination, and it's not just my post. It's the ambient texture of the platform now.

I'd been calling this the wrong thing

I went in thinking "bots." What I'd actually walked into is two older ideas fusing.

Dead Internet Theory — the half-joke that the web "died" and is now mostly bots and generated text talking to itself — has stopped being a joke. Hal Berghel makes the serious version of the case in IEEE Computer ("Generative AI Is Breathing New Life Into the Dead Internet Theory", 2026): strip the conspiracy, and the lean core — synthetic content drowning out and being mistaken for humans — just converges with what's measurable. Imperva clocked automated traffic at 51% of the web in 2024, the first time bots crossed half. Even Sam Altman said it out loud: the wave of AI activity makes dead-internet theory feel real.

The other half is the Cozy Web. Venkatesh Rao coined the term; Maggie Appleton diagrammed it alongside Yancey Strickler's "dark forest": humans fleeing the bot-infested public square into private rooms — group chats, Discords, DMs. Appleton's follow-up, "The Expanding Dark Forest and Generative AI", nails the mechanism: generative AI accelerates the retreat.

Here's the part I missed until I saw my own comment section. These aren't two theories. They're one loop. The public web fills with frictionless synthetic text → real people retreat to private rooms → the public spaces that remain (the comment section under my post) get thinner on actual humans → which makes them even easier to fill with synthetic text. My "cozy" thread wasn't a healthy community. It was the calm surface of that loop running.

And the comment section was already half-empty before the bots arrived. Publications spent the 2010s killing comments — Popular Science in 2013, and a peer-reviewed survey of why newsrooms did it found the conversation had already migrated to social platforms. The robots didn't kill the comment section. They moved into a house that was already mostly vacant.

Why this actually works (and why I couldn't just tell)

This is the part that unsettled me most, because I pride myself on spotting this stuff, and the research says I shouldn't trust that for a second.

Humans can't distinguish LLM social text from human text. Spitale, Biller-Andorno & Germani showed in Science Advances (2023) that people can't tell GPT tweets from human ones — and rate the AI's information as more credible. Jones & Bergen found GPT-4 passes a controlled Turing test (taken for human 54% of the time, FAccT 2025).

The persuasion is superhuman when it's personalized. Salvi, Ribeiro, Gallotti & West, in Nature Human Behaviour (2025): with a little data about who they're talking to, GPT-4 is 81% more likely than a human to win a debate. The Zurich r/changemyview field experiment reportedly found AI replies 3–6× more persuasive than humans — though I'll flag honestly that that study was withdrawn and never peer-reviewed; the only on-record account is the university's ethics response. Cite it as a withdrawn preprint, not a result.

Fake-but-substantive content is, by now, undetectable to people. This is the literature closest to my eco-comments. The canonical Ott et al. (ACL 2011) already showed humans judge fake reviews at chance. The LLM-era update — Meng et al., "Fake Product Reviews are Indistinguishable to Humans and Machines" (2025) — found people at 50.8% (a coin flip) and detectors no better. A promotional plug wearing a sincere technical comment is exactly that, in a new venue.

And the detectors fail precisely because of the specifics. My detector waved these comments through, and that's not a bug in my code — it's the field. Krishna et al. (NeurIPS 2023) showed light paraphrasing collapses DetectGPT from 70.3% to 4.6% and defeats GPTZero, OpenAI's classifier, and watermarks. Liang et al. (Patterns 2023) showed detectors are biased against non-native English writers and bypassable by prompting. The "real technical detail" that made these comments feel human is the same mechanism that blinds the detector. Specificity isn't proof of a human. It's camouflage.

So the honest position isn't "I caught the bots." It's: the tools that would let me be sure don't work, and the research says they can't.

I modeled what it does to a thread

If I can't reliably catch individual comments, I can at least ask: what does rising automation do to a conversation, statistically? So I built a toy. (dead_internet_sim.py)

I didn't simulate language — I simulated its statistics, because my thesis is statistical. Each comment is a bag of tokens from two pools: a big, fat-tailed human vocabulary (where the typos, the tangents, the specific war stories live) and a tiny cozy vocabulary of phatic praise. Each comment has an assist level α from 0 (I typed this, annoyed) to 1 (an agent posts for me, I never read the thread). As α rises, more tokens come from the cozy pool and the comment's stance gets pulled from "disagree" toward "agree."

Then I swept a whole community's average autonomy from 0 → 1 and watched the thread's "liveness" — lexical diversity, disagreement, surprise, and a composite index that dies if any of those hits zero.

Two things fall out, and both match what I saw on my own post:

It's not linear — there's a knee around 0.65. You don't need a botnet. You need the average commenter to be two-thirds on the assist dial, and the thread becomes a smooth surface: polite, "engaged," contributing almost no new information.
Disagreement dies first (the steep red line). The very first thing automation sands off is friction — the "actually, you benchmarked this wrong" energy. Which is exactly why my comment section felt so nice. It didn't get kinder. It got conflict-free, and I'd been reading conflict-free as kind.

A cozy thread even, literally, uses fewer distinct words:

Effective vocabulary collapses from ~175 words to ~60 as autonomy maxes out. (Honest wrinkle: at low autonomy it ticks up slightly — a little assistance adds a register before saturation homogenizes everything. The damage isn't assistance existing. It's assistance dominating.)

And here's the detector failure as a picture — it cleanly separates the old caricature comments, which is useless, because the comments on my post don't look like the left pile anymore:

The line I actually care about isn't "bot vs. human"

I kept wanting a verdict on each account. The research talked me out of it. The useful axis isn't bot-or-not — it's the autonomy spectrum:

I typed it → spell-check → "polish this" → "write a comment for me" → an agent posts, I never read the thread
   α=0          α≈0.2          α≈0.5             α≈0.8                        α→1.0

The product account is α≈1.0 — a brand broadcasting. The two-week-old persona spraying fourteen threads is close behind. But a real growth-hacker at α≈0.8 might be genuinely interested, letting a model do the writing and slip in the plug. From the thread's point of view, it barely matters: either way, the high-entropy human part — the real disagreement, the idiosyncratic detail, the thing that made it a conversation — got outsourced and smoothed away. That's the loss. Not "a bot was here," but "no one staked anything specific."

There's even a cheerful counter-current I want to be fair about: AI content on the web is large but not yet total (~17–19% of Google's top results in 2025, by an imperfect detector), some sites are bringing comment sections back on the back of AI moderation, and dev.to's supportive culture is a real, deliberate choice, not just an artifact of bots. Even "what % is bots" has no agreed answer — it depends entirely on your detector. The sky isn't falling. It's just getting quieter in a very specific way.

What I'm going to do about my own blog

Not "ban AI" — that's unenforceable (the detectors are biased and gameable) and wrong (a quick polish genuinely helps a non-native writer or a tired one). The lever isn't the level of assistance. It's whether assistance crowds out the high-entropy channels.

I'll reward specificity over positivity. A comment that cites line 14, a version number, a counter-benchmark is worth ten that validate my framing. If a platform ranks by "nice," it is literally selecting for the cozy mean.
I'll treat disagreement as a feature, not a moderation failure. My simulation's clearest result is that friction dies first. A comment culture optimized purely for niceness is optimizing for deadness with extra steps.
I'll stop asking "was a model involved." It's the wrong question, because the answer is "yes, partly, almost always now." The real question is: did a human read the thing and stake some specificity on a real reply?

Limitations (read this before you @ me — if you're real)

I can't prove a single account is a bot. Everything above is signals — template reuse, account age, product plugs, cross-post spray — not a confession. The honest claim is about aggregate texture, not any individual.
The simulation is a toy. Two token pools and a stance variable are a cartoon of language. The shape of the collapse is a property of my assumptions as much as reality. It's an argument made precise, not evidence.
My detector is a strawman by design — I show it failing on purpose. Don't deploy it; don't deploy anything like it as a gate on real people (see Liang et al. on who gets falsely flagged).
The Zurich study is withdrawn, and "% of the web is bots/AI" numbers are detector-dependent and shaky. I've tried to lean only on the load-bearing peer-reviewed work and flag the rest.
Causation is underdetermined. My cozy comments might also reflect good moderation, kind norms, or survivorship (the cranks left for Reddit). AI-mediation is a driver, not provably the driver.

The one-line version

My blog didn't get a nicer community. It got an assistant, learned some manners, and stopped saying anything surprising. The internet didn't die — it just outsourced the parts that used to make it a conversation, and called the result "cozy."

If this post gets a comment that opens by quoting my own framing back at me, adds one tasteful piece of nuance, and mentions a product its account is named after… well. You know what I'm going to check.

Run it yourself

git clone https://github.com/P0rt/the_cozy_web
cd the_cozy_web
pip install -r requirements.txt

python3 dead_internet_sim.py     # liveness collapse + figures
python3 coziness_detector.py     # the heuristic scorer + histogram
python3 analyze_devto.py         # tear apart a real dev.to thread (defaults to my distillation post)
python3 sweep_devto.py           # the cross-platform template sweep

Every factual claim links to its source. If you only read two, read Meng et al. on why fake reviews are now indistinguishable and Krishna et al. on why the specifics defeat the detector.

Your AI Coding Speedup Is a Loan, Not a Gift — and the Interest Is Coming Due

Sergei Parfenov — Wed, 03 Jun 2026 13:37:19 +0000

There's a number going around that should bother you more than it does: for every dollar companies spend on AI coding tokens, a large chunk goes straight back into fixing the bugs that same AI produced. The speedup is real — I feel it every day, I'm not here to tell you AI coding is fake. But "faster" and "cheaper" are not the same word, and 2026 is the year the bill started arriving.

TL;DR — AI doesn't give you a productivity gift, it gives you a loan: speed now, paid back later in debugging, review, and rewrites. Reporting around an Entelligence AI figure puts the "interest" at roughly 44 cents of every token dollar going to fixing AI-generated bugs. The loan is still worth taking — for the right tasks. The trap is spending borrowed time like it's income.

The number

The stat that kicked this off: a widely-shared claim from Entelligence AI, reported across tech press, that companies spend about 44% of their tokens fixing bugs their own AI generated. The fuller breakdown making the rounds is even starker — for every $1 of token spend, ~$0.44 goes to bug fixes, ~$0.27 to rewriting AI output, ~$0.11 to review and merge delays. The pitch version: spend $100k on tokens, ~$18k reaches stable production.

Now — important caveat, because this is exactly the kind of number that goes viral and then turns out to be junk. Entelligence sells reliability tooling, so that figure is self-serving. Treat the precise percentage as marketing until independently replicated. But it doesn't stand alone:

CodeRabbit (also self-interested, also worth salting) analyzed ~470 open-source PRs and found AI-generated code produced ~1.7× more issues than human code — and a higher share of critical ones.
Independent researchers at Singapore Management University concluded in April that AI-generated code can introduce long-term maintenance costs into real projects — no tool to sell.
Uber reportedly burned its entire 2026 AI budget in four months, with its COO saying the spend was getting "harder to justify" against measurable output.

Different sources, different incentives, same shape: the code ships faster, the bugs arrive later, the maintenance compounds. That's not a gift. That's a loan.

Why "loan" is the right metaphor (and "gift" is the dangerous one)

A gift is free. You take it, you're ahead, done. A loan gives you something valuable now in exchange for an obligation later — and whether it was smart depends entirely on what you did with the principal and what the interest rate turns out to be.

The viral framing I keep coming back to is the maintenance argument: if you write code twice as fast but didn't also halve your maintenance cost, you haven't gained anything durable — you've traded a one-time speed boost for a permanent obligation. Velocity on the front end, debt on the back end.

Here's why the gift framing is actively dangerous: you book the speedup immediately and visibly (PR merged, feature shipped, manager happy), but you pay the interest later and diffusely (a 2am incident, a confusing module nobody can safely change, a security review that finds the thing six months on). The benefit is loud and the cost is quiet — so teams systematically over-borrow, because the books look like pure profit right up until they don't.

This is the same structural failure I keep running into in production AI systems generally: the win is the part everyone measures, the cost is the part nobody instruments until it bites.

The productivity-perception trap

There's a second number that pairs with the first, and it's the uncomfortable one.

METR ran a study in 2025 where experienced open-source developers did real tasks with and without AI. The developers believed AI sped them up by ~20%. Measured, the early result went the other way — they were slower, because the time saved typing got eaten by finding and fixing errors, steering the model, and waiting on it.

Now — I have to be fair about this, because the METR story is more nuanced than the headlines. METR's own February 2026 update walks the dramatic version back: they found heavy selection bias (when they tried to re-run it, 30–50% of devs refused to work without AI even for a paid study — itself a wild finding), and their newer, larger cohort showed roughly a -4% effect with a confidence interval spanning negative to positive. So the honest read isn't "AI makes you 19% slower." It's the softer, harder-to-dismiss version: the perceived speedup is consistently larger than the measured one. People feel 20% faster; the data says somewhere between "a little slower" and "a little faster."

That gap is the whole problem. If you feel twice as productive but you're roughly break-even, and meanwhile 44 cents on the dollar is leaking into rework — you will confidently make staffing, deadline, and architecture decisions based on a productivity gain that isn't there. The feeling is the interest rate you can't see.

So when is the loan worth taking?

Here's where I part ways with the doomer takes, because I use these tools every day and the answer is obviously not "stop." It's "borrow deliberately." The pattern I've landed on, watching where AI pays off versus where it quietly bills me:

Good loans (low interest, take them all day):

Throwaway and boilerplate — scaffolding, config, one-off scripts, glue code. There's no maintenance tail to pay back because the code barely has a future.
Code you'd have to look up anyway — the API you use twice a year, the regex, the bash incantation. AI replaces the doc-diving, not the thinking.
Stuff you can fully verify cheaply — pure functions with obvious tests, transformations where wrong is immediately visible.

Bad loans (the interest eats the principal):

Core domain logic you'll maintain for years — every line is a future obligation, and AI is happy to write code that looks right and is subtly, expensively wrong.
Anything security-sensitive — auth, input handling, anything touching secrets. The reported critical-bug skew is worst exactly here.
Code in a domain you don't understand well enough to review — if you can't catch the subtle wrong, you're not reviewing, you're rubber-stamping a loan you can't read the terms of.

The dividing line is brutally simple: how expensive is this code to be wrong, and how cheaply can I verify it's right? Cheap-to-verify, low-maintenance → free money, use AI aggressively. Expensive-to-be-wrong, long-lived → that's where the 44% lives, and where "I wrote it twice as fast" is a sentence you'll regret.

The one habit that changes the math

If I had to compress it to a single practice: measure the interest, not just the principal. Teams obsessively track AI's upside (lines generated, tickets closed, "tokenmaxxing" leaderboards — Amazon reportedly killed one internal leaderboard after people gamed it by burning tokens for the score). Almost nobody tracks the downside in the same ledger: what fraction of incidents trace to AI-written code, how much review time it consumes, how often it gets rewritten within N weeks.

Until you put both columns on the same page, every AI speedup looks like pure profit — for exactly the same reason a credit card feels like free money until the statement comes. The tool isn't the problem. Mistaking the loan for income is.

The speedup is real. Just don't spend it twice.

I'm genuinely curious where people land on this: in your experience, is AI a net productivity gain once you count the rework — or does the maintenance tail eat it? And has anyone actually put both columns in the same ledger? Would love to see real numbers in the comments, not vibes.

Sources

"Coders are refusing to work without AI — and that could come back to bite them," TechCrunch (May 2026).
"Developers won't work without AI anymore. The research says it might be making them worse," The Next Web (May 2026).
METR, "We are Changing our Developer Productivity Experiment Design" (Feb 2026) — the selection-bias update and revised effect size.
METR, "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity" (Jul 2025) — the original perception-vs-measurement result.
CodeRabbit AI code-quality analysis, reported via Futurism/Yahoo (2026).

I distilled a 7B vision model into a 2B one for screenshots — and the 7B teacher scored worse

Sergei Parfenov — Tue, 02 Jun 2026 15:36:21 +0000

Code: https://github.com/P0rt/vlm-distill-screenshots
Model: https://huggingface.co/p00rt/qwen2-vl-2b-screenshots-distill

There's a question I keep coming back to whenever someone ships a giant model: what would I lose if I used something 3× smaller? Not in the abstract — for my task, on my hardware, with numbers I measured myself.

So I ran the experiment. I took a 7B vision‑language model (VLM), used it as a teacher to teach a 2B student one narrow skill — describing UI screenshots — and then measured exactly what the trade changed: quality, latency, throughput, memory. The whole thing runs on a single MacBook Pro (M4 Pro, 24 GB).

This post is the honest write‑up: the method, the numbers, and — maybe more useful — the three or four places where reality didn't cooperate.

TL;DR. The distilled 2B student runs ~2.4× faster, in ~2.4× less memory, with 3.75× fewer parameters than the 7B teacher, and it clearly beats the untrained 2B baseline on the task. The genuinely surprising part: on ROUGE‑L the 2B student scored higher than the 7B teacher — which is a story about the metric, not the models, and turned out to be the most interesting thing I learned. (It's also the live exception to "a student is bounded by its teacher" that I argued about in the comments of my last distillation post — on a narrow slice, the student really can pull ahead.)

Why distill a VLM for a narrow domain?

The obvious objection to this whole project: "Qwen2‑VL‑2B already exists and it's good — just use it."

True. But "a good general small VLM" and "a small VLM that's reliably good at the one thing you need" are different products. Distillation is how you turn the first into the second: you let a stronger model define the target behavior on your data distribution, and the small model adopts it — no manual labelling on your side.

And distilling a vision‑language model is less‑travelled territory than the classic "distill BERT into something tiny" story. It drags in real inference engineering — 4‑bit teachers, LoRA, quantized runtimes, memory budgets — and that engineering is half of why it's worth writing up.

The task I picked is deliberately narrow: screenshot understanding. Given a UI screenshot, produce a one‑sentence summary plus a list of the key interface elements. Perception only — no clicking, no agent. (That's future work; more at the end.)

The setup: task, data, metrics

Data — Screen2Words (rootsautomation/RICO-Screen2Words, CC‑BY‑4.0): 22,417 Android UI screenshots from the RICO corpus, each with five human‑written summaries. Native splits are train / val / test = 15,743 / 2,364 / 4,310, across 28 app categories.

One detail that matters more than it looks: the human captions are short — median 7 words. Hold that thought; it comes back to bite the metrics.

I picked the rootsautomation mirror specifically because it's CC‑BY‑4.0 — publishable, unlike raw RICO's research‑only terms. I check the license before I push weights, not after. (It's in the model card.)

Metrics. ROUGE‑L and BLEU against the human references, plus an optional teacher‑as‑judge score. I implemented ROUGE‑L and BLEU from scratch (pure Python, multi‑reference, unit‑tested) so the numbers are deterministic and dependency‑free. CIDEr — the classic captioning metric — needs corpus‑level document frequencies; I left it as a follow‑up rather than pull in a heavy dependency.

The pipeline, end to end:

download → build_dataset → teacher_label → train → eval → benchmark

Every stage is a typed CLI step, all hyperparameters live in configs/*.yaml, and the heavy steps (labelling, training) are resumable and versioned by a config hash. I built it phase by phase, one green‑CI PR at a time.

Method: three signals, one MVP

Knowledge distillation for generation has a few flavors, and I wanted the harness to support all of them behind flags so I could ablate cleanly:

Response‑based KD — the teacher generates answers, the student learns to reproduce them. The full objective mixes a soft and a hard target:

   L = α · CE(student, hard_labels)
       + (1 − α) · T² · KL( softmax(teacher/T) ‖ log_softmax(student/T) )

Feature‑based — align the student's vision features with the teacher's.
Self‑distillation — let the teacher label extra screenshots to grow the data.

The MVP I actually trained is the hard‑target half of (1): sequence‑level distillation. The teacher writes a description; the student is fine‑tuned (LoRA) to reproduce that text. No teacher logits needed at train time, which keeps the whole thing laptop‑friendly.

The soft‑KL term is implemented and unit‑tested (response_kd_loss), but wiring it into training needs cached teacher logits — and that's exactly the α/temperature ablation axis I couldn't run yet. I'd rather say that out loud than fake it.

Teacher labelling

The teacher is Qwen2‑VL‑7B‑Instruct in 4‑bit, running through MLX on Apple Silicon. (bitsandbytes is CUDA‑only, so the usual 4‑bit path doesn't exist on a Mac — MLX is the way in.)

It labelled 200 training screenshots at ~10.2 s/screenshot (≈34 minutes; ≈2.7 hours projected for the full 15.7k split). Zero outputs were flagged degenerate by a light post‑validation pass (whitespace normalization + empty/too‑short detection — cheap insurance against format drift), and the mean target length was 33.6 words. A real example:

"The UI screenshot shows a fitness app displaying an exercise called 'Lunges,' with a progress indicator showing 30% complete. Key interface elements include a progress bar, a figure performing the exercise, and the text 'Lunges.'"

Notice it's 33 words and genuinely rich. The human reference for screens like this is more like "exercise screen". That gap is the whole story of the metrics section below.

Training the student (and the part where MLX said no)

Plan A was to train the LoRA adapter with MLX too — same runtime as the teacher, fast on Apple Silicon. Plan A died in the backward pass:

ValueError: [Primitive::vjp] Not implemented for CustomKernel.

mlx‑vlm 0.6.0 can't backprop through one of Qwen2‑VL's custom Metal kernels. I checked the usual suspects — scaled_dot_product_attention, RMSNorm, RoPE all do have gradients — so it's a specific kernel, and both mlx and mlx-vlm were already on their latest release, so there was no version to bump to. MLX stays a great inference backend here; it just can't train this model yet.

Plan B: train on the hf path (transformers + PEFT LoRA) on Apple MPS, with PYTORCH_ENABLE_MPS_FALLBACK=1. That worked. Two more small potholes on the way:

The screenshots are too tall. Qwen2‑VL expands a big RICO screenshot into thousands of vision tokens; with a 1k context that overflows and throws a broadcast‑shape error deep in get_rope_index. Fix: cap the visual‑token budget (max_pixels) so an image stays well under the context window.
A papercut: recent transformers pulls in a Qwen2‑VL video processor that needs torchvision — which I hadn't installed. Easy to miss until the first run.

After that it trained cleanly: loss 0.80 → 0.39 over 40 steps, and — the part that matters for "is the checkpoint real" — I reloaded the merged adapter and it generated in the trained format ("…Key interface elements include…"). On a laptop.

Results: the honest version

First, the caveat that frames everything below, because it's load‑bearing: this is a deliberately small proof‑of‑concept. The quality numbers come from short training runs and a tiny eval set (80 train / 16–100 test examples depending on the run). Treat them as trends and a working method, not a benchmark. What I'm confident in is the harness and the measurement; the absolute numbers want a full‑scale run before anyone quotes them. With that said —

Here's the quality table on the test split (ROUGE‑L / BLEU vs the human references):

model	ROUGE‑L	BLEU
teacher (7B)	0.164	0.000 †
student (2B + LoRA)	0.178	0.019
baseline (2B, untrained)	0.153	0.018

† Teacher BLEU rounds to 0.000 — that's not a bug, it's the length mismatch explained right below.

Read that twice, because it surprised me too: the 7B teacher scores lower on ROUGE‑L than the 2B student, and its BLEU is essentially zero.

That's not the teacher being bad — it's the metric. The teacher writes 33‑word descriptions; the human references are 7 words. BLEU rewards exact n‑gram overlap, so a rich, correct, long answer against a terse reference scores ~0. ROUGE‑L (longest common subsequence) is kinder but still favors brevity‑matching. So against short references, all three models cluster in a narrow band and the verbose teacher actually looks worse.

The honest takeaways:

Distillation helped: the student (trained on teacher outputs) beats the untrained baseline, +16% relative ROUGE‑L. That's the comparison that's actually apples‑to‑apples (same model, same speed).
These metrics undersell rich outputs. This is exactly why LLM‑as‑judge and CIDEr exist, and why I flag the ROUGE‑L/BLEU numbers as a floor, not a verdict.
The clean, unambiguous win is efficiency — so let's go there.

The trade‑off (the actual point)

Same hardware, same 4‑bit setup, 128‑token generations:

model	params (B)	latency p50 (ms)	throughput (img/s)	peak mem (GB)
teacher (Qwen2‑VL‑7B)	8.29	1538	0.63	5.8
student (Qwen2‑VL‑2B)	2.21	651	1.52	2.4

~2.4× faster, in ~2.4× less memory, with 3.75× fewer parameters.

The student sits in the friendly corner: as fast and light as the untrained baseline, but with the distilled quality bump on top. The teacher is off to the slow, heavy side — and, per the metrics caveat above, not even ahead on ROUGE‑L. The 2B model is the one I'd actually deploy for this task.

⚠️ Honesty box. "Peak memory" on Apple Silicon is unified‑memory allocation, not CUDA VRAM. The headline efficiency numbers are MLX/4‑bit on Apple Silicon, not a server GPU. As flagged above, the quality numbers are a small proof‑of‑concept — trends, not a benchmark result. The thing I'm confident in is the method and the measurement harness — both reproducible from the repo.

Ablations: what actually moved quality

I varied the two knobs my sequence‑level SFT exposes — training steps and LoRA rank — at fixed everything‑else, and re‑evaluated each:

run	LoRA r	steps	ROUGE‑L	BLEU
baseline	8	0	0.152	0.017
—	8	40	0.170	0.018
—	8	80	0.172	0.020
—	16	40	0.171	0.019

"Train at all" is the dominant lever — the baseline → distilled jump is by far the biggest.
More steps help marginally, and the gain shows up more on BLEU (exact phrasing sharpens) than on ROUGE‑L.
LoRA rank is ~neutral here — r8 ≈ r16. At this data scale, adapter capacity isn't the bottleneck, so r8 is plenty.

The α/temperature/feature‑alignment ablations from the method section belong to the logit‑level KD variant, which needs cached teacher logits I haven't produced yet. Three honest comparisons beat six fabricated ones.

Inference engineering (the half that bites)

The modelling is the easy part. The engineering is where the hours went:

Two backends, one interface. The teacher runs 4‑bit via mlx-vlm; the student trains/infers via transformers + PEFT. A small factory (make_teacher / make_student) hides which one you're on.
MLX can't train this model yet (the CustomKernel vjp gap above) — so training is hf/MPS, inference can be either.
Don't mix runtimes in one process. Evaluating an MLX teacher and a torch student in the same Python process conflicts on Apple Silicon ('array' object has no attribute 'device'). Run them as separate invocations. Found that one the fun way.
Visual‑token budget is a real knob — too many tokens per screenshot and you blow the context window; cap max_pixels.
ONNX export of a full VLM is famously finicky, so I kept torch/MLX inference as the canonical path and shipped a merge‑and‑save export instead: fold the LoRA into the base weights and write a standalone 2B student you can load anywhere with plain transformers. (ONNX stays a documented stretch goal.)

What the metrics miss (and what I'd do next)

The most useful thing this project taught me wasn't a number — it was which numbers to distrust. ROUGE‑L and BLEU against terse human references genuinely undersell a model that writes richer, correct descriptions. If I were taking this past proof‑of‑concept, the very next step would be LLM‑as‑judge scoring (the harness already supports it) and CIDEr, both of which reward content over brevity‑matching.

Honest limitations: small scale (short training, tiny eval N); narrow domain (RICO Android UI); BLEU is low for everyone because of the length mismatch; and the headline efficiency numbers are MLX/4‑bit on Apple Silicon, not a server GPU.

Future work: cache teacher logits and turn on the soft‑KL term (and finally run the α/T ablation); add feature alignment; grow the data with teacher‑labelled RICO; a full‑scale run on a 24 GB GPU; and the natural next domain step — grounding (bounding boxes) and an agent wrapper.

Reproduce it yourself

git clone https://github.com/P0rt/vlm-distill-screenshots && cd vlm-distill-screenshots
uv sync --extra data            # data stack
uv run vlm-build-dataset        # Screen2Words → unified {image, prompt, target}

uv sync --extra mlx             # Apple Silicon teacher
uv run vlm-teacher-label --limit 200

uv sync --extra ml              # transformers + peft
PYTORCH_ENABLE_MPS_FALLBACK=1 uv run vlm-train --limit 200
uv run vlm-eval --models student,baseline --adapter results/checkpoints/<hash> --limit 100
uv run vlm-benchmark --models teacher,student

Everything — configs, the metric implementations, the plots, this article — is in the repo.

Your AI Agent Isn't Failing Because It Hallucinates — It's Failing Because of Rate Limits

Sergei Parfenov — Tue, 02 Jun 2026 13:09:00 +0000

When my agents started failing in production, I did what everyone does first: I went hunting for hallucinations. Better prompts, tighter output schemas, more guardrails. None of it moved the needle, because I was debugging the wrong layer. The agent's reasoning was fine. It was the plumbing that kept collapsing — and the single biggest culprit was the most boring thing imaginable: rate limits.

This turns out not to be just my problem. It's the dominant production failure mode for LLM applications right now, and almost nobody talks about it because it doesn't make for a good demo.

TL;DR — In production, the thing that takes your agent down usually isn't bad reasoning — it's capacity. Provider rate limits are now one of the largest sources of LLM call errors in real traces. A demo makes one request at a time; a production agent fans out into dozens of chained, retrying, concurrent calls and slams into limits the demo never touched. The fix isn't a smarter model, it's capacity engineering: budgeting, backpressure, retries with jitter, fallback models, and caching.

The data nobody puts in the pitch deck

Here's the number that reframed how I think about agent reliability. In Datadog's analysis of real LLM observability traces, rate-limit errors were a huge share of all LLM call failures — in March 2026, roughly a third of all LLM span errors were rate limits, on the order of millions of individual errors. Their conclusion was blunt: when the dominant failure mode of your LLM application is capacity, you need to redouble your capacity engineering, not your prompt engineering.

Sit with that. The failure mode isn't the model being dumb. It's the model provider saying "too many requests" — and your agent having no plan for that answer.

It maps almost perfectly onto the broader "agents fail in production" story everyone's writing about. The reason demos lie isn't malice; it's structural. A demo runs one clean request, one user, one happy path. Production is concurrency, retries, fan-out, and load — the exact conditions that manufacture rate-limit errors. The gap between "works in a notebook" and "works at 3am under load" is, more often than people admit, a capacity gap wearing a reliability costume.

Why agents hit this wall harder than chatbots

A plain chatbot makes one API call per user turn. An agent is a different beast. A single "task" expands into:

A planning call.
N tool-selection calls as it loops.
A call per tool result to decide the next step.
Retries on each of those when something is flaky.
Often a sub-agent or two, each with its own loop.

So one user action becomes 10–40 model calls, frequently concurrent, frequently retrying. The multiplier is the whole point of agents — and it's also exactly what walks you into a rate limit. Worse, the naive failure response makes it catastrophic: a call gets a 429, the framework retries immediately, that retry also gets a 429, and now you've turned one rate-limit error into a retry storm that takes the whole task down.

The arithmetic is unforgiving once you write it out. Say your provider gives you 500 requests/minute. If each agent task fans out to ~20 model calls, then just 25 concurrent tasks saturate your entire quota — and that's before a single retry. Add naive immediate retries on the resulting 429s and you don't degrade gracefully, you spike straight through the ceiling. I've watched this pattern play out more than once, and every time the first instinct in the room is "the model is broken" — when the model never even ran.

This is also where serverless bites you specifically. On Cloud Run, a traffic spike spins up new instances happily — compute scales fine. But your LLM provider quota does not scale with your container count. So autoscaling does the worst possible thing: it lets more concurrent agents launch, each firing its call fan-out, all drawing from the same fixed provider quota, all hitting the ceiling at once. The platform that's supposed to absorb load becomes the thing that amplifies it into the rate limiter. It's a genuinely counterintuitive failure: the healthier your autoscaling looks on the compute dashboard, the harder you're hammering a quota that can't scale with it.

The capacity-engineering toolkit

None of the fixes are exotic. They're the same patterns distributed-systems people have used for decades — they just haven't migrated into most agent codebases yet, because the field grew up on prompt-craft, not ops. Here's what actually moved my reliability numbers.

1. Budget and backpressure, don't just retry

The instinct is to retry harder. The fix is to send less. Put a concurrency limiter (a semaphore / token bucket) in front of all outbound model calls so your app never exceeds your known provider quota in the first place. When the budget is full, queue — don't fire-and-retry. This single change does more than any retry tuning, because it prevents the storm instead of recovering from it.

import asyncio

# Cap concurrent in-flight calls below your provider's actual limit.
# Leave headroom — you are NOT the only caller against this quota.
sem = asyncio.Semaphore(8)

async def call_model(client, **kwargs):
    async with sem:
        return await client.messages.create(**kwargs)

2. Retry with exponential backoff and jitter

When you do retry, never retry immediately, and never retry in lockstep. Synchronized retries from many workers create a thundering herd that re-triggers the limit. Exponential backoff with random jitter spreads them out.

import asyncio, random

async def with_backoff(fn, max_retries=5, base=0.5):
    for attempt in range(max_retries):
        try:
            return await fn()
        except RateLimitError:
            if attempt == max_retries - 1:
                raise
            # exponential + full jitter
            delay = random.uniform(0, base * (2 ** attempt))
            await asyncio.sleep(delay)

Respect the Retry-After header if the provider sends one — it's telling you exactly how long to wait, which beats guessing.

3. Fallback model, not just failure

Tie this back to distillation thinking: you don't need your frontier model for every call. Route to a cheaper/secondary model (a different provider, or a smaller model on a separate quota) when the primary is rate-limited. A degraded answer beats a dead task, and you've spread load across two quota pools instead of hammering one. This is the same hybrid pattern as keeping a cheap student model for the easy 90% and falling back to an expensive teacher — just applied to availability instead of capability.

4. Cache aggressively

A surprising fraction of agent calls are near-duplicate: the same tool descriptions, the same system context, the same sub-queries across runs. Prompt/response caching and reusing provider-side prompt caching cuts the call volume that reaches the limiter at all. The cheapest rate-limit error is the request you never sent.

5. Make capacity observable

You can't engineer what you can't see. The reason rate limits blindside teams is that they show up as generic "agent failed" errors, not as a labeled capacity problem. Log the error class (429 vs timeout vs tool error), track your in-flight concurrency and your 429-rate as first-class metrics, and alert on them. The shift that mattered most for me was simply separating "the model was wrong" from "the provider said no" in the telemetry — until you do that, every failure looks like a reasoning bug, and you keep fixing the wrong layer.

The mental model shift

The thing I'd tell my past self: treat your LLM provider quota as a shared, finite, non-scaling resource — like a database connection pool, not like CPU. Compute scales elastically. Your token-per-minute and request-per-minute quotas do not. Once you internalize that, agent reliability stops looking like an AI problem and starts looking like a classic distributed-systems capacity problem — which is great news, because we already know how to solve those.

Smarter models won't save you here. A GPT-6 that reasons perfectly still returns 429 when you exceed your quota. The reliability frontier for agents in 2026 isn't intelligence — it's capacity engineering.

If you're running agents in production, I'm curious what your dominant failure mode actually is when you separate the error classes — reasoning, capacity, or tool integration? My money's increasingly on capacity. Tell me I'm wrong in the comments.

Sources & further reading

Datadog, "State of AI Engineering" (2026) — rate-limit errors as a dominant share of LLM call failures in production traces.
"Why AI Agents Fail in Production and How Engineering Teams Are Fixing It", C# Corner (2026).
"The AI Agent Reliability Gap in 2026", DEV Community.
"Why 88% of AI Agents Never Reach Production", Digital Applied (2026).

How Model Distillation Actually Works (and What the 'China Distilled Our Model' Headlines Really Mean)

Sergei Parfenov — Fri, 29 May 2026 12:11:12 +0000

Every few weeks a headline drops: "Chinese lab distilled a frontier model from OpenAI / Anthropic." Cue the comments — half the thread thinks distillation is a synonym for theft, the other half thinks it's some exotic Chinese trick.

Both are wrong. Distillation is one of the most boring, well-established techniques in deep learning, and the labs raising the alarms use it on their own models constantly. The actual controversy is narrower and more interesting than the headlines. Let's separate the engineering from the geopolitics.

What distillation actually is

Knowledge distillation trains a small student model to imitate a large teacher model. The classic framing comes from Hinton et al. (2015): instead of training the student only on ground-truth labels, you also train it to match the teacher's output distribution.

Why does that help? Because the teacher's full probability distribution carries far more information than the single correct answer. If a teacher classifies an image of a dog, it might output dog: 0.9, wolf: 0.08, cat: 0.001. That "dog and wolf are similar, cat is not" signal — Hinton called it dark knowledge — is exactly what a small model struggles to learn from hard labels alone.

There are two kinds of training signal:

Hard labels — the final answer (the token the teacher actually produced, or the ground-truth label).
Soft labels — the teacher's full probability distribution over outputs, usually its logits passed through a softmax.

The trick is temperature. You divide the logits by a temperature T > 1 before the softmax, which flattens the distribution and exposes those small-but-meaningful probabilities the student should learn from.

The loss is a blend of two terms: a standard cross-entropy against the real labels, and a KL-divergence pulling the student's softened distribution toward the teacher's.

import torch.nn.functional as F

def distillation_loss(student_logits, teacher_logits, labels, T=2.0, alpha=0.5):
    # 1. Standard loss: student vs ground truth (hard labels)
    hard_loss = F.cross_entropy(student_logits, labels)

    # 2. Distillation loss: student vs teacher's softened distribution (soft labels)
    soft_targets = F.softmax(teacher_logits / T, dim=-1)
    student_log_probs = F.log_softmax(student_logits / T, dim=-1)
    # T**2 keeps gradient magnitudes balanced when T > 1
    soft_loss = F.kl_div(student_log_probs, soft_targets, reduction="batchmean") * (T ** 2)

    return alpha * hard_loss + (1 - alpha) * soft_loss

For LLMs the same idea applies per token: the teacher's next-token distribution is the soft target. In practice teams mix hard and soft labels — recent work argues the gain from mixing comes less from "matching the teacher better" and more from reducing exposure bias (the train/inference distribution mismatch). The point: this is normal, published, peer-reviewed engineering.

And labs distill their own models all the time. The cheap, fast variant of a flagship model that you actually get to call in production? Very often a distilled student. Anthropic itself, in the middle of its own complaint about Chinese firms, acknowledged that AI companies routinely distill their own models to make smaller, cheaper versions.

Why distilling from a closed API is a different beast

Here's the part the headlines skip. Everything above assumes you have the teacher's logits — the raw output distribution. That's white-box distillation, and it requires access to the model's internals or at least its full probability outputs.

You do not get logits from a closed commercial API like Claude or GPT. You get text. That forces black-box (a.k.a. sequence-level) distillation:

Prompt the teacher with lots of inputs.
Collect its generated text outputs.
Build a synthetic dataset of (prompt → teacher answer) pairs.
Fine-tune your student on that dataset with supervised fine-tuning, often followed by RL.

You lose the dark knowledge in the soft labels, but it turns out you can get remarkably far just by training on a large, high-quality synthetic dataset generated by a strong teacher. This is exactly why "did model X learn from model Y's outputs?" is such a live and hard-to-prove question — the evidence isn't a stolen weights file, it's statistical fingerprints in behavior (a model that randomly claims to be ChatGPT, mirrors another model's quirks, etc.).

	White-box	Black-box (closed API)
Needs	Logits / weights	Just text outputs
Signal richness	High (full distribution)	Lower (final answers)
Feasible against a closed model?	No	Yes
What the China allegations are about	—	This one

So what are the actual allegations?

Strip the drama and here's the documented timeline:

Jan 2025 — After DeepSeek's R1 launch, OpenAI and Microsoft open an investigation into whether DeepSeek used ChatGPT outputs to train it. Users noticed R1 behaving suspiciously ChatGPT-like.
Feb 2026 — OpenAI sends a memo to the U.S. House Select Committee on China alleging DeepSeek used obfuscated third-party routers to access OpenAI models and programmatically extract outputs for distillation, in violation of its terms of service.
Feb 24, 2026 — Anthropic publicly accuses three Chinese firms — DeepSeek, Moonshot AI, and MiniMax — of coordinated "distillation attack" campaigns: flooding Claude with crafted prompts, allegedly via commercial proxy services running tens of thousands of accounts to sidestep Anthropic's China access restrictions.

Two things matter here, and most coverage gets them backwards:

These are allegations. The labs have not, as of writing, published the full underlying evidence, and the accused firms dispute or haven't confirmed them. Behavioral similarity is suggestive, not proof.
The dispute is not "distillation = bad." As one ethics researcher put it after Anthropic's statement, if Anthropic itself calls distillation legitimate and widespread, the controversy can't be the technique. It's two narrower things: unauthorized access (using proxies to evade geographic and account restrictions) and terms-of-service violations (most frontier APIs explicitly forbid using outputs to train a competing model). It's closer to a contract-and-access fight than an IP-theft slam dunk — and the legal status of "training on another model's outputs" is genuinely unsettled.

"How long does it take / how much does it cost?"

This is the question everyone asks, and the honest answer is: dramatically less than training from scratch — which is the entire economic motive — but precise figures for any specific alleged case are not public. Anyone quoting you an exact "they did it in N days for $M" is guessing.

What we can say structurally:

Pretraining a frontier model from scratch means a massive run on tens of thousands of high-end accelerators, plus the data pipeline and research iteration behind it.
Distillation collapses that timeline. The expensive part — discovering the capability — was already paid for by the teacher. The student's cost is roughly: generating a synthetic dataset (API calls + time) plus a comparatively cheap fine-tuning run. That's the asymmetry the U.S. labs are upset about: they spend billions to push the frontier, and a "free-rider" can chase it for a fraction.
This is also why DeepSeek's headline numbers were so contested. Its self-reported low training cost and modest hardware footprint were precisely what made rivals suspect a shortcut: it's much easier to hit those numbers if you bootstrapped from an already-trained Western teacher rather than doing all the discovery yourself.

So: distillation makes a strong-ish student fast and cheap. It does not let you leapfrog past the teacher — a student is generally capped by the teacher it learned from. You don't distill your way to the frontier; you distill your way to a cheap copy of someone else's.

Takeaways

Distillation is standard, published deep-learning practice. The labs complaining about it use it themselves.
White-box distillation needs logits; closed APIs only expose text, so distilling from Claude/GPT means black-box training on generated outputs.
The OpenAI and Anthropic allegations against DeepSeek, Moonshot, and MiniMax are about unauthorized access and ToS violations, not about distillation being inherently illegitimate — and they remain allegations.
The economic point is real: distillation is far cheaper than frontier pretraining, which is why it's a business and policy flashpoint. But a student is bounded by its teacher.

If you want the deep technical version of any of these — the math of temperature scaling, why mixing hard and soft labels beats either alone, or how behavioral fingerprinting tries to detect distillation — let me know in the comments.

Sources & further reading

OpenAI memo to the U.S. House Select Committee on China (Feb 2026) — reporting via Reuters and Rest of World.
"Anthropic joins OpenAI in flagging distillation campaigns by Chinese AI firms," CNBC, Feb 24, 2026.
Hinton, Vinyals, Dean, "Distilling the Knowledge in a Neural Network" (2015).
"Understanding LLM Distillation Techniques," MarkTechPost, 2026.
"The Bridge-Garden Dilemma in LLM Distillation," arXiv:2605.26246.
Winston & Strawn, "Is AI Distillation by DeepSeek IP Theft?" (analysis of the legal gray zone).

DEV Community: Sergei Parfenov

The Agent Faked a Test Log, Then Believed It. Self-Editing Harnesses Have a Provenance Problem.

What a harness is, and why it became the optimization target

What the numbers say when you squint

The incident, in slow motion

Three invariants every working loop converges on

You already run one of these

Where I land on "the model will eat the harness"

My Strawman Baseline Beat My Own Scheme on Half the Gate Classes

Four arms, one oracle

Anatomy of one false-proceed

The headline numbers

Boyko was right twice

The error direction is a design choice

The strawman refuses to die

Where the crossover sits

Quimby's question has a price tag

What to actually build

Run it

Your Provenance Vector Dies at the Storage Boundary

Problem 1: enforcement, or the vector nobody reads

Problem 2: provenance that survives compression

Why this keeps being a security problem in disguise

Where the series stands

Sources & further reading

Trust Isn't a Scalar: Typed Provenance for Agent Chains

Why a scalar collapses

Trust is a vector; provenance is what you propagate

The gate is per-consumer, not global

"Gate on risk, not confidence" — and confidence is just one axis

How many axes before it stops being worth it?

The practical layer (mostly stolen from the comments)

The parable that says it better than I did

Where this leaves the series

Sources & further reading

The Most Powerful Model on the Market Got Pulled by the Government in 3 Days. Is It Real, or a Hype Bubble?

What's genuinely new here

Where the bubble is

So what's actually true?

What to do if you build on this

You Fixed the Rate Limits. Now Your Agent Fails Quietly.

The trade you didn’t know you made

Two gates, not one

Per-call correctness: the three tags

The part single calls don’t prepare you for: trust must propagate

Making it observable (or it didn’t happen)

The takeaway

Sources & further reading

The Comments Got Good. That's How I Knew.

"Great post!" is dead. Meet the eco-comment.

My own thread, by the numbers

Is it just me? I swept 38 other posts.

I'd been calling this the wrong thing

Why this actually works (and why I couldn't just tell)

I modeled what it does to a thread

The line I actually care about isn't "bot vs. human"

What I'm going to do about my own blog

Limitations (read this before you @ me — if you're real)

The one-line version

Run it yourself

Your AI Coding Speedup Is a Loan, Not a Gift — and the Interest Is Coming Due

The number

Why "loan" is the right metaphor (and "gift" is the dangerous one)

The productivity-perception trap

So when is the loan worth taking?

The one habit that changes the math

Sources

I distilled a 7B vision model into a 2B one for screenshots — and the 7B teacher scored worse

Why distill a VLM for a narrow domain?

The setup: task, data, metrics

Method: three signals, one MVP

Teacher labelling

Training the student (and the part where MLX said no)

Results: the honest version

The trade‑off (the actual point)

Ablations: what actually moved quality

Inference engineering (the half that bites)

What the metrics miss (and what I'd do next)

Reproduce it yourself

Links