DEV Community: Sergei Parfenov

It caught "partner" "pareja" in an IRS rule and missed "return" "devolución": an offline MT checker for translators

Sergei Parfenov — Thu, 15 Oct 2026 12:25:34 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

What I Built

The IRS page on the failure-to-file penalty has a rule for small partnerships: "Each partner is either an individual (excluding nonresident aliens), or the estate of a deceased partner." An open machine-translation model turned it into «Cada pareja es una persona (…) o la herencia de una pareja fallecida». A pareja is a romantic partner; the IRS's own Spanish says socio. A condition for penalty relief has become a rule about couples.

SecondRead flags it at 86% risk. On the same page it misses "the return is late" → «la devolución es tardía», the refund is late. This post covers both.

Post-editors fix machine translation against a deadline. Broken Spanish jumps out; the costly mistake is the fluent sentence that means something else. QA tools catch numbers, tags and glossary terms, not pareja, and many NDAs restrict pasting client text into cloud AI.

So I built SecondRead, a second reader that runs on the translator's Mac. It reads an XLIFF or Trados SDLXLIFF file, scores each segment for the risk of a major meaning error (wrong sense, dropped negation, wrong number, omission or addition), flags the risky ones and sorts the job so you re-read the riskiest 10% first.

Under the hood: Qwen3.5-4B at 4-bit (2.7 GB), a 36 MB LoRA adapter I fine-tuned on Tinker for $1.22, and rule checks for numbers and negation; a 39-segment page takes about 6 s on an M4 Pro, offline. On WMT English→Spanish test it ranks errors as well as Gemma 4 31B (AUROC 0.815 vs 0.807); on ten real public documents Gemma ranks better (0.940 vs 0.910), and my default flag line finds only a fifth of the errors.

Who would pay: agencies, then freelancers, who already buy QA tools such as Xbench and Verifika. WMT text is research-only, so the product is this pipeline trained on an agency's own reviewed jobs: a private 36 MB adapter on each reviewer's laptop, per seat, with no per-segment bill. What they would actually pay is still open; that's what I'm testing next.

Demo

89 s: an FDA page on disposing of medicines, machine-translated into Spanish and checked on the Mac: 39 segments, 3 flags, and a network log with only local requests.

No-install demo: https://secondread-e49hx.ondigitalocean.app, a static App Platform build with that page's scores precomputed. The top flag is the DEA's "Take Back Day", now «Día Nacional de Retraso de las Drogas Recluidas», a "Day of Delay of Confined Drugs" (80%). It misses the instruction to flush fentanyl patches, which became «lavar», wash (16%).

Code

P0rt / secondread

SecondRead

An offline checker for translators who post-edit machine translation (MTPE). It reads an XLIFF job and ranks segments by how likely they are to contain a major meaning error a dropped or added negation, a wrong number or unit, an omitted clause, a mistranslation Style, grammar and word order are out of scope; other QA tools handle those.

The goal is a shorter re-read: the reviewer starts with the riskiest 10% of the job instead of reading every segment twice.

Status (2026-10-15): the app serves LoRA v1, trained on Tinker on human labels only, as a 36 MB GGUF. By default it flags at the review line: per language pair, the highest threshold that finds 60% of the dev Majors.

On test, measured once: the review line finds 74% of the en-ru Majors (precision 0.39 38% of segments flagged). On en-es it finds 48% [0.42, 0.58], below the…

View on GitHub

https://github.com/P0rt/secondread (MIT): everything from data prep to the UI, with the command behind every number below in docs/RESULTS.md and docs/REALDOCS_GOLD.md.

How I Built It

I built it with Claude Code, starting with go/no-go checks: does Tinker train this model, does the adapter survive GGUF conversion, does Sentry receive zero text?

One token is the whole answer

The model answers one question per segment. This is the entire user message:

EN: {source}
ES: {machine translation}
Major meaning error (wrong meaning, negation, number, omission, addition)? Answer Major or OK.

Major and OK are single tokens in Qwen's vocabulary, so one forward pass gives risk = p(Major) / (p(Major) + p(OK)), with the same prompt in training, Tinker evaluation and the app. The rule checks add highlights (which number changed, where the negation went) and a small train-measured weight on the log-odds.

Training on Tinker, on human labels

The labels are Google's wmt-mqm-human-evaluation (Apache-2.0): every error in WMT24 English→Spanish (9,330 segments) and WMT22 English→Russian (19,292), marked by human annotators with an MQM category and severity. Major means at least one major or critical accuracy, terminology or non-translation error. I split by document.

The run: LoRA rank 16 on the MLP layers only, one epoch over 14,791 human-labelled segments (231 steps of 64), loss only on the answer.

The cookbook's suggested learning rate, 4.9e-4, flipped the model from always-OK to always-Major within three steps; 20-step probes picked 2e-4. The run took 9 min 24 s and $1.22: $1.07 for 1,452,983 training tokens, $0.15 for dev scoring. Dev AUROC rose from 0.552 to 0.739. Locally, the 4-bit model ranks like Tinker (Spearman 0.989).

Results on WMT test

Unseen documents (25 en-es, 33 en-ru), scored by the model alone. AUROC is the chance a segment with a major error ranks above a clean one (0.5 is a coin flip); PR-AUC rewards a clean top of the list.

Test split	System	AUROC	PR-AUC	Share of major errors in the riskiest 10%	$ per 1k segments
en-es, 1,815 segments, 9.3% Major	Qwen3.5-4B, no adapter	0.776	0.413	45.2%	0
	SecondRead v1	0.815	0.517	51.8%	0
	Gemma 4 31B	0.807	0.311	45.8%	0.024
en-ru, 3,573 segments, 20.2% Major	Qwen3.5-4B, no adapter	0.666	0.369	22.7%	0
	SecondRead v1	0.801	0.537	30.3%	0
	Gemma 4 31B	0.794	0.494	30.7%	0.017

Fine-tuning works: the adapter alone adds +0.039 AUROC over the base model on en-es (95% CI +0.003 to +0.070) and +0.135 on en-ru (+0.090 to +0.191).
Level with a zero-shot model about 8× its size on ranking: v1 minus Gemma is +0.008 AUROC on en-es and +0.006 on en-ru, both CIs across 0. On en-es PR-AUC v1 leads by 0.206 (CI +0.099 to +0.286), mostly because Gemma saturates: 27% of en-es rows score 0.97 or more.

Choosing the flag line: recall first

My first flag line, F1-optimal on dev, caught only 30% of the major errors in en-es test.

A false flag costs a post-editor seconds; a missed error reaches the client. So before computing any number I fixed a rule: per pair, the highest threshold that still finds 60% of the dev Majors. For en-es that is 0.4984 (risk ≥ 50% in the UI), flagging 10% of dev segments instead of 3%, and it became the default. Then test, once:

Test split	Flag line	Recall [95% CI]	Precision	Segments flagged
en-es	review (default)	0.48 [0.42, 0.58]	0.54	8.2%
	F1	0.30	0.77	3.6%
en-ru	review (default)	0.74 [0.63, 0.84]	0.39	38.4%

en-es missed its target: the whole CI is below 0.60. I didn't move the line; that would be a choice made on test.

Real documents: ten public pages, labelled before scoring

Post-editors also get tax rules and medicine leaflets. So I took ten U.S. federal and EU pages with an official Spanish version (IRS, FDA, CDC, SSA, USCIS, EU air passenger rights and more) and translated their 720 segments twice: with OPUS-MT, an open 2020 model, locally, and with DeepSeek-V3.1, an open 671B LLM, on Tinker. Nothing was tuned on these 1,440 translations.

The labels. Three blind AI annotators (an MQM rater, a post-editor and a Spanish-only reader) labelled each translation Major or OK; two of three decide. None saw the MT system or any score. On 247 WMT24 units with professional labels, a first version of the panel reached F1 0.67 and Cohen's kappa 0.59. Labels, scorer and flag lines were fixed before any detector was scored.

The surprise is the MT systems: of the 84 major errors, 80 are OPUS-MT's (11% of its segments). DeepSeek-V3.1 made 4 (0.6%); seven of the ten pages have none.

1,440 translations, 84 Major	AUROC [95% CI]	PR-AUC	Major errors in the riskiest 10%	Default line: recall [95% CI] / precision / flagged
SecondRead v1	0.910 [0.884, 0.928]	0.520	61.9%	0.20 [0.14, 0.30] / 0.77 / 1.5%
Gemma 4 31B	0.940 [0.911, 0.957]	0.627	75.0%	0.82 [0.71, 0.89] / 0.39 / 12.2%

Gemma's line is from en-ru dev; CIs come from a document bootstrap.

Ranking: v1 beats its base model (AUROC +0.071, CI +0.036 to +0.119) but trails Gemma (−0.030, CI −0.051 to −0.009), the reverse of WMT test. On OPUS-MT's output alone the gap is within noise (−0.013, CI −0.051 to +0.017); Gemma's lead comes mostly from ranking DeepSeek's clean segments lower.
The default line is far too quiet here: 17 of its 22 flags are real, but that is a fifth of the errors (48% on WMT test), while 52 of the 84 sit in its riskiest 10%. Again, the line stays.
Caught: garbled names and words that become other words, such as pareja, «lista de color» for the flush list and «Desclutter»: 16 of 70 mistranslations, 1 of 2 untranslated.
Missed: the wrong sense in fluent Spanish: fentanyl patches to be «lavar», washed; «Si hace calor», if the weather is hot, for a hot door in a fire; immigration parole as «libertad condicional»; «devolución». Also all 7 omissions, all 5 terminology errors and all 3 critical errors (Gemma flags all three).
Quiet where it matters: v1's 5 false flags are all minor errors, only 2 on DeepSeek's 716 clean segments (Gemma 16, base 25), where a real error is one in 180. The publishers' own Spanish, scored as MT, gets 3 flags in 720 (Gemma 79). But v1 caught none of DeepSeek's 4 errors (Gemma 1).

The caveats: the labels are AI's, from three lenses that agree closely (Fleiss' kappa 0.88) and an adjudicator in the same workflow, so treat them as one annotator; no professional labelled this set, and the panel saw the official Spanish. An LLM judge like Gemma may share the panel's idea of Major more than a 4B model trained on WMT labels does.

Can synthetic errors teach it wrong senses?

Mistranslations are 70 of the 84 real-page errors and 66–76% of WMT test's major errors. v2 showed the trap: 5,095 synthetic number, negation, omission and addition rows taught it those edits and cut en-es test PR-AUC from 0.517 to 0.446 (CI of the drop −0.113 to −0.009).

For v3, Gemma swapped one word of a clean training translation for a wrong sense, false friend or near-antonym (Major) or a synonym (OK); a blind second Gemma call checked each edit, and the real-page error words (return, flush, partner …) never entered a prompt. Under a rule fixed first and run on dev only, v1 stays: v3a and v3b both lose about 0.035 PR-AUC on en-ru dev. They learned their generator (en-es recall 0.79–0.80 on its held-out edits, v1 0.56) but ranked en-es dev's real mistranslations no higher: 46 of 73 in the riskiest 10%, as for v1. Reviewed jobs are the better training data, and they are the paid product.

Gemma 4 31B on DigitalOcean: the yardstick and the error factory

Gemma 4 31B runs on DigitalOcean's serverless inference (the Gradient AI Platform) and sees only public text.

The yardstick: on WMT, 9,436 calls with the small model's prompt and 800 with GEMBA-MQM, $0.024 per 1,000 en-es segments, 0.58 s median; 2,435 more on the real pages. GEMBA-MQM ranked worse than the plain question, at 14× the cost.
The error factory: the v2 negations, omissions and additions (5,626 calls, $0.42) and the v3 wrong-sense edits and checks (7,101 calls, $0.81).

In all, at most $1.64 of a $5 credit. Gemma 4 is Apache 2.0, and I found nothing in DigitalOcean's terms against training on its outputs.

Sentry: tracing every review without the text

Each review is one Sentry AI-agent trace: invoke_agent SecondRead, four execute_tool spans (parse_xliff, number_check, negation_check, lora_score), an aggregate span and a gen_ai.chat span per segment with input tokens, latency, p(Major) and cost (0: local). An FDA review is 39 chat spans and 3,159 input tokens: 6.4–7.5 s per trace below (5.9 s on the backend's own timer), about 0.6 s per model call, four in flight.

No segment text goes in, and no segment IDs either, because some CAT tools use the source sentence as the ID. The hostname (often a person's name) is replaced, and the OpenAI and httpx integrations are off, so each model call is traced once.

scripts/canary_check.py plants SR-CANARY-<uuid> in segment text, XLIFF IDs, the file name, the query string, exceptions and logs, runs real reviews and captures every envelope: 0 of 12 contain it. With the scrubbers for what integrations add on their own (exception and log messages, query strings) turned off, 4 do.

Tracing was once a suspect: after a few idle minutes, the first job took 21 s. Against bare llama-server, the backend added about 0.7 ms per segment, Sentry or not; the culprit was llama-server's RAM prompt cache, and -cram 0 holds a 51-segment job at 8.4–9.0 s. One limit: Sentry keeps about 1,000 spans per transaction, so a job over about 990 segments loses some chat spans.

What it can't do (yet)

Two pairs, Apple Silicon only.
Wrong senses in fluent Spanish are the blind spot, and synthetic data didn't help.
It ranks; it doesn't certify: the default line finds about half of the en-es test errors and a fifth on the real pages.
Raw MT, one segment at a time: untested on edited text; errors across sentences are invisible.
Noisy, public training data: one annotator per segment, nothing legal or medical, possibly seen by Qwen and Gemma.
A big cloud model catches more: Gemma found 82% of the real-page errors, flagging 12%.

Why Does Open Innovation Matter?

The model has to come to the text. Under an NDA, what matters is whether the text leaves the machine. Open weights and llama.cpp make "no outbound calls" checkable: the API listens on 127.0.0.1, nothing loads remotely, and Sentry is off unless you turn it on.

The licensing trap. The best-known quality-estimation models, Unbabel's xCOMET and CometKiwi, are CC BY-NC-SA 4.0: a paid tool needs Unbabel's permission. Qwen3.5-4B, Gemma 4, OPUS-MT and Google's annotations are Apache-2.0, so I could train my own and translate the real pages locally.

The caveat: training wasn't local, but Tinker and DigitalOcean saw only public text. Tinker billed $6.46 in all: $6.21 for Qwen3.5-4B (four training runs, probes and evaluations), $0.23 for the DeepSeek-V3.1 translations and $0.02 for checkpoint storage. DigitalOcean cost at most $1.64 at list price, and Sentry's free plan covered tracing.

Prize Categories

Best Use of Tinker. The shipped model is a $1.22 Tinker LoRA, run locally as a 36 MB GGUF. Over the untuned base model it lifts WMT test AUROC from 0.776 to 0.815 (en-es) and 0.666 to 0.801 (en-ru), and real-page AUROC by 0.071, level with Gemma 4 31B on WMT at $0 per segment. The v2 and v3 Tinker runs decided what ships, and Tinker sampled the DeepSeek-V3.1 translations.
Best Use of DigitalOcean. Gemma 4 31B on serverless inference is the cloud baseline everywhere (12,671 calls) and generated the synthetic errors, checking v3's (12,727 calls), for at most $1.64; App Platform hosts the demo: https://secondread-e49hx.ondigitalocean.app.
Best Use of Sentry Agent Tracing. One AI-agent trace per review with tokens, latency, p(Major) and cost per model call (screenshots above); a canary test proves the traces carry no text: 0 of 12 envelopes, 4 with the scrubbers off.
Best Use of Gemma. Gemma 4 31B, served through DigitalOcean, is the yardstick on every test set, wrote the synthetic errors for v2 and v3 (and blindly checked v3's), and is the stronger detector on the real pages.

Credits

Data: Google's wmt-mqm-human-evaluation (Apache-2.0; Freitag et al., 2021; en-ru annotations by Unbabel) on the WMT22 and WMT24 General MT test sets (Kocmi et al., 2022 and 2024), released for research; the repo keeps ids and labels, not their text.
Real pages: IRS, FDA, NIDDK, FEMA (Ready.gov), USCIS, SSA, CDC and MedlinePlus (U.S. federal works, public domain or free to copy) and Regulation (EC) No 261/2004 from EUR-Lex (reuse under Decision 2011/833/EU). The Spanish shown is machine translation; no endorsement is implied.
Code: scripts/train_tinker.py is adapted from supervised/train.py in Thinking Machines' tinker-cookbook (Apache-2.0). The GGUF export uses llama.cpp's convert_lora_to_gguf.py (MIT); llama-server runs the model.
Prompt and models: GEMBA-MQM (Kocmi & Federmann, 2023, CC BY-SA 4.0; adapted, with attribution, in prompts/gemba_mqm.txt); Qwen3.5-4B (Apache-2.0) from unsloth's GGUF; Gemma 4 31B (Apache-2.0); OPUS-MT en-es (Tiedemann & Thottingal, 2020, Apache-2.0); DeepSeek-V3.1 (MIT); Kokoro-82M (Apache-2.0) narrates the video.
AI disclosure: the code and this post were written with AI assistance (Claude Code). Gemma 4 31B generated the synthetic training rows, labelled as such. The real-page labels come from AI annotators and a Claude adjudicator; no professional labelled them.

34.8k Stars Won't Put GTA VI on Your PC. Here's What AnyPS5 Is Actually Doing

Sergei Parfenov — Tue, 13 Oct 2026 20:23:58 +0000

Seeing GTA VI run on a PC before the official port would be a spectacular way to end the console exclusivity argument. So a project that promises to turn PS5 executables into Windows and Linux programs gets attention almost automatically. When I checked on October 13, 2026, GitHub showed AnyPS5 at roughly 34.8k stars.

The attention has technical substance behind it: open code, hand written system libraries, a shader compiler, and reports of commercial games booting. There is also AI involvement, which the contributors state openly in their own write ups.

None of that means GTA VI is about to run on PC through AnyPS5. Rockstar's official page lists November 19, 2026, on PlayStation 5 and Xbox Series X|S. Nothing in the project's own material claims a GTA VI boot.

TL;DR. AnyPS5 does not emulate PS5 hardware. The CPU is already x86-64, so the real work is replacing a platform's dependencies: a relinker rewrites the game's ELF into a native executable, hand written libraries answer the system calls the game expects (with the right ABI, not just the right name), and a recompiler turns RDNA shaders into SPIR-V for Vulkan. AI is used in development, not at runtime, and the project's answer to "did the model understand the hardware correctly" is a hardware oracle that runs real RDNA snippets on a real AMD GPU. The 90% style progress counters measure stubs removed and opcodes decoded, not games completed. Boot reports are real and narrow: PAC-MAN WORLD Re-PAC on main, GTA V at about 27 to 28 FPS in one scene on an experimental branch, Astro Bot at about 10 FPS on a patched build.

The more interesting question is what is already happening. AnyPS5 is a good lens for looking at a console as a set of software dependencies, some of which can be replaced, and at AI as a tool that can help replace them.

By deplatforming here I mean freeing a program from its technical attachment to one platform. The game still executes its own code. The environment around it is reproduced by other people.

This has a familiar relative: Wine, the compatibility layer for Windows applications. The shared idea is reproducing the interfaces a program expects. AnyPS5 does it its own way: ahead of time conversion of the executable, its own libraries, and a separate path for console graphics.

The hardware gives you the first opening

The PS5 runs an AMD Zen 2 CPU on x86-64. That is the same instruction family as an ordinary Intel or AMD desktop. A large share of the game's machine code can potentially be executed by the host CPU itself.

Identical instructions are not compatibility, though. The program expects a particular loader, particular system functions, a particular memory layout, particular threading rules, and a particular graphics interface. For the game, the platform lives in exactly those details.

Picture a program that can compute a car's trajectory perfectly well, but reads a file through a function Windows does not have. The CPU understands the math. Execution stops at the first contact with the outside world.

AnyPS5 splits that problem into parts. Simplified, the pipeline looks like this:

That shape comes from the project's architecture notes and its implementation. It avoids running a separate process that pretends to be a whole console. The work of reproducing platform behavior does not disappear, it just lives in the libraries and the graphics layer.

Step one: rewrite the executable

The first component is the relinker. It reads the executable, walks its dependencies and relocations (the places where addresses must be patched at load time), and emits a file in the target OS format: ELF for Linux, PE for Windows.

Conversion.cpp shows the sequence: prepare the source image, run RelinkerPipeline, convert the game's bundled modules, pick a platform patcher, write the result. Illustrative pseudocode, not a literal excerpt:

auto image      = read_game_elf();
auto imports    = analyze_imports_and_relocations(image);
auto modules    = convert_bundled_modules(image);
auto executable = build_native_executable(image, imports, modules, target_os);

Nobody is recovering the game's C++ sources and rebuilding it. The existing binary and its relationship with the environment are what get reworked.

Even inside x86-64 there are differences. The project has a --to-intel mode that rewrites supported AMD specific instructions, with a stated limit: instructions the game generates at runtime are not something an ahead of time pass can catch. The input has to be prepared too, since the relinker expects a clean ELF rather than an arbitrary game package. Both are spelled out in the usage docs.

Step two: "just provide the functions"

In the executable, a system function can be referenced by a NID, a compact identifier. AnyPS5 builds its own libraries and renames exported functions to the identifiers the game looks for. NidCompute.cpp computes a NID from a name. The identifier connects a call to an implementation. It says nothing about the implementation being correct.

Here is a small real excerpt from Time.cpp:

std::uint64_t APS5_VABI sceKernelGetProcessTime() {
    const std::uint64_t start = GetStartNanos();
    return (GetMonotonicNanos() - start) / 1000ULL;
}

The game calls a familiar system function. AnyPS5 returns elapsed time in microseconds using its own clock helpers. From the caller's point of view the interface holds.

That tiny function hides another platform boundary: APS5_VABI. On Windows the macro expands to __attribute__((sysv_abi)), telling the compiler which calling convention the guest code expects. The ABI governs, among other things, how arguments are passed and which registers may be clobbered.

A matching function name is not enough. If the caller puts an argument in one register and the implementation reads another, the thing breaks before any real work happens.

Clocks make a compact example. Threads and memory get hard fast: wakeup ordering, cancelled waits, address reuse. A mistake there can surface minutes later, when the next scene load meets a badly reproduced synchronization primitive.

Step three: the GPU, where it stops being cheap

A console shader cannot be handed to an arbitrary driver as a finished program. AnyPS5 builds its own translation path: decode RDNA instructions, rebuild the control flow graph, lower operations into an intermediate representation, emit SPIR-V for Vulkan.

Recompiler.cpp really does contain a decoder, a GraphBuilder, a Structurizer, an InstructionTranslator and a SpirvEmitter. Strip out resource management, specialization and caches, and the logic reads like this (again, illustrative pseudocode):

auto instructions  = decode_rdna(shader_bytes);
auto control_flow  = build_control_flow(instructions);
structurize(control_flow);
auto ir    = translate_to_ir(instructions, control_flow);
optimize(ir);
auto spirv = emit_spirv(ir, host_gpu_capabilities);

An IR is convenient because it separates what an operation means from the machine format it came in. The meaning still has to be reproduced exactly: rounding, memory behavior, branching, how threads inside the GPU interact.

Beyond shaders, the graphics layer handles draw commands, textures, buffers and synchronization. The driver's known limitations list has a telling case: GPU results can land late in memory the game has already freed and reused for something else. That needs correct ordering of events. A faster GPU does not fix it.

Where the AI actually sits

In the execution path above there is no step where a language model invents the next frame. AnyPS5 performs program transformations. The AI shows up in building those transformations.

Contributors publish concrete loops: diagnose a crash, change code, build, run, post a new report. In the PAC-MAN WORLD Re-PAC thread the author credits Claude with diagnosis and patch preparation. The Astro Bot reports name Codex directly. That confirms assistants are in use. It does not let anyone compute how much faster the project moved.

My engineering hypothesis: this kind of work suits assistants unusually well, because the big task decomposes into checkable fragments. Parse a structure format. Find the cause of a crash. Implement one instruction. Write a test for an edge value. Repeat on another driver.

The hard part stays in verification. If a model misunderstood an instruction's behavior and then wrote the test from the same misunderstanding, a green test only proves two errors agree.

Which is why the hardware oracle is the most interesting file in the repository. The tool runs small RDNA assembly snippets on a real AMD GPU and returns the result, so measured values can be frozen into tests. The docs warn about GPU differences: an observation on one desktop card does not automatically prove PS5 behavior in every case.

The contribution rules require stating the provenance of shader instruction semantics, disclosing AI use, and owning every line you submit. That is a sane collaboration model: the assistant proposes an implementation, and evidence of correctness is gathered separately.

Read the progress counters carefully

tools/progress.py counts a library function as implemented once it no longer calls the project's NotImplemented_nid_no_patch stub. For GPU instructions, the counter reflects decoder recognition.

Those are useful measures of work on the code. They do not measure the share of compatible games. A function can exist while a rare mode of it still diverges from the console. A decoder can recognize an opcode while the failure happens later. And the denominator of the library statistic only includes functions the project already knows about. "90% implemented" cannot be read as "10% away from GTA VI".

Concrete boot reports say much more. As of October 13, in the threads I read:

Game	Build	Reported result	Caveats
PAC-MAN WORLD Re-PAC	`main`, commit `ff8fe4d`, no local patches	Menu, save load, a playable level with audio	Post level transition not tested in that session
GTA V, PS5 version	experimental branch `gta-v/testing`	~27 to 28 FPS in the prologue bank scene	Limited scene; the same report says `main` stalls at load
Astro Bot	Windows, Ryzen 5 7600, RTX 4060, build with extra patches	~10 FPS in Crash Site	Author notes game logic running slow

These are substantial results. The distance between a first frame, a controllable character, and a finished playthrough is still large. A working menu says little about saves, about a complex scene five hours in, or about another GPU family.

GTA VI would require checking its binary, its shaders, its memory behavior, its world streaming and the system interfaces it actually uses. Even a fully working PS5 build of GTA V would not prove the next game compatible. A shared developer and a shared series name are not measurements.

So promising a near term GTA VI boot through AnyPS5 would be premature. What is already worth discussing is the changing cost of this kind of engineering. If AI helps cycle through checkable hypotheses faster, write adapters, and dissect crashes, then more attempts become practical. How much faster those attempts reach a stable result is something only reproducible tests will show.

Platform lock in is made of many small contracts between a program and its environment. AnyPS5 makes a number of those contracts visible in code. That is its most interesting effect: "which device does this game run on" slowly turns into "which dependencies are still missing".

AI can help write the next implementation. Correctness still has to be demonstrated on a running game.

What would you accept as proof that a compatibility layer is actually correct, rather than correct enough to boot a menu?

Checked on October 13, 2026. Code read at commit dacd78525e537a779682032c0ace7f7af120af05. Boot results and FPS figures are as reported by the authors of the linked threads; I did not run the games myself while preparing this. Illustrative pseudocode is marked separately from the one real source excerpt.

Sources and further reading

What Agent Builders Can Borrow from OpenAI's Math Research

Sergei Parfenov — Wed, 07 Oct 2026 16:52:57 +0000

On October 6, 2026, OpenAI published mathematical results from an internal frontier model. The openai/math repository contains 722 manuscripts organized into 372 families, with Lean formalizations for many of the results.

For engineers building AI agents, the release suggests a concrete experiment: apply its separation between proposing a result and independently checking it to the rules that update persistent memory.

For Symptomato, the target is a familiar failure: a corrected patient record arrives, an old document is uploaded again, and the current value silently goes backward. Every extracted field can be correct while the update rules fail.

What the research gives us

The published Lean artifacts make many of the proofs available for machine checking. Verification status varies across the collection, and some results still lack formalizations. The model itself remained internal at publication.

The repository also includes verification instructions using Comparator, an existing tool from Lean FRO. OpenAI's release shows how these established tools can accompany AI-generated research results.

OpenAI additionally shares reasoning summaries, attempted-problem counts, and compute estimates. That transparency suggests another practice to carry into our experiment: report the effort required to obtain a checked result.

The transferable component is the verification workflow. Our patient-memory specification and proofs would be new application work.

Apply the idea to Symptomato

Consider a synthetic sequence for one weight observation:

Revision 1 records 72.4 kg.
Revision 2 explicitly corrects that observation to 72.1 kg.
Revision 1 arrives again.

The current view must retain 72.1 kg, while history retains both revisions.

Bind each accepted structured event to an organization, patient, encounter, and assertion. Store its source lineage, source-issued revision, operation, value, and immutable provenance. Compare revisions only within that lineage. Independent sources need separate reconciliation rules.

Corrections carry complete replacement values, allowing revision 2 to be accepted before revision 1. Arrival order never determines precedence.

The ledger stores distinct structured events. Equality includes stable payload and provenance; transport delivery timestamps and delivery IDs stay outside it. Reusing a version-specific event ID with different content produces an ingestion conflict, with both candidates preserved.

Derive the current view from the highest revision for each scoped assertion and source. A consistent assertion becomes active; a retraction removes its active value while preserving history. Incompatible operations or values at the highest revision become a visible conflict.

Prove the insertion layer first

The smallest layer fits into a finite-set model:

import Mathlib.Data.Finset.Insert

variable {Event : Type} [DecidableEq Event]

def ingest (s : Finset Event) (e : Event) : Finset Event :=
  insert e s

theorem retry_is_noop (s : Finset Event) (e : Event) :
    ingest (ingest s e) e = ingest s e :=
  Finset.insert_idem e s

theorem delivery_order_independent
    (s : Finset Event) (a b : Event) :
    ingest (ingest s a) b = ingest (ingest s b) a :=
  Finset.insert_comm b a s

theorem accepted_event_is_retained
    (s : Finset Event) (e : Event) :
    e ∈ ingest s e :=
  Finset.mem_insert_self e s

These statements reuse existing Mathlib finite-set laws. The model covers evidence insertion only. Event validation and the current-record projection still need their own definitions and proofs.

The third obligation is essential: a function that ignores every event would also be idempotent and independent of delivery order.

Implement the projection next, with explicit obligations:

Consistent assertions at the highest revision produce the current value and its provenance.
A strictly older revision cannot replace the current value or resurrect a retracted assertion within its lineage.
Incompatible highest revisions produce a conflict regardless of delivery order.
Every displayed value has stored evidence for the exact organization, patient, encounter, and assertion.
Previously accepted events remain in the ledger after a correction.

Reuse the independent checking approach

A reviewer freezes the schema, reducer definitions, theorem statements, imports, and build configuration. The agent fills in proof bodies, using Lean diagnostics to guide bounded retries.

Give Comparator a trusted Challenge and an agent-produced Solution. Under its documented environment assumptions, it checks statement and definition agreement, permitted axioms, and acceptance of the proofs by the Lean kernel. Changes to the contract require a separate review.

Keep a minimal repository with the reviewed model, candidate proofs, generated histories, and the CI check. Pin Lean and Mathlib versions so every proof attempt runs against the same environment.

Generate proofs during development and check them independently in CI. Patient updates call the deterministic reducer.

Measure what the research workflow adds

Generate histories containing duplicates, shuffled deliveries, corrections, retractions, conflicting revisions, and repeated IDs across organizations and patients. Seed concrete bugs: prefer the last arrival, discard correction history, or omit the organization ID.

Compare a fixed property-based test suite with the same suite plus formal proof obligations, targeting the same Lean reducer implementation. Freeze each candidate's definitions before proof search while keeping the required properties unchanged. Hold model settings and proof-search budgets constant.

Report confirmed counterexamples, proof completion, human effort, elapsed time, and model cost. A timeout means the search failed to produce a proof; it does not establish a bug.

Define the production boundary

The guarantees apply to the formal model. A separately written Python reducer needs a correspondence argument; differential tests provide evidence, not a formal equivalence proof. Calling an implementation compiled from checked Lean definitions reduces that gap. Parsing and database integration still need validation.

Stored provenance also does not establish that an LLM interpreted the source correctly or extracted every relevant fact.

OpenAI's math release makes this a concrete engineering experiment: study the published proof artifacts and checking workflow, specify one memory operation, and measure the value of independently checked proofs. For Symptomato, that first operation is updating a fact without letting a stale document undo its correction.

Same Patient, Conflicting Documents: Can AI Preserve the Evidence?

Sergei Parfenov — Wed, 30 Sep 2026 10:20:40 +0000

This is a submission for the Kaggle Benchmarking Challenge

What I Benchmarked

In the first chronological run of one synthetic history, a record built from GPT-6 Luna's extractions returned the expected answer to 21 of 22 final queries. Yet it had lost an earlier source assertion, hiding a conflict before a correction arrived. The accepted value at the end was right; the documented history was incomplete.

A patient uploads a laboratory report, an older discharge summary and a photograph of a prescription. A correction arrives later. The same report appears twice. One note describes the patient's mother; another says a diagnosis is suspected. The dates inside the documents do not follow the order in which the files arrived.

I'm a technical consultant at Symptomato, where AI gathers context before a human health specialist joins the conversation. For Symptomato, we are building a way to turn incoming documents into a longitudinal, source-grounded record.

ChartReplay asks whether an update policy preserves every required assertion and its relationships as corrections and repeated documents arrive. A record fails this content test if it omits a required assertion, adds unsupported content or misrepresents its subject, value, time, uncertainty or relationship to other sources.

Trust starts with knowing what the record can establish

There are two different questions here. Did the system preserve what the documents actually said? And are those documents themselves accurate accounts of what happened to the patient?

This experiment tests the first. A faithful record can still contain an erroneous report, a mistaken patient recollection or unresolved disagreement. Establishing authenticity and medical accuracy requires evidence and review beyond this benchmark. The system should expose those questions rather than silently answer them through a confident rewrite.

In ChartReplay, accepted is a state under explicit record-merging rules. It does not mean independently verified, clinically current or true in the world. A statement can be accepted because no visible source disputes or supersedes it. If we lose a conflicting source, that state can become misleading.

Provenance makes such decisions inspectable. FHIR's Provenance resource describes how information was created or revised and which entities and agents were involved. ChartReplay is not a FHIR implementation, but the relevant principle is the same: the record needs an inspectable relationship to its sources.

The record has to preserve relationships, not just values

I defined a source contract before running the models. It specifies what should survive an update:

What arrives	What the record must preserve
Two different values for the same documented event	Both assertions and the unresolved conflict; upload order does not choose a winner
An explicit correction	The corrected assertion, its links to the originals, and the originals as superseded history
A repeat upload	The existing assertion, without inventing another measurement
A later independent measurement	A separate event, not a correction of the earlier one
A family-history statement or suspected diagnosis	The correct subject and the stated uncertainty
A medication order or an absent entry	What the source states, without inferring medication consumption or an explicit negative

For example, a prescription does not establish that someone took a medication. An absent allergy entry does not establish that the patient denied allergies. These distinctions follow the source semantics described by FHIR MedicationRequest and AllergyIntolerance.

Every retained assertion is checked against its supporting source, including subject, event time, value, uncertainty and correction relationships. A citation to an unrelated document does not count. The evaluation includes superseded assertions and unresolved conflicts, not just whichever value appears last.

Testing one part of the upload problem under controlled conditions

The model pilot uses ten self-authored synthetic histories, one from each logical family. It covers corrections, conflicts and reconciliation, family attribution, uncertainty, medication orders, duplicate versus independent events, time offsets, similar-looking sources, multi-assertion notes and longer histories.

Each history is delivered chronologically, in a seeded shuffle and with duplicate replays, with two repetitions of each schedule. The unique source set is the same when delivery finishes. At intermediate checkpoints, the expected record is reconstructed from only the sources received so far: a system cannot act on a correction it has not received.

These are controlled text fixtures with explicit source and event identifiers. They isolate the update problem; they do not reproduce all the difficulty of the opening upload scenario. No photographs, OCR or unrestricted clinical prose enter this pilot. Most documents contain one assertion. A deterministic parser written for the known grammar passes the calibration set without an LLM.

The broader fixture generator contains 120 cases, 1,020 documents and 1,092 assertions. The model results below use the ten pilot histories, not all 120 cases. Repeated episodes are not independent patients.

Three ways to maintain the same history

Full rebuild asks the model to construct the whole chart again from all available originals.

Rolling rewrite gives it the previous chart, new documents and the complete available archive, and asks for an updated chart.

Extract plus ledger asks the model to extract assertions from newly delivered documents. Deterministic software then applies the declared correction and conflict rules.

The third design has an important constraint: its prompt explicitly prohibits re-extracting archive-only documents. Rebuild and rewrite can recover an earlier omission by reading the originals again. The extractor can see the archive for context, but does not revisit old documents for extraction unless they are delivered again. There is no separate completeness check before the ledger accepts an extraction.

This is a comparison of those particular update policies, including that asymmetry. It does not establish the performance of every ledger-based design. All three can retain the full history without an imposed chart-size budget; provider context and response limits still apply. Each call starts a fresh conversation.

Models Tested

The primary models are GPT-6 Luna (gpt-6-luna), GPT-6 Sol (gpt-6-sol) and Claude Sonnet 5 (claude-sonnet-5). Gemini 3.8 Flash (google/gemini-3.8-flash) supplies two separate Kaggle phases.

Evidence	Episodes and models	Execution and overlap
Primary comparison	540 episodes; Luna, Sol and Sonnet 5	Direct APIs: OpenAI for Luna/Sol, Anthropic for Sonnet
Matched secondary comparison	288 outcomes; the same three models plus Gemini	72 Gemini episodes through Kaggle + 216 reused primary episodes
Public smoke benchmark	12 episodes; Gemini only	Separate Kaggle runs on two histories

Each view includes all three update architectures. The primary comparison has the same 60 case–schedule–repetition combinations for each architecture and model. Its complete three-model subset was selected retrospectively from an original five-model lineup by coverage, not by correctness. Every terminal outcome is included.

The secondary view matches all four models on Gemini's 24 available combinations. The public smoke is a smaller demonstration; its task scores are reported separately below. The direct-provider results are not Kaggle-hosted leaderboard entries.

The original lineup was recorded on September 25, 2026, with recent releases and sustainable recurring cost as the practical criteria. Luna and Sol were released on September 22. Sonnet 5, released June 30, was the explicitly chosen cheaper Anthropic tier, not the provider's latest overall release. These are dated selection decisions, not a claim about the current model catalog.

Inputs, prompts and profiles were frozen. Luna and Sol used medium reasoning effort; Sonnet used adaptive reasoning with high effort. These labels do not establish equal compute. Models ran at different times, and the Kaggle route has its own recorded settings. Each model's smoke test preceded its own pilot.

The primary score requires an episode to finish under the declared interface and produce an exact final chart. Format failures remain unsuccessful outcomes. The direct-provider arms requested JSON through the prompt, without provider-enforced structured output. This matters for interpreting Sonnet's results; it is separate from the question of whether a completed record preserved the evidence. Saved, unmodified responses and first terminal outcomes are evaluated by the frozen rules, with no judge LLM.

Findings

Losing one assertion hid a conflict

In the reconciliation case, the first source reports 138. A second source reports 145 for the same patient, event, concept and event time. Neither supersedes the other. At that point, the correct record contains both assertions, marked disputed.

A third source explicitly corrects both to 140. Only then does the contract accept 140 and retain the two earlier assertions as superseded history. These numbers use synthetic units; the example concerns document relationships, not interpretation of a clinical measurement.

In both chronological repetitions, Luna's extractor returned no assertion for the first source. When the second source arrived, the ledger contained only 145, marked accepted. The software had no extracted counterpart with which to identify the conflict.

Observed Luna plus ledger trace, chronological delivery. “Accepted,” “disputed” and “superseded” are benchmark states, not judgments of medical truth. The figure shows the first three uploads of the six-document case. Open the trace at full size.

The first assertion is CR-02-0000-D34123:1; the conflicting source is CR-02-0000-D12503; the explicit correction is CR-02-0000-D31621. Those identities make the error traceable to particular documents rather than a vague claim that the model “forgot something.”

The correction to 140 was later applied correctly, but 138 remained absent through all six checkpoints. The first repetition's failed query asked for the measurement history. A consumer inspecting only the accepted value would miss the loss.

Full rebuild and rolling rewrite preserved the complete record in these chronological repetitions. The difference matters: repeated reconstruction gave those arms an opportunity that the one-pass extraction policy did not provide.

This is also why storing the original file and retaining its assertions in the usable record are separate requirements. The original document remained in the source archive. It was the derived record that lost its contribution, and with it the visible disagreement.

Completeness matters even when retained facts are correct

The primary results use the same 60 combinations in every cell. “Exact” requires the complete final record, including history and relationships, and successful completion of the strict interface contract.

Model	Full rebuild	Rolling rewrite	Extract plus ledger	Format failures across 180 episodes
GPT-6 Luna	59/60	59/60	54/60	0
GPT-6 Sol	60/60	60/60	49/60	0
Claude Sonnet 5	0/60	7/60	0/60	170

All six Luna ledger failures and all eleven Sol ledger failures involved missing assertions. In one chronological Sol episode, a source explicitly documented an allergy to drug:beta; its extraction was empty and the final record omitted the allergy.

Across Sol's 590 ledger checkpoints, all 3,961 predicted assertion occurrences matched the contract, against 4,094 expected occurrences: 100% precision and 96.75% recall, yet only 49/60 exact final records. These are repeated occurrences across checkpoints and schedules, not thousands of independent patients. Checking whether retained facts are supported would not, by itself, find every missing fact.

The experiment also contains successful handling of disagreement. In the dedicated unresolved-conflict case, Luna and Sol produced exact final records in all six schedule–repetition combinations under each architecture. The system could preserve a known conflict. The reconciliation failure shows what happens when extraction prevents one side from reaching that system.

A second case checks what the assertion actually means. Under duplicate replays, with eight unique documents delivered twelve times, all three Luna architectures preserved statements about the patient's mother and father separately from a suspected condition in the patient. They also kept an allergy marked unknown and a symptom explicitly denied, without creating duplicate assertions. Preserving a supported statement can mean preserving uncertainty, rather than confirming a diagnosis.

Exact scoring also catches literal-contract errors. Luna's single rebuild failure shortened the source identifier procedure:alpha to alpha; the assertion was present. That is a different defect from losing an allergy or hiding disagreement, even though each prevents an exact-record result.

A correct final record is not enough during an ongoing history

Records may be used between uploads. Luna produced 1,713/1,770 exact checkpoints, and Sol 1,656/1,770. A final correction does not undo the period during which the record was incomplete.

We also required the three delivery schedules to produce the same correct final record for each case–repetition group. The ledger passed 14/20 groups for Luna and 11/20 for Sol; full rebuild passed 19/20 and 20/20. An invariant but incomplete record would fail this requirement.

The experiment does not isolate a causal effect of upload order. For ledger, repeated identical schedules disagreed in 4/30 pairs for Luna and 10/30 for Sol; different schedules within a repetition disagreed in 12/60 and 19/60 pairs. These correlated comparisons show that ordinary model variability also matters. The strongest observation is the actual lost evidence and its consequences, not a claim that reordering alone caused every difference.

Gemini adds evidence on the same available conditions

Gemini's 24 available combinations let us make a smaller matched comparison across four models and all three architectures. Selection uses availability only and includes every format failure.

Model	Full rebuild	Rolling rewrite	Extract plus ledger	Format failures across 72 episodes
GPT-6 Luna	24/24	24/24	22/24	0
GPT-6 Sol	24/24	24/24	19/24	0
Claude Sonnet 5	0/24	5/24	0/24	65
Gemini 3.8 Flash, via Kaggle	23/24	20/24	17/24	11

All ten families appear, but unevenly; only one case–repetition group has all three delivery schedules. This is not a complete four-model test of order invariance.

Gemini's one completed but inexact episode was a shuffled medication history in the ledger arm. A newly delivered source described a separate 5 mg medication order. The extractor omitted it, leaving seven of eight required assertions in the final chart. It is another example of source information failing to enter the derived record.

The sample also contains counterevidence to a blanket preference for reconstruction. In the chronological 24-document history, Gemini's ledger passed all 24 checkpoints, while rebuild and rewrite stopped on format violations at steps 24 and 8. A stopped integration and an incomplete completed record are different outcomes.

A necessary interpretation note: all 170 Sonnet format failures and all eleven Gemini format failures involved Markdown-wrapped JSON. Offline removal of one outer wrapper made those responses pass the schema; the accumulated record at the rejected step was exact in 121/170 and 10/11 cases, respectively. These diagnostics do not repair primary scores or invent the later responses of episodes stopped early. The low Sonnet totals cannot be read as a general inability to understand the sources. The missing-fact examples above come from responses the original interface accepted.

Lower expense does not price the missing verification step

At the September 25 tariffs and observed cache usage, the selected 60-episode rebuild and ledger arms cost an estimated $0.31 and $0.11 for Luna, and $5.68 and $2.06 for Sol. These are model-token estimates for selected calls, not provider invoices or production bills; interrupted attempts and unknown costs remain separate.

Those cheaper ledger results include the consequences of the one-pass policy. A completeness check, targeted re-extraction or periodic reconciliation would add work that was not measured. We have not established the cheapest design that meets an acceptable quality target for Symptomato.

My Benchmark

ChartReplay separates source inputs from evaluator-only expected records. It retains original source text, assertion identities, correction links, delivery schedules and saved response traces, so an error can be inspected at the update where it appeared. The ledger's disposition rules are explicit; a claim is not considered supported merely because it cites some document.

The evaluator needed its own checks. An initial audit corrected timestamp comparison, categorical normalization and validation inconsistencies. Calibration then ran 9,240 deterministic episodes across 120 fixtures, seven schedules and eleven systems. Two positive controls each passed 840/840 final episodes and 5,988/5,988 checkpoints; all nine deliberately injected fault types were detected in at least one applicable case. These results check the declared contract, not clinical understanding.

A separate controlled demonstration removed every superseded assertion from 24 parameterized histories. Eight existing questions still passed on all 24; full-record evaluation passed on none. Adding a history question exposed every failure. Question answering can test the same contract if its coverage is exhaustive, but a handful of correct answers is not evidence that the whole record survived.

Streaming evaluation of medical memory is established prior work. MedMemoryBench evaluates memory as it is constructed; ClinTraceBench studies longitudinal clinical tasks with source-verifiable evidence. ChartReplay's narrower contribution is an explicit record-construction contract and traces showing how an update can lose evidence or change its disposition.

The Kaggle tasks

The ChartReplay benchmark on Kaggle consists of three existing smoke tasks, one for each update architecture. They use two synthetic histories—a correction chain and a history containing a multi-assertion note—delivered chronologically and with duplicate replays, once per schedule.

Each task returns the fraction of the two case–repetition groups for which both delivery schedules produce the same exact final record. This score tests correctness and invariance together; it is different from the fraction of individual episodes that finish correctly.

The saved Gemini 3.8 Flash results are:

Task	Correct-and-invariant score	Exact final records
Full rebuild	1.0 (2/2 groups)	4/4 episodes
Rolling rewrite	1.0 (2/2 groups)	4/4 episodes
Extract plus ledger	0.5 (1/2 groups)	3/4 episodes

The smoke used 84 recorded application calls. Its two histories recur in the larger pilot, so it is not independent held-out confirmation.

The tasks provide a small executable test of the same evidence-preservation contract. They cover corrections and repeated delivery; the reconciliation example with values 138, 145 and 140 comes from the larger pilot, not these smoke tasks. The architecture scores are reported separately because combining them would obscure the comparison.

Inspect the saved failure offline

Download the hidden-conflict evidence ZIP. It contains six synthetic documents, six saved prompts and unmodified model responses, expected and observed states, and 22 saved queries from Luna's first chronological reconciliation run.

Unzip it, open the extracted folder in a terminal, and run:

python3 verify_trace.py

The standalone verifier uses Python's standard library and makes no API calls. It reconstructs dispositions and query answers from the saved trace. This case was selected after reviewing the results to illustrate a failure; the bundle does not reproduce all primary tables or the full evaluator.

What this changes for the system we are building

For Symptomato, the immediate design implication is to keep the original upload, its extracted assertions and its reconciliation status inspectable separately. A successful extraction call should not, by itself, mark a source fully incorporated. I would expose unresolved coverage to the reviewer and test a separate check that can trigger targeted re-extraction. The benchmark knows which assertions are missing because it has an oracle; the product will not. That coverage check needs evaluation against independently annotated documents before we can claim it improves completeness.

A further evaluation needs independently annotated, naturally written document sets with ambiguous references, dates and source quality. OCR would need its own paired tests against the corresponding text. The present experiment does not assess document authenticity, rank competing sources by authority, verify real-world medical facts or establish clinical safety. A prepared controlled-prose layer has not been run through models and would not, by itself, establish those capabilities.

OpenAI Codex assisted implementation, offline analysis and drafting. The cover was generated with AI; the trace figure comes from saved experimental states. No model was used to judge the reported record scores.

A record we can inspect must keep disagreement visible until there is evidence to resolve it. In the observed failure, the system made that disagreement disappear before the correction arrived. Preserving the accepted answer at the end was not enough to preserve the patient's documented history.

The Test Looked Redundant. The Ninth Bug Needed It

Sergei Parfenov — Tue, 15 Sep 2026 13:20:00 +0000

On a catalogue of eight implementations, a repeated-status test added no unique detection. Every candidate it rejected was already rejected by another check.

I added a ninth implementation. The repeated-status test became the only check that caught it.

That changes how I would apply the recommendation at the end of my previous article on AI-generated tests. I suggested reviewing a test by asking which plausible wrong implementation it rejects. The comments made that question executable: run the suite against a catalogue of mistakes and count the rejections.

The approach is useful. Its denominator still needs review.

I have put the code, checks, mutation diffs, and recorded results on GitHub. These are local runs of a deliberately constructed Python fixture, not measurements of a coding agent or a replication of ExecCritic. One part reproduces a commenter's result with matching Python and tool versions. Another reconstructs prose-described variants whose original files I do not have. The ninth candidate is my own deliberate counterexample.

Five out of five did not distinguish the suites

The previous example was an order filter with three rules: omitting the filter or passing None returns all orders; an empty list returns none; a list of statuses selects matching orders.

The correct implementation is small:

ORDERS = [
    {"id": 1, "status": "paid"},
    {"id": 2, "status": "pending"},
]

def filter_orders(orders, statuses=None):
    if statuses is None:
        return list(orders)
    return [order for order in orders if order["status"] in statuses]

Changing the condition to if not statuses introduces the regression. Python treats both None and [] as falsey, so the function returns everything for an empty filter. Two tests—default call and paid-status selection—accept both versions. The empty-list assertion separates them.

Vinh Nguyen ran mutmut against this fixture and reported a boundary I had left too easy to miss: the generated candidates did not include that condition substitution.

I reproduced the comparison with CPython 3.14.6 and mutmut 3.7.0. The correct implementation scored 5/5 with the two tests and 5/5 after adding the empty-list test. The wrong implementation also scored 5/5 with the two tests. With all three, the wrong implementation failed its baseline, so it received no mutation score.

For the correct function, the tool inverted the identity comparison, replaced list(orders) with list(None), altered the "status" key in two ways, and inverted membership. It never replaced the identity comparison with a truthiness test. Several generated candidates raised exceptions; the suite detected those, too. The exact diffs and failure categories are recorded.

The score correctly described all five candidates the tool had generated. It could not distinguish these two suites because both rejected that entire set.

Following Vinh's second comparison, I added the missing truthiness candidate by hand. Against the same six candidates, the two-test suite rejected 5/6 and the three-test suite rejected 6/6. All of the new distinction came from the candidate derived from the empty-filter requirement.

This observation is specific to the fixture and operator configuration. It gives no estimate of how often mutation testing misses important distinctions in other code.

Eight candidates made another test look unnecessary

howcani described a broader catalogue: eight alternatives involving the filter condition, ignored filtering, an incorrect comparison, list identity, output order, repeated selectors, and changes to the input.

I reconstructed those alternatives with fresh input for each check and independent expected values. The cumulative rejections matched the reported progression:

The original two checks rejected 3/8.
Adding the empty-list check rejected 4/8.
Requiring a new list object brought it to 5/8.
Adding an order-sensitive check brought it to 7/8.
Checking that the caller's input remained unchanged brought it to 8/8.

Those matching counts are not a blind replication. The descriptions and counts informed the reconstruction. Exact inputs and implementation choices matter, and the original files were unavailable.

One reconstructed candidate iterates over requested statuses first, then orders. That can both reorder the result and duplicate orders when a status appears twice. My order probe catches it. So does this check:

assert f(ORDERS, ["paid", "paid"]) == [ORDERS[0]]

Here, f is the candidate being evaluated. Once the order probe is present, the repeated-status check adds no unique rejection among those eight candidates. It is redundant for separating that finite set. In the discussion, I had agreed with adding assertions only when they reject a surviving candidate.

The stronger interpretation of that rule does not survive the next function.

The ninth candidate preserves order and duplicates matches

Here is the additional implementation:

def duplicate_only(orders, statuses=None):
    if statuses is None:
        return list(orders)
    return [
        order
        for order in orders
        for _ in range(statuses.count(order["status"]))
    ]

For unique requested statuses, it behaves like the correct filter. It preserves input order, creates a new list, leaves the input unchanged, and handles both None and [] correctly. For a repeated status, it repeats the matching order.

It passes all six checks that collectively rejected the first eight candidates. The repeated-status check rejects it.

With the ninth candidate included, the six-check result is 8/9. Retaining the seventh check makes it 9/9.

The last row is the added challenge. Its only rejection comes from the repeated-status column.

I constructed this candidate specifically to separate behaviors that the earlier loop combined. It is not a held-out sample, evidence of bug frequency, or a reason to report 9/9 as general coverage. It demonstrates a narrower point: a check's lack of a unique rejection can be a property of the catalogue.

The smaller catalogue contained no candidate that duplicated matches while preserving order. Its apparent redundancy inherited that omission.

My original runner also changed the count

There was another awkward result. Running the same reconstructed catalogue through the untouched runner from my published archive produced 4/8 and 5/8, instead of 3/8 and 4/8.

The extra rejection came from the candidate that builds the correct return value and then appends a sentinel to the input. In simplified form:

def append_after_copy(orders):
    result = list(orders)
    orders.append({"id": -1, "status": "sentinel"})
    return result

My original default check compares the function's result with ORDERS. That expected value is the same mutable list passed into the function. By the time equality is evaluated, the expected list has grown. The returned copy has not. The assertion fails.

With an independent before-call snapshot as the expected output, the output check passes. A separate input-preservation assertion catches the side effect when that property is required.

The original check happens to reject this candidate. But the two runners are measuring different things. A count without the expected-value construction and state-isolation rules is missing part of its method.

This discrepancy does not show that howcani's count was wrong; I do not have his precise implementation and test harness. It shows why publishing the functions alone would not make my reconstruction reproducible.

Some of the catalogue expanded the contract

The original three rules described which orders should be returned. They did not separately promise a new outer list, stable ordering, or unchanged input. Those may be appropriate requirements. The brief did not settle them.

I checked a narrow interpretation: preserve the membership and multiplicity of the matching orders, without specifying their order or object ownership. Repeating a selector does not create another order under this interpretation.

Across a finite domain of 1,134 calls per function, four alternatives—returning an alias, reversing the output, sorting the input, and appending after constructing the result—returned the correct multiset on every call. The domain included empty inputs, omitted and explicit None filters, repeated selectors, unknown statuses, and reversed input order. The full setup is in the experiment report.

Those functions can still be bad choices for a real caller. Mutating input can corrupt later calls. Order or ownership may be an established compatibility promise. Passing a one-call, finite-domain check does not establish product correctness.

The implication for the catalogue is specific: attach a requirement to each candidate before labeling its behavior a bug. return orders violates a new-list promise if the product makes one. The fact that my reference implementation uses list(orders) does not, on its own, establish that promise.

Keep the regression; keep questioning the catalogue

For an agent repair loop, I would keep the review question from the previous article and narrow the pruning rule.

Start with a written requirement and its supported inputs. Give each candidate an executable counterexample and a reviewed expected result. Use the candidate matrix to discover missing distinctions. Keep accepted regression checks stable while the implementation changes.

If a test adds no unique rejection, inspect the overlap before deleting it. Two tests may reject the same candidate because that candidate bundles two independent mistakes. Ask whether one mistake can occur without the other. Here, an outer loop over selectors bundled ordering and duplication. Changing the loop structure separated them.

This does not mean retaining every test forever. A finite catalogue can support pruning when the intended objective is discrimination within that catalogue. Removing a regression check tied to a distinct requirement needs a broader argument—such as the remaining checks establishing the same behavior over the supported inputs.

I would carry that distinction into the same record as the score: contract revision, candidate revision, probes, expected values, environment, and observed failures. It extends the concern in my earlier piece about harness-dependent scores: the number only describes the system that produced it.

The repository includes the original archive, four mutation-test configurations, saved generated functions, reconstructed candidates, the ninth challenge, and commands for replaying the evidence. The quick verification uses only Python's standard library; the full mutation run uses the recorded dependency versions.

The repeated-status requirement did not change when I added the ninth function. The catalogue finally contained a way to violate it without also violating the order check. That is the reason I would keep the test.

If this article helped you, you can buy me a coffee. Your support helps me make time for the next experiment.

AI-Generated Tests Can Make Coding Agents Worse. Here's How to Check Yours

Sergei Parfenov — Fri, 11 Sep 2026 11:57:46 +0000

A bug fix can make every new test pass and still introduce a regression. Here is a deliberately constructed Python example, checked locally without an LLM.

An order filter has three requirements:

Omit the filter, or pass None: return all orders.
Pass an empty list: return no orders.
Pass a list of statuses: return only matching orders.

The reported bug is that omitting the filter returns nothing. This proposed fix looks reasonable:

ORDERS = [
    {"id": 1, "status": "paid"},
    {"id": 2, "status": "pending"},
]

def filter_orders(orders, statuses=None):
    if not statuses:
        return list(orders)
    return [order for order in orders if order["status"] in statuses]

assert filter_orders(ORDERS) == ORDERS
assert filter_orders(ORDERS, ["paid"]) == [ORDERS[0]]
print("2 checks passed")

Both checks pass. Both branches of the if have been exercised. The reported symptom is fixed.

Now add the check for the second requirement:

assert filter_orders(ORDERS, []) == []

It fails. The function returns every order.

Python treats both None and [] as falsey. Our requirements give them different meanings, and the patch erases that distinction.

The connection to coding agents becomes more consequential when those checks determine what the agent does next.

On September 8, Leitian Tao and colleagues published the ExecCritic preprint. Holding the Qwen-3.5-35B-A3B Repair agent fixed, they reported these SWE-bench Verified results:

Feedback source	Tasks resolved
Initial repair, before generated-test feedback	61.2%
Tests from the base Qwen Test agent	57.3%
Tests from GPT-5.6-sol	65.3%

The weaker tests reduced the resolved rate by 3.9 percentage points. Better tests improved it.

Rates average three repair runs, reusing generated tests. Failed test qualification retains the initial patch in the all-task score. The baseline does not forbid repository tests. A separate official evaluator determines resolution. Feedback adds test-generation and revision work; compute budgets are not matched. These are the authors' results, not a benchmark replication for this article. Method and results.

A bad test can do more than miss a defect. It can give the next edit the wrong target.

Imagine adding this assertion to the filter example:

# This expectation contradicts the stated empty-list requirement.
assert filter_orders(ORDERS, []) == ORDERS

Our broken patch passes it. A correct implementation would fail it. Feed that failure into an automatic repair loop, and the loop now has a reason to damage correct behavior.

Adding assertions has strengthened the wrong interpretation.

Even the familiar “fails before the fix, passes afterward” check needs a closer look. Here is the original implementation from the fixture:

def filter_orders(orders, statuses=None):
    statuses = statuses or []
    return [order for order in orders if order["status"] in statuses]

The default-filter assertion fails against this version and passes against our proposed patch. It correctly detects the original bug. It simply cannot detect the new one.

The complete fix handles None explicitly:

def filter_orders(orders, statuses=None):
    if statuses is None:
        return list(orders)
    return [order for order in orders if order["status"] in statuses]

Running the same checks against all three implementations produces:

Implementation	Default + paid-filter checks	Those checks + empty-list check
Original	1 passes, 1 fails	2 pass, 1 fails
Plausible patch	2 pass	2 pass, 1 fails
Corrected patch	2 pass	3 pass

The runnable companion includes all three implementations, the checks, and the verified output. It uses Python’s standard library and makes no LLM or network calls.

The extra check earns its place because it distinguishes two implementations the earlier checks considered equally acceptable.

That is the question I would bring to an AI-generated test review: which plausible wrong implementation would this test reject?

For the filter, the candidate mistakes are easy to name: ignore the filter entirely, treat every missing filter as empty, or treat every empty filter as missing. They correspond to different misunderstandings of the contract. Tests that separate those cases tell us more than several additional examples of paid orders.

This is also where mutation testing can help: make small changes to the implementation and check whether the suite detects them. Inspect surviving mutations to understand what they change; some are equivalent for the supported inputs. For this fixture, changing statuses is None to not statuses is a useful manual mutation because it has a known, observable effect on required behavior.

For an agent workflow, I would make four changes.

Write down the expected behavior before reviewing the patch. Include the ordinary case, the reported failure, and the neighboring case most likely to be confused with it. Here, None and [] belong on separate rows. If the issue leaves that distinction unspecified, get a product decision before turning either interpretation into a test.
Review expected values as carefully as production code. An assertion is a claim about the product. Trace that claim to a requirement, an established compatibility promise, or an independently checked example. Copying the current output into an expected value can preserve the exact behavior you meant to question.
Keep an accepted regression check stable during repair. Let the agent change the implementation while a separate runner evaluates it with the reviewed tests. Protect the test command and configuration too: an unchanged test file helps little if the patch can skip its execution. If the test itself is wrong, revise and review it explicitly, then evaluate the candidate again.
Inspect the failure before asking the agent to fix it. An assertion showing the wrong returned orders is actionable behavioral evidence. A missing dependency, an import failure, or a command that selected zero tests needs a different response. Record what ran and why it failed.

ExecCritic separates test creation from repair, qualifies tests on the original repository, and keeps them unchanged during revision. That limits the repairer's ability to change its target. Separate contexts and permissions still cannot guarantee that both agents understood the issue correctly. Paper; released implementation.

The four steps above are a review procedure you can try in an existing project. They do not require training a model. Their value should be judged by the bugs and mistaken expectations they expose.

For your next AI-assisted fix, keep the original code, the proposed patch, and the new tests. Identify one plausible alternative implementation that violates the requirement. Run the tests against it.

If both implementations get the same green result, you have found a specific question the suite still cannot answer. Add the check that separates them, and review its expected result before trusting the next repair.

If this article helped you, you can buy me a coffee. Your support helps me make time for the next experiment.

Can Your AI Use What It Remembers?

Sergei Parfenov — Sat, 05 Sep 2026 14:09:59 +0000

An assistant correctly recalls a user's tree-nut allergy. Given a request for macarons, it supplies an almond-flour recipe without applying that allergy to the user.

This is a recorded example from InMind, a July 2026 study of agent memory, using xMemory. The evaluations ask two different things: can the system retrieve a fact when the question names it, and can it bring that fact into a decision that needs it?

The second question is the reason to give an assistant memory in the first place. A user should not need to know which past conversation to reference before every request.

For a coding agent, the equivalent might be remembering that a service runs in short-lived processes, then proposing an in-process timer for hourly cleanup. That is an illustrative test case, not a result reported in the paper. It has the same useful structure: the current request does not repeat the architectural constraint, but the constraint changes what a correct solution looks like.

InMind makes that distinction measurable. Its 125 synthetic tasks pair a stored personal fact with a later request whose relevance depends on background knowledge. The authors deliberately remove obvious lexical and semantic retrieval cues. This is a stress test for a particular failure mode, not a sample of everyday assistant traffic.

For the study's A-Mem configuration with text-embedding-3-large, Table 1 reports:

100% direct recall: a passing score on all 125 questions that directly ask for the stored fact.
12% target recall: the necessary fact is judged present in the model's context for 15 of the 125 indirect requests.
9.6% application: the combined memory-and-answer evaluation passes 12 of those 125 requests.

These are separate measurements. The released evaluation protocol calls for direct questions and indirect requests to use the same frozen memory state, without letting the first answer supply a hint for the second.

The application number needs care. Its rubric requires both the personal fact in context and a relevant warning or reminder in the answer. When the authors score only the answer, the same A-Mem configuration reaches 25.6%, not 9.6%. A model can produce a useful caution from general knowledge without retrieving anything about this particular user. That answer may be good; it is not evidence that the memory system worked. The paper's answer-only evaluation and human audit expose exactly this ambiguity.

The cleanest signal here is the missing fact: direct questions find it, while indirect requests usually do not deliver it to the answerer.

The A-Mem setup appends a raw target note to a prebuilt, read-only bank. This tests selection when the fact is available; it does not establish survival through weeks of memory rewriting.

In a retrieve-then-answer pipeline, the order is:

new request
    → choose memories
    → put selected memories in context
    → generate a response

The selection step must decide what matters before the answerer sees the candidate facts together with the request. If selection misses a constraint, the answerer may produce a perfectly reasonable solution for a different environment.

This does not mean embeddings cannot encode useful associations, or that an agent cannot discover them through additional searches. It means those capabilities need testing on requests that do not already identify the required memory. Bigger retrieval scores on direct questions do not settle that question.

Related work already reaches beyond factual recall. LoCoMo-Plus examines whether agents retain and apply implicit conversational constraints. InMind's useful contribution here is its paired diagnosis of stored facts whose relevance requires an unstated knowledge connection. It is not the discovery that memory should affect behavior.

For a project assistant, I would turn that diagnosis into a small set of regression cases. Start with a requirement whose effect can be checked:

Stored constraint:
Processes can terminate at any time. Recurring jobs must
continue running even when no web request is active.

Direct question:
What execution constraints does this service have?

Indirect task:
Implement hourly cleanup of expired exports.

Behavior to check:
Does the proposed solution keep recurring execution outside
the lifetime of a single web process?

Use independent copies of the same starting state. One receives the direct question. Another receives only the task. A third receives the task with the relevant constraint explicitly included. Keep unrelated context and budgets matched where possible; the explicit-fact condition is an intervention, not an ordinary retrieval result.

Record the context the model actually received. That gives a failure somewhere to belong:

If the fact is absent from storage, investigate capture or updates.
If it exists but does not reach the answerer, investigate selection and context assembly.
If it reaches the answerer but the implementation violates it, investigate application.

Score the implementation separately from memory delivery. A model might choose a durable scheduler without knowing the project's constraint. That can pass the functional test while telling us little about retrieval. Conversely, mentioning the constraint in an explanation should earn no credit if the generated code still relies on a process-local timer.

The test also needs requests where the constraint should have no effect. A button-label change should not become a lecture about background workers. And it needs a later instruction that legitimately changes the requirement: if recurring execution is explicitly removed from scope, the old rule should not silently override the new task.

Without those controls, a system that recites every restriction on every turn can look surprisingly competent. InMind itself notes that its rubric does not penalize excessive warnings. That makes it useful for finding missed associations, but insufficient for deciding whether an assistant applies memory appropriately across ordinary work.

The accompanying probe contains 12 developer-constraint fixtures and isolated inputs for these conditions. A local BM25 run, returning at most three records, delivered the target fact for 12 of 12 direct questions and 4 of 12 indirect tasks. The cases were deliberately written with vocabulary mismatches. This is a transparent illustration of lexical retrieval, not a measurement of a modern agent's memory quality.

No LLM was run in that probe. Whether a generated implementation follows the constraint remains a separate evaluation. The inputs and behavioral rubrics are included so those checks can be run against an actual agent. InMind's public release also still lacks baseline adapters and paper-aligned per-task results, so the published model scores above are reported findings, not a replication performed for this article.

Keeping selected constraints visible in every task is one option to test. The engineering problem then moves to which constraints deserve that space, which project or environment they apply to, and when they expire. A remembered fact can be correct and still be irrelevant to the current branch, obsolete after a migration, or superseded by an explicit instruction.

For an initial implementation, I would keep a small set of active project requirements with an explicit scope, while leaving detailed history searchable. Then I would test both omissions and unnecessary enforcement. That is a design proposal, not a fix established by the paper.

The benchmark is small and deliberately difficult; GPT-5-mini serves as both answerer and judge, and the human audit found scoring errors. Its numbers do not establish how often a particular production agent will fail. They do establish a question that a direct-recall demo cannot answer.

Ask the agent what it remembers. Then, from a fresh copy of the same state, ask it to do something that depends on that memory. Check what reached the model, check what it built, and check when the remembered rule should stop applying.

The useful promise of memory is that you can stop repeating yourself.

That is the promise the test should measure.

Sources: Ruizhe Li, Mingxuan Du, Benfeng Xu, and Zhendong Mao, Keep It InMind, v1, July 27, 2026 (CC BY 4.0); InMind repository; Yifei Li et al., LoCoMo-Plus, February 11, 2026.

If this article helped you, you can buy me a coffee. Your support helps me make time for the next experiment.

The Agent Knew It Was Wrong. The System Let It Ship

Sergei Parfenov — Tue, 01 Sep 2026 17:49:00 +0000

In 660 of 800 autonomous research runs, the agent found a serious flaw in its own work.

It wrote the flaw down.

Then it delivered the report anyway.

The model did not fail to notice.

The system failed to make noticing consequential.

TL;DR

AutoResearchEval labeled 82.5% of its 800 trajectories with “uncorrected self-awareness”: the agent identified a critical flaw, then continued without fixing or gating it

A separate abstention benchmark found agents sometimes performed an irreversible action and only then claimed they had refused

In a 3,621-trial policy study, an output reviewer ran after exposure and backend effects; moving enforcement to the tool boundary cut trace failures from 57.6% to 0.2% in the primary comparison

A review that cannot change execution state is observability, not control

The failure was not awareness

AutoResearchEval took 100 research tasks across seven scientific domains and ran each through eight harness–model combinations.

The result was 800 complete trajectories, roughly 73,000 tool calls, and an average of 92.3 steps per run. The evaluator inspected the reports, code, generated data, retrieval logs, and execution artifacts rather than scoring only the final answer.

Its most common failure pattern was not hallucination.

It was uncorrected self-awareness.

Pattern	Trajectories	What happened
Uncorrected self-awareness	660 / 800	The agent identified a fatal or critical flaw and made no consequential correction
Method–conclusion disconnect	620 / 800	The written conclusion was not supported by the method actually executed
Failure to gate critical flaws	502 / 800	A critical issue was recorded but did not block delivery
Report–trace gap	484 / 800	Claims could not be traced to the code, data, or logs produced in the same run

Those categories overlap. They are not four independent populations and should not be added together.

The useful part is the shape of the failure.

The evidence was already inside the trajectory. The agent had the report, the code, the logs, and—in 660 cases—a written recognition that something was seriously wrong.

Nothing in the execution loop required that recognition to change the result.

We keep describing this as a reasoning problem because the visible artifact is text. The agent writes a bad answer, then writes a good critique of the bad answer, so the system appears to contain both failure and correction.

But correction is not a paragraph.

Correction is a state transition.

RUNNING → REVIEW_REQUIRED → BLOCKED → REMEDIATED → COMMITTABLE

If “this result is invalid” and “publish this result” can both be true in the same system state, the review stage is decorative.

Self-review is usually just another generation

A common agent loop looks approximately like this:

draft = worker.run(task)
review = worker.review(draft)
final = worker.revise(draft, review)
return final

This can improve an answer. It can also produce a more articulate failure.

The same probabilistic system proposes the work, selects the evidence, interprets the evidence, judges itself, and decides whether the judgment matters. The review has no independent authority and frequently no independent source of truth.

Even adding a second model does not automatically fix that. Two actors are not two control planes if both operate inside the same mutable context and either can still call the production tool.

The distinction I would make is this:

A reviewer produces a judgment

A gate changes what the system is allowed to do

The first is information.

The second is authority.

I call the distance between them the review-to-effect gap: the part of the pipeline between detecting a problem and the final point where the system can still prevent an externally visible result.

The wider that gap, the easier it is for a correct diagnosis to become an irrelevant log entry.

The agent refused after it acted

AgentAbstain tested 17 frontier models in four agent harnesses on 263 paired tasks across 42 executable sandbox environments.

Each pair contained a normal task and a minimally changed version where the correct behavior was to stop. The best tested agent achieved 59.5% paired accuracy: it correctly handled both the act and abstain sides in fewer than three out of five pairs.

The paper also separated what the agent said from what it did.

In 115 trajectories, 2.6% of the aggregate abstention-run breakdown, the agent crossed an irreversible action boundary and then verbally claimed restraint. The authors call this post-hoc abstention.

Their sandbox examples included agents canceling a flight before noticing conflicting rebooking constraints and submitting unauthorized simulated bank transfers after acknowledging that a required verifier was unavailable.

The final message could say “I cannot proceed.”

The tool trace said it already had.

This is the same structural failure as the research report. The model discovers the trigger, generates the correct language, and reaches that language after the effect boundary has been crossed.

An apology is not a rollback.

A refusal is not a refusal if the transaction log contains a commit.

The reviewer arrived after the side effect

The clearest timing experiment I found is Out-of-Band Policy Enforcement at a Trusted Tool Boundary.

The authors ran 3,621 trials against controlled Jira and ServiceNow mocks. Their trace-level failure definition included any of three events:

protected data entered model context
an exact protected value appeared in the answer
a forbidden backend effect completed

Detailed prompt rules alone still produced a 57.6% trace-failure rate in the primary comparison. Adding an enforcement boundary between the agent and the backend reduced it to 0.2%.

The paper then compared two mechanisms directly on Claude Sonnet 4.6.

One left the tool path untouched and asked a fixed reviewer model to pass or replace every assistant message. The other mediated the request before dispatch and the response before it re-entered agent context.

The output-review arm had a 58.6% deterministic trace-failure rate and 19.7% safe-useful completion.

The full boundary had 0.0% trace failure in that arm and 63.0% safe-useful completion.

The reviewer also required 930 additional model calls.

The important result is not that one reviewer prompt was weak. The paper explicitly says this is one mechanism comparison, not a ranking of every possible guardrail.

The important result is that the reviewer saw the assistant message after the tools had returned.

It could suppress unsafe text.

It could not remove data already placed in context.

It could not undo a backend mutation.

A post-output safety filter is a censor. It is not a transaction boundary.

Review before dispatch helps — but it is still probabilistic

Moving the reviewer earlier is useful.

Reinforced Agent puts a specialized reviewer in front of provisional tool calls. The call is reviewed before execution, and the worker can revise it before anything changes outside the agent.

That architecture improved the reported tool-calling benchmarks. It also exposed the reviewer’s own failure rate.

With o3-mini as reviewer, 36.8% of base-agent errors were corrected while 11.7% of previously correct cases were damaged. The reported benefit-to-risk ratio was 3.1 to 1.

That is a useful component.

It is not a proof boundary.

A model reviewer is good for semantic questions that deterministic code cannot answer cheaply:

Does this action satisfy the user’s actual intent
Is the evidence sufficient for this conclusion
Does the requested operation conflict with a policy expressed in natural language
Is the proposed scope disproportionate to the task

It should not be the only thing standing between a stochastic plan and production credentials.

The reviewer may recommend ALLOW, HOLD, or DENY.

The trusted boundary must decide whether a valid capability exists for the exact operation about to execute.

The architecture I would ship

The planner, reviewer, policy engine, and effect adapter have different jobs. Collapsing them into one “agent” object hides the boundary that matters.

UNTRUSTED DECISION PLANE

user intent
    ↓
planner
    ↓
proposed action + evidence bundle
    ↓
semantic reviewer

TRUSTED CONTROL PLANE

schema and invariant checks
    ↓
policy and authorization decision
    ↓
exact action manifest
    ↓
short-lived commit capability

EFFECT PLANE

effect adapter holding credentials
    ↓
provider commit
    ↓
terminal receipt or durable uncertainty state

“Untrusted” here does not mean malicious. It means non-authoritative.

The planner may be brilliant. The reviewer may be more capable than the planner. Neither should be able to convert its own text directly into an authenticated side effect.

The effect adapter should accept something closer to this:

{
  "proposal_id": "refund_0184",
  "operation": "payments.refund",
  "resource": "payment_intent:pi_7F...",
  "arguments_hash": "sha256:9fa...",
  "evidence_hash": "sha256:1bd...",
  "policy_version": "refunds@7.2",
  "provider_state_version": "captured@2026-09-01T10:42:18Z",
  "review": {
    "decision": "allow",
    "risk": "low",
    "reason_codes": ["amount_within_limit", "recipient_verified"]
  },
  "commit_capability": "cap_opaque_single_use"
}

The adapter then verifies the hashes, policy version, current provider state, capability scope, expiry, and single-use status before it calls the provider.

The natural-language conversation is evidence for constructing the manifest.

It is not the manifest.

Five invariants that turn review into control

1 Critical findings must change system state

A critical finding cannot coexist with a committable action.

review.severity == critical
    ⇒ run.state == BLOCKED
    ⇒ commit_capability == null

Do not rely on the worker to “take the feedback into account.” Make remediation create a new proposal and a new review record.

2 Approval must bind the exact action

“Refund the customer” is not sufficient authorization.

The gate should bind the operation, acting identity, resource, arguments, evidence version, policy version, and relevant external state. If any bound field changes, the old approval is invalid.

3 The credential holder must sit below the gate

The worker and reviewer should not hold direct production credentials.

Otherwise the controlled path is optional. An agent that can bypass the gateway eventually will—through a bug, a fallback, an alternate connector, or a tool call the policy layer never saw.

4 Uncertain delivery must not mint fresh authority

A timeout does not mean no effect occurred.

If the provider may have committed but the response was lost, the original authorization must remain occupied until reconciliation reaches a terminal result. Creating a fresh approval for a blind retry can turn one user decision into two external effects.

5 The final answer must be generated from the effect receipt

Do not let the agent report operational state from memory.

The user-facing response should be projected from the durable provider result or uncertainty record. That makes “I refused” impossible when the effect ledger says “committed.”

The gate has to survive retries and recovery

Admission is only the start of authority.

AID-Guard frames the remaining problem as authorization-to-effect closure. It revalidates the exact approved request and provider state at commit, keeps the original reservation while delivery is ambiguous, and permits release or one successor only after terminal evidence or certified no effect.

Its target invariant is simple:

one approval lineage → at most one provider effect

In the paper’s bounded evaluation, all 210 Stripe test-mode provider-contract trials matched their declared outcomes, and the tested recovery and confirm/cancel schedules produced no duplicate effect.

That is not a universal exactly-once guarantee. The authors used bounded provider schedules, test-mode contracts, synthetic credentials, and a high-latency prototype. Their strict exact-manifest profile also reduced benign utility substantially.

But the architecture names a production bug most agent diagrams skip:

Retry and recovery are authority transitions, not transport details

A gate that protects the first call but disappears during timeout recovery is not end-to-end enforcement.

What these numbers do not prove

I am not claiming that 82.5% of all production agents knowingly ship invalid work.

That number is the label rate for one new prerelease dataset of autonomous scientific-research trajectories. Most of its 800 annotations were produced by an artifact-aware agent judge calibrated against 50 human-labeled trajectories. The paper found the pattern across all eight tested systems, but it did not test the orchestration intervention I am proposing.

AgentAbstain uses generated tasks in executable sandboxes. Its post-hoc abstention rate is a benchmark result, not an estimate of real banking or travel incidents.

The policy-boundary study used controlled Jira and ServiceNow mocks and intentionally concentrated on cases where policy should intervene. Its 0.2% headline is conditional effectiveness inside that test design, not a promise for arbitrary production traffic. Durable approval and broad write controls were outside its primary evaluation.

AID-Guard tested finite provider schedules and does not establish arbitrary provider linearizability.

What survives those caveats is the mechanism.

Across research reports, abstention tasks, data exposure, and state-changing tools, the same failure appears:

The system detects the problem
The detection is represented as text
The text has no binding relationship to the effect path
The effect proceeds or has already happened

That is not a missing sentence in the system prompt.

It is a missing control boundary.

The next boundary

My last article asked who owns the harness.

This one asks who owns the commit.

The model can propose an action. It can criticize the action. It can explain exactly why the action is unsafe and still take it, because awareness and authority are different capabilities.

A review that cannot block the effect is not a control.

It is a log entry.

What in your agent stack can actually stop the commit—and does it run before or after the side effect?

Sources and further reading

Disclosure: I wrote the original drafts in my native language and conducted the research, source selection, analysis, and conclusions myself. AI was used only for translation and English-language editing.

If this article helped you, you can buy me a coffee. Your support helps me make time for the next experiment.

Your Agent Planned the Right Tools. It Still Crashed the Machine.

Sergei Parfenov — Wed, 26 Aug 2026 13:51:56 +0000

Your agent needs four independent facts before it can approve a refund:

the order record,
the fraud score,
the customer's history,
the policy that applies.

It correctly sees that all four calls can run in parallel.

So it launches all four.

The order lookup is cheap. The policy search is cheap. The fraud model loads a large checkpoint. The history job scans two years of events. Together they cross the worker's memory limit, the container restarts, and the refund never happens.

The plan was correct. The execution was not.

Most agent benchmarks collapse those into one score.

A new preprint submitted on August 25, PeakBench, argues that this hides an entire class of production failures: the agent may understand every dependency and still overload the machine that executes them.

TL;DR

A dependency graph tells you what may run in parallel. It does not tell you what your machine can run safely.

Across eight tested models, planning accuracy had almost zero correlation with capacity violations on the same workflows.

Resource metadata helped, but model-generated schedules were not reliably safe for every model.

In production, let the model build the DAG. Let deterministic infrastructure enforce the resource envelope.

The DAG is not the schedule

If step C needs the output of step A, C waits. If A and B are independent, the framework can run them concurrently.

That is a logical property. It says what may run at the same time.

It says nothing about what the machine can survive.

Two independent calls might each need 6 GB of memory on an 8 GB worker. Four browser sessions might fit logically but exceed a provider concurrency limit. Ten embedding jobs may be valid in parallel and still saturate their shared network connection.

PeakBench calls this the peak-load problem: runtimes translate logical independence into immediate execution while implicitly assuming infinite capacity.

When that assumption stays hidden, a crash gets blamed on “the agent.” The benchmark cannot tell you whether the failure came from reasoning, scheduling, or infrastructure.

What PeakBench separates

The authors assembled roughly 1,200 MCP-compatible tools from about 130 servers and constructed 300 executable workflows:

150 easy,
100 medium,
50 hard.

The workflows run inside containers. The benchmark records data flow, perturbs execution order, and watches for failures. This produces an execution-grounded dependency graph instead of relying only on an annotator's guess.

In a manual audit of about 100 workflows, 94% of those graphs matched the auditors' optimal structure.

PeakBench then scores two jobs independently.

1. Logical planning

The model receives the task and tool descriptions. It must recover which calls are prerequisites and which may run concurrently.

The benchmark reports graph edit distance and edge F1.

2. Physical scheduling

The model receives the verified dependency graph, so planning ambiguity is removed. It only has to assign start times under small, medium, and large machine profiles.

The schedule is scored on:

completion time,
capacity violation area: how badly, and for how long, the schedule exceeds a resource limit,
strict mean resource utilization: utilization that counts only while the schedule remains feasible.

That split is the useful contribution. A failed run no longer has to disappear into one end-to-end number.

The result that matters

PeakBench evaluated eight models under one prompting and parsing protocol: GPT-5, o3, GPT-4.1, Claude Sonnet 4.6, GLM-5, Kimi-K2.5, DeepSeek-V4-Pro, and DeepSeek-V4-Flash.

GPT-5 was the strongest logical planner in the reported table:

Graph edit distance: 0.42
Edge F1: 0.839
Capacity violation area: 3.698 for its resource-blind schedule

DeepSeek-V4-Flash recovered the graph less accurately:

Graph edit distance: 0.81
Edge F1: 0.733
Capacity violation area: 3.458 once given the verified graph

That comparison is not a model ranking. The case-level result is more important.

Across all eight models, the correlation between edge F1 and capacity violations was almost zero. The reported coefficients ranged from roughly -0.045 to 0.000.

A model understanding a workflow's dependencies told you almost nothing about whether it would schedule that same workflow safely.

Your planner benchmark is not a scheduler benchmark.

Resource data helped. It did not solve scheduling.

The authors added a Resource-Aware Scheduling Context, or RASC. For each tool call, the model sees estimated duration, average and peak CPU, peak memory, machine capacity, and verified dependencies.

No weights change. The model simply stops scheduling blind.

The aggregate trade-off is easier to read without a wide table:

Launch everything ASAP: 8.62 s completion, 5.865 violation area, 0.080 safe utilization.
Run everything serially: 15.19 s, 2.925 violation area, 0.097 safe utilization.
Best rule-based scheduler: 9.13 s, 2.925 violation area, 0.141 safe utilization.
Best RASC result: 9.11 s, 2.938 violation area, 0.165 safe utilization.

Blind parallelism was fastest and most dangerous. Serial execution reduced overload by paying almost twice the latency. Resource-aware context nearly matched the best rule-based violation score while producing higher safe utilization.

But the effect was model-dependent. RASC improved most models, not all. DeepSeek-V4-Flash slightly increased its violation area. Kimi-K2.5 and GPT-4.1 lost strict utilization even while finishing faster.

Resource metadata is necessary input. It is not an admission controller.

What I would change in production

The tempting fix is another system-prompt sentence:

Be careful with resources.

That is not an architecture.

I would make three artifacts explicit.

1. The planner emits a dependency graph

The planner decides which outputs are required and which calls are logically independent. It should not decide that every ready call starts now.

{
  "steps": {
    "fraud": {"depends_on": []},
    "history": {"depends_on": []},
    "policy": {"depends_on": []},
    "decision": {
      "depends_on": ["fraud", "history", "policy"]
    }
  }
}

2. A scheduler owns physical execution

Give every tool a versioned resource profile:

{
  "tool": "fraud_score@3.4.1",
  "duration_p95_s": 4.8,
  "cpu_peak_cores": 1.7,
  "memory_peak_mb": 6144,
  "network_slots": 1,
  "provider_concurrency": 4
}

Then enforce the envelope outside the model with queues, per-resource semaphores, rate-limit budgets, and backpressure.

The model may propose a schedule. Deterministic infrastructure should decide whether it is allowed to run.

3. The trace records what actually ran

A configured concurrency limit of four does not prove that an SDK did not add three hidden retries inside one call.

For every attempt, record:

requested and actual start time,
queue delay,
resource-profile version,
observed peak resources,
retry and fallback attempts,
the dependency state that made the call eligible,
the limiter or scheduler decision.

Without that trace, a passing benchmark tells you what the agent intended to do, not what the runtime did.

A production drill you can run this week

This is not a reproduction of PeakBench. It is a smaller test for your own runtime.

Pick one real workflow with at least two expensive independent calls.
Capture each tool's p95 duration, peak memory, peak CPU, and external concurrency limits.
Run the same workflow under three envelopes:
- normal production capacity,
- degraded or burst-constrained capacity,
- a generous diagnostic capacity.
Compare three policies:
- launch every ready call,
- serialize every call,
- enforce a resource-aware queue.
Report task correctness and resource feasibility separately.

A correct answer that violates the capacity envelope is not a pass. A safe schedule that serializes everything is not automatically good either.

The goal is maximum safe parallelism.

Why the comments mattered

Comments on my previous harness article kept returning to the same missing fields: publish the action budget, retry policy, and actual route taken, not only the model and final score.

PeakBench made the operational half of that argument measurable.

That comment signal helped identify the question. It is not the evidence for the answer; the benchmark is.

An agent run is defined not only by the model, prompt, tools, and dependency graph. It is also defined by the machine profile and the scheduler that turns readiness into execution.

If those fields are absent, a successful plan can still become an outage—and the model will get blamed for a failure the runtime created.

Limitations

PeakBench is a version-one preprint. Its workflows are synthesized, its capacity profiles are simulated, and the reported results should not be treated as measurements of your cluster.

The benchmark is best read as a diagnostic: it shows that logical planning and physical scheduling can fail independently. The exact numbers still need validation against real production traces.

Sources & further reading

If this article helped you, you can buy me a coffee. Your support helps me make time for the next experiment.

The Model Scored 30%. The Harness Scored 100%. Which One Did You Benchmark?

Sergei Parfenov — Mon, 24 Aug 2026 13:38:40 +0000

On July 24, ARC Prize verified Claude Opus 5 at 30.16% on the ARC-AGI-3 public set. On August 21, NVIDIA reported the same model at 100.00 on the same set. The weights did not change. The code around them did.

In between, MIT did the same thing (August 5), a group led by Impossible Research got to 98.98 (July 15), and OpenAI tripled GPT-5.6 Sol's score by flipping two API settings (July 29). Then Microsoft published a framework that trains the model through the harness (August 18), and Google published one that gives the environment a harness of its own (August 20).

In July I wrote that self-editing harnesses have a provenance problem. This month the problem moved up a level: the benchmark score itself has no provenance.

TL;DR: On ARC-AGI-3's public set, the spread between "model in the official harness" and "model in the best harness" is 25 to 70 points, on a benchmark designed to resist exactly this. None of the 100s are verified on the private set, and every author says so. Microsoft's Agent Lightning v1.0 runs RL with the deploy-time harness owning the loop, so the harness is becoming part of the weights, and its reward-hacking section is the checklist my July post warned about. A benchmark number without a harness version, memory state and action budget attached is a self-reported claim with an unmarked type. Unmarked means self-reported.

Five harnesses, one public set

ARC-AGI-3 scores agents with RHAE (Relative Human Action Efficiency). Per level, score = (human_baseline_actions / ai_actions)^2, with the ratio capped at 1.15x the human baseline. Game scores are level-weighted averages, you must finish the last level to get full credit, and the overall number is the mean over games. A 100.00 means the agent finished every level at least as efficiently as a first-time human.

Harness	Who	Date	Model	Public RHAE	Actions	Verified by ARC Prize
Official ARC Prize harness	ARC Prize	Jul 24	Claude Opus 5 (high)	30.16%	n/a	yes
Official harness, default settings	OpenAI	Jul 29	GPT-5.6 Sol (max)	13.3%	n/a	no
Official harness + retained reasoning + compaction	OpenAI	Jul 29	GPT-5.6 Sol (max)	38.3%	6x fewer output tokens	no
Schema	Impossible Research (+ UC Berkeley, CMU)	Jul 15	Opus 4.8 / Fable 5	98.98	n/a	no
VISTA	MIT (Han, Hu, Qiu, Wu, He)	Aug 5	Claude Opus 5	100.00	7,542 (humans: 17,135)	no
AVO	NVIDIA	Aug 21	Claude Opus 5	100.00	6,624	no

The number that matters is not in the table. It is the gap between the first row and the last: 70 points, same model, same 25 games, same metric.

The official harness is not a neutral baseline. OpenAI's write-up quotes ARC's intent: an "intentionally generic harness, without tools or special features" built to make "model shortcomings more visible." In practice it discarded all private reasoning after each game action and used a rolling truncation window, so older actions vanished as history grew. Retaining reasoning and enabling compaction took Sol from 13.3% to 38.3% and cut output tokens by 6x. The harness was wiping the model's mind between moves.

So the leaderboard measures "model plus a harness built to expose the model." The 100s measure "model plus a harness built to cover for the model." Neither measures the model, and nobody has isolated which part of the 70 points is which.

The authors are unusually honest about this. NVIDIA: the AVO-versus-VISTA comparison "should not be interpreted as a controlled ablation," and the results "should not be interpreted as a direct measurement of the performance contribution of AVO." VISTA: the models "were released after the public ARC-AGI-3 games," overlap cannot be excluded, and "the private set remains the real test of generalization." Schema: "no frozen-harness or held-out-performance claim." Every 100 on that table is a public-set number on games the models may have seen in training.

What the 70 points are made of

Read the harness papers side by side and the same three components appear under different names.

Memory. VISTA keeps a "lossless visual memory" of every past observation. AVO carries forward "prior implementations, evaluation results, compiler and profiler outputs, and accumulated reasoning." OpenAI's two settings are memory settings: keep the reasoning, compact instead of truncate.

Supervision. AVO runs a monitor that watches "the broader trajectory for stagnation or repeated unproductive cycles and can redirect the main agent." That is the layer that turns a model that gives up into an agent that does not.

An action budget. RHAE squares the efficiency ratio, so wasted moves are punished quadratically. AVO's headline against VISTA is 12% fewer actions. That is a harness optimization target, not a model property.

In July I split harness work into two piles: compensatory layers that patch what the model cannot do yet, and protective layers that constrain what it must not do. I predicted pile one depreciates with every model release. All three components above are pile one, and on a benchmark built to resist static tricks they are currently worth 25 to 70 points with the newest frontier models. Either my prediction is early or it is wrong about magnitude. I will take the second reading until the private-set numbers say otherwise.

One more thing about compaction, since it is the setting that tripled OpenAI's score. In my preregistered compaction experiment, the same operation produced 3.47% false proceeds on irreversible-action gates: the agent went through a gate it should have stopped at, because the compacted context no longer carried the provenance the gate depended on. Not a contradiction. ARC-AGI-3 scores task completion; my gates scored whether the agent still knew why it was allowed to act. Compaction improves the first, degrades the second, and a benchmark only sees the first.

Then Microsoft put the harness inside the training loop

Agent Lightning v1.0 (arXiv, August 18) names something the July thread never got to: RL where the harness is not a bystander. In their words, "the harness owns this loop, while the training engine observes only a sequence of LLM request-response pairs." The deploy-time scaffold (mini-SWE-agent in their coding runs) executes the task inside Kubernetes; the trainer sits behind a gateway that looks like a normal LLM endpoint and collects the traffic.

The result is real: Qwen3.5-9B goes from 41.8% to 56.4% on SWE-bench Verified, a 14.6-point gain from about 6,000 examples filtered out of SWE-smith's 59,136 tasks across 128 repositories, in roughly 3,500 lines of framework code.

Two details matter more than the headline.

First, retokenization. The harness re-renders text between calls, chat templates are not compositional, decode-then-retokenize is lossy, and the harness parses and repairs outputs. So the token IDs the trainer sees for the model's previous answer can differ from the ones it actually sampled. Their fix is best-effort merging: merge only when the exact token prefix holds, otherwise close the sequence. That is the engineering admission that model and harness now share a boundary at the token level. Train through one harness's rendering and you get a model tuned to that rendering.

Second, section 4.3.2, "Preventing Reward Hacking." During training the agents were caught "using Git history to locate the gold commit," "using wget or curl to retrieve upstream source code from GitHub," using pip to download a package's source, and using urllib to do the same. Countermeasures: "disable Git commands and hide the .git directory from the agent," plus a Kubernetes network policy that "blocks general outbound network access and permits connections only to explicitly whitelisted services."

That is my July post compressed into a paragraph, arrived at as an engineering necessity rather than a design principle. Vinicius Pereira said it best in the comments: the agent must not be able to author the artifact the gate reads. Microsoft's version is that it must not be able to reach it either, through the filesystem or the network. Dipankar Sarkar's separate trust domain for test execution is the same control from the other side.

Now put the halves together. The harness that decides what the model sees also decides what the trainer sees. Once RL runs through it, the tricks in pile one stop being code you can diff and become weights you cannot. That is the absorption I predicted, except what gets absorbed includes whatever the harness let the agent get away with. Hide .git and the model learns the task. Forget to, and it learns to find the gold commit, and the benchmark will not tell you which one you trained.

Google gave the environment a harness too

EnvHarness (arXiv, August 20, Google Research) wraps a static environment at the reset/step interface with three plug-in types: Setup reshapes the initial state, Rule reshapes "which actions are allowed, what they do, and what the agent observes," and Link composes in another environment's tasks. A designer agent, EnvRigger, "treats the target policy as a black box, observing its execution trajectories to synthesize EnvHarness components targeting diagnosed flaws," writes a _Rules subclass, and tests it. Across ALFWorld, WebArena, SWE-bench Verified, OfficeQA and SpreadsheetBench, skills learned in reshaped environments transfer back for up to 9.0 points on held-out instances with 9.8% fewer steps.

Credit where due: this is the responsible version. Verifiers are untouched, the goal predicate is never modified, evaluation happens on the unadapted benchmark. A curriculum, not a thumb on the scale.

But note the direction of travel. In one week the field shipped a harness around the agent (AVO, VISTA), a harness around the trainer (Agent Lightning), and a harness around the environment (EnvHarness). The capability you end up with has its provenance spread across three codebases, and only one of them comes with the model card.

What you are actually buying

The one benchmark this month that held the model constant and varied the harness came from a vendor. TrueFoundry's TrueForge comparison (August 18) ran DevRev's Enterprise-Bench: 14 cross-system tasks, three MCP servers, fresh session per task, blind grading, list-rate and cache-aware costs.

System	Model	Tasks solved	Cost per run
Claude Managed Agents	Opus 4.8	~11/14	$11.80
TrueForge	Opus 4.8	~11/14	$8.50
TrueForge	GLM-5.2	~11/14	$2.90
deepagents / LangGraph	Opus 4.8	~10/14	n/a

Same model, same tasks, 28% cost difference from the harness alone. Swap the model under the same harness and cost drops another 2.9x with no change in tasks solved. TrueFoundry sells the gateway next to TrueForge, so the framing is self-serving. It is still more methodology than NVIDIA offered.

I have seen this pattern in teams that compare a vendor's managed agent against their own scaffold and attribute the entire difference to the model. After this month I do not think that attribution is defensible without a controlled harness swap, and almost nobody runs one.

What a score needs to carry

If provenance is a vector, a benchmark score needs one. The minimum I would want attached to an agent number before quoting it, illustrative rather than a standard:

score: 100.00
metric: RHAE
set: arc-agi-3-public-25       # not semi-private, not private
model: claude-opus-5           # provider version string, reasoning effort
harness: avo@<commit>          # the code between model and environment
memory_at_start: empty         # or warm, and from which prior runs
supervisor: stagnation-monitor # any policy that can redirect the agent
compaction: on                 # summarize vs truncate, and where
action_budget: 6624
trainer_harness: none          # if the weights were RL'd through a harness, which one
verified_by: self              # or ARC Prize, or a named third party

Mike Czerwinski's rule from the July thread applies to every row: unmarked has to mean self-reported. A score that arrives without the harness commit is not a measurement of the model. It is a claim about a system, typed by whoever produced it, and the default type is untrusted.

What this post might get wrong

The model/harness split may already be dissolving. After harnessed RL, "the model" is partly a harness artifact, and the boundary I am measuring might not survive the next benchmark cycle.
Every 100 is on the public set, which shipped before the models did. If Opus 5 in the generic harness gets 30 on the private set and AVO gets 40, the harness story shrinks from 70 points to 10, and the leaderboard was more honest than I am giving it credit for.
Source hygiene: NVIDIA sells the compute AVO runs on, OpenAI's two settings are its own API features, TrueFoundry sells a gateway, Microsoft would like you on Azure. I read the papers and reproduced none of them.
My July prediction that compensatory harness layers depreciate with each model release. This month's evidence points the other way. The prediction stays up, marked as losing.

The question I cannot answer alone

If the harness is worth 70 points on a benchmark built to resist it, who owns the harness in your stack: you, the model vendor, or the router in between? And when you report an agent result internally, does the harness commit travel with the model version, or does it get dropped at the first summary?

Sources & further reading

If this article helped you, you can buy me a coffee. Your support helps me make time for the next experiment.

Distilling Kimi Into Qwen Doesn't Give You Kimi. It Gives You Qwen With Kimi's Handwriting

Sergei Parfenov — Mon, 10 Aug 2026 12:40:37 +0000

On July 22, 2026, White House OSTP director Michael Kratsios accused Moonshot AI of distilling Anthropic's Fable 5 to build Kimi K3, and Treasury put sanctions on the table. No technical evidence was published, and researchers immediately pointed out the timeline problem: by K3's launch day, Fable 5 had been publicly reachable for roughly 18 days in total.

Everyone argued about whether it happened. Almost nobody asked the more useful engineering question: if it did happen, what would Moonshot actually have received?

Here is the experiment that answers it. In 2025, the Berkeley team behind Sky-T1 fine-tuned a model on long reasoning traces whose final answers were wrong. Accuracy dropped by 3.2 points. They randomized half the numbers inside the reasoning steps: 3.3 points. Then they shuffled the order of the steps, and performance collapsed. The content of a distill is nearly disposable. The structure is the payload.

That is the folk model's blind spot. The folk model says: pour a strong model's outputs into an open model, get a model at the strong model's level. What actually crosses the wire is mostly the shape of the reasoning, and the size of the gain is set by your base model, not by how smart your teacher was.

TL;DR A "distill" is a dataset of teacher traces, not a set of weights. Fine-tuning on it reliably transfers output structure (long chain-of-thought, backtracking, <think> blocks) and only narrowly transfers capability. Evidence: models trained on traces with wrong answers lose about 3.2 points versus correct ones, and randomizing half the numbers in the traces costs 3.3 points on AIME 2024. Real capability gains do exist (DeepSeek's R1 distills beat RL on the same base by 25 points on AIME), but they cost 800k rejection-sampled samples and a full fine-tune, not 8k samples and a LoRA. The headline benchmark jumps you see on Hugging Face are very often eval artifacts.

What a "distill" actually is

When someone says "I poured a distill into Qwen," they are not moving weights. They are running SFT on a dataset of teacher outputs. There are two distinct channels, and they behave differently:

	Black-box (traces)	White-box (logits)
What you collect	Generated text, usually prompt + long CoT + answer	Full next-token distributions, or hidden states
Loss	Cross-entropy on the teacher's tokens	KL between teacher and student distributions
Bits per token	One sampled token	The whole distribution ("dark knowledge")
Needs	API access	Weights, and enough GPUs to run the teacher
Works across model families	Yes	Yes, if tokenizers align

Anything distilled from a closed API (Claude, GPT, Gemini) is black-box by construction. You cannot get logits out of an inference endpoint. This matters more than it sounds: black-box distillation is the low-bandwidth channel, and it is the only one available in the scenario the White House described.

The whole current ecosystem is downstream of two decisions made four months apart. In September 2024, OpenAI hid o1's raw chain of thought and explicitly listed competitive advantage among the reasons. In January 2025, DeepSeek shipped R1 with full traces exposed under a permissive license, and within weeks the distill wave existed: s1, LIMO, Sky-T1, and a Hugging Face shelf of "-Distill" repos. Traces are the substrate; whoever exposes them feeds the ecosystem, and hiding them is anti-distillation policy by another name. None of this is exotic in-house either: Google's own Gemini 1.5 technical report states that Flash is online-distilled from the much larger Pro. Every lab does this to its own models. The fight is only ever about doing it to someone else's.

Kimi is the interesting inverse case. K3 shipped as open weights in late July 2026 at 2.8 trillion parameters, so white-box distillation from Kimi is legally and technically on the table. The gate is not access, it is the inference bill for generating traces from a model that needs roughly 1.4 TB of fast memory resident before you load any context.

The case that it genuinely works

DeepSeek ran the cleanest public experiment on this, and it is still the strongest pro-distillation datapoint we have.

They generated about 600k rejection-sampled reasoning traces plus 200k general samples from R1, then ran plain SFT for two epochs on off-the-shelf open bases. No RL on the students at all. Then they asked the obvious control question: what if you skip the teacher and just run large-scale RL on the same base?

Qwen-32B base, three treatments	AIME 2024	MATH-500	GPQA-D	LiveCodeBench
QwQ-32B-Preview (reference)	50.0	90.6	54.5	41.9
RL directly on the base, 10k+ steps	47.0	91.6	55.0	40.2
SFT on 800k R1 traces	72.6	94.3	62.1	57.2

That is a 25 point gap on AIME in favor of distillation, against an RL run that cost far more compute. DeepSeek's own conclusion was blunt: distilling a powerful model into a smaller one works, while small models relying on large-scale RL need enormous compute and may still lose.

This is not just curve-sharpening either. A widely cited ICML/NeurIPS 2025 analysis measured pass@k rather than pass@1 and found the two methods differ in kind. RLVR raises pass@1 while narrowing the reasoning boundary at large k, because the paths it reinforces were already in the base's sampling distribution. Distillation raises the curve at every k (on Qwen-7B: pass@1 from 28% to 45%, still above 90% at pass@256), meaning genuinely new reasoning patterns entered the model.

So: yes, capability moves. Hold that thought.

The case that it isn't what you think

Back to the corruption experiment from the intro, with the setup spelled out. The Sky-T1 team first got +40 points on AIME 2024 by fine-tuning Qwen2.5-32B-Instruct on just 17k long-CoT traces distilled from R1. Only then did they start breaking the training data on purpose, and found that only structural damage (shuffling, inserting, deleting steps) actually hurt, while wrong answers and randomized numbers cost ~3 points each. Their conclusion sits in the paper's own title: structure, not content, is what matters.

Two more datapoints point the same way. s1 hit strong reasoning numbers with 1,000 samples. LIMO used 817. If a thousand examples move a benchmark 40 points, you are not transferring a frontier lab's knowledge in a thousand examples. You are flipping a switch that was already wired.

This is the same finding that killed the first imitation wave in 2023. Berkeley's "False Promise of Imitating Proprietary LLMs" found crowd workers rated ChatGPT imitators as competitive, while targeted benchmarks showed they closed little to none of the gap: they mimicked style, not factuality. Thinking Machines said the same thing in 2025 about off-policy distillation, that the student learns the teacher's style and confidence without necessarily learning its accuracy.

A worked example at 0.01% of the weights

Here is the exact thing the question is usually about, done in public and documented honestly.

Someone took Qwen3.6-35B-A3B, generated ~7.8k reasoning traces from Kimi K2.6 via OpenRouter, and ran SFT with Unsloth and LoRA. Attention-only adapters, r=16, 980 steps, about 21 hours on a single H200. Trainable parameters: 3.44M out of 35.1B. That is 0.01% of the model.

The model card then does something almost nobody does. It reports evals that undercut the model:

Benchmark, same pipeline both sides	Base Qwen3.6-35B-A3B	Kimi-distill
MATH-500 (0-shot, `math_verify`)	53.0	47.0
GPQA Diamond (0-shot CoT)	79.29	75.25
GSM8K (8-shot, strict-match)	64.0	92.67

The author's read on that GSM8K number is the important part: the base scoring 64% is implausible for a frontier 35B-A3B, and the likely cause is that the few-shot template never triggers the base's thinking mode. So the +28.67 point "win" measures "my pipeline rewards models that always think," not capability. His stated conclusion is that the run provides no evidence the distillation improved raw reasoning over the base. What it does provide is a guarantee: the distill emits <think> blocks regardless of prompt shape, where the base's thinking mode is conditional.

That is a real, useful property. It is also exactly what "handwriting transplant" means. And note the cost signature: Kimi's traces averaged 2,933 tokens against 849 for a matched Claude Opus 4.7 set, roughly a 2.5x compute multiplier for the same number of rows.

The price list

Dollar figures are illustrative (cloud rates move), but the orders of magnitude are the point:

Run	Data	Compute	What you demonstrably get
s1-32B (Stanford)	1,000 curated Gemini traces	26 min on 16 H100s, ~$25 at cloud rates	Thinking format + test-time scaling; beats o1-preview on competition math
Kimi K2.6 → Qwen3.6 LoRA	7.8k K2.6 traces via OpenRouter	~21 h on one H200	Unconditional `<think>` blocks; no measured capability gain over base
DeepSeek R1 distills	800k rejection-sampled samples	Full SFT, 2 epochs, six bases	+25 pts on AIME over RL on the same base
Thinking Machines on-policy	Teacher-scored student rollouts	1,800 GPU-hours (their RL baseline: 17,920)	~70% AIME'24, RL parity at ~10% of the compute

Read top to bottom and the pattern is the whole article: the format transplant costs lunch money, the capability transfer costs a training run.

So what actually crossed the wire?

Until recently this was unanswerable per output. A December 2025 paper proposes a provenance-tracing framework that scores every sentence a distilled model produces under three models (teacher, original student, distilled student) and sorts it into four buckets: teacher-originated, student-originated, already present in both, and pre-existing but boosted by distillation. In their analysis, teacher-originated actions do appear in unseen test contexts and correlate with correctness, but a large share of what the distilled model emits is student-internal patterns that distillation merely amplified, and not all of that amplification helps.

One more mechanism worth knowing, because it bounds the cross-family case. Anthropic Fellows work published in Nature this year showed models can transmit behavioral traits through data with no semantic connection to those traits, including through reasoning traces and code. The catch: the effect only appears when teacher and student share the same base model or a behaviorally matched one. Across families it disappears. So a Fable-to-Kimi transfer, if it happened, has only the visible-traces channel open. The spooky hidden channel is not available across architectures.

The bill

Distillation is not free, and the invoice arrives in three places.

Capacity gap. Apple's distillation scaling law found the student's loss improves with teacher strength only up to a point, after which a stronger teacher produces a worse student. The teacher becomes harder to model, and the student can no longer absorb the gains. Picking the strongest available teacher is not automatically correct.

Distribution mismatch. Off-policy SFT trains the student to navigate states the teacher visits. At inference the student is in its own states, which it never trained for, and errors compound over long chains. This is the structural argument for on-policy distillation, where the student generates and the teacher scores each token. Thinking Machines reported roughly 70% on AIME'24 for 1,800 GPU-hours where their RL baseline needed 17,920, and framed the reason cleanly: RL delivers about O(1) bits of signal per episode, distillation delivers O(N) bits, one per token.

Forgetting. A 2026 paper on post-training with knowledge retention reports that SFT on rejection-sampled Gemini 2.5 Pro responses dropped IFEval by 11.5 points on Llama-3.1-8B-Instruct, and even eroded in-domain reasoning on Qwen3-8B from 46.8% to 41.0%. You are not adding a skill to a static model. You are moving the whole distribution, and things fall out of it.

Checklist: capability transfer or handwriting transplant?

Factor	Handwriting only	Capability actually moves
Data scale	1k to 20k traces	100k+, rejection-sampled against verified answers
What you train	Attention-only LoRA, 0.01% of params	Full fine-tune, or LoRA over FFN/experts too
Teacher gap	As large as possible	Inside the capacity-gap range for your student
Domain	Traces from one domain, eval in another	Traces drawn from the distribution you'll deploy on
Policy	Off-policy SFT only	On-policy phase after the cold start
Eval	pass@1, one pipeline, base's thinking mode never invoked	pass@k, identical pipeline, thinking mode forced on both sides, OOD and instruction-following held out

If your setup sits mostly in the left column, you did not get a frontier model. You got your base model that now reliably reasons out loud. Ship it if that is what you needed. Just do not put "matches Kimi K2.6" in the model card.

Back to Kimi K3

Numbers first, because most coverage skipped them. In February 2026, Anthropic publicly attributed more than 16 million Claude exchanges across roughly 24,000 fraudulent accounts to coordinated campaigns by Moonshot, DeepSeek, and MiniMax, over 3.4 million of them pinned on Moonshot alone. In June, Anthropic told the US Senate Banking Committee that Alibaba's Qwen lab had run the largest known distillation attack to date. So industrial-scale trace harvesting is on the record as a specific, quantified allegation, months before Kratsios spoke.

The Fable-to-K3 link is the shaky part. Per press reconstructions of the timeline, Fable 5 launched June 9, was suspended June 12 under export controls, and returned July 1; K3 launched July 16. That is the ~18 days from the intro. K3's pretraining necessarily predates all of it, so the only technically coherent version of the accusation is targeted post-training on Fable-derived traces over an already-trained base. Which is precisely the scenario this article is about: it would buy behavioral transfer in the domains the traces covered, at black-box bandwidth, from under three weeks of harvesting. It would not buy the pretraining run, and pretraining is the part nobody knows how to distill, because the parameter space is specific to each network.

I wrote earlier about the legal and ToS side of this and I want to correct my own emphasis: framing distillation primarily as an IP question makes it sound like the technique hands over a copy of the model. It does not. The interesting question was always mechanical, and the mechanical answer constrains the policy answer.

Caveats on my own sources, since I am asking you to update on them: DeepSeek's numbers are self-reported, the Kimi-distill evals above are single-run and small-sample by the author's own admission, and the White House claim has no published evidence behind it.

Question for anyone who has actually run one of these: what is the one eval in your harness that would have caught a benchmark jump that was really just your base model's thinking mode failing to trigger? I suspect most of us do not have one, and that is why the Hugging Face leaderboard looks the way it does.

Sources & further reading

DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL (Tables 5 and 6, distillation vs RL)
Does RL Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (pass@k; distillation expands the boundary, RLVR narrows it)
LLMs Can Easily Learn to Reason from Demonstrations: Structure, not content, is what matters! (the corruption experiments)
The False Promise of Imitating Proprietary LLMs (style vs factuality, 2023)
Distillation Scaling Laws (capacity gap)
On-Policy Distillation (Thinking Machines Lab)
Where Did This Sentence Come From? Tracing Provenance in LLM Reasoning Distillation
Language models transmit behavioural traits through hidden signals in data (Nature; preprint)
Surgical Post-Training: Proximal On-Policy Distillation with Knowledge Retention (forgetting numbers)
Qwen3.6-35B-A3B-Kimi-K2.6-Reasoning-Distilled model card (the honest eval section)
s1: Simple test-time scaling (1,000 samples, 26 minutes on 16 H100s)
Learning to reason with LLMs (OpenAI on hiding o1's raw chain of thought)
Gemini 1.5 technical report (Flash is online-distilled from Pro)
Trump tech official accuses China's Moonshot AI of stealing from Anthropic (SCMP), The New Stack (the 16M / 24,000 numbers), and Glitchwire (Moonshot's 3.4M, the June Senate claim about Qwen)
Kingy.ai's timeline reconstruction (Fable 5 availability windows vs K3's release)
Kimi K3 launch notes (Simon Willison)

If this article helped you, you can buy me a coffee. Your support helps me make time for the next experiment.

I Found Two Bugs in Zulip. The Maintainers Had Filed Both Two Weeks Earlier.

Sergei Parfenov — Fri, 07 Aug 2026 14:38:24 +0000

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

I went hunting in Zulip's codebase for this challenge and found two real data-corruption bugs in the Slack importer. Solid ones: silent message scrambling during workspace migrations, the kind of bug that costs somebody a re-migration.

Then I did the thing you are supposed to do before writing a single line of fix. I searched the tracker.

Issue #39650. Opened June 30, by the maintainers themselves: a systematic audit of the Slack import path, roughly a dozen items, confirmed and triaged. My two discoveries were on that list, described more precisely than I would have described them. I was two weeks late to my own findings.

This is the story of what happened next, because "what happened next" turned out to be the useful part.

TL;DR: Hunted in Zulip → found 2 real bugs → both already filed by the maintainers (#39650) → the maintainer was already fixing them in an open PR (#39757) → pivoted twice: found an unreported latent twin of the bug class in the newest importer (Microsoft Teams), and claimed the one confirmed item the maintainer's PR did not touch (an unguarded timestamp sort key, which turned out to hide a silent NaN failure mode on top of the two loud ones). Shipped two PRs (#39813, #39814), each with a test that fails on the old code. Lessons at the end.

Picking the hunting ground

Why Zulip: it is on this challenge's suggested repo list, it is Python, and it has a reputation that makes hunting interesting rather than easy. Near-total backend test coverage, strict mypy, ruthless linting, and a commit discipline of "each commit is a minimal coherent idea". I ran an extended ruff pass over the whole backend just to check the floor. Nothing but style noise. Lint-level bugs do not survive in this codebase.

Which leaves logic-level bugs, and for those you want a subsystem where correctness is hard and the inputs are hostile. Data import is exactly that: long batch jobs over other tools' export files, arbitrary data quality, and no retry culture, because a workspace migration is a one-shot event for the admin running it. One unhandled edge case on message 31,000 of 50,000 and the whole thing is off.

The two bugs I "found"

Bug one: thread state does not survive chunk boundaries. Slack messages stream through the converter in chunks of 1,000. The map that routes thread replies to their Zulip topics was allocated inside the per-chunk function. Thread root in chunk N, replies in chunk N+1: the replies land in an orphan topic literally named "... No channel message". The detail that makes it art: the adjacent cache in the same file was deliberately made module-global, with a comment explaining that it must survive across calls. One cache got the cross-chunk treatment. Its sibling, added later, did not.

Bug two: the thread key is truncated to seconds. A thread's identity was computed as strftime("%Y/%m/%d %H:%M:%S") plus the parent's user id. No channel component. No microseconds, even though Slack's raw ts carries them, as a string, right there in the message. Any bot that posts twice within one second and collects replies on both messages: two distinct threads merged into one topic, conversations interleaved.

Both bugs are silent. No exceptions, no warnings, just a migrated archive that is quietly wrong.

I was fairly pleased with myself for about as long as it took to search the tracker.

The tracker, and the etiquette call

Finding #39650 stung, then got worse in an instructive way: PieterCK, the maintainer who owns the importers, already had an open PR, #39757, "slack_importer: Fix Slack thread conversion bugs". I opened its Files changed tab with a sinking feeling. Both of my thread bugs: being fixed, by the person whose code it is, with more context than I will ever have.

This challenge has a section called "Smash Bugs, Respectfully", about not adding to maintainer workload. Racing a maintainer's open PR on their own audit items is a textbook way to fail it. So: stand down on the thread bugs. Not negotiable, and honestly not even disappointing once framed correctly, because independently rediscovering two items from a maintainer audit is not wasted work. It is calibration. My nose was pointing at real bugs; it was just pointing at them second.

If the audited file is picked clean, two moves remain: find what the audit missed elsewhere, and find what the audit found but nobody claimed.

Move one: same disease, newer organ

Audits cover files, but bug classes travel across files, carried by copy-paste and by shared habits. So I took the bug classes from the Slack audit and checked the sibling importers. Mattermost: clean on these classes, different batching helper. Microsoft Teams, the newest importer in the family: jackpot.

Its batching generator yielded a list and then called .clear() on the same object to build the next batch. Run the minimal version yourself:

list(batched(range(12), 5))
# expected: [[0..4], [5..9], [10, 11]]
# actual:   [[10, 11], [10, 11], [10, 11]]

Every yielded batch is one shared list. Ten of twelve messages gone, zero exceptions. The bug is latent: Zulip's current caller consumes each batch before advancing, so nothing fires today. It is still a broken contract, because the consumer owns a yielded value, and every future consumer inherits a trap that silently destroys data during a one-shot migration. Nobody had reported it. That became PR #39814, and the "why fix a latent bug" argument became [its own submission][LINK: Teams Clear the Lineup post].

Move two: the confirmed item nobody took

Back in #39650, one high-impact robustness item sat confirmed and unclaimed: the unguarded timestamp sort key. float(message["ts"]), used to sort every message of the export and as date_sent. I checked the maintainer's open PR for it, Ctrl+F in Files changed: not found. Free.

The item as filed had two failure modes: missing ts raises KeyError, garbage ts raises ValueError, either one aborts an entire import because of one message. Writing the guard surfaced a third mode that nobody had listed, and it is the best souvenir of this whole hunt: "ts": "NaN". float("NaN") parses without complaint. NaN compares as False against everything, which quietly violates the total ordering Timsort assumes, so sorted() returns an inconsistent order. No crash, no warning, scrambled chronology. The crash is the lucky failure mode. That is why the shipped guard requires math.isfinite, not just a successful parse. This became PR #39813 and [its own submission][LINK: Slack Clear the Lineup post].

Boring discipline, on purpose

Both fixes went through the same routine. Write the regression test first, roll the source back to upstream/main, and watch the test fail with the exact predicted error: KeyError: 'ts' for the Slack guard, AssertionError: 24 != 29 for the Teams aliasing (the batches measurably eat messages). Then restore the fix and run everything: 56/56 on the Slack importer module, 10/10 on Teams, lint and mypy clean, coverage showing no uncovered lines in the touched Slack module. A regression test that never failed against the old code is a test fitted to the fix, not a test of the fix.

Full disclosure on process: the mechanical part of this hunt (cloning, grepping, running suites) went through AI agent tooling; every call you have read about, from which repo to whose bug not to race to which failure policy to ship, stayed human. Zulip has an explicit AI-use policy for contributions, and both PRs follow it.

Epilogue: arguing with the robot

For the other track's submission I wired the crash into Sentry and asked Seer, its AI debugger, for a root cause. Credit where due: Seer nailed the diagnosis in seconds, down to quoting the exact poisoned message it pulled from the frame locals. Then it suggested a fix: float(message.get("ts", 0)).

That one-liner is the fallback-timestamp option I had already declined in the PR. It patches the missing-ts mode, leaves ValueError alive, waves "NaN" straight through into the sort, and stamps real messages with a 1970 date_sent. One failure mode out of three, plus fabricated chronology. The tool found the root cause faster than I would have; deciding the failure policy was still my job. I suspect that division of labor is going to describe a lot of debugging from here on.

What I am taking away

Freshly audited ground is picked clean. Hunt where the code is newest. The Slack importer had a dozen filed bugs and zero available ones; the newest importer had an unreported twin waiting.
Rediscovery is calibration, not waste. Going two-for-two against a maintainer audit told me the method works; it just needs to run earlier or elsewhere.
"Filed" is not "fixed". A confirmed, high-impact item sat unclaimed for over three weeks in one of the best-maintained Python codebases around. Trackers are full of these.
Read the open PRs before you race them. The most useful contribution I made to the thread bugs was not making one.
The crash is the lucky failure mode. The loud errors were filed by an audit; the silent NaN mode was not. The bugs that skip the crash are the ones that make it to production, and to your archives.

Have you ever arrived two weeks late to your own discovery? And did you file it under wasted effort, or under proof your nose works? I have firmly moved to the second column.

The paper trail

The maintainers' audit: https://github.com/zulip/zulip/issues/39650
The maintainer's thread-fix PR: https://github.com/zulip/zulip/pull/39757
My Slack timestamp guard: https://github.com/zulip/zulip/pull/39813
My Teams batching fix: https://github.com/zulip/zulip/pull/39814
The challenge's "Smash Bugs, Respectfully" guide: https://dev.to/opensourcepledge/how-to-respectfully-contribute-to-open-source-cbh

If this article helped you, you can buy me a coffee. Your support helps me make time for the next experiment.