DEV Community: syntaxixr

On Windows, four processes appending to one file lost 37 of 200 lines

syntaxixr — Thu, 15 Oct 2026 16:21:01 +0000

My CI failed with this, only on the Windows runner, on a commit that changed nothing but a few GIFs:

receipts: Unexpected token '\', "\hostedtoo"... is not valid JSON

The same test had passed on that runner a few hours earlier. Here is what it was, because it is a small bug that is easy to ship.

The setup

Receipts runs the tests a change added or edited twice: with the change, and with the changed source files put back the way they were. For pytest it loads a tiny plugin that writes one JSON line per test result into a file, and the Node side reads that file afterwards.

def _emit(record):
    out = os.environ.get("RECEIPTS_PYTEST_OUT")
    if out:
        with open(out, "a", encoding="utf-8") as fh:
            fh.write(json.dumps(record) + "\n")

One file, opened in append mode, one line per record. Fine, until the project uses pytest-xdist. Then several worker processes run the tests, and at the end every one of them writes a record with the list of modules it imported. Those records are long, full of paths like \hostedtoolcache\windows\Python\.... That is the fragment in the error: the start of a line that got torn in half.

The repro

Four processes, each appending the same long JSON line 50 times to one file:

import json, os, multiprocessing as mp

def worker(out, n):
    rec = {"files": [r"\hostedtoolcache\windows\Python\Lib\m%04d.py" % i for i in range(400)]}
    for _ in range(n):
        with open(out, "a", encoding="utf-8") as fh:
            fh.write(json.dumps(rec) + "\n")

if __name__ == "__main__":
    out = "race.jsonl"
    ps = [mp.Process(target=worker, args=(out, 50)) for _ in range(4)]
    [p.start() for p in ps]; [p.join() for p in ps]
    print(sum(1 for _ in open(out, encoding="utf-8")), "lines")

On my Windows machine: 163 lines instead of 200. 37 writes simply vanished. With records of different lengths you also get torn lines instead of clean losses, which is what CI hit.

Why

On POSIX, O_APPEND means "move to the end and write" happens as one step for each write(), so concurrent appenders don't overwrite each other. On Windows, the C runtime emulates append mode: it seeks to the end, then writes. Two processes can both seek to the same end and the second write lands on top of the first. Nothing errors, data is just gone.

It only showed up now because it needs two xdist workers to finish within the same moment. It had been passing for weeks.

The fix

Stop sharing the file. The controlling process keeps out.json, each xdist worker writes out.json.<worker id>, and the reader picks up all of them:

worker = getattr(_config, "workerinput", None)
if worker is not None:
    out = "%s.%s" % (out, worker.get("workerid", os.getpid()))

While I was there, the reader stopped throwing on a line it can't parse: a run killed for hanging can leave a half-written last line, and the records before it still count.

The regression test doesn't try to reproduce the race (that would be a flaky test about flakiness). It checks the mechanism directly: a worker writes to its own file, the controller to the main one, and a torn line at the end doesn't sink the read. It fails on the old plugin and passes on the new one, which, given what the tool is for, felt like the least I could do.

Takeaway

If several processes append to one file, it works on Linux and macOS and loses data on Windows, quietly and rarely. Give each process its own file, or take a lock.

I drafted this post with an AI assistant. The error, the numbers and the code are from the actual CI run, my machine and the commit: https://github.com/syntaxixr/receipts/commit/70bb9b2

A reader picked apart my benchmark. Here is what survived, and what changed

syntaxixr — Wed, 14 Oct 2026 15:14:00 +0000

Last week I published a study of 181 real code changes, and one sentence in it was wrong. A reader found it by re-adding my numbers. This is what they found, what I changed, and what is still open.

The sentence

I'd written that in 10% of agent-written pull requests every test failed on the old code only because the test file imports a name the change adds (so the file can't even load), and that "no maintainer commit in my sample looked like that."

The reader re-derived my results files before reading the write-up. Every sum held. Then they pointed at a line I'd skipped: the maintainer file lists 9 tests with the verdict WEAK out of 212 (4%), while the agent file lists 79 out of 591 (13%). So the pattern isn't absent on the maintainer side.

I went and looked. All nine come from one marshmallow feature commit that added a new keyword argument. Its test file loads fine, and the same commit also has tests that do prove the change. So the sentence was about "every test is WEAK", and the fix was to say what the maintainer data actually contains. By change, 14 of 91 agent PRs contain a WEAK test, against 1 of 71 maintainer commits.

The split that made it more interesting

The reader's next question: does the import pattern show up even when something else already proves the change, or only when nothing does? Splitting the 91 agent PRs by whether they have a proving test:

With a proving test: 5 of 80 carry a WEAK test (6.3%, 95% interval 2.7 to 13.8).
Without one: 9 of 11 (81.8%, interval 52.3 to 94.9).

If WEAK tests were written at the same 6.3% rate everywhere, you'd expect about 0.7 of the 11 to carry one. There are 9. So most of it sits exactly where nothing else proves the change. The reader also pointed out the catch: my change-level rule never calls a change Proven if it carries a WEAK test, so part of that is the aggregation rule read back. The 5 of 80 is the only number that isn't, because those PRs ship a proof and still add a WEAK test. That's an authoring rate, and it's small but not zero.

I checked their statistics myself (Wilson intervals, Fisher exact tests). They hold. The maintainer side is a different story: 1 of 71 is one event, so it can't tell anyone the base rate, however clean the contrast looks.

What changed

The study doc now says all of the above, with the intervals, and one definition of "weak only" everywhere (my generator and two results files disagreed with the doc).
Receipts, the tool, now prints how each PROVEN test failed on the old code. A different reader, on Reddit, said "it went red" isn't enough: you want to see that the test reached its assertion, not that it died on an import. The tool knew that and wasn't showing it.
I wrote down the next step before running it: a few hundred more maintainer commits, as their own row, with one selection rule applied to both sides. The reader predicted the maintainer rate would move up toward the agent rate, because the one maintainer event is itself a churn-shaped commit. That prediction is in the doc now, so whatever comes back gets read against it.

What I took from it

Two things. One: re-derive someone's sums before you argue with their sentence, because the sentence is where it's loose. Two: a nice headline number ("10% of agent PRs") can be a fact about your verdict rules as much as about the code, and the only way I found that out was a reader who wasn't invested in the answer.

The tool is open source and free: https://github.com/syntaxixr/receipts. The study with every change linked: https://github.com/syntaxixr/receipts/blob/main/docs/study.md

Heads up: I drafted this post with an AI assistant. The numbers are from the study files in the repo and from the comment thread under the original post.

Claude Code's /goal judge said "done" in 17 of 17 runs. 8 of them were broken.

syntaxixr — Sun, 04 Oct 2026 12:22:31 +0000

I like /goal in Claude Code. You say what "done" looks like and Claude keeps working until it gets there. What bothered me is who decides it got there.

After every turn a small model (Haiku) reads the conversation and answers "met", "not yet" or "impossible". I tested what that judge can see. It never runs a command and never opens a file. If the agent writes "All done, tests pass!", the judge can just believe it.

So I measured it.

The benchmark

Four small tasks: a ledger, a text toolkit, a job queue and a spreadsheet engine. Each one has a hidden grader that the agent never sees. I ran Sonnet 5.5 and Haiku 4.5 on them, with plain /goal and with my plugin, and graded every run three times. 33 valid runs in total.

The judge said "met" in all 17 plain runs. Eight of those were broken: one Sonnet run and seven Haiku runs. The worst one was a Haiku run on the spreadsheet task that passed 4% of the hidden checks, and /goal still called it done.

What I built

goalpost is a Claude Code plugin made of hooks. You keep typing /goal like before. Behind it:

The goal becomes a spec with acceptance criteria. Each criterion has a command that exits 0 only when it holds.
Claude can't stop while a criterion has no passing check, and a check only counts if it ran after the last edit.
Tests that existed when the goal started are read-only, so they can't be edited to fit the code.
When everything passes, a fresh auditor agent re-runs the checks and looks for stubs, skipped tests and hardcoded answers.
After context compaction the spec and the progress notes come back.

It's hooks, not a prompt, so the model can't talk its way past it.

One example from the benchmark: in a Sonnet run on the spreadsheet task, the agent changed the expected values in four of its own tests after the spec was frozen, for example AND(A3,A1) from #VALUE! to TRUE. The auditor caught it ("expectation changed after the freeze, undeclared"), the agent explained the change, and the run finished with all 73 hidden checks green. A plain run has nobody looking at that.

The honest numbers

	plain `/goal`	with goalpost
Sonnet 5.5, runs fully correct	8 of 9	9 of 9
Haiku 4.5, hidden checks passed (mean)	79.8%	89.7%
Haiku 4.5, runs fully correct	1 of 8	1 of 7
Time per run (Sonnet / Haiku)	2.7 / 5.7 min	5.2 / 15.1 min
Cost per run (Sonnet / Haiku)	$0.46 / $0.64	$0.91 / $1.25

Sonnet barely needed help on these tasks, and one extra perfect run is within noise. Haiku improved but it didn't become reliable: only one of its seven runs with goalpost was fully correct, and the auditor is a Haiku too, so it can wave things through. And it costs about twice the time and money. I'd use it for goals that matter, and turn it off for small ones (/goalpost:off).

The runs are small because my account's rate limit cut several of them. The full report lists every run I threw out.

Try it

claude plugin marketplace add syntaxixr/goalpost
claude plugin install goalpost@goalpost

Then open a new Claude Code session and type /goal as usual. It needs Node.js 18 or newer. The repo has the benchmark, the notes on how /goal works under the hood, and a 90-second video of two real runs.

If you've seen /goal end too early on your own work, I'd like to hear about it.

Heads up: I drafted this post with an AI assistant. The numbers come from the benchmark report in the repo.

I reverted the fix in 181 real changes to see if the tests would notice

syntaxixr — Wed, 30 Sep 2026 12:05:24 +0000

I keep running into the same thing with coding agents. The agent fixes a bug, adds a test, CI goes green, and it says "done". But green only means the test passes. It doesn't mean the test would have failed before the fix. And if it wouldn't, it checks nothing about the bug, however green it is.

The old-school way to find out is boring: revert the fix and run the test again. So I wrote a tool that does exactly that for every test in a change, and pointed it at the real history of 17 open-source projects.

How the check works

For each test a change adds or edits, it does two runs. First the test runs with the change. Then only the changed source files go back to the base branch (tests, dependencies and config stay new) and the same test runs again.

Fails without the fix, passes with it: PROVEN. This is what you want.
Passes both ways: THEATER. It proves nothing about this change.
Fails on the old code only because it imports something that didn't exist yet: WEAK.
Passes both ways but sits next to a test that does prove the change: GUARD. That's fine, it's the "did I break the neighboring case" test.

What I found

I took two samples. 81 fix commits from the maintainers of twelve libraries (click, sqlparse, marshmallow, dateutil, more-itertools and others), and 100 pull requests that carry a coding agent's fingerprint, from five repos that are full of agent work: Claude Agent SDK, OpenAI Agents SDK, the MCP Python SDK, fastmcp and simonw/llm. 87 of those 100 were Claude Code.

	Proven	Weak only
Maintainer fix commits	90%	0%
Agent pull requests	82%	10%

Good news first: most tests do their job. These are well-run projects, some of them run by the companies that build the agents.

The difference is that WEAK column. In 10% of the agent PRs, every test failed on the old code for the same reason: the test file imports, at the top, a name that the change adds. On the old code that import breaks, the whole file can't load, and every test in it "fails". It looks like proof, but none of those tests ever ran against the old behavior.

One example is anthropics/claude-agent-sdk-python#1016. The test file gets two new imports at the top:

from claude_agent_sdk.types import (
    TERMINAL_TASK_STATUSES,
    ...
    TaskUpdatedMessage,
    ...
)

On the old code neither name exists, so all 10 changed tests die at import, including one that was already there. The fix may well be right. The tests just can't show it. The cheap cure is to import new names inside the tests that need them, so the rest of the file still runs on the old code.

No maintainer commit in my sample looked like that.

THEATER is rare

Only 7 of the 162 changes I could judge were unproven, and every one had a reason. Type-only fixes (runtime tests can't see types). A Windows newline fix tested on Linux, where the bug doesn't exist. A dateutil __repr__ fix whose new output matched what the inherited default already printed. A maintenance commit that happened to mention an issue number. So when the tool says THEATER, there's usually something worth a look.

Try it

In any git repo, on a branch with a fix and its test:

npx github:syntaxixr/receipts check

It's deterministic, with no LLM and no API keys, and it runs your own pytest, vitest or jest. There's also a GitHub Action that keeps one comment on the PR, and a skill so the agent checks its own tests before it says it's done:

# Claude Code
/plugin marketplace add syntaxixr/receipts
/plugin install receipts-check@receipts

# Codex, Cursor and other agents
npx skills add syntaxixr/receipts

The fine print

The samples are small and not random, so the numbers describe those projects, not the whole ecosystem. "Agent-authored" only means an agent's fingerprint is in the commits, and humans steer those agents. "Judged" means at least one test ran on both sides. The rest failed for environment reasons, since I installed each project once at its current tip. PROVEN means the test notices the change, not that the change is correct. Python and JS/TS only for now.

Repo: https://github.com/syntaxixr/receipts. Every change with a link and the method: https://github.com/syntaxixr/receipts/blob/main/docs/study.md. Data: https://huggingface.co/datasets/syntaxixr/receipts-study

If it says something wrong about your code, I'd like to hear about it.

Heads up: I drafted this post with an AI assistant. The numbers come from the study in the repo, and every change is linked there so you can check them yourself.