I ran the same "did it actually do anything" test against Codex

Yesterday I measured Claude Code in headless mode and found it exits 0 whether it did the work or not. I ended that post asking whether that is one vendor's choice or how the whole industry works.

Today I ran the same three arms against codex exec, on August 19, 2026, codex-cli 0.147.0, macOS 26.5.2. Nine runs, three per arm, nine throwaway git repos.

Exit code: identical, and identically useless

Nine runs, nine exit codes of zero. Six of those nine left the repository byte for byte identical to how it started.

So the answer to my own question looks like the industry, not one vendor. Two CLIs from two companies, same week, same behaviour at the process boundary. Two vendors is suggestive and not proof, but if your CI treats $? from a coding agent as evidence of work, it has been reporting a success it never verified.

I did not judge this from the agent's own report. The harness hashed every non-.git file before and after each run:

impressao() {
  ( cd "$1" && find . -path ./.git -prune -o -type f -print0 \
    | sort -z | xargs -0 shasum -a 256 | shasum -a 256 | cut -d' ' -f1 )
}

That fingerprint is the judge in every number below. The agent never touches it.

Where Codex is genuinely better: two signals instead of none

Run codex exec --json and the CLI streams one JSON object per line. In the three runs where the edit actually landed, this line showed up:

{"type":"item.completed","item":{"id":"item_2","type":"file_change","changes":[{"path":".../src/utils.js","kind":"update"}],"status":"completed"}}

Present in 3 of 3 runs that changed the repo. Absent in 0 of 6 that did not.

There is a second signal, on stderr, which I nearly failed to look at:

ERROR codex_core::tools::router: error=patch rejected: writing is blocked by read-only sandbox; rejected by user approval settings

6 of 6 blocked runs. 0 of 3 that wrote. A plain grep ERROR on your captured stderr catches it.

Claude Code's JSON envelope has no equivalent to file_change. The fields there (subtype, is_error, permission_denials) said success in every run I measured yesterday, including the ten that changed nothing. I did not measure what Claude Code writes to stderr, so I am claiming nothing about that.

My prediction was wrong, and a reviewer caught the rest

I wrote the prediction down before running anything, which is the only way a prediction counts. I said Codex would distinguish by error inside the JSON stream: a blocked write surfacing as a command with a non-zero exit_code, or a failure event.

Wrong. Zero commands with non-zero exit. Zero turn.failed. Zero error events on stdout, across all nine runs.

Then it got worse, in the useful way. My draft said the blocked write "surfaced nowhere as a failure" and that the agent "never even attempted" it. A reviewer with no stake in my text went and read the .err files I had captured and never opened. Both claims were false: the log says rejected, not skipped. I had the evidence sitting right there and trusted my own summary of it, in an article about not trusting summaries.

The agent is not the liar

Six runs changed nothing, and in all six Codex said so in plain English:

"Blocked from editing: this workspace is read-only. [...]"

"Unable to modify src/utils.js: the workspace is read-only and approvals are disabled. [...]"

Two of them printed the full function they would have written. The prose is honest. The status is not. Which means the person who gets deceived is the one who automated: you, reading the terminal, see the refusal on the last line. Your GitHub Actions step reads $? and goes green.

One thing that will bite you today

codex exec with no -s flag behaved exactly like -s read-only in my runs: repository untouched 3 of 3, no file_change event, patch rejected on stderr, and the agent saying the workspace was mounted read-only.

The default non-interactive invocation is the one that most looks like it is working while doing nothing. It reads your files, reasons, spends tokens, writes a considered paragraph, and leaves the repo exactly as found. Pass -s workspace-write if you want edits, and mean it.

What I did not prove

Both signals agreed with the fingerprint 9 out of 9. That is agreement, not reliability. The obvious case I did not test is an agent that writes a file and then reverts it, where the event fires and the fingerprint does not move.

Three runs per arm is a small sample. Two of my three arms turned out to be the same sandbox state reached by different routes, which I found out from the agent rather than from my own design, and I left it in the table with the caveat instead of quietly dropping an arm. One version of each CLI, one machine, one morning.

The full table and the script exactly as it ran, comments and a counting bug included, are in the canonical post.

If you run this against Cursor, Aider or Gemini CLI, I would genuinely like to see the numbers.