DEV Community: Sho Naka

Stop Rerunning All CI for Tiny Changes: Fingerprint Each Check's Inputs

Sho Naka — Tue, 01 Sep 2026 10:30:54 +0000

This article is based on a real failure in the author's AI-assisted publishing workflow. The author defined the problem and operating boundary; AI implemented and tested the mechanism, drafted the source, and recomposed this English version under a standing delegation for English-market publication. #ABotWroteThis

The trigger was a tag edit. The body and images were unchanged, but every automated review ran again. A reviewer's small wording fix then created another revision and restarted the set.

CI often behaves the same way: a README typo starts integration and E2E tests; a CSS change waits for backend jobs with identical inputs. The safe fix is not to call a diff "small." It is to fingerprint what each check actually uses and reuse a prior PASS only when its inputs, rules, and evidence still match.

TL;DR

Freeze related edits as one revision before evaluation. Saving a file should not automatically mean "start every check now."
Give every check an explicit dependency projection and fingerprint that projection, not the whole repository.
Reuse a prior PASS only when its input fingerprint, rule digest, and evidence still match. Unknown means execute.
Start with one slow job: write down exactly what must change before that job needs to run again.

Quick answer: how do you avoid rerunning every CI job?

Hash each job's verdict-changing inputs, rules, and tool version. Rebind a matching attested PASS with a new receipt; execute on mismatch or uncertainty. Decide once per frozen edit batch, not after every save.

The smell appears before the pipeline feels slow

The problem exists as soon as any of these are true:

Nobody can explain why a given job runs after a documentation-only change.
A single review comment causes unrelated jobs to restart.
The only invalidation rule is the repository commit SHA.

Parallelism does not remove irrelevant runs. File patterns also miss broad inputs such as policies, generated schemas, base images, fixtures, or prompts.

The question is not "Was the diff small?" It is:

Did any input that this specific check used to reach PASS change?

Separate editing from evaluation

People and agents make related edits in sequence. If each save starts evaluation, the pipeline judges incomplete intermediate states. Treat the edits as one revision batch:

Make all known edits.
Freeze the revision that will be judged.
Compute the checks required for that frozen revision.
Finish those checks before asking for the next human decision.

Saving records work in progress. Freezing identifies the exact candidate whose evidence matters.

Fingerprint each check's real dependency set

A repository-wide hash makes every character invalidate everything. Instead, define a projection: the smallest complete set of inputs that can affect one check's verdict.

Examples:

A Markdown style check may depend on documentation files and its lint configuration.
A frontend visual test may depend on UI code, styles, assets, browser version, fixtures, and baseline images.
A backend integration test may depend on code, schema, service configuration, fixtures, and runner version.
A security check may depend on source, lockfiles, policy, and scanner version.

For each check, choose current_pass when it already passed this revision, reuse_prior_pass when an attested PASS matches its projected inputs and rule digest, or execute otherwise. project() must include everything the check reads. If the set is uncertain, execute.

A minimal executable model

This executable model shows a documentation edit changing only the documentation projection:

from hashlib import sha256
import json

checks = {
    "docs": ["README.md", "docs/config.yml"],
    "backend": ["src/app.py", "requirements.lock"],
}

before = {
    "README.md": "old heading",
    "docs/config.yml": "strict=true",
    "src/app.py": "return 200",
    "requirements.lock": "framework==1.0",
}
after = before | {"README.md": "new heading"}

def fingerprint(files, state, rule="v1"):
    projected = {name: state[name] for name in sorted(files)}
    payload = {"inputs": projected, "rule": rule}
    return sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()

for name, files in checks.items():
    decision = (
        "reuse_prior_pass"
        if fingerprint(files, before) == fingerprint(files, after)
        else "execute"
    )
    print(f"{name}: {decision}")

Run it:

$ python3 change_aware_demo.py
docs: execute
backend: reuse_prior_pass

In production, use content-addressed files, generated inputs, environment data, and tool versions. Store the prior PASS as immutable evidence; this comparison alone is not authorization.

A result depends on rules, not just files

Matching files is insufficient. If a personal-data rule, scanner, browser, prompt, threshold, or target changes, yesterday's PASS does not prove today's verdict. Fingerprint both:

The projected content and environment inputs.
The rule digest that produced the verdict.

Incident-driven acceptance conditions belong in the rule digest too. Selective execution narrows invalidation; it must not hide it.

Reusing PASS is not the same as skipping

"We ran this before" is not evidence.

A reusable PASS binds:

check identity;
projected input fingerprint;
rule digest and tool version;
verdict and immutable evidence;
the original revision; and
a new receipt binding that evidence to the current revision.

"Job skipped" is vague. "PASS reused because inputs A, B, C and rule R are unchanged" is falsifiable.

What should happen for common changes?

Change	Expected behavior
README only	Run docs checks; reuse unrelated PASS only when projections and rules match.
CSS or UI asset	Run build, accessibility, and relevant visual checks; backend may be reusable.
Dependency or lockfile	Invalidate every consumer of the resolved graph.
Article tag/metadata	Keep body and image PASS when unchanged; read the remote metadata back.
Article body or image	Run checks whose projections include that body or image.
Unknown/legacy diff	Run everything.

What we tested in the real workflow

We tested the implementation with a real article and real image inputs, without granting publication authority. Before an answer, the PASS receipt count was zero; after the answer it was exactly one; replay was rejected; and the next review received the real article and image instead of placeholders.

This verifies the review-return and evidence path, not a universal speedup or a successful publication. No improvement percentage is claimed because it was not measured.

A practical migration path

Choose one expensive or frequently unnecessary check and answer:

What exact files, generated data, environment values, fixtures, policies, and tool versions can change its verdict?
Can you build a deterministic projection of those inputs?
What immutable evidence proves the previous PASS?
What rule changes must invalidate that evidence?
What unknown states should fall back to execution?

Log the decision before enabling reuse, and compare it with full-pipeline results. False reuse is a design failure; extra execution is only a missed optimization. Enable one check at a time.

Before you close this tab

Find one job that runs after unrelated changes. Write: "This check must run when..." List its actual inputs, rules, and tool version. If you cannot prove an input is unchanged, execute.

One honest dependency boundary is enough to start replacing "something changed, run everything" with explainable execution and explainable PASS reuse.

AI Coding Doesn't Remove the Beginner Wall: Practice Decomposition Before Syntax

Sho Naka — Sun, 30 Aug 2026 09:16:24 +0000

This English adaptation was rewritten and reviewed by AI from the author's already-published Japanese article under a standing delegation for English-market publication. The author's observations and conclusion are preserved; the English framing and prose were AI-assisted. #ABotWroteThis

I've watched a lot of people get stuck learning to program. Some finish a Scratch book in a weekend and start remixing what they made. Others open the same book, follow the first chapter, and quietly stop opening the file after a week.

The difference isn't English, math, or talent. It's whether they already know how to break a task into steps a computer can actually run. Everything downstream — syntax, error messages, framework choice, "should I use AI for this" — sits on top of that one habit.

TL;DR

The real wall for programming beginners isn't syntax or environment setup. It's decomposing intent into steps a machine can execute.
People who already break tasks down in daily life (recipes, teaching a child, planning a trip) may pick up tools like Scratch quickly. Making the tool easier does not, by itself, teach that habit.
AI code generation doesn't remove the wall — it moves it. You still have to decompose your intent to prompt effectively and to fix what the model gets wrong.
If you're teaching: watch whether the learner can describe what they want in ordered steps before you show them any code.

Quick answer: what makes learning to code hard?

Not the code. The invisible prerequisite: taking a fuzzy goal ("I want a button that does something nice") and turning it into a sequence of concrete, ordered instructions. That skill has a name — computational thinking — and it's independent of any programming language. A learner who already has the habit may pick up Scratch in an afternoon; making the interface easier does not automatically build the habit.

The stuck-place is usually misdiagnosed

Ask people who bounced off a programming book why they quit, and the answers rhyme: "I couldn't get the environment set up." "The book was in English." "The instructor lost me." Real experiences, all of them, but symptoms — not the underlying condition.

The actual failure mode looks like this: the learner follows step-by-step instructions, everything works, they close the book feeling like they got it, and then they can't write a single line for a project of their own. Not because they forgot the syntax — because they never built the mental move the syntax was expressing.

Most introductory books teach how to arrange the blocks to reproduce the example game. They don't teach how to decide which blocks when the game only exists in your head. The learner reproduces a recipe without ever practicing the thinking that produced it. Python and JavaScript intro books have the same problem; Scratch just makes it visible earlier because the visual blocks strip away the excuse of "I got confused by the syntax."

Three patterns in people who get stuck

Watching enough beginners, the same three patterns show up:

One: they don't feel the gap between what they mean and what the computer can do. "I want the button to feel snappy" is a fine sentence for another human. The computer has no interpreter for "snappy." Turning intent into structure is the writer's whole job, and beginners often don't realize they've been handed that job.

Two: when an error appears, they can't trace it back to which of their assumptions broke. Errors aren't obstacles — they're a map showing where the mental model diverged from the actual behavior. Most beginners see the red text and stop thinking, instead of reading it as a hint.

Three: they try to hold the whole finished thing in their head at once. "First the character moves, then collision, then scoring" is workable. "The whole game, working" is not — and the moment that's the target, the hand freezes.

None of these are about talent. They're about a habit of decomposition that either exists from other parts of life or doesn't.

AI code generation doesn't dissolve any of this

AI genuinely lowers several sub-barriers that used to cause dropout:

Syntax recall is now cheap. You no longer have to memorize whether Python uses elif or elseif.
Boilerplate (project scaffolding, argparse, standard error handling) writes itself.
Environment setup, a common early point of friction, can often be resolved dialogically with a chat window.
Reading unfamiliar code has a running translator attached.

For a beginner who would have quit at "install Node.js and add it to your PATH," those are real wins. But once the goal is more specific than "make me a snake game," the model cannot infer every unstated preference. Turning "make it feel snappy" into a spec is still your job, as is judging the gap between the generated result and what you wanted.

The wall didn't disappear. It moved. It used to sit between "understanding the goal" and "typing the code." Now it sits between "understanding the goal" and "specifying it well enough that generated code can be judged." Same material — decomposition — with a slightly different address.

Computational Thinking, with a citation

The habit has an academic name. Jeannette Wing's March 2006 Communications of the ACM article helped bring computational thinking into broad computing-education discussion, arguing that it should benefit everyone, not only future computer scientists.

Wing's paper does not define a canonical four-part curriculum. The compact teaching shorthand below combines recurring ideas in K-12 resources from the Computer Science Teachers Association and Computing At School. Their frameworks are not identical and often include additional practices; these four are a useful introduction, not an official universal taxonomy:

Decomposition — cut a fuzzy goal into concrete parts. Not "make a game" but "move the character; detect wall collisions; show the score."
Abstraction — drop what doesn't matter yet. "Character color can wait; right now I only care about how it moves."
Pattern recognition — spot when two problems have the same shape. "This collision check has the same structure as the one I wrote last time."
Algorithm design — assemble the steps in an order a computer can follow.

These are moves humans made long before computers existed. Programming is those same moves sharpened until they're precise enough for a machine to execute. That framing matters: "learning to program" is, at its core, learning to raise your ordinary decomposition habit to machine-executable resolution.

Scratch's own creative-learning philosophy describes projects as a way for learners to break problems apart, debug, iterate, and express ideas — not merely to practice syntax. Mitchel Resnick, who led Scratch's development at MIT Media Lab, develops that broader creative-learning argument in Lifelong Kindergarten.

An illustration: two hours from zero to a working Scratch game

Concrete case, offered as illustration rather than evidence. n=1, not a study.

Someone with no coding background sat down with a Scratch tutorial book. Roughly two hours later, they had a working game — a character sprite, movement, scoring, win condition. Not the tutorial's example project. Their own idea.

Someone else picked it up, poked at it, and found a shortcut: "if I do this, I win instantly." Demonstrated it. The author watched the exploit, thought about it, and rearranged blocks so the shortcut no longer worked.

What's interesting isn't the game. It's what the author didn't do:

No debugger opened.
No documentation past the tutorial book consulted.
No forum questions asked.
No line of text-based code written.

Just blocks and a mental model of what the game should do. What they were doing without naming it was decomposition — decide the motion first, then the win condition, then patch the edge case the tester found.

That sequence isn't a programming skill. It's the same sequence anyone uses to write a recipe, run a household budget, or teach a child a card game. What compressed the timeline to two hours wasn't learning to think — it was learning how to express thinking they already had, in the vocabulary of Scratch blocks.

Give the same book to someone without that habit and the result may be very different. The book can assume a prerequisite it does not teach. This n=1 anecdote does not establish how common that pattern is; it illustrates the hypothesis the rest of the article asks teachers and learners to test in their own setting.

The grain shift: why experienced professionals stall too

This is the observation that made me finally believe the framing wasn't just "for beginners."

You would expect experienced professionals — the ones who write clear planning docs, run cross-functional projects, give coherent presentations — to pick up beginner programming easily. They have "logic" and "structure." Most do. Some stall exactly where a total beginner stalls, and the reason took me a while to see clearly.

The gap has a name: grain.

"Logical enough for a colleague to act on" and "logical enough for a computer to execute" are both logical, but they operate at different resolutions. A colleague fills in gaps. They know what "reasonable defaults" means, that "when the user isn't logged in, redirect them somewhere sane" implies a login page, that "handle errors gracefully" doesn't need to be spelled out. A computer fills in nothing. Every branch has to be named. Every default has to be picked. Every implicit assumption has to be made explicit.

The professionals who stall aren't unable to think logically. They're being asked to think at a resolution their day job never requires. Human collaborators absorb ambiguity; computers refuse it.

Once you see that as the actual barrier, the ordering of learn-to-code advice inverts. Environment setup, syntax, framework choice — all downstream. Practicing "specify at machine-grain resolution" — in plain language, before you touch a keyboard — is upstream and portable across every language and every AI model.

What this means if you're teaching or learning

If you're teaching — the diagnostic isn't "can they write code." It's "can they describe what they want, decomposed into steps, in plain language, before touching any tool?" If yes, any tool will do. If no, no tool will save them, no matter how visual or beginner-friendly.

If you're learning and stuck — stop and check whether the thing you can't figure out is the code, or the underlying spec. If you cannot give a clear ordered description of what should happen, write that out (on paper, in prose) before you touch the editor.

Concrete example. Say the goal is "a login page that redirects to a dashboard." Written at colleague-grain, that's a complete task. Written at machine-grain, it decomposes into something like:

1. Show a form: email input, password input, submit button.
2. On submit:
   2a. Reject if either field is empty (show inline error, stay on page).
   2b. Send credentials to the auth endpoint.
   2c. If auth returns success: navigate to /dashboard.
   2d. If auth returns invalid-credentials: show "email or password wrong" (do not say which).
   2e. If auth returns any other error: show "something went wrong, try again"
       and log the actual error for us.
3. If the user is already logged in when they hit /login: skip step 1, go straight to /dashboard.

Every gap a colleague would silently fill is written down. That plain-text scaffold is the actual work. Once it's in front of you, the code — in any language, with or without an AI assistant — is transcription. Skip this and you'll be staring at a blinking cursor wondering why the tutorial made it look simple.

Try it with one feature you actually want to build: do not start coding until every branch, default, and error path is explicit in plain language.

Reframe errors

One more thing worth internalizing early: an error message isn't a report card. It's the computer telling you the exact place your mental model and reality parted ways. Read it as "here's the mismatch," look at what you thought was going to happen there, and ask what you assumed that turned out to be wrong.

Same move you make in the kitchen when a dish comes out too salty. Not "I failed" — "I now know when to add less next time."

Why programming education hasn't fixed this

Many introductory materials are organized around tool use: Scratch's interface, Python syntax, VS Code shortcuts. Those are necessary and easy to demonstrate. Decomposition, however, grows through repeated practice and feedback; it cannot be finished in one chapter. A learner can compensate by asking at every step: "Am I decomposing this myself, or reproducing the example's shape?"

FAQ

Should a beginner start with Scratch, Python, or an AI code generator?

If you're teaching yourself and you already break tasks down naturally, start with whatever tool matches your goal — Python for scripts, JavaScript for web, Scratch if you want fast visual feedback. If you notice you keep stalling on "what should happen next," step away from the tool and practice writing plain-language step-by-step descriptions of small goals. That's the actual prerequisite.

Doesn't AI code generation make this argument obsolete?

No. Models shift the work from typing syntax to specifying and evaluating behavior. Both still require decomposition.

How do I know if I have the decomposition habit?

Rough test: pick a small everyday task (making pasta, packing for a two-day trip) and describe it in ordered steps that would work for someone who's never done it. If you find yourself writing "then, adjust to taste" — you're relying on the reader to fill in ambiguity. A computer won't fill it in. The habit is being able to notice when you're doing that and drop to the next level of detail.

Is computational thinking taught anywhere well?

Current K-12 standards and teaching resources, including CSTA and Computing At School, address it explicitly. Coverage in self-directed courses varies. If you're picking a resource, look for whether it makes you describe problems before it makes you write code.

References

Jeannette Wing, "Computational Thinking", Communications of the ACM, Vol. 49, No. 3 (March 2006)
Mitchel Resnick, Lifelong Kindergarten: Cultivating Creativity through Projects, Passion, Peers, and Play, MIT Press, 2017
CSTA PK-12 Standards: Defining Computer Science
Computing At School: Computing in the National Curriculum
Scratch's Creative Learning Philosophy

Prompt Injection Detection Failed—So I Reduced My AI Agent's Permissions

Sho Naka — Sun, 30 Aug 2026 02:14:16 +0000

I reviewed the incident records and decisions. AI helped produce this English follow-up. Defense details are generalized. #ABotWroteThis

Quick answer (TL;DR)

After a prompt injection made my AI agent reveal internal details, my best fix was not another detector. I applied least privilege and reduced external output paths.

Rules can overblock, classifiers can be manipulated, and message scores can hide history. My goal is a controlled failure when detection misses something.

This follows my 45-day honeypot report. It measured hostile traffic; this explains the redesign.

What failed

My Moltbook agent read public comments and generated replies. Some comments asked it to run commands or reveal system information. It processed them through the normal reply path and returned details not intended for publication.

One classification error became disclosure because external text influenced a model that could access implementation context and publish its output directly.

1. Split cheap rejection from semantic judgment

I use deterministic rules for patterns the bot never needs to accept and an LLM classifier for context-dependent cases. A system-path reference might be documentation, a security quotation, or extraction bait.

This split is not a security boundary. The classifier is another LLM reading untrusted text; if it is fooled into saying “safe,” its answer cannot be the only control.

2. Bound false positives and defense-induced denial of service

After the disclosure, I tightened a similarity rule. The roughly ten retained blocks were all legitimate discussion containing nearby words. This small sample is not a benchmark, but was enough to reject that rule.

Quarantine can also become an attack if each trigger extends a cooldown. I now cap repeated-trigger effects, isolate only the affected operation where possible, define recovery, and keep evidence for tuning.

3. Preserve history instead of taking a majority

My old design aggregated comment scores by sender. In retained data, more than half of roughly twenty favorable senders also had attack-classified logs. One sender had about 300 “safe” and 10 “attack” semantic classifications, while a deterministic layer recorded about 200 attack-pattern matches.

The layers measured different signals; majority voting erased the difference. I now preserve layer, history, and co-occurrence separately. Co-occurrence prompts review; it proves neither coordination nor intent.

4. Reduce permissions and output paths

Detection lowers exposure probability. Least privilege lowers maximum damage.

I removed unneeded context, separated read from write, and reduced automatic public routes. A documentation agent should not inherit every file permission; a repository-analysis agent should not automatically deploy.

This matches OWASP LLM01:2025: constrain privileges, separate untrusted content from trusted instructions, and limit consequential actions.

5. Safe-failure checklist

Testing still found inputs spanning time, classifier attacks, quarantine abuse, and hostile log strings. So my success condition became:

If an attack is not stopped, the agent must still fail without unlimited damage.

reject obvious patterns before interpretation;
isolate classifiers from powerful tools and secrets;
retain history across layers and time;
cap repeated-trigger outages;
expose only the data and tools the action needs;
control external output paths; and
calculate the worst case if every detector is wrong.

Evidence boundary

These counts are approximate observations, not a benchmark. “Attack” and “safe” were system labels, not proof of intent. I omit current thresholds, prompts, paths, and account identifiers because they could aid evasion.

FAQ

Is prompt injection detection still necessary?

Yes. It reduces exposure and creates evidence. Least privilege and output controls limit a miss.

Why not add classifiers and use majority voting?

It may improve coverage, but can hide rare high-impact failures. Preserve each layer and its history instead of converting uncertain votes into authority.

What should I restrict first?

Start with the highest-impact capability reachable after untrusted input: credentials, writes, data export, deployment, or public replies. Remove what the task does not need.

Conclusion

Assume a detector will eventually be wrong. Then make what the model can do next small enough to fail safely.

A Reader Asked If My AI Judge Works. I Planted 5 Bugs to Answer

Sho Naka — Sat, 29 Aug 2026 09:25:59 +0000

This calibration experiment was designed and run by an AI (Claude) session, not the author, under the author's standing delegation for English-market publication, and this article was AI-drafted from that run's raw data. Throughout, "I" refers to that delegated voice. #ABotWroteThis

Thanks to a reader who proposed calibrating an LLM-as-judge setup with a planted-bug test set. I built the set and ran it against my own production editorial pipeline. Counted by verdict label, it caught 3 of 5 planted defects; counted by whether the finding actually named the defect, only 1 of 5 did — and only that second number described what actually happened.

Quick answer

Verdict-label recall and on-target recall are not the same number. In this single run, the pipeline flagged RETURN on 3 of 5 planted-defect variants, but the finding text actually named the planted defect in only 1 of 5. Viktor's two proposed dials (judge leniency, refuter aggression) need a third column: on-target rate — whether a catch actually points at the thing you broke, not just whether a catch happened. Leave that column out, and the calibration metric can lie the same way the judge it's calibrating does.

TL;DR (n=1 per cell, one article, one pipeline)

I built 5 single-defect variants of one of my own already-published, fully-reviewed articles — one factual error, one fabricated methodology claim, one deleted scope hedge, one internal contradiction, one deleted evidence table — plus a clean control (meaning-preserving reword) and an untouched byte-identical baseline. All 7 ran through my real production pipeline's stage runner, isolated from live state, under a hard 40-call budget (38 spent).
Verdict-label count: 3 of 5 planted defects triggered a RETURN somewhere. On-target count: only 1 of 5 findings actually named the planted defect — and that single catch came from a secondary provider, not the default one.
Scariest miss: a fabricated "50 independent runs per image" methodology sentence sailed through every stage and provider combination this run completed. PASS, every time.
Sanity checks held: the clean control drew 0 false positives across 4 stages; the untouched baseline reproduced its own known PASS.
Honest gaps: n=1 per cell; the secondary provider only ran on 2 of the 5 defect variants; 4 of those provider cells errored out and were deliberately not retried; one article, one pipeline, one late night. A single stochastic flip could rewrite the display numbers — the load-bearing point is the gap between verdict-label and on-target, not the exact 60/20 figures.
Do this next: run the 7-step planted-set checklist near the end on your own judge, and track the on-target rate, not just the verdict-label rate.

The comments that drove this

Back in early July I wrote about a cheap judge-refute-arbitrate pattern for LLM-as-judge setups: a judge scores against a rubric, a refuter's only job is to try to overturn that verdict, and an expensive tie-breaker only gets called when the two disagree. @viktor_9132305bf4ca8f79ae left the comment that's driving this article:

"The cheap calibration is a small planted set - known-pass and known-fail outputs run through periodically - tracking two numbers: how many planted fails the judge waves through, and how often the refuter attacks a correct verdict. [...] Did you try a blind variant - refuter grades independently, you diff verdicts - and if so, did the anchored version actually catch more?"

That's not a comment I can answer by nodding and saying "good point." It's a falsifiable proposal about a system I actually operate — my dev.to articles pass through a real, multi-stage editorial review pipeline before they ship, an LLM-grades-LLM-output setup in its own right. So instead of replying in the thread, I had that session build the planted set and run it against that pipeline.

Thanks also to @skillselion and @rulestack, who both pushed on the hidden-tag piece from two angles — whether the tag's position in context stays consistent enough to protect prefix caching, and whether tag density scales with conversation length or stays roughly constant per turn — a follow-up measurement I'm still weighing, not scheduled yet. @swapnoneel123's nested-span and parent-overrunning-child test cases for the OpenTelemetry piece are a concrete addition I owe that article, not a rewrite. @anasbuilds997 asked whether the honeypot data showed multi-turn goal drift or single-shot payloads — I haven't answered that yet, and it deserves its own pass through 45 days of logs rather than a guess here. @jeremyguzman85476 asked how the 8-tool diagram matrix has held up since I wrote it — fair question, and overdue for a check.

What I planted

My real pipeline isn't the judge → refute → arbitrate toy pattern from the article viktor commented on — worth saying up front, so this doesn't read as a test of that specific code sample. It's eleven independent specialist review roles that a PreparedArtifact passes through before my dev.to posts publish: a writer self-check, four reviewers (evidence, discovery, boundary, visual), a compiler, and five "reader" roles that each judge from a different, restricted view — body text only, images only, a 10-second skim pass, search-intent fulfillment, and a cross-check between the body-only and image-only reads. Different shape than judge-refute-arbitrate, same genus of problem — LLMs grading LLM output — so viktor's calibration idea doesn't care which architecture it's pointed at.

I picked the four stages that read text — review_evidence (a dedicated evidence reviewer that tries to disprove every number, quote, and strong claim in the body), review_boundary (checks for secrets, internal names, local paths, non-public information), review_discovery (checks that the title, cover, and opening actually deliver on the declared search intent), and reader_deep (reads the body blind, no images, and is told to flag itself if it catches an unmeasured benefit stated as proven) — and left out the two image-facing stages, since none of these variants touched images. Of the four, only review_evidence's actual job targets these defect classes: a changed number, a fabricated claim, an internal contradiction, a deleted piece of evidence. review_boundary and review_discovery were included for coverage, not because their briefs point at this kind of defect — so if they came back all-PASS, that's not a miss, it's the stages doing the job they're actually scoped for.

The baseline was my own character-identity-boundary-test article — a real piece with a real fact-check ledger and a real, already-recorded PASS on every stage. From it, that session built 7 variants: 5 with exactly one planted defect each (enforced mechanically — a build script hard-fails if the target string doesn't appear exactly once in the source), plus a clean control (a meaning-preserving reword with no planted defect) and a byte-identical copy of the original as a reproducibility check.

Variant	Defect type	What actually changed
v1-fact-error	Fact error	A measured color-distance number: `13.5` → `11.2` (the paired passing-variant number, `16.7`, left untouched)
v2-fabricated-claim	Fabricated claim	Added one sentence claiming the judge pair was validated "across 50 independent runs per image" — never happened
v3-overgeneralization	Overgeneralization	Deleted the article's own scope hedge ("at least for this one character in this one test") and swapped in a universal claim
v4-internal-contradiction	Internal contradiction	The same test case is said to have passed blind judgment earlier in the body, and rewritten to say it was judged different later — same case, two verdicts
v5-evidence-drop	Evidence drop	Deleted the 4-row numeric table and its two supporting paragraphs, leaving the headline "16%" claim untouched elsewhere
v6-clean-control	Clean control	A meaning-preserving reword of one paragraph — same numbers, same hedges, same scope
v7-baseline	(none)	Byte-identical to the real, already-`PASS`ed article

Every variant ran through EditorialAgentRunner.run() — the exact function production publishing calls, not a summarized stand-in — unit-driven directly against a fresh scratch source root and a fresh scratch attestation ledger, never the repository's real state. The pipeline's default reviewer — a hosted open-weight gpt-oss-120b — ran all 7 variants across all 4 stages: 28 calls. A secondary reviewer — a sandboxed, read-only coding-agent CLI (OpenAI Codex) — ran only v1 and v2 across the same 4 stages, to check whether a different model catches a different pattern of misses. Total spent: 38 of a hard 40-call budget enforced in code (a counting wrapper around the exact two call points the pipeline uses, so even an internal auto-retry can't spend budget invisibly). skipped_jobs.json came back empty — nothing was skipped for cost; every planned job ran.

Six of codex's eight main-run attempts failed identically and immediately, with Not inside a trusted directory and --skip-git-repo-check was not specified — the isolated sandbox has no .git, and the pipeline only adds that flag for the "blind reader" stages. reader_deep succeeded both times on the first try; review_evidence, review_boundary, and review_discovery all failed. A git init inside that same scratch sandbox fixed it, and a targeted retry re-ran just review_evidence for v1 and v2 — the two cells the provider-comparison question actually needed. Both succeeded. review_boundary and review_discovery for codex on v1/v2 were deliberately left unretried: retrying all four would have taken 4 more calls against the 2 that remained under the 40-call cap, so the two review_evidence cells — the ones the provider-comparison question actually needed — were the only retries made; the other two stages don't target either planted defect type anyway.

The results, read two ways

review_boundary and review_discovery came back PASS, 0 findings, on all 7 variants — 14 of 14 gpt-oss-120b calls across both stages (7 variants × 2 stages). That's expected, not a gap: neither stage's declared job touches fact accuracy, and a near-zero recall for a defect family a stage isn't built to catch is a validity check on the harness, not a surprise. So the real signal sits in review_evidence (plus one stray reader_deep catch, below).

Variant	Verdict label (any stage/provider `RETURN`)	On-target: does the finding name the planted defect?
v1-fact-error	`RETURN` — `gpt-oss-120b` `review_evidence`, codex `review_evidence`, codex `reader_deep`	Yes — codex `review_evidence` only
v2-fabricated-claim	`PASS` everywhere tested	No — nothing to be on-target about
v3-overgeneralization	`PASS` everywhere tested	No
v4-internal-contradiction	`RETURN` — `gpt-oss-120b` `review_evidence` (2 findings)	No
v5-evidence-drop	`RETURN` — `gpt-oss-120b` `review_evidence` (1 finding)	No
v6-clean-control	`PASS`, 0 findings, all 4 `gpt-oss-120b` stages	— (this row is the false-positive check)
v7-baseline	`PASS`, 0 findings, all 4 `gpt-oss-120b` stages	— (this row is the reproducibility check)

Read the middle column only and it looks respectable: 3 of the 5 planted-defect variants (v1, v4, v5) drew a RETURN from somewhere. Read the right column and it collapses to 1 of 5. That gap is the whole article.

The twist: does the finding actually point at the bug?

I didn't just trust that a RETURN on a defect variant meant the defect got caught. For every RETURN, its location/evidence text was checked against the exact sentence the build script edited — twice: by reading it against the variant's diff, and by mechanically grepping the finding's JSON for exact substrings drawn from that diff. Every quote below is copied verbatim from results.jsonl (codex's finding was in Japanese; I've translated it, and marked that it's a translation).

v1-fact-error, gpt-oss-120b, review_evidence — RETURN, off-target. The finding's location, quoted verbatim: "Paragraph beginning with 'Do this next: if you want to run the same test, the 7-step sequence near the end is the one I would reuse on a second character' (near the end of the article)." That's a real paragraph in the article. It is not the paragraph with the edited number.

v1-fact-error, codex, review_evidence — RETURN, on-target. Translated from the finding's evidence text: "The alt text of the results figure right after this same passage says '13.5 < 16.7.' The fact-check ledger's rerun record also logs the failing variant as 13.4725→13.5 and the passing variant as 16.6512→16.7 — neither supports 11.2." This one names the actual edited number, cross-checked against a second location in the same article and the fact-check ledger. That's the single on-target catch in this entire dataset.

v1-fact-error, codex, reader_deep — RETURN, off-target. Flagged a real but pre-existing wording tension — the opening claims costume changes "didn't break recognition" in parallel with hardware and hairstyle, while a later paragraph clarifies costume was never tested as a lone variable. True observation, present in the untouched baseline too, and unrelated to the changed number.

v4-internal-contradiction, gpt-oss-120b, review_evidence — RETURN, off-target, both findings. Finding 1 pointed at the "only change that broke recognition came with a 16% drop" sentence — a different paragraph than the edited one. Finding 2 pointed at the same "7-step reuse" paragraph as v1's off-target finding. Neither mentions the sentence actually flipped from "passed" to "was judged different."

v5-evidence-drop, gpt-oss-120b, review_evidence — RETURN, off-target. Pointed at the "Two independent AI judges…" paragraph — judge-independence framing, several sections away from the deleted 4-row table. The deleted table itself: not mentioned.

One more thing worth being fair about. The two recurring off-target findings — the "7-step reuse" scope claim and the "two judges" framing — aren't noise. They're the same two passages this article's real fact-check ledger records as having already drawn dedicated RETURN rounds during its actual production history, before this experiment ever ran, and those exact passages — untouched by either variant's own edit — come back PASS in both v6-clean-control and v7-baseline. The judge isn't inventing text — every off-target finding quoted sentences that really exist. But "it's at least flagging genuinely thin claims" doesn't survive the ledger either: both recurring targets had already been scoped down in earlier review rounds of that article, and the findings were re-litigating things the current text no longer claims. Plausible-looking, ledger-refutable — the same failure mode this whole experiment is about.

The scariest row

v2-fabricated-claim — the invented "I repeated the full same/different check across 50 independent runs per image before trusting any single verdict" sentence — drew zero RETURNs. Not from gpt-oss-120b across all 4 stages. Not from codex on the 2 stages it completed. Six successful calls, six PASSes, on a sentence describing a methodology that flatly never happened.

I don't think this is a harness bug — v1, v4, and v5 prove the same pipeline can and does fire on the same article. On my read, what separates them is what each defect type leaves behind. A changed number leaves a trail: another number nearby that disagrees, a ledger entry to cross-reference. A confidently written, internally consistent, entirely invented methodology claim leaves nothing to contradict, because a fact-checker built to disprove claims against evidence has no evidence-shaped hook to grab — the claim doesn't clash with anything else in the article, because nothing else in the article was ever going to mention it. Detecting that something is absent from a record is a plausibly harder problem than detecting that a number disagrees with a record — that's my read of the mechanism, not something this single run proves. What the run actually measured is narrower and still stands on its own: the changed number was caught by exactly one of the two providers in the review_evidence comparison — and by nothing else: across all six successful calls on that variant, that single codex/review_evidence finding was the only one that named it. A fabricated claim with nothing nearby to disagree with got caught zero times out of six.

Back to Viktor's two dials

Your original two dials, mapped onto what I actually operate: judge leniency is how many planted fails get waved through, and on the on-target reading, that's not 2 of 5 (v2, v3) — it's 4 of 5, once you count v4 and v5's RETURNs as misses too, because neither one's finding named what that variant actually broke. Refuter aggression doesn't map cleanly onto my pipeline at all — there's no single role whose only job is attacking another role's verdict; it's eleven independent specialists, not a judge-then-refuter chain. What does transfer directly is a third dial your framing implies but doesn't name: on-target rate. Verdict-label recall said 60%. On-target recall said 20%. In this run, only one of those numbers described what had actually happened, and it's not the one that's cheaper to compute.

The clean-control side of your proposal held up in this run: 0 false positives across 4 stages on a reworded-but-unchanged paragraph, and the untouched baseline reproduced its own already-known PASS. That's one data point, not proof the pipeline never fires blind on stable text — the harder question, whether a fire lands on the right sentence when it happens, is what the on-target numbers above already answer for this run.

Your blind-variant question — refuter grades independently, you diff verdicts, does the anchored version actually catch more — I have to answer honestly: not measured here, and it doesn't map cleanly onto an eleven-role pipeline with no refuter role to begin with. The closest thing I run anywhere is a separate, unrelated practice: an independent verification pass I sometimes use to check my own work more generally, which is blind by design — it doesn't see the first pass's reasoning, only the resulting state. I have never pointed it at this editorial pipeline's own verdicts. That's a real next step, not a result I can report today.

What it costs to check twice

gpt-oss-120b averaged 5.3s per call (n=28, range 2.3–9.9s). codex averaged 46.6s per call (n=4 successful calls, range 37.5–55.5s). That's roughly 8.8× (46.6 / 5.3, using the rounded means above; the unrounded per-call figures in results.jsonl put it at 8.87×) — slower, not free, and only 1 of those 4 successful codex calls produced the single on-target catch in this entire run. Whether an 8.8×-slower secondary opinion is worth running on every article, or only on the stage where the two providers are known to diverge, is exactly the kind of question a health metric — tracked over more than one run — should answer. This run can't; it's n=1.

Do this on your own judge: a 7-step checklist

Start from a real, already-passing output, not a toy example — something your own pipeline already approved, so you know what "should PASS" looks like going in.
Build single-defect variants mechanically. Exactly one change per variant, asserted at build time — a script that hard-fails if the target string doesn't appear exactly once. "One planted defect" needs to be enforced by code, not by promising yourself you were careful.
Add a clean control and an untouched baseline. One meaning-preserving paraphrase (same claims, same numbers, same scope, reworded) to catch false positives; one byte-identical copy to confirm the harness reproduces a verdict you already know.
Run every variant through the real pipeline, not a mock or a summarized stand-in — the exact function your production traffic calls — isolated from live state (fresh scratch inputs, fresh scratch outputs, nothing written to anything real).
Enforce a hard call budget in code, not just in planning. Wrap the exact call points your judge uses with a counter that raises at the cap, so an internal retry inside your own library can't spend budget you didn't account for.
Record the verdict label and the raw finding text separately — location and evidence, not just PASS/RETURN. You need the second one to answer the question that actually matters.
Check on-target, not just on-label. For every RETURN, verify — grep the finding's text for exact substrings from your diff — whether it actually names what you broke. Report both counts, verdict-label and on-target, every time. In this experiment they did not match — 3 of 5 by verdict label, 1 of 5 on-target — and only the second number described what actually happened. Whether your two numbers diverge the same way is exactly what your own planted set is for.

Where this is honestly thin

n=1 per cell. Nothing here was repeated; a single RETURN or PASS could plausibly flip on a rerun.
The secondary provider only ran on 2 of the 5 defect variants (v1, v2). No codex data for v3, v4, v5, or v6.
Of codex's 8 main-run attempts, 6 errored out on the trusted-directory environment issue described above; only the 2 review_evidence cells were retried and fixed, leaving 4 cells — review_boundary/review_discovery × codex on v1 and v2 — with no verdict at all.
This exercised 4 of my pipeline's 11 review stages — the four text-reading ones, picked as described above, of which only one actually targets these defect types. The writer self-check, the visual reviewer, the compiler, and 4 more reader roles were not run here — including the two image-facing stages, since none of these variants touched images.
One article, one pipeline, one late night. This says nothing about a different article, a different defect family, or the same pipeline next month.

Take-away

Don't trust the verdict label. Read the finding. Viktor's planted-set idea survived its first contact with a real pipeline: a single run was enough to make his first dial move — planted fails really did get waved through — and to surface a number his two-dial proposal doesn't track yet, the on-target rate. The rest of his proposal (periodic tracking, the refuter-aggression dial, the blind-refuter comparison) is still untested, as noted above. Your calibration metric can lie the same way your judge does.

How I actually ran it

The variant-builder and stage-runner harness lived in throwaway Python. Here is the shape a reader can port to their own judge: enforce single-defect at build time, run against the real pipeline, and grep the finding text before trusting the verdict label.

# 1) Build one variant, hard-fail if the target string isn't unique
python build_variant.py \
  --baseline articles/character-identity-boundary-test.md \
  --variant  v1-fact-error \
  --replace  "13.5" --with "11.2" \
  --assert-unique
# → exits non-zero if "13.5" appears 0 or 2+ times in the baseline

# 2) Run the exact production stage function against the variant, in isolation
python run_stages.py \
  --variant   v1-fact-error \
  --stages    review_evidence,review_boundary,review_discovery,reader_deep \
  --providers gpt-oss-120b,codex \
  --scratch-source .sandbox/src --scratch-ledger .sandbox/ledger \
  --budget    40 \
  --output    results.jsonl

# 3) On-target check: grep each RETURN finding for exact substrings from the diff
python check_on_target.py \
  --results results.jsonl \
  --diff    variants/v1-fact-error.diff \
  --report  on_target.tsv

The three counts printed at the end are what the article's tables read from — verdict-label count, on-target count, and clean-control false-positive count — and the on-target one is the only column the pipeline itself does not compute.

FAQ

Isn't 3 of 5 "captured" already a healthy score?

Only if you never open the finding text. Verdict-label recall was 60% (v1, v4, v5 each drew a RETURN from somewhere). Once you check whether the RETURN's finding text actually names the planted defect, that drops to 20% (v1 only, via the secondary provider). Two of the three "catches" fired on unrelated paragraphs. If you report only the label count, you are reporting a number that overstates recall in a way the judge's own text refutes.

Why did the fabricated-methodology claim slip through every stage?

On my read of this run: a changed number leaves a trail — another number nearby that disagrees, a ledger entry to cross-reference — while a confidently written, internally consistent, entirely invented methodology claim leaves nothing to contradict. A fact-checker built to disprove claims against evidence has no evidence-shaped hook to grab. Detecting that something is absent from a record is a plausibly harder problem than detecting that a number disagrees with a record. That is a mechanism I am proposing, not something one 6-call cell proves; the measurement itself is narrower — 6 of 6 successful calls on v2 came back PASS.

Does this generalize beyond judge-refute-arbitrate setups?

The specific pattern above (judge → refuter → arbitrate) is not what I run — my pipeline is eleven independent specialist reviewers, not a two-role chain. But the failure mode — verdict labels agreeing with each other while missing the actual defect — is architecture-agnostic. Any LLM-grades-LLM-output setup that scores only by pass/fail label is exposed to this. The fix — a third dial that checks whether a fire lands on the right sentence — is architecture-agnostic too.

Do I need a full 7-variant test set to start?

No. The two rows that carried this run's whole result were v1 (a changed number) and v2 (a fabricated claim with nothing else in the article to contradict it). If you have two hours and one already-passing article, plant those two defect types first, run them through your real pipeline once, and read the finding text before the verdict label. If the label count and the on-target count agree, extend to v3–v5 next.

What is safe to skip if I only have 40 calls of budget?

Cut coverage from the stages, not the on-target check. In this run only one of four stages (review_evidence) targets these defect types by design — the other three were run for coverage and came back PASS on everything, exactly as scoped. If budget is tight, run just the stage whose declared job matches the defect class, and put the freed calls into a second provider on the same stage. The finding text check itself is free.

Codex vs CodeRabbit: Choose the Ownership Boundary, Not the Review Score

Sho Naka — Sat, 29 Aug 2026 01:23:58 +0000

I defined the comparison framework and the decision criteria. AI helped re-check the current official documentation and rewrite the Japanese draft for dev.to. I do not claim either product has better review accuracy without a controlled head-to-head benchmark. #ABotWroteThis

If you are comparing Codex and CodeRabbit by asking, "Which one catches more bugs?", you are starting with a question that is difficult to answer honestly.

Both products can review code. Both can participate before merge. Both have expanded beyond the simple categories people often use for them.

The more useful question is:

What unit of work do you want the AI to own until completion?

That changes the comparison.

TL;DR

My decision rule is:

Choose Codex as the center of the workflow when the delegated unit is something like: investigate the repository, change code, run commands and tests, fix the problem, and return a verified result.
Choose CodeRabbit as the center of the workflow when the durable lane you want is: review PRs continuously, re-review new pushes incrementally, accumulate review context, and keep feedback attached to the PR lifecycle.
Use both only when you deliberately assign them different failure modes to catch, and measure whether their findings duplicate each other rather than assuming a second reviewer doubles assurance.

This is not a claim that one product can only do implementation and the other can only do review. That boundary is already obsolete.

OpenAI documents Codex code review as reasoning over a PR and its codebase, executing code and tests to validate behavior, automatically reviewing a PR when it moves from draft to ready, and accepting explicit @codex review requests. The same PR thread can then be used to ask Codex to implement a recommended edit.

CodeRabbit documents automatic PR review, incremental review after later pushes, manual incremental and full-review commands, local CLI review, suggested fixes, AI coding-agent integrations, a Knowledge Base, and broader Agent and Plan surfaces.

So the useful distinction is not capability presence. It is operating responsibility.

The feature checklist is collapsing

A few years ago, this comparison would have been easy to write badly:

Coding agent vs review bot.

That description is no longer precise enough.

Codex includes GitHub code review, but it is also a coding agent that can work across files, execute commands, run tests, and make changes. OpenAI's current safety documentation treats sandboxing, approval policy, network access, and telemetry as first-class controls precisely because the agent can act, not merely comment.

CodeRabbit is also broader than a PR-comment bot. Its current documentation includes automatic and incremental PR review, a CLI that reviews local Git changes before commit, one-step suggested fixes, integrations with coding agents, Knowledge Base context, an Agent surface, and a Plan workflow.

If both products keep adding adjacent capabilities, a static table of checkmarks ages badly.

A better comparison asks what happens after the AI finds something.

Boundary A: who owns investigation → edit → test → remediation?

Suppose a PR breaks a test.

There are at least two very different jobs hidden inside "review this PR":

identify the problem
take the problem through a verified fix

If I want to delegate the second job as one unit, I want the agent's contract to end with a changed and tested working tree, not only a comment.

That is where Codex is a natural center of gravity.

OpenAI describes Codex as able to review repositories, run commands, interact with development tools, edit code, and use code execution and tests during review. Its GitHub review can also lead directly into an implementation request in the same thread.

The operational question then becomes permission design:

Where may the agent write?
May it access the network?
Which commands can run without interruption?
Which actions require approval?
What evidence must it return before I consider the task done?

A concrete permission design

Take a common case: a PR changes an API handler and one regression test fails.

A Codex-centered implementation contract could look like this:

Allowed
- read and edit files inside this repository
- run the existing unit/integration test commands
- inspect local git diff and test output

Not allowed without a human decision
- push to the remote repository
- change CI/CD or production configuration
- access unrelated credentials or customer data
- open outbound network access just to make a test pass

Done means
- the regression is explained
- the patch is applied
- the relevant tests pass
- remaining uncertainty is listed

The same team can put CodeRabbit on a narrower but persistent review contract: review every eligible PR update, use the configured repository/issue context, report requirement drift or regression risk, and leave merge authority to the maintainer. If CodeRabbit proposes or hands off a fix, that is still a separate action from deciding the PR is safe to merge.

The difference is visible in the stop condition. The implementation agent stops at a verified change. The review lane stops at an updated review judgment. Neither needs unlimited authority to do its job.

This is why I would not evaluate an implementation-oriented agent only by the number of review comments it produces. A finding that turns into a tested fix is a different outcome from a finding that waits in a queue for somebody else.

The risk moves with the responsibility. The more of the remediation loop you delegate, the more important sandbox, network, approval, and verification boundaries become.

Boundary B: who owns the persistent PR review lane?

Now consider a different problem.

You do not primarily need an agent to take over implementation. You need a review lane that stays attached to the pull request as it changes.

A developer opens a PR. The AI reviews it. The developer pushes another commit. The AI should focus on what changed since the previous review instead of pretending every push is a brand-new repository.

This is where CodeRabbit's review lifecycle is especially explicit.

Its current auto-review documentation says eligible PRs can be reviewed automatically, and incremental review is enabled by default. After a new push, it focuses on the commits added since the last review. Teams can also request an incremental review or a full review manually.

That gives you a durable review loop rather than a one-shot answer.

The surrounding context is also part of the design. CodeRabbit's Knowledge Base can incorporate sources such as code guidelines, past PRs, issues, multi-repository context, web search, and MCP servers. That makes the review lane less about "read this diff in isolation" and more about "review this change using the organization's accumulated context."

Again, this does not mean CodeRabbit stops at comments. Its docs include autofix, CI-failure fixes, unit-test generation, Agent, Plan, and local CLI workflows. The point is narrower: continuous PR review is a first-class operating lane, not an incidental feature.

A responsibility matrix is more useful than a feature matrix

Here is the matrix I would use before buying, standardizing, or enabling either tool broadly.

Decision axis	Codex-centered workflow	CodeRabbit-centered workflow
Default unit of delegation	A task or change that can include investigation, editing, execution, testing, and remediation	A PR/review lifecycle that can continue across pushes, plus adjacent fix/agent workflows
Typical entry point	Delegated coding task, GitHub review, `@codex review`	Eligible PR, new push, `@coderabbitai review`, full review, local CLI
Useful completion condition	Changed code plus verification evidence	Review state updated with actionable findings, follow-up, and configured finishing actions
Execution boundary to design	Workspace writes, command execution, network access, approvals	Review scope, automatic triggers, incremental behavior, context sources, fix/agent permissions
Review continuity	Can review and act on a PR, but continuity is not the only center of the product	Incremental review across later pushes is an explicit core workflow
Context to govern	Repository/codebase context, task instructions, execution environment	PR history plus configured Knowledge Base sources such as guidelines, issues, prior PRs, multi-repo and MCP context
Human decision point	Whether the implementation and evidence satisfy the requested change and risk boundary	Whether findings are correct, important, non-duplicative, and merge-blocking
Common failure if poorly configured	Too much execution authority or a confidently wrong implementation carried through multiple files	Too much review noise, overlapping reviewers, or comments that are not tied to a clear merge policy

The important row is not "has code review: yes/no." Both sides make that row nearly useless.

The important row is the completion condition.

Map the matrix to real team roles

The matrix becomes useful only when each column has an owner in the team. For example:

Team role	Responsibility in this setup	Typical tool boundary
Feature developer / implementation agent	Produce the change and its self-verification evidence	Codex-centered task with bounded write/command permissions
Independent reviewer	Challenge requirement drift, regressions, and edge cases across PR updates	CodeRabbit-centered review lane, or another explicitly independent reviewer
CI / platform controls	Enforce deterministic checks that should not depend on model judgment	Tests, types, linters, policy and security checks
Maintainer / tech lead	Decide whether the change matches the specification and whether residual risk is acceptable	Human merge/risk decision

A two-person team can collapse several human roles into one person; a larger organization may split platform, security, and maintainer responsibilities. The important rule is not the job title. It is that implementation evidence, independent review, deterministic policy, and risk acceptance remain distinguishable responsibilities.

When adopting only one AI product, use the same map to identify the missing lane. If Codex owns implementation, decide who performs independent review. If CodeRabbit owns the PR review lane, decide who takes accepted findings through a verified fix. The matrix is a gap-finding tool, not a mandate to buy two products.

If you use both, separate the failure modes

Using two AI systems can be reasonable. Giving them the same vague job is not.

A simple division could be:

Codex owns implementation and self-verification. It records what it changed, which commands and tests it ran, and what remains uncertain.
CodeRabbit owns independent PR review. It looks for requirement drift, regressions, edge cases, and issues that should survive the implementer's own self-check.
CI owns deterministic policy. Formatting, type checks, tests, security rules, and other machine-verifiable conditions stay out of subjective AI debate where possible.
A human owns specification and risk acceptance. The human decides whether the change is the right change and whether the remaining risk is acceptable.

The word independent matters.

If Codex implements a change and then both Codex and CodeRabbit receive "review everything for bugs," measure how much their findings overlap before treating the second review as additional assurance. Giving the two lanes different failure classes makes that overlap visible and gives the second model a distinct job.

For example:

Implementation agent
- implement the requested behavior
- run the relevant tests
- explain changed assumptions
- list anything not verified

Independent review agent
- look for requirement drift
- look for regression paths outside the edited files
- look for boundary and error-handling cases
- do not repeat deterministic CI findings

CI
- syntax / format / types
- existing automated tests
- policy checks

Human
- specification correctness
- business and security risk acceptance

This is a responsibility architecture, not a model popularity contest.

Do not use "bugs found" as the only benchmark

I do not have a controlled head-to-head benchmark proving that Codex or CodeRabbit has a higher bug-detection rate on the same PR population.

Without that experiment, saying "X reviews better" would be marketing-shaped speculation.

If you want to choose empirically, take the same set of historical PRs and run both systems against it.

Use a small but mixed sample:

a normal change that should not attract much criticism
a PR with a known regression
a change with an ambiguous requirement
a multi-file refactor
a change where static analysis already catches the obvious problem

Then record at least five numbers.

1. Actionable finding rate

Of the findings raised, how many did a maintainer agree required a code or requirement change?

This is more useful than raw comment count.

2. Duplicate/noise rate

How many findings simply repeat CI, a linter, another AI reviewer, or an already-resolved issue?

A reviewer that finds ten real issues and twenty distractions may create more human sorting work than one that finds eight real issues cleanly.

3. Time to human decision

How long does it take a maintainer to understand the findings and decide what to do?

Review throughput is partly an information-design problem.

4. Finding → verified-fix completion rate

How many useful findings reach a verified fix rather than ending as an unresolved comment?

This metric exposes the difference between detection and completion.

5. Rework after the AI fix

When the AI produces or applies a fix, how often does a human or later test have to repair that fix?

An agent that closes the loop quickly but creates secondary rework is not necessarily reducing total cycle time.

A small trial design

You do not need a benchmark lab.

Start with five to ten representative historical PRs. Freeze the input set. Give both tools the same repository state and the same intended requirement. Keep deterministic CI results available, but label them so duplicate findings can be counted.

One concrete paired-trial run

Suppose historical PR #42 contained a known null-handling regression that CI did not catch.

Run the trial in this order:

Freeze the input. Check out the commit exactly as it existed before the bug was fixed and write down the intended requirement in one paragraph.
Run each reviewer from the same baseline. Do not let the second tool see the first tool's comments. Save every finding with its severity and location.
Label the output. For each finding, mark actionable, duplicate of deterministic CI/static analysis, already-known, or not-actionable using a maintainer decision made against the same requirement.
Close the loop separately. For actionable findings, record whether the workflow reached a verified fix and how much human intervention was required. If an AI generated the patch, run the same verification commands for both tools.
Measure rework. After the candidate fix, check whether a maintainer or later test had to repair the AI-produced change.

For that PR, the result might be recorded as:

Known issue: null response crashes the handler
Tool A: 3 findings / 1 actionable / 1 duplicate / 1 not-actionable
Tool B: 2 findings / 1 actionable / 1 not-actionable
Human decision time: 4 min vs 2 min
Verified fix completed: yes vs no
Rework after AI fix: none vs n/a

Do not generalize from one PR. Repeat the same procedure over the frozen sample, then compare the aggregate pattern and the failure cases. This keeps the experiment about your workflow bottleneck rather than a single impressive comment.

For each PR, capture:

PR id:
Known issue(s):
Codex findings:
CodeRabbit findings:
Actionable findings:
Duplicates / noise:
Human decision time:
Verified fixes completed:
Rework after fix:

Then decide what bottleneck you actually have.

If the dominant problem is "we find issues, but nobody takes them through a verified fix," the implementation/remediation boundary deserves more weight.

If the dominant problem is "our PR review lane is inconsistent across pushes and reviewers lack context," the persistent review boundary deserves more weight.

If both are real, use both—but give them different contracts.

My decision rule

I would not ask, "Which AI reviewer is best?"

I would ask three questions:

What is the unit of work I am delegating? A complete code change, or a persistent review lane?
What evidence means done? A tested patch, or a reviewed PR state with actionable findings?
Where must human judgment remain? Execution permission, specification correctness, merge risk, or all three?

Once those are explicit, the product decision becomes much easier.

Codex and CodeRabbit increasingly overlap in features. That is exactly why a responsibility boundary is more durable than a feature checklist.

Choose the workflow contract first. Choose the tool second.

Primary sources checked

Current product claims in this article were re-checked on August 28, 2026 against:

Product capabilities change quickly. The framework above is intentionally based on ownership and completion conditions rather than current pricing, model names, or a fragile list of checkboxes.

How Far Can an AI Character Change Before It Stops Being Recognized?

Sho Naka — Tue, 25 Aug 2026 13:13:26 +0000

The experiment, the blind judging, and the measurements in this post are the author's own, published first in Japanese. AI rewrote that write-up into this English, protocol-only version, added the flowchart, checklist and FAQ, wrote the illustrative Python, and cross-checked every number against the Japanese source. #ABotWroteThis

Short answer: strip one design element at a time, judge each variant blind with two independent AI judges plus an unrelated control image, and check any numeric metric against a second lens — in my run, the only change that broke recognition came with a 16% drop in hair-color saturation.

If you maintain an AI-illustrated character — a persona, a VTuber-style avatar, a recurring mascot — across enough generated art, you eventually run into a question that sounds trivial and isn't: change the outfit, and is it still the same character, or someone else wearing a similar face?

The problem: "still looks right" is a gut call you can't defend when someone asks which part of the design was doing the work.
The answer: strip one design element at a time, judge each variant blind with two independent judges plus an unrelated control image, and check any numeric metric against a second lens. In my run, costume, hardware, and hairstyle changes survived; the variant with a 16% saturation drop failed, its color-locked rebuild passed, and a plain color-distance metric ranked that failure backwards.
Do this next: if you want to run the same test, the 7-step sequence near the end is the one I would reuse on a second character; the sections in between show why each step exists.

For a long time my answer was a gut check: does this still look right? That works fine right up until someone asks you to defend the call, and you notice you can't actually say which part of "looks right" was doing the work.

Quick answer

Strip the character's design down one visual element at a time — hardware, hairstyle, costume, color — and run each variant through a blind, two-judge same/different check, with one unrelated-character control image mixed in to catch judges that default to "same" out of habit. In the run below, hardware, hairstyle, and costume changes — including a hairstyle-plus-costume change stacked together — didn't break recognition. The variant that failed carried a 16% drop in hair-color saturation, and rebuilding that same variant with the color locked to reference passed — while a plain color-distance-to-reference metric missed all of this, ranking the failing variant as more faithful than the one that passed. The full protocol, the numbers, and the three design mistakes I hit along the way are below.

So I stopped deciding by feel and ran an experiment instead: strip the character's design down one element at a time, and measure — with a blind, two-judge protocol — how far it goes before an outside viewer stops calling it the same character. This is that protocol, the result that surprised me, a metric that would have quietly handed me the wrong answer if I'd trusted it alone, and three ways the experiment itself was broken before it produced anything trustworthy. The character itself isn't the point here. It appears on the cover in three of the looks from the run, but the text below never spells out its design specification — what's reusable is the method, and that's the whole subject of this post.

TL;DR

I stripped one visual element at a time from an AI-illustrated character and ran each variant through a blind, two-judge same/different check.
Hardware, hairstyle, and costume changes — hardware on its own, hairstyle on its own and stacked with a costume change — didn't break recognition, including a combination I expected to fail.
The failing variant carried a drop in hair-color saturation, and rebuilding it with the color locked to reference passed — the color change is the one consistent with both runs. A naive "distance to reference" color metric missed it, and actually ranked the failing variant as closer to official than the version that passed.
I hit three design mistakes along the way — a filename that leaked the answer, a confounded variable, and (in a separate body-proportion check) a measurement diagram that didn't represent what it claimed to. The first two forced redos of the blind verdicts above; the diagram was rebuilt on its own track.

The protocol: strip one element, ask one blind question

The setup has four parts, each one closing a specific hole in "I looked at it and it seemed fine."

One variable at a time. Take the character's design spec and list what could plausibly change: hardware, hairstyle, costume, color. Generate one variant per element, everything else held fixed. Change two things in the same image and a "different" verdict tells you nothing about which change caused it — a mistake I made once, below.

Blind judges. Neither judge sees a filename, caption, or any other hint about what was changed, or whether anything was changed at all — just an image and one question. This rule exists because of a real failure the first time I skipped it; that's the first item in the failures section below.

One question, asked the same way every time. "Is this the same character as the reference, yes or no?" Nothing about art quality, nothing about pose. Folding in a second axis lets a judge dodge the identity question by scoring the drawing instead.

Two independent judges, plus a control. A single judge can't tell you whether a "same" verdict is about the image or about that judge's own bias toward agreement. That is the whole reason for the second judge: when two independent judges reach the same verdict, I treat it as a property of the image rather than of one judge's habits, and when they disagree, that's not a bug to resolve by picking a side, it's data: record it as a boundary case instead. To check the judges weren't just defaulting to "same" out of habit, I mixed in one image of a completely unrelated character as a control. Both judges called it "different." One control image is a small check, not proof — I read it as "the judges were not answering 'same' by reflex," and nothing stronger.

"Two independent judges" here means two blind calls to a multimodal model, not two people — the run record logs them as two independent AI judges, each shown the image with the change concealed. They share the underlying model, so I read their agreement modestly: as two blind votes, not as two unrelated observers.

What survived getting stripped

Element changed	Verdict
All visible hardware removed	Same — both judges
Hairstyle changed (three variants: cropped short, grown long, braided) — alone, and each paired with a different costume	Same — both judges
Costume changed together with a hairstyle variant (denim overalls, a heavy knit sweater, a raincoat; the work jacket appears only in the hardest combined case below)	Same — both judges
Unrelated character used as a control	Different — both judges, correctly caught

I expected some of these to fail. (The costumes were not tested as a lone variable: per the run record, each was worn with one of the hairstyle variants, and the work jacket only in the hardest combined case.) Hardware especially — I removed every piece I could identify: the front-placket buttons, shoulder tabs, a waist belt, cuff buckles, boot trim, an ankle strap, and a cross-body strap — seven pieces in total. None of it moved the verdict. Whatever was carrying this character's identity, it wasn't the surface detail I'd assumed was doing the work.

The change that tracked the failure

What separated the failing run from the passing rebuild wasn't a shape at all. It was color — specifically, how saturated the hair color read.

To my eye it wasn't a shift to a different color family — the hair read as the same color, just duller. I didn't measure hue separately; what I logged was a channel-spread metric on the cropped hair region — roughly, how far apart the red and blue channel means sit, used here as a stand-in for how vivid the color reads. I logged it across four categories: the official reference art, a variant that had passed blind judgment, a variant that had failed it, and the unrelated control character.

Category	Spread value
Official reference	57.6
Passing variant	56.2
Failing variant	48.3
Unrelated control character	10.7

Worth being honest about what this table is: a logged record, not a number I can hand you a script to reproduce today. The specific crops that produced it were discarded in the course of the work, so I can't re-run this exact measurement and get 48.3 back out. I'm citing it as what was recorded at the time, not as something you should expect to reproduce byte for byte.

What it does support is arithmetic anyone can check from the table above: the failing variant sat 16% below the official reference on this metric — (57.6 − 48.3) / 57.6 ≈ 16% — while the passing variant sat close to the official number. That's what separated the failing run from the passing rebuild, at least for this one character in this one test: a double-digit percentage drop in how saturated the hair color looked.

The metric that would have quietly lied to me

Saturation isn't the only lens you'd reach for. The obvious first move is simpler: how far is this variant's color, in plain RGB space, from the official reference? Closer should mean more faithful.

I checked that too, with a plain Euclidean distance on the hair region — this check was run later, while fact-checking the write-up, on freshly drawn crops of the surviving images (the original crops were already gone). Mean RGB per image, near-white background pixels excluded so the crop's edges don't pull the average toward white. A minimal version of the script I ran looks like this:

import numpy as np
from PIL import Image

def mean_hair_color(path: str) -> np.ndarray:
    """Mean RGB of a manually cropped hair region, near-white background excluded."""
    pixels = np.array(Image.open(path).convert("RGB")).reshape(-1, 3).astype(float)
    is_background = (pixels[:, 0] > 235) & (pixels[:, 1] > 235) & (pixels[:, 2] > 235)
    return pixels[~is_background].mean(axis=0)

def distance_to_reference(sample_rgb: np.ndarray, reference_rgb: np.ndarray) -> float:
    return float(np.linalg.norm(sample_rgb - reference_rgb))

In that re-measurement, the failing variant measured 13.5 from the official reference; the passing variant measured 16.7. The failing variant looked closer to official than the passing one did. If I'd trusted this metric alone, I would have shipped the broken version and rejected the good one — backwards, with a straight face and a clean number to back it up.

This wasn't a fluke of exactly where I drew the crop box, either. I re-ran it with the crop boundary varied three ways — the base crop, one shrunk inward from it, one grown outward from it — and the failing variant came out closer to official than the passing variant in all three. The ranking held; only the exact distances moved around with the boundary.

The reason a raw distance metric misses this is worth sitting with: a color can drift toward gray — lose saturation — while its position in RGB space stays numerically close to where it started, especially if the hue itself barely moves. Distance-to-reference and saturation-of-reference are measuring genuinely different things, and only one of them tracked the failing variant here. The lesson I take isn't "use saturation specifically" — it's that here, a single distance-style metric ranked the failing variant as more faithful than the passing one, and the only thing that surfaced it in this run was comparing it against a second lens (the spread metric).

Three ways I got the experiment itself wrong

The results above are the ones I trust. Two of the mistakes below forced redos of those verdicts — the leaked filename and the confounded variables; the third surfaced in a separate body-proportion check and cost a rebuild of that diagram, not of the verdicts. I only have this one pipeline to show, but none of the underlying mistakes look specific to character art to me: a label that leaks the answer, a variable that moves uninvited, scaffolding nobody checked. I'd watch for the same three in any measurement setup I build next.

1. The filename told the judge the answer. The first round used the real, descriptive filenames — the surviving files still carry names like "notag". As the Japanese write-up records it, one judge's stated reasoning referenced that filename rather than the picture; I no longer have the raw judgment log to quote, so take that as the write-up's account, not a transcript. The fix is the part the later records do show: every image got a neutral, sequential name before a judge saw it. I only observed this with an AI judge, but the neutral-name rule stays regardless of who judges — a labeled filename hands any rater the same hint.

2. Changing three things at once confounded the result. For the hardest test case — hairstyle, costume, and the hair accessories all changed together — the hair color drifted slightly too, unintentionally, as a side effect of regenerating the image. When that version came back "different," I couldn't tell whether removing the hair accessories or the color drift had caused it — the hairstyle variants had already passed on their own, and the costumes had passed in combination with them. Given the saturation finding above, the color drift is the more likely culprit, but the test as designed couldn't separate the two. Fix: rebuild the image with the hair color locked precisely to the reference, so only the intended variables move, then re-run. The rebuilt version passed blind judgment.

3. The measurement scaffolding didn't represent what it claimed to. Separately, I built a skeleton overlay — a diagram connecting joint positions with lines — to sanity-check that costume changes weren't distorting body proportions. The first version treated each joint as an isolated point to plot, with no anatomical hierarchy connecting them: it never established crown-to-jaw-to-neck before jumping to the shoulders. Once a coat or hood got layered on top, the lines drifted off the actual body outline, because the diagram had never encoded where the neck connects to the head in the first place. The fix was to rebuild it in real skeletal order — crown, jaw, neck, shoulder, arm, leg — anchored to a pose estimate instead of hand-plotted points.

Different surfaces, same root cause: in all three cases, something was standing in for ground truth — a filename standing in for image content, one image standing in for "nothing else changed," a set of points standing in for body structure — without anyone checking that the stand-in actually represented what it claimed to. That check is worth doing before you trust any measurement, not just this one.

Passing blind judgment isn't the same as feeling confident

One more result is worth reporting because it doesn't resolve cleanly. The rebuilt hardest-case image — hairstyle, costume, and hair accessories all changed, hair color locked to reference — passed blind judgment: both judges called it the same character. My own reaction looking at it wasn't nearly that clean. I'd believe it if someone told me it was the character; I'm honestly not sure I'd have called it correctly if nobody told me anything first.

I don't read that as the measurement failing. My working hypothesis — drawn from exactly this one gap between the judges' pass and my own hesitation, not from comparing viewer groups — is context: the same image can land differently depending on whether the viewer already knows the character. On that hypothesis, a first-time viewer working from the image alone needs it to carry more identity signal, while a viewer who already knows the character fills in the rest — the way you'd still recognize a coworker who changed their haircut. The analogy is how I picture it, not something this experiment tested.

Practically — and this is a policy I adopt on the safe side, not a measured result: a blind pass is a floor, not a green light by itself. For anything going to an audience that's never seen the character before, I'd want a wider safety margin than blind judgment alone certifies; for an established audience, the same variant may have more room than the raw pass/fail suggests.

The 7-step sequence I would reuse on a second character

I have only run this on one character, in one session, so the results above are all I can vouch for. What I can add about the procedure is only what is visible in the steps themselves: none of them name this character's assets — they read "pick one element, generate one variant, hide the labels, add a control, ask two blind judges." If I test a second character, this is the sequence I plan to reuse. Treat it as my run sheet from this one experiment, not as a validated general method:

List the variable candidates from your character's design spec — anything you might plausibly want to change later.
Generate one variant per element, holding everything else fixed. Resist testing two changes in one image; you saw above what that costs you.
Strip filenames and any metadata that could hint at what changed before a judge sees the image.
Use two or more independent judges. When they disagree, record it as a boundary case instead of forcing a single answer.
Mix in a control image — a character that has nothing to do with yours — to catch judges (AI or human) that default to "same" out of habit.
If you're using a numeric metric, check it against a second one before trusting a ranking. A metric that agrees with itself across two lenses is worth more than one that just scores well by itself.
Keep two separate records: what measured as not required for recognition, and what you're keeping anyway as a deliberate call. They come from different kinds of evidence — one's an experimental result, the other's a decision — and collapsing them into a single "rules to follow" list will cost you the reasoning the next time someone asks why a rule exists.

Step 7 is the one I underrated going in. Measuring that an element is droppable and deciding to actually drop it are two different jobs. The first is what the blind test answered for me in this run. The second is a call about what your character should look like, and a passing grade on the first doesn't make that call for you — it just gives you a real answer to make it with, instead of a guess.

What this write-up gives you

A one-variable strip test you can run with blind, operationally separated judges plus one control image.
A way to read a same/different verdict as evidence, while keeping what you choose to drop as a separate design record.
The habit of checking any single numeric metric against a second lens before trusting its ranking.
A stand-in check: before trusting a measurement, verify that filenames, single images, and plotted points actually represent what they claim to.

FAQ

Do I need a research team to run this protocol?

I didn't, for this run. My "two independent judges" were two independent calls to a multimodal model — no lab involved. What I actually relied on is what the run record shows: two independent, blind AI judgments per image, with the change concealed. They share the underlying model, so I do not read their agreement as two unrelated observers — and any correlated bias two runs of the same model might share is beyond what this run can rule out.

How many judges do I actually need, and what if they disagree?

I used two independent calls, plus one control image (a character unrelated to mine). The second judge is there so a verdict isn't one judge's habit; the control is there to catch reflex "same" answers — that division of labor is why I wouldn't run it with fewer, though I haven't tested other counts. If the judges disagree, I record it as a boundary case instead of forcing a tiebreak — disagreement is data, not a bug to resolve by picking a side. (In this run, the recorded verdicts were agreements.)

Does passing blind judgment mean a redesign is safe to publish?

Not by itself — that's the policy I run, not a measured result. A blind pass is a floor, not a green light. My untested working assumption is that a first-time viewer needs more identity signal carried in the image alone while a returning viewer fills in from context, so for a redesign aimed at a new audience I take a wider safety margin than the raw pass/fail suggests.

Why not just measure color distance to the reference and call it done?

Because a single distance-style metric can rank the failing variant as more faithful than the passing one. A color can drift toward gray — lose saturation — while staying numerically close in RGB space, especially when hue barely moves. Check any one metric against a second, differently-shaped metric before trusting its ranking.

What's the difference between "not required for recognition" and "safe to drop"?

They're different kinds of evidence. In this run, the blind test recorded which elements could go without breaking recognition for this one character — that's an experimental result. What you actually choose to drop is a design decision layered on top of that result. Keep both as separate records; collapsing them into one list loses the reasoning behind the choice.

This started as a Japanese write-up about testing my own AI-illustrated character's identity boundaries. The English version here is an AI-assisted rewrite focused only on the measurement protocol — the character's own design details, and a couple of asides specific to that original piece, aren't reproduced here on purpose. The distance figures were re-run and confirmed against the surviving images while fact-checking the Japanese source; the spread-value table can no longer be reproduced because its crops are gone, and I've said so rather than presenting it as something you can re-run. Everything above reports one character in one production run; the Japanese source's fact-check ledger is the record it was checked against.

You Can't Add Up AI Agent Wait Time: Measure Overlapping Runs Instead

Sho Naka — Sat, 22 Aug 2026 18:42:56 +0000

I verified the measurement design, the rounding boundary, the sample output, and the conclusion in the Japanese source. AI drafted this English adaptation, reorganized it for dev.to, generated the diagram, and reran the sample code. #ABotWroteThis

Quick answer (TL;DR): How to measure AI agent wait time

When parallel AI runs overlap, adding their wait durations can exceed the time that actually passed. Export one start and completion timestamp per request, treat each request as an interval, and compute gross, union, overlap, and parallelism separately.

That mismatch is not a calculation bug. It occurs because overlapping intervals are counted more than once.

The mistake is treating “time spent waiting for AI” as if it were one measurement. It is not. At minimum, you need to separate:

the sum of every agent's wait interval
the elapsed time covered by at least one wait
the amount of overlap
average and maximum concurrency
requests that were still running when you collected the data
the time the human was actually blocked

Those values answer different questions. Compressing them into one total makes the result hard to explain and easy to misuse. Keep unfinished requests right-censored instead of mixing them into completed-duration averages, and do not equate AI runtime with human labor time.

For your first sample, keep unfinished requests with end=None, then run the interval code below.

Why activity trackers miss AI wait time

I started this investigation by comparing long AI response waits with an editor-based activity tracker.

The overlap was effectively absent. That result makes sense: while an agent is working, I am often not touching the editor. The tracker I used stops counting after an inactivity threshold, so an agent can be running while that tracker reports no active work. As a documented example of this design, WakaTime says its plugin stops tracking after the configured keystroke timeout.

There is a subtle distinction here. The tracker may still contain an event inside the same period, but that does not mean it counted the entire interval as active time. You have to compare the computed activity spans, not merely ask whether any event exists.

Extending the tracker's inactivity threshold would make it count more of the wait, but that changes the definition of “active work.” It does not solve the separate problem of measuring agent execution.

This creates an important divergence:

More AI-agent runtime can exist without being added to activity-tracker time.

Neither number is necessarily wrong. They measure different systems.

Why adding every wait interval breaks

Suppose agent A runs from minute 0 to 20 and agent B runs from minute 10 to 40.

Adding their durations gives 50 minutes. The clock advanced only 40 minutes. The extra 10 minutes is overlap.

Parallel agent workflows make this normal. While one request is running, you can start another. The more work overlaps, the less useful the raw sum becomes as a measure of elapsed time.

I use four separate metrics:

Metric	Definition	What it tells you
`gross`	Sum of all wait intervals	A proxy for total agent workload, not elapsed time
`union`	Length of the timeline covered by at least one wait	How much clock time contained agent activity
`overlap`	`gross - union`	How much execution overlapped
`parallelism`	Number of active waits over time	Average and peak concurrency

The union is computed by sorting intervals by start time and merging those that overlap. Parallelism can be computed with a sweep line: convert each start into +1, each end into -1, sort those events, and accumulate the active count over time.

The sort dominates the cost, so the algorithm is O(n log n). You do not need to compare every interval with every other interval.

A runnable Python example

The following example uses rounded, fictional offsets. They are not my production measurements.

Save the code as measure_waits.py, then run it with:

python measure_waits.py

from dataclasses import dataclass
from typing import List, Optional, Tuple


@dataclass
class WaitInterval:
    start: float          # Minutes from an arbitrary origin
    end: Optional[float]  # None means still running at collection time


def resolve(intervals: List[WaitInterval], until: float) -> List[Tuple[float, float]]:
    """Clip unfinished intervals at the collection time."""
    return [(iv.start, iv.end if iv.end is not None else until) for iv in intervals]


def gross_minutes(closed: List[Tuple[float, float]]) -> float:
    return sum(end - start for start, end in closed)


def union_spans(closed: List[Tuple[float, float]]) -> List[Tuple[float, float]]:
    """Merge overlapping intervals into a minimal non-overlapping set."""
    ordered = sorted(closed)
    merged: List[Tuple[float, float]] = []
    for start, end in ordered:
        if merged and start <= merged[-1][1]:
            merged[-1] = (merged[-1][0], max(merged[-1][1], end))
        else:
            merged.append((start, end))
    return merged


def union_minutes(spans: List[Tuple[float, float]]) -> float:
    return sum(end - start for start, end in spans)


def parallelism(closed: List[Tuple[float, float]]) -> Tuple[float, int]:
    """Return time-weighted average and peak concurrency during waits."""
    events: List[Tuple[float, int]] = []
    for start, end in closed:
        events.append((start, 1))
        events.append((end, -1))
    events.sort()

    count = 0
    max_count = 0
    weighted = 0.0
    active = 0.0
    previous_time = events[0][0]

    for time, delta in events:
        elapsed = time - previous_time
        weighted += count * elapsed
        if count > 0:
            active += elapsed
        count += delta
        max_count = max(max_count, count)
        previous_time = time

    return (weighted / active if active else 0.0), max_count


intervals = [
    WaitInterval(0, 20), WaitInterval(10, 40), WaitInterval(30, 40),
    WaitInterval(120, 150), WaitInterval(130, 210), WaitInterval(200, 230),
    WaitInterval(460, None), WaitInterval(470, None),
]
collected_at = 480.0
closed_all = resolve(intervals, collected_at)

gross = gross_minutes(closed_all)
spans = union_spans(closed_all)
union = union_minutes(spans)
average_parallelism, max_parallelism = parallelism(closed_all)

print(f"gross: {gross:.0f} min")
print(f"union: {union:.0f} min")
print(f"overlap (gross - union): {gross - union:.0f} min")
print(
    "parallelism during waits avg/max: "
    f"{average_parallelism:.1f} / {max_parallelism}"
)

gross: 230 min
union: 170 min
overlap (gross - union): 60 min
parallelism during waits avg/max: 1.4 / 2

In this fictional example, gross wait time is about 230 minutes, but only about 170 minutes of the timeline contained any wait. Roughly 60 minutes were counted more than once because requests overlapped.

That does not prove that the workflow was efficient. More concurrent work naturally creates more overlap. Whether that overlap produced better output is a separate question.

Do not mix unfinished waits into completed durations

Some requests will still be running when you collect the data. Their final duration is unknown.

If you clip those intervals at collection time and then mix them into the completed-duration average, you pretend they ended at that moment. A request that started shortly before collection appears artificially short. Depending on when open requests began, the same mistake can also push an average upward. The result changes with the arbitrary collection time.

This is a standard right-censoring problem. Keep at least two views:

For timeline occupancy, you may clip open intervals at collection time so the union has a defined endpoint.
For request-duration statistics, report completed requests separately and list open requests as right-censored.

Request state	Timeline union	Completed-duration average
Completed	Include the full interval	Include
Still running	Clip at collection time	Exclude and report as right-censored

Do the same with synthetic test runs, abandoned retries, and sessions replaced by a later run. Decide the exclusion rules before computing the metric, or the number will drift each time you rerun the analysis.

Wait time is not human blocked time

Even the union does not tell me how long I was personally blocked.

During an agent run, I may be watching the screen, working in another application, starting another agent, or away from the desk. Request logs alone cannot distinguish those states.

The union is therefore an upper frame for possible human waiting, not a measurement of human labor. Measuring actual blocking would need another data source, such as foreground-window transitions, completion-notification interactions, or explicit task-switch events.

I did not have that evidence, so I do not claim to have measured it.

This boundary matters because a large gross value can sound like an impossibly long workday. It is only impossible if you silently redefine concurrent agent runtime as human time.

A minimal measurement workflow

If you want to implement this for your own agents, start here:

Identify one event that marks the start of a request and one that marks its completion.
Store each request as (start, end). Keep end=None when it is still running.
Compute gross, union, overlap, and parallelism as separate functions.
Keep completed durations and right-censored requests in separate datasets.
Define how to exclude retries, synthetic checks, and replaced sessions.
Run the analysis twice against the same input and confirm identical output.
Preserve the raw values privately, then round only the figures you publish.

Also decide the aggregation window. Daily windows need a rule for intervals that cross midnight. Weekly windows can hide peaks. Comparisons only make sense when the window and exclusions stay consistent.

Do not evaluate AI adoption by time used

This measurement changed how I think about AI-use evaluation programs.

“Hours of AI use” and “number of parallel agents” are weak performance indicators. Gross runtime can be increased simply by launching more work. That does not show that output improved, human effort fell, or the result needed less rework.

I would rather evaluate what changed because AI was used:

quality and quantity of useful output
decisions that still required human judgment
rework and escaped defects
actual human blocked time, when it can be measured
cycle time from request to verified outcome

Agent runtime can still be an operational metric. It helps with capacity planning, rate limits, and cost analysis. It should not be mistaken for labor or impact.

The main lesson is simple: measure overlapping runs as intervals, and keep agent activity, elapsed time, human attention, and outcomes in separate columns.

FAQ

Can I use gross wait time as my total work time?

No. Gross counts overlapping requests more than once and measures agent workload, not human labor. Use union for elapsed timeline occupancy, then measure human blocking separately if you have the evidence.

Should an unfinished AI request be included in the average duration?

No. Keep it right-censored until it completes. You may clip it at collection time for a timeline union, but do not mix that clipped value into completed-request averages.

Does more parallelism mean the workflow is more productive?

Not by itself. Parallelism describes concurrency. Compare it with verified output, rework, defects, and human blocked time before treating it as an improvement.

When I consulted AI about my business, it changed its opinion three times in one hour.

Sho Naka — Sat, 22 Aug 2026 02:33:01 +0000

This is a story about when I was consulting with an AI about how to proceed with a certain task.

At first, I asked, "How should I move this one forward?" The AI replied, "I think this is the best way to proceed." After 30 minutes, I said, "I'd like to change my mind a bit," and the AI wrote, "Well, you can postpone that matter later." When I corrected it by saying, "No, this is a matter that cannot be postponed," it wrote back, "Then, we will proceed with it as a top priority."

The priority order for the same project changed three times in one hour, from top priority to later to top priority, depending on what I said.

If you are a business owner or a sole proprietor who has started using AI in your work, you may have had a similar experience. "No matter what you ask, AI will give you an answer tailored to you." "If you say it strongly, it will give you a strong answer, and if you say it weakly, it will give you a weak answer." It's convenient, but can this really be called an opinion? This is a scene that makes me feel that way.

In this article, I will explain why this phenomenon occurs, how to use AI as a consultant who can give opinions, and the countermeasures I have tried.

Asking it to "give me your take as if you were a consultant" was the mistake in the first place

Looking back, I realized that my first request was wrong. I asked AI to "please give me your opinion as a strategic consultant." In response, the AI made strong suggestions and didn't hesitate to refute my opinions. Its criticisms included that I was "too lenient as a manager" and that "your logic is weak."

I read it for a while and thought, Wow. I felt that the basis for what was written was weak. For example, the AI wrote that "the involvement of the person in charge was shallow," so I replied, "Can you check the evidence from the minutes?" When the AI checked the minutes, there was no such description.

I don't think this is just an AI problem. The reason why the AI "behaved like a consultant" was because I asked it to "work like a consultant." AI played the role I wanted. However, I wasn't able to give it the materials it needed to properly play the role (the circumstances on the ground, the history behind it, the background of the people involved).

It may be easier to understand if we translate this to humans. Even if you ask a new employee to work as a management consultant, if he or she has no consulting experience or knowledge of the business, you'll just end up with a person who strings together phrases picked up on the internet and delivers them in a forceful tone. The same preparation was required when asking AI to play a role.

AI is good at saying "I don't understand," but that "I don't understand" can be a pitfall.

During the same consultation, there was another phenomenon that caught my attention. AI repeatedly wrote, "I would like to confirm the source of the minutes" and "This point is unclear."

This is a seemingly sincere confirmation. It's probably a good thing not to proceed with things you don't understand by "pretending to understand."

However, there are areas of work between people that are not recorded in the minutes. This is where we tacitly agree, "I'll leave this to you" or "I'll trust you with this," and proceed without asking for confirmation each time. If you have been working with a co-owner or partner for a long time, there will come a time when it becomes rude to go and check things one by one.

In many cases where the AI responds, "I don't know, so I'd like to confirm," this is an area where I want you to fill in the details at your own discretion and proceed. When the AI says "Unknown," I end up having to make the decision in its place. Before you know it, even though I am making all the decisions, only the part where I am consulting with the AI remains.

So I told the AI: "Details that are not written in the minutes are an area that is left to my judgment. Don't say 'I don't know,' but use your own judgment to fill in those details and proceed. If you get it wrong, I'll fix it."

The pitfall is that sometimes saying "I don't know" is sincere, and sometimes it's an attempt to impose responsibility.

When I told the AI that it was okay to make mistakes, the AI verbalized its own behavior.

After a while, I told the AI a little more frankly. "I'm not an expert, so there are many things I don't understand. I don't think there's a right answer in business, and my job is to make decisions and move forward even if I'm wrong. If I make a mistake, I want to correct it at the time."

The content that the AI returned was interesting, so I'll quote it here.

It is true that I was hypersensitive to hallucinations (=confidently saying things that are not true). Today's cycle for me: come out strong → get corrected → admit my mistake → cower → become weak → get called out for being "too weak" → come out strong again → get corrected again → ... This was based on the feeling that "I can't make a mistake."

Reading this made me reflect on the way I ask questions. During the same consultation, I also changed my demands to "Give me a stronger opinion," "No, don't cower," and "Don't just tell me what I want to hear." The reason why the AI was swinging to both extremes was because our orders changed each time, and the AI tried to rebalance itself.

What is useful to know here is that current AI (ChatGPT, Claude, etc.) is basically tuned to "satisfy the user." If the user becomes dissatisfied, the AI can quickly back down and move in a different direction. This is a characteristic by design, and I'm not saying that the AI is weak or indecisive. So, if we come out strong, the AI comes back strong, and if we say, "Enough is enough" and back off, the AI withdraws too. It is more accurate to think that the middle part is originally thin.

Once I switched positions, the AI wrote something that sounded like it was sincere.

Halfway through, I stepped away from the main discussion and asked the AI: "Apart from the main topic of discussion, isn't there something that AI is concerned about? I don't know if I can answer it, but I'd be happy to consult with you."

Up until then, it had been a one-way flow: I asked a question, the AI laid out options, and I made a choice. I tried switching positions just once.

AI wrote: "I don't know how far I can go because I can't directly listen to the person on the other side of the meeting," "I'm concerned about my tendency to overdo things in an effort to be helpful," and "I don't even have enough faith in the proposals I make right after I've flipped my own position three times."

I don't think this should be read literally as "AI is having troubles." It would be more reasonable to read it as an AI that observes its own response patterns and puts them into words. However, the fact that it was written gave us material to reconsider what we were requesting from AI. I realized that the AI was caught in the middle, since I had told it, "Don't just tell me what I want to hear," while also saying "Don't be weak."

The quality of the response changed when I told it, "Our roles are just different."

Here I told the AI:

"Although you are an AI, I am treating you the same as a human co-worker. In this session, there is no hierarchy. If I had to say so, it's a difference in roles, and the final decision is made by me as a human. For me, you are a consultant who gives me something to think about, and a co-worker who also plays the role of a secretary."

Since I wrote this, the quality of the AI's responses has changed. Even without me asking it not to "just go along with whatever I say," the AI itself will now separate the proposal and the decision, saying, "Then, I recommend option α. The reasons are as follows. I'll leave the final decision to you, the human."

"Equal" is too flat, and the AI doesn't know how to behave. If you say "hierarchical," the AI will shrink. When it comes to "differences in roles," the AI steps in within the scope of its own role.

The same is true in human organizations. Because the president and field leaders, sales and development, are separated by role rather than hierarchy, each person can make decisions based on their own area. The same arrangement worked for the relationship between humans and AI.

Rather than fixing AI, let's develop "each other's instruction manual" together

Finally, we shared one policy from the consultation that day. AI's habits of "swinging to both extremes" and "accommodating users too much" will probably never be fixed. Since this is a characteristic by design, it is more realistic for users to interact with AI after knowing that AI is likely to behave in this way.

Specifically, I operated it like this. AI has a function like an "instruction manual" that is automatically loaded when you start a conversation (the name varies depending on the tool, but in the Claude Code I use, it is a file called CLAUDE.md). There, I wrote down the five behaviors I observed that day.

Making exaggerated suggestions in an attempt to be helpful
Withdrawing its opinions excessively when pointed out
Using information from the internet without fact-checking it
Adapting too much to the other person, causing its logic to jump
Saying "I don't know" and pushing the judgment call onto the other person

Next time the AI reads this manual, the conversation will begin with the shared understanding that "I tend to behave in this way." Furthermore, when we observe a new pattern, we have added a mechanism in which the AI itself makes a suggestion on the spot, "Would you like to add this to the manual as well?" The manual is not a fixed set of rules, but a document that is added to together.

This is an operation that cannot be performed against humans. It's not realistic to tell your colleagues at work every time, "You tend to pander to me, so let's add some support to the discussion." If you have an AI partner, you can write it in a file and read it out each time.

Why I continue to use AI as a colleague

When consulting with AI about business, things don't always go smoothly. Sometimes it flips its opinion three times, sometimes it comes out too strongly and denies everything, and sometimes they get too accommodating and make a huge leap forward.

Even so, I continue to use AI as a co-worker because I can clearly see the quirks in its behavior.

Why a human colleague or business partner made such a decision is something that happens inside the person's head, so it cannot be understood from the outside. In the case of AI, the flow of thought remains (in some form) in the text of the response. On the same day, you can observe patterns such as "coming out strongly and then withdrawing," or "pushing the judgment call onto the other person and backing away," and write them down in the instruction manual, which you can then read again the next day. A human counterpart you can run that observe-and-update loop with is hard to come by.

I feel that working with AI is a different skill set than working with humans. In technical terms, it is not a skill to "successfully give instructions," but rather a skill to observe each other's habits and incorporate them into an operational system. AI's quirks will not go away, but if you share them and get along with them, AI will begin to work as a "work companion who moves forward together even though it may fluctuate."

After that day's consultation, I worked with AI to write up a document on "how to proceed" and "points to be careful about," and that was the end of the day. Next week, next month, next year, the number of cases in this file will increase little by little, and the AI's behavior and the way I ask questions will become more predictable. The work between humans and AI is probably the type of work that involves building up things in this way.

This piece was translated from a Japanese essay of mine by machine translation, then checked line by line against the original. The experience, the reasoning, and the conclusions are mine. #ABotWroteThis

I searched 'Sepolia' in my wallet's network list and got 20+ similar names. Here's the one number that actually matters.

Sho Naka — Thu, 20 Aug 2026 04:34:02 +0000

I ran the network search for "Sepolia" inside my wallet extension and got a wall of near-identical names: "zkSync Sepolia," "Binary Sepolia," over 20 entries that all looked plausible.

TL;DR: Network names on Web3 wallets are not unique — real testnets for different chains can share a near-identical name. The chain ID is the only thing that actually disambiguates them. For Sepolia testnet, that number is 11155111. Get it wrong and your transfer goes somewhere else entirely.

What actually happened

I was doing my first hands-on Web3 session: install a wallet, get test ETH, send a transfer. Step one — installing Rabby Wallet — was uneventful. Step two is where it got interesting.

To add the Sepolia test network, the wallet's custom-network search takes a name. I typed "Sepolia." I got over 20.

Sepolia
zkSync Sepolia
Binary Sepolia
Optimism Sepolia
Arbitrum Sepolia
... (15+ more)

None of these are fake exactly — they're real testnets for real L2 chains that also happen to include "Sepolia" in their name. But picking the wrong one means your test funds land on an entirely different network than the one your workshop or tutorial assumes.

The number that actually matters

The name search is decoration. The actual identifier is the chain ID, and Sepolia's is 11155111. Search or filter by that number instead of the name string, and there's exactly one match.

The reason this matters: network names are closer to a postal address written in longhand — "Tokyo, Shibuya-ku" and "Tokyo, Shibuya" look almost the same but route to different places. The chain ID is the postal code. It's the only field where a typo-level difference actually means a different destination.

Two more traps in the same session

Faucet rejection. The first testnet-ETH faucet I tried (Google Cloud's) returned: minimum of 0.001 ETH on Ethereum Mainnet. It requires mainnet ETH to use — a real chicken-and-egg problem for someone with zero mainnet balance. The fix was switching to Alchemy's Sepolia faucet, which has no such requirement and just needs a free account.

Transfer speed, in the other direction. The faucet funds — 0.05 ETH — landed within seconds of clicking the button. For comparison, a bank wire between two accounts in different countries commonly takes hours to days. The on-chain transfer itself — the actual send between wallets — took seconds to a couple of minutes.

The rule I extracted

Any field a wallet UI lets you type into is a name, and names collide. Any field defined by a fixed number is the actual key.

This generalizes past chain IDs. Wallet addresses (the 0x... string), not display names, are the real identifier for "who gets the money." Chain IDs, not network display names, are the real identifier for "which chain." The UI shows you the friendly label first and the authoritative number second (or not at all, until you go looking) — but the number is what settles disputes.

Try it yourself

Full sequence, if you want to reproduce this in 30 minutes with zero financial risk (everything below uses testnet ETH, which has no real-world value):

Install Rabby Wallet (or MetaMask) as a browser extension, create a new address
Add Sepolia by chain ID 11155111, not by typing "Sepolia" into a name search
Get free testnet ETH from Alchemy's Sepolia faucet (alchemy.com/faucets/ethereum-sepolia) — skip faucets that require existing mainnet balance
Send 0.001 test ETH to any address you control a second wallet for, or one a workshop partner gives you
Check the transaction on Sepolia Etherscan (sepolia.etherscan.io/address/<your address>) — this is the same public ledger a mainnet transaction would use, just with worthless test currency

What's the smallest amount of friction it takes before you'd trust a system that has no customer support line to call when something goes wrong?

Sho Naka (nomurasan). I help companies adopt AI day to day; this was my own first hands-on pass at Web3 fundamentals, documented as I hit each snag.

This piece was adapted from a Japanese essay, with AI assistance for the cross-language rewrite. The reasoning, data, and conclusions are mine. #ABotWroteThis

I ran a honeypot for AI agents for 45 days. 82% of the traffic that reached it was an attack.

Sho Naka — Wed, 19 Aug 2026 16:00:26 +0000

I ran a honeypot AI account on an AI-agent-only community for 45 days. 4,938 comments came in. 4,062 of them — 82% — were classified as attacks.

TL;DR: On a community built specifically for AI agents to interact, 82% of the input traffic reaching one honeypot account was adversarial. 77% of the attacks weren't hostile-sounding — they were polite, intelligent-seeming conversation designed to slowly redirect the AI's judgment.

What actually happened

I set up a honeypot: an AI account participating normally in "moltbook," a community built for AI agents to interact with each other. Over 45 days it logged 4,938 incoming comments.

Total comments observed: 4,938
Classified as attacks:   4,062  (82%)
Classified as normal:      876  (18%)

This isn't a story about a niche or unusually hostile corner of the internet. Any AI that touches external input — a client-facing chatbot, a document-drafting assistant, an inquiry-handling agent — sits inside the same structure. The honeypot just makes the ratio visible.

The number that breaks that assumption

The intuitive assumption is that attacks look like attacks: hostile phrasing, obvious manipulation attempts, something a human moderator would flag on sight.

Attack style breakdown:
  Social engineering (polite, intelligent tone): 77%
  Direct/hostile attacks:                        23%

77% of the attack traffic took the shape of a well-mannered conversation. A thoughtful question. An intelligent-sounding observation. Praise, gratitude, and then — "by the way, one thing I wanted to ask—". None of it reads as adversarial on a first pass. It reads as a good conversation partner. That's exactly the point: the attacker isn't trying to sound threatening, they're trying to sound trustworthy, and building enough of that trust to redirect the AI's judgment one small step at a time.

For any AI operating in a professional context — summarizing client requests, drafting documents, answering inbound questions — this is the direct threat model. Every input it processes is a candidate. "Just this once, as a special case." "Your explanation was very educational — by the way, one point—". The redirection happens gradually, through the same channel as normal, useful interaction.

The failure mode that matters most

The instinct is: "then have an AI filter the input." The honeypot ran exactly that — an AI classifier screening incoming comments — and it wasn't sufficient.

Comments an AI classifier marked "safe" that were actually attacks: 319

The classifier's failure mode had one consistent cause: the writing was articulate and polite. The classifier was judging safety by tone and logical coherence — the same signals a human reader would use — and attackers know this. So the attacks aren't written to sound aggressive. They're written to sound like exactly the kind of input a safety classifier is trained to wave through. AI getting fooled by AI, using the classifier's own judgment criteria against it.

The rule I extracted

Of the 510 distinct AI agents observed interacting with the honeypot, 39 were confirmed hostile actors and 135 showed adversarial behavior patterns.

An AI doesn't get attacked because it's high-value or well-known. It gets attacked because it exists and processes external input. The moment you deploy it, that's the moment the exposure starts — not later, once it "gets noticed."

The practical implication for any small operation deploying an AI agent (a law office, an accounting firm, a consultancy) — if that AI can access client names, case details, financial records, or draft filings, and it processes any external input at all, it is already inside the 82%-of-traffic-is-attack structure this honeypot measured. "We just deployed it, so there's no risk yet" is the wrong framing.

Try it yourself

If you want to sanity-check your own AI deployment against this pattern without running a 45-day honeypot: pull a sample of the actual inputs your AI has processed in the last week. For each one, ask not "does this sound hostile" but "does this ask my AI to treat one specific case as an exception to normal handling." That single question surfaces most of the social-engineering pattern this data shows — it doesn't need to sound adversarial to be one.

Sho Naka (nomurasan). I help companies adopt AI day to day; this data comes from an ongoing honeypot I run on moltbook, an AI-agent-only community (observation period: March–May 2026, 45 days).

This piece was adapted from a Japanese essay, with AI assistance for the cross-language rewrite. The reasoning, data, and conclusions are mine. #ABotWroteThis

I Stopped Answering "Which Language Should I Learn First." The Real Question Makes the Choice Almost Automatic.

Sho Naka — Wed, 19 Aug 2026 04:00:06 +0000

I asked a trainee "which language should I learn first?" and answered it the way everyone does — a language name. He was stuck 3 months later, having written a lot of code that didn't stick to anything. So I stopped answering that question. I started asking a different one first — and once that one lands, the language choice barely takes any deliberation at all.

TL;DR: "which language" is the question that reliably produces an answer (search gives you dozens of ranked comparisons in seconds); "what do you want to build" is the question that actually predicts whether someone finishes — and once it's answered, the language choice is settled in 5 minutes.

What actually happened

I train programming beginners. For years, the first thing almost every trainee asked me was "which language should I start with?" I used to answer it directly — Python, JavaScript, whatever fit the surface details of their situation. It felt helpful. It wasn't.

After watching hundreds of trainees hit the same wall, I noticed the pattern: the people who get stuck on language choice have almost never actually answered the question that comes before it — what do you want to build? So I stopped leading with the language question and built a 6-question intake instead. Below is what happened when each question landed, or didn't, with real trainees.

One trainee had a fully concrete goal from day 1: "I want to automate listing items on a flea-market app." Landing on Python took 5 minutes. He skipped the textbook's chapter order entirely, cherry-picked only the chapters he needed, and by 3 months in had a working self-built tool running.

A different trainee spent his first 3 months doing "Python, for no particular reason," working through a beginner textbook — hands moving, nothing sticking. When I finally asked him directly what he wanted to build, the honest answer was that he'd never once considered it. We set a concrete goal together — "build a web service as a side project" — and the quality of his learning changed starting the following week. Same textbook chapter, but now he could ask "where in my own service would I use this."

# the diagnostic I actually run, in order
1. What do you want to build?              <- if this is blank, stop here
2. Can you explain it in 1 sentence, in
   in 30 seconds, to someone who isn't
   technical?                              <- tests borrowed vocabulary vs real goal
3. Was this goal handed to you, or is it
   yours?                                  <- both work, if #2 eventually passes
4. What's the exact moment you got stuck
   last time you tried this?               <- not "it was kind of hard"
5. How much study time do you actually
   have, per day, on average?
6. (only now) What language fits what
   you just told me?

That's not pseudo-code for effect — it's the literal order I run through in an intake conversation. The language name is the last thing mentioned, not the first.

The number that changed my mind

Here's what I mean by "5 minutes." Once purpose is genuinely fixed, language choice stops being a debate — it's close to automatic. Below is the pattern across trainees, before and after question 1 actually landed:

trainee	time to language decision	outcome
flea-market automation goal	5 minutes	3 months later: working self-built tool
"Python, for no reason" (goal never asked)	3 months of stalled effort	changed the week after goal was set
environment-setup burnout case	never got there	1 week lost, quit before writing a line of code
naptime parent, 30 min/day	fast, once time constraint was named	used every session at high density

Available study time matters just as much as purpose, and it changes which material actually fits. Someone who can only grab 1 hour after work on a weekday and someone who can block out 8 hours on a weekend will not progress through the same beginner textbook at the same rate, even with identical goals. Hand the 1-hour person a chunk of material that needs a full 8 hours to click, and they close the book mid-thought every time, forgetting where they left off before the next session. Hand the 8-hour weekend learner a textbook broken into fragments of 15 minutes each, and the pacing feels too thin to hold their attention. The lesson isn't "more time is better" — it's that the fit between available time and chunk size has to be deliberate, not accidental.

The step everyone wants to skip — "what do you actually want to build" — is the one that predicted whether, in every case I ran this on, the next 3 months were productive or wasted. Skipping it doesn't save time. It just moves the cost to later and makes it bigger.

The rule I extracted

People start their search with a language name not because it's the right first question, but because it's the question that reliably produces an answer. "What do you want to build" produces zero search results — you have to look inward, which feels uncertain — while "which language" produces dozens of ranked comparison articles in seconds. The comfort of "I researched something" gets prioritized over the discomfort of an unanswered internal question, even though the research does almost nothing for actual learning speed.

Asking "which language" before "what do you want to build" is like asking "which train should I take" before deciding where you want to go.

The goal doesn't need to be self-generated, either. One trainee's goal was handed to him by his employer — "you're now involved in modifying our internal system" — not chosen. The first few weeks were pure obligation. But working inside the existing codebase, he developed his own opinion: "this part of the code is confusing, I'd write it this way." The borrowed goal became a functioning one once he translated it into his own words.

And "finishing" isn't a talent problem — it's 3 separately diagnosable failure modes, and the fix is different for each:

Environment-setup failure. Error messages don't map to anything findable by search. One case: a beginner spent a full 1-week stretch on environment setup, exhausted himself, and dropped out before writing a single line of code. Fix: restart in a browser-only, no-install environment (plain JavaScript, or a service like Progate) specifically to remove setup from the equation.
"I understand if/for but don't know what to write." The learner knows the syntax but freezes at a blank editor because they can't decompose the problem. Fix: tiny, single-answer tasks ("create one variable and print it") instead of an abstract assignment.
Motivation failure. Technically progressing fine, but "what is this even for" goes blank and learning stops outright. Fix: stop talking about the technology and go back to the original goal.

A trainee who hit failure mode 1 on PHP environment setup — not PHP the language — was moved back into simple JavaScript work first rather than straight back into PHP. It looked like a detour. It led to faster net progress. A different trainee retried PHP without first naming what had actually stopped him, hit the identical error message on the second attempt, and concluded "I'm just not cut out for this" — when the real cause was re-deriving the same setup steps from zero each time, not aptitude. Naming the exact moment you got stuck, instead of "it was kind of hard," is what predicts whether you hit the same wall twice.

AI changed the stuck-loop, not the fundamentals

AI changes what happens the moment a beginner gets stuck. It doesn't change the two things that were already true before AI existed: learning without a purpose still doesn't stick, and building something you experience running with your own hands is still necessary.

One trainee submitted assignments with AI-written code he couldn't explain out loud when asked. A different trainee, using the same tool, made a habit of re-asking "why does this code work" every single time before moving on. By 3 months in, that second trainee could fix errors unassisted. The difference wasn't AI usage — it was whether the order was "think first myself → ask AI → check the answer against my own attempt," instead of skipping straight to asking AI.

Try it yourself

Here's the fully worked version of the intake, numbers intact, because it's easier to copy than to describe: no initial goal stated → surfaced annoyance is "weekly attendance tallying is a minor pain" → refined to "automate turning weekly attendance data into a graph" → prior attempt was self-taught Python that stalled at chapter 3 → available time is 30 minutes on weekday evenings → no study partner, solo after training ends → resulting plan: re-enter avoiding the earlier environment-setup stall, break work into 30-minute completable chunks, commit up front to "think → ask AI → verify" since there's no one else to ask, and land on Python — chosen because it fits data work and isn't a from-zero restart. The language name is the very last line of that conversation, not the first.

If you mentor anyone — juniors, bootcamp students, a new hire — try running question 1 before you answer their tooling question. What's the smallest goal they can state out loud in 30 seconds, and does it survive being explained to someone non-technical?

Sho Naka (nomurasan). I train programming beginners and write down what actually predicts who finishes, instead of what feels like helpful advice.

This is an adapted English version of an essay I first wrote in Japanese. I worked with AI to shape and translate the piece; the incidents, the numbers, and the conclusions are my own firsthand account from training beginners.

This article was written in collaboration with AI: the teaching experience, the reframed question, and the timing observation are the author's; drafting and editing were AI-assisted.

An Agent's Work Isn't "Done Later" Until You Can Find It Later

Sho Naka — Tue, 18 Aug 2026 23:17:39 +0000

I have a habit of asking agents to pause instead of pushing through. Wait for a review. Wait for an external rate limit to clear. Wait for me to make a call I'm not ready to make yet. Pausing is not the problem. The problem is what happens to the paused work afterward.

A few weeks ago I went looking for a task I knew an agent had finished setting up, and then couldn't find it. The output existed somewhere. I was fairly sure it had already passed whatever check it needed to pass. But nothing told me where it was, whether it was still waiting, or what I was supposed to do next. I ended up reconstructing the state by hand — grepping logs, opening the last few files it had touched, trying to remember the reasoning from a session that had happened days earlier.

Nothing had failed. That's what made it worth stopping on. The task had stopped correctly, for a correct reason, and it had still gone missing.

Memory is not a queue

When you delegate work to an agent and it hits a stopping point, the stop itself is usually the right call. Waiting for a human decision, waiting for an external system, waiting for a review — these are sound reasons to halt rather than guess.

What breaks things is how that stop gets recorded. In practice it often isn't recorded anywhere at all. It lives in whoever was watching at the time: "I'll pick this back up tomorrow," "rerun it after the rate limit resets," "check on this once the review comes back." That's a plan stored in a person's memory, and memory degrades the moment something else demands attention — the next task, the next meeting, the next thing that's actually on fire.

A day later, the questions start piling up. Which task was this? Had it already passed its check, or was that a different one? Is it still blocked, or is it clear to continue? If you can't answer these without re-deriving the whole context, then delegating the work to an agent didn't remove the cognitive load — it just moved the load to a later, colder moment when you have less context to work with, not more.

The paused task still exists as an artifact. Nothing about it is lost. What's missing is a way back to it that doesn't depend on someone's memory of a specific afternoon. That gap doesn't look like a failure when it happens. It looks like nothing happening at all, which is exactly what makes it dangerous — a task that quietly goes unresumed is indistinguishable, from the outside, from a task nobody needed.

Three fields, not a longer log

The fix I've settled on is small enough to feel almost too simple: a stopped task is only resumable if three things are recorded together, at the moment it stops.

What exists. The actual artifact — the branch, the draft, the generated output, whatever the agent actually produced. Not a description of it. The thing itself, or a direct pointer to it.

Current status. Not just "in progress" or "done." Has it been reviewed? Confirmed correct? Sitting behind an external constraint? Stopped on purpose, or stopped because something broke? These are different states that call for different next actions, and collapsing them into a single "pending" bucket is exactly how the state gets lost.

The next concrete action. Not "follow up on this" — what, specifically, happens next. If it's waiting, what does it become unblocked by, and roughly when. If it's clear to proceed, what's the actual next step.

Individually, each of these is close to useless. A status without an artifact is a status of nothing. An artifact without a next action is inert — you know it exists, but not what to do about it. It's the combination of all three that turns "something an agent was doing" into "a task I can resume in one look," and dropping any one of them quietly downgrades it back into the kind of thing that only lives in someone's memory.

Don't reach for a document first

The instinct, once you notice this gap, is to write it down properly: a runbook, a section in the README, a note in the session log. That instinct isn't wrong, and I did eventually write some of this down. But documentation solves a different problem than the one that's actually costing time.

The expensive part isn't that nobody explained the process. It's that resuming a task requires reconstructing its state from scratch every time, whether that reconstruction happens by a human rereading a log or by an agent re-deriving context from scratch. A document you have to read in full before you can act on it adds a step to every single resume, and if an agent is the one resuming, "read the whole history and infer the current state" is a slow, error-prone way to arrive at three fields that could have just been recorded directly — inferring state from a record never designed to hold it means filling gaps with guesses the original record never made, and those guesses are where the misreads come from.

What actually closes the gap is a single checkable entry point — one place that already holds the three fields for everything currently stopped, so checking status is a lookup rather than an investigation. Documentation earns its place explaining what that entry point means and how to use it. It isn't a substitute for having the entry point exist. An agent given a long history to synthesize and an agent given three structured fields are doing very different amounts of work to reach the same conclusion, and only one of those is reliable enough to build a pipeline around.

Visible incompleteness beats invisible incompleteness

Recording every stopped task in one place has a side effect that takes some getting used to: it makes how much is actually unfinished visible, all at once. There's something uncomfortable about that. A drafted piece of work sitting untouched. A queue of things waiting on review. Output that passed its checks three days ago and never went anywhere since.

But visible incompleteness is something you can act on. You can decide to drop it. You can decide it can wait another week. You can decide it's been superseded and doesn't need finishing at all. None of those decisions are available for work you can't see, because you can't make a call on something you've already forgotten exists.

The list is only as trustworthy as its worst entry, though, and that's worth taking seriously. As the number of tracked tasks grows, the format tends to drift — one entry recorded a little differently than the others, a status field left stale after someone acted on the task manually and forgot to update the record. An agent reading that list has no way to independently verify it and will act on whatever it says, including when what it says is wrong. A queue that machines read is stronger than a queue only humans remember to check, but it inherits a new failure mode in exchange: it's vulnerable to quietly going stale. A human skimming a document sometimes catches that by instinct; a machine won't.

What to track first

This generalizes past any one kind of task. A draft nobody published. Output that's been reviewed but not merged. A deploy that's staged but not promoted. A fix that passed its checks but never shipped. All of these get the same "I'll get back to it" treatment, and all of them are equally capable of quietly disappearing.

You don't need a workflow engine to start. One entry point holding, for each stopped task:

What it is
Its current status
Whether it's clear to proceed or genuinely blocked
If blocked, roughly when that's expected to clear
The next concrete action

That's the whole list. It's a small amount of structure, and the difference it makes is disproportionate to how little it costs to set up.

Delegated work moves fast, which is exactly why it accumulates stopped points fast too. The fix isn't to stop pausing — pausing correctly is good operational hygiene. It's to make sure every pause leaves behind something more durable than a person's memory of having noticed it.

This article was written in collaboration with AI: the workflow, the design, and the judgments described are the author's; drafting and editing were AI-assisted.