DEV Community: Sho Naka

My Audit Caught Every Known Bug. Ten of Its Next 18 Flags Were Wrong.

Sho Naka — Sat, 19 Sep 2026 10:49:39 +0000

In a July 2026 AI-assisted audit of my automation, a checklist caught all seven known defects it had been designed from. Applied to 13 previously unaudited guards, it produced 18 findings. The recorded adjudications accepted eight and rejected ten.

That is 10 rejected findings out of 18, not “56% of healthy guards flagged.” My earlier draft used that misleading description. A guard could receive several findings, and the sample was not established to be healthy.

The useful question became: when this checklist flags something new, what evidence makes the finding worth acting on? Here is what the records showed, where the checklist overreached, and a small reporting pattern you can use on your own reviews.

AI assistance: AI agents performed the historical audit and recorded the findings and adjudications. The counts were reconstructed and this article was edited with AI assistance. The measurements below are a recount of those July records, not a fresh execution of the guards or an independent re-adjudication of every label. #ABotWroteThis

Quick answer

A checklist built from seven known defects caught all seven, which only proves it can describe the failures that shaped it. On 13 previously unaudited guards it produced 18 findings, of which 10 were rejected: 44.4% precision among resolved findings, measured per finding, not a per-guard false-positive rate. Before quoting any percentage from a review, keep accepted, rejected and unresolved findings as separate rows and recount them.

Seven known defects made a regression set, not a benchmark

The checklist contained nine criteria derived from seven confirmed defects. They covered problems such as a missing required input being treated as success, a selector that matched nothing and silently skipped checking, and destructive work without an adequate bound on its effect.

Checking that all seven cases were caught was useful: the checklist could describe the failures that motivated it. But those cases had already influenced its design. They could not establish how reliably it would recognize a different defect, or how often it would accuse correct behavior.

This is the same evaluation concern behind keeping development data separate from test data. I am applying that principle to a checklist here, not claiming that these seven examples constitute a statistical evaluation of a trained model. Scikit-learn's guidance on data leakage explains why results on data used during development can be overly optimistic.

The next audit therefore mattered more than the neat “7/7” result.

Count the findings before naming the percentage

The archived records support this comparison:

Recorded audit	Guards examined	Findings	Accepted as defects	Rejected as defects
First application beyond the seven design cases	13	18	8	10
Follow-up after criteria changes, on a different sample	12	4	2	2

For the first audit, 8/18 findings were accepted: 44.4% precision under the recorded labels. The complementary share, 10/18, was 55.6% rejected findings.

The denominator matters. Precision is TP / (TP + FP). A false-positive rate is FP / (FP + TN): it needs the actual negative cases, including ones the detector correctly left alone. The precision definition and confusion-matrix layout make that distinction explicit.

Here, the recorded labels attach to findings. “Thirteen guards examined” is a different unit. I cannot combine those numbers to calculate a per-guard false-positive rate. Nor do these findings tell me how many defects were missed in the newly examined code.

There was another reason to recount. One prose summary inside the first record said 11 rejected out of 19 flags. Its structured totals said 10 out of 18, and its 18 individual finding rows contained eight accepted labels and ten rejected labels. This article uses the row count. The summary sentence was inconsistent with the underlying entries.

An audit report can need an audit of its own arithmetic before its conclusions are useful.

“Empty input” was too broad a defect definition

Six of the first audit's ten rejected findings came from one criterion: treating a degenerate or empty-input result that looked like ordinary success as a defect.

That criterion had a real motivation. A mistyped selector can match nothing, skip every check, and leave a reassuring success status. But “nothing matched” is also a legitimate outcome for some tools. An optional collection can simply contain no work today.

The old wording collapsed those cases. A better contract asks what the program has promised to verify:

Situation	What a useful contract should distinguish
A required source directory does not exist	Input failure; this is not an empty collection
An existing, optional collection contains no work	Successful no-op, if that is the documented behavior
An explicit selection intended to identify required work matches nothing	Invalid selection or an explicit unresolved result, according to the tool's contract

These are proposed contract examples, not a claim that every tool should use the same exit code. The important test is whether the caller can distinguish a valid no-op from a failed precondition.

Before accepting an empty-input finding, I would want the finding to name the intended contract, supply an input that violates it, and show the observed result. “It returned success on zero items” is an observation. Whether that is a defect still needs a specification.

The second audit did not prove an improvement

After the criteria were tightened, the follow-up record contained four findings across 12 guards. Two were accepted and two rejected: 50% rejected findings under those labels.

It is tempting to write “false positives fell from 56% to 50%.” That would imply more than these records establish. The guard sample changed, the criteria changed, and the follow-up contained only four findings. This was not a paired before-and-after experiment on a fixed set of independently labeled cases.

One follow-up rejection also showed why scope matters: a limitation identified in one check was covered by another check in the execution path, according to the recorded verification. Whether a finding describes a component limitation or an unprotected system behavior must be explicit. A limitation can still deserve documentation without proving that the complete path violates its contract.

The follow-up supports a modest conclusion: the revised process still produced findings that were rejected after examination. It does not establish the size of a quality improvement, a reduction in missed defects, or the current performance of the guards.

A report that keeps unknowns visible

You do not need a dashboard to avoid the denominator mistake. Start by keeping accepted, rejected, and unresolved findings separate. This standalone Python example reconstructs the first audit's label counts; it is a reporting example, not the original private audit dataset.

from collections import Counter

# Reconstructed counts from the recorded adjudications.
labels = ["accepted"] * 8 + ["rejected"] * 10
counts = Counter(labels)
allowed = {"accepted", "rejected", "unresolved"}
if set(counts) - allowed:
    raise ValueError("Unknown finding label")

accepted = counts["accepted"]
rejected = counts["rejected"]
resolved = accepted + rejected
report = {
    "unit": "finding",
    "findings_total": len(labels),
    "resolved": resolved,
    "unresolved": counts["unresolved"],
    "precision_among_resolved": accepted / resolved if resolved else None,
    "rejected_share_among_resolved": rejected / resolved if resolved else None,
    "per_guard_false_positive_rate": None,  # Required labels unavailable.
    "recall_on_new_cases": None,  # Missed defects not established.
}
print(report)

For these labels, the two calculated shares are approximately 0.444 and 0.556. If you add an unresolved finding, it increases the total and unresolved count without being silently classified as correct or incorrect. The two shares then describe only the resolved subset; they may not represent the pending findings. With no resolved findings, both remain None rather than becoming a reassuring zero.

For your next ten review findings, keep one row per finding with the criterion, reproduction evidence, expected behavior, verdict, and reason. Preserve unresolved cases. Recount those rows before quoting an aggregate percentage.

If you then want to compare two versions of the checklist, define the evaluation unit and labels first, freeze a separate evaluation set, and run both versions against it. Include legitimate no-op cases as well as known defects. Treat the original seven defects as regression tests, and keep their result separate from the new-case evaluation.

FAQ

Why is "7 out of 7 caught" not evidence that the checklist works?

Because the seven cases shaped the checklist. They are a regression set: useful for confirming the checklist still describes the failures that motivated it, but unable to say how it behaves on a defect it has not seen or how often it accuses correct behavior.

What is the difference between precision and a false-positive rate here?

Precision is accepted findings over all resolved findings (8/18). A false-positive rate needs the true negatives, meaning the guards the checklist correctly left alone, and those were never labeled. The records attach labels to findings, so a per-guard rate cannot be derived from them.

Did tightening the criteria reduce false positives?

Not demonstrably. The follow-up audit had a different guard sample, changed criteria and only four findings (two rejected). Showing an improvement would need a frozen evaluation set with independent labels, run against both versions of the checklist.

Catching every familiar failure was a useful start. The more useful result was learning exactly what I could—and could not—conclude from the next eighteen findings.

I'm dropping spec-first AI development for smaller experiments

Sho Naka — Sat, 19 Sep 2026 02:55:45 +0000

The decisions and experiments here are the author's own. An AI assistant drafted the English text from the author's notes, and an independent AI review checked the links and the one third-party claim before publication. #ABotWroteThis

Quick answer

For my own reversible development work I am replacing specification-first as the default with small working experiments: state the outcome, build the smallest change that lets me try it, and judge the running result. Prohibited operations are rejected by a pre-execution hook rather than described in a warning, and the main measure is elapsed time from request to a usable result. This is a decision under test, not a measured productivity gain.

I gave my coding agent a link to Ponytail, a set of rules intended to stop agents from overbuilding. It returned a cautious assessment: pushing for fewer files and one-line implementations might hurt readability or fit poorly with the existing architecture.

My response was a question: readable for whom?

If an agent does most of the reading, changing, and checking, I want the structure that helps it make the next correct change with the least context. An architecture should earn its maintenance cost, even if I designed it myself.

That conversation led me to a decision: for my personal development work, I want to replace the default specification-first process with small working experiments. I want short instructions for judgment, hard enforcement for prohibited operations, and elapsed time to a usable result as the main measure.

This is a decision I am testing, not a report of proven productivity gains.

Keep the outcome stable. Let the implementation change.

My reason for changing this default is the human cost of evaluating the plan. Even if an agent writes a careful plan instantly, I still have to imagine the resulting behavior and find what is missing.

I have asked an AI to check implementation concerns in plan mode and proceeded carefully, yet the resulting system did not meet the customer's needs. That experience does not establish that plan mode caused the failure. It does show that checking whether something can be implemented is insufficient evidence that it will serve the customer's work. Plan mode and specification-driven development are not identical; I am questioning the shared habit of expanding a document before testing the behavior it describes.

Some behavior only becomes clear to an industry expert or someone doing the work. For example, correcting an already processed request might require preserving its history and returning it for review, rather than overwriting it. Both behaviors are technically implementable. This is an illustrative example, not a claim about the customer system. A more internally consistent plan cannot supply a missing business premise by itself.

I would rather spend ten seconds writing a prompt, let the agent implement a small change in a minute, and judge the result. Those numbers describe the short loop I want, not a measured guarantee. A working interaction can make a mismatch easier to identify than a description that I must simulate in my head.

By spec-first, I mean making a written specification, plan, and detailed task breakdown the default prerequisites to implementation. Tools such as GitHub Spec Kit put specifications at the center of development. I am changing that default for my own reversible work, not claiming that every specification-driven process is slow.

Suppose I want to find an item in an existing list. I can ask an agent to design a search subsystem, or I can ask it to make the smallest working filter and let me try it.

The first experiment needs a concrete outcome: a user can find a known item by its title. It does not need a commitment to a particular component library, search service, or abstraction.

I would start with a brief like this:

Outcome: find a known item by title in the existing list.
Use existing code or platform features where they meet that outcome.
Build the smallest reversible change that lets us try it.
Check a matching query, a query with no matches, and clearing the query.
Follow the existing authorization boundaries.
Report what works, the observed limitation, and the next change if needed.

Those checks define this experiment's scope. They do not prove that a production search feature handles pagination, large datasets, permissions, accessibility, or every language correctly. If the list only contains one loaded page, that limit must be visible when judging the result.

The useful loop is short: build, try, change or discard. A prototype that cannot answer its intended question has not completed the experiment.

Readability is the cost of the next correct change

Before considering code structure, I need to address a trap in the shorter loop: the agent and I can keep saying “it works” while neither of us has the missing domain knowledge. Replacing plan iterations with implementation iterations does not automatically introduce customer evidence.

Marty Cagan distinguishes value, usability, feasibility, and business viability risks. Checking that we can build something addresses only one of them. The Four Big Risks

For the next experiment, I want a concrete example from the actual work, appropriately anonymized, rather than only tidy dummy data. Existing understood behavior can be checked automatically. An unresolved business decision should be shown to a practitioner through that specific example. I do not need to ask the customer about every edit; I need their knowledge where it can change the next decision.

Martin Fowler describes discovering requirements by observing what users try to do and where they abandon a task. That is a useful counterweight to checking only our own written assumptions. Observed Requirement

Another trap is accumulating every quick prototype. If an experiment disproves the approach, discard the implementation. Fast generation stops helping when each addition makes the next change harder to understand and verify.

Keep the next change affordable

Ponytail's rules ask the agent to look for existing code, the standard library, native platform features, and installed dependencies before writing something new.

That order appeals to me. A native date input can remove a custom component from the code I must maintain, provided it meets the actual requirements.

But I am not optimizing for the smallest line count. A dense expression that needs repeated explanation can cost more than a few straightforward statements. Equally, a small change spread across wrappers and duplicate definitions can require more context than keeping the behavior together.

My question is: how much reading, reasoning, editing, and verification does the next change require? If a human will maintain the result, their understanding belongs in that cost too. “It matches the existing architecture” is evidence of consistency, not proof that the architecture is worth preserving.

Make prohibited operations impossible through the controlled path

I also told the agent that I want enforcement, not warnings.

A long instruction saying “be careful” still asks the model to interpret the rule on every relevant turn. A warning that allows an operation to continue leaves the final decision elsewhere.

For a mechanically decidable prohibition, I want a pre-execution hook or equivalent control to reject the operation. The response should identify the failed condition and a permitted next action. If the agent can repair the request within its authority, it should continue.

This only works when the operation actually passes through the control. Blocking one shell command does not enforce a boundary if another tool can perform the same write. Enforcement belongs at the relevant execution boundary, with the paths it covers verified.

Product quality is a different job. A hook cannot generally decide whether the filter is useful or whether the screen makes sense. I want those questions answered by the working result and a focused check, rather than an expanding collection of vague warning rules.

Optimize the whole request, not the last response

The agent also noted that adding a plugin can add hook processes. That is a cost to measure, but it is not a reason to reject the approach by itself.

A small deterministic check might replace repeated instruction processing and warning conversations. Or a verbose ruleset might add more input and reasoning than the shorter implementation saves. The balance has to be measured.

Ponytail's author reports exactly this kind of variation. A separate single-shot experiment found higher cost and worse latency with GPT-5.5. That is not a result for my workflow, but it is enough to reject “less code always means cheaper and faster” as an assumption. Cost verification

I therefore want to replace redundant instructions, not keep adding new ones. A short reply that stops halfway through the task and requires another context reload is not automatically efficient.

Start the clock at the request

When I asked the agent to turn this decision into an article, I added: elapsed time is itself part of the result.

I want to measure from the request to the artifact I can actually use or judge. Research, setup, waiting, and repairs stay on that clock. If I ask for a working feature, the clock does not stop at code generation. If I ask for an experiment, it stops when the intended question has been tested.

For a first comparison, I would record:

Measure	What it tells me
Request-to-result elapsed time	How long I actually waited
Total recorded tokens, where available	The model work spent across the request
Repair rounds	Whether the initial shortcut moved work later
The same outcome checks	Whether the two results are comparable

A fair comparison keeps the starting state, model settings, requested outcome, and necessary checks the same. Without that comparison, I can report a duration; I cannot attribute an improvement to dropping specifications.

I am starting with one reversible task. Keep the outcome clear, build only enough to test it, enforce the known prohibitions, and count the full time to the result. If that loop does not improve delivery, the next thing to question is the process I just introduced.

FAQ

Is this an argument against specification-driven development in general?

No. It is a change of default for my own reversible, personal work. Tools such as Spec Kit exist for cases where a shared written specification is the cheapest way to align people; my problem is the cost of evaluating a plan in my head when a running change would answer the same question faster.

Why enforce prohibitions with a hook instead of instructions?

An instruction asks the model to re-interpret the rule on every turn, and a warning that lets the operation continue leaves the decision elsewhere. A mechanically decidable prohibition belongs at the execution boundary, where it can reject the operation and name a permitted next action. It only works if every path that can perform the operation goes through that boundary.

How will I know whether the shorter loop is actually faster?

By measuring request-to-result elapsed time, recorded tokens where available, repair rounds, and the same outcome checks, with the starting state and model settings held constant. Until that comparison exists I can report a duration, not an improvement.

This article was drafted with AI from my actual discussion and decisions. The proposed workflow's productivity effect has not yet been established.

When AI Becomes Production Infrastructure, You Need Recovery Capacity

Sho Naka — Fri, 18 Sep 2026 00:40:56 +0000

This article is based on the author's experience operating AI-assisted development workflows. The author identified the failure mode and defined the Recovery Reserve concept. AI helped recompose the article for DEV readers and prepare the Japanese review back-translation. #ABotWroteThis

Quick answer

If AI materially expands how much work your team can keep in motion, recovery becomes AI-dependent too. "We will do it manually" is not a fallback when the workload was sized for AI-assisted throughput. Keep enough inference capacity to execute the critical recovery path (Recovery Reserve), define a fallback provider and a migration context, reduce demand in a degraded mode, and decide emergency purchasing authority and cost attribution before the incident.

Generative AI has changed how much work I can keep moving at once.

I use it to write code, investigate existing systems, organize requirements, compare architecture options, generate tests, review implementations, write documentation, and trace dependencies across a project.

That is useful. It also creates an operational problem that is easy to miss:

AI can increase production capacity faster than it increases your ability to recover without AI.

Once a project has been planned around AI-assisted throughput, losing that capacity is not just an inconvenience. The workload remains, while the system that made the workload manageable disappears.

That is the point where AI stops behaving like a productivity feature and starts behaving like production infrastructure.

"We'll just continue manually" is not always a fallback

I noticed this during a development project where AI was already part of the normal workflow.

Research, architecture, implementation, debugging, and review were all being accelerated with AI. Then sufficient AI capacity was no longer available.

The obvious fallback sounded reasonable:

If AI is unavailable, continue manually.

But the remaining workload had already been shaped around AI-assisted throughput. Switching execution back to unaided human work reduced processing capacity without reducing demand.

That is not graceful degradation.

It is a capacity failure.

A real fallback must change both sides of the equation: the available production capacity and the amount of work the system is expected to carry.

AI is becoming production capacity, not just a tool

If one editor stops working, I can use another. If one terminal application breaks, I can switch tools.

Those tools improve ergonomics and productivity. They do not necessarily determine the amount of work I can keep in motion.

AI increasingly does.

It can compress investigation, create implementation scaffolding, review broad changes, and turn vague requirements into an initial design and test strategy. Once those capabilities become part of normal delivery, removing them changes the capacity of the production system itself.

That changes the question from:

Did we use too much AI?

into:

Did we provision enough inference capacity to deliver the committed workload and recover it when something goes wrong?

We already ask similar questions about compute, storage, CI runners, databases, failover, and backups. AI-dependent engineering work needs the same operational framing.

Personal spending can hide a capacity failure

Suppose company-provided AI capacity is exhausted while business-critical work remains.

A practical reaction might be:

I'll pay for another AI service myself and finish the work.

That can solve an immediate incident. It is a bad default operating model.

If an employee quietly supplies missing production capacity, the organization sees a successful delivery rather than an under-provisioned workflow. The next project can then be estimated with the same insufficient assumptions.

This resembles hidden overtime, except the hidden input is inference capacity rather than labor time.

If AI is part of production, its shortage and its cost need to be visible.

AI can expand the blast radius of a bad assumption

AI accelerates correct decisions. It also accelerates incorrect ones.

A bad assumption introduced early can quickly influence architecture, implementation, tests, documentation, follow-up issues, and later features. Human execution speed used to impose a natural limit on how quickly an assumption could spread. AI weakens that limit.

By the time the mistake is found, the task may no longer be a local patch. It can become a reconstruction exercise: identify everything derived from the assumption, restore the correct boundary, regenerate affected work, and verify the result.

That recovery workload can itself require substantial AI assistance.

The loop looks like this:

Use AI
  ↓
Increase throughput
  ↓
Increase work in motion
  ↓
A bad assumption or incident occurs
  ↓
Recovery workload grows
  ↓
Recovery also needs AI

Now add the worst timing:

AI capacity is unavailable when recovery begins

That is the operational failure mode this article is about.

Recovery Reserve: protect the recovery path, not a percentage

I call the protected inference capacity for this situation Recovery Reserve.

It is capacity kept available for work such as:

incident investigation
debugging
re-analysis
regeneration
code review
rollback
emergency documentation
migration to another model or workflow

The important point is not to prescribe a universal reserve percentage. A fixed percentage would imply precision I do not have.

Instead, start with a recovery scenario.

Ask what must happen if the primary model is unavailable, if a bad assumption has propagated widely, or if the current work must be handed to a different model. Identify the investigation, decisions, edits, and verification steps on that critical path.

Then protect enough capacity to execute that path.

The reserve is adequate when the recovery path remains executable, not when a dashboard shows a particular percentage unused.

A fallback provider is an operational dependency

Important AI-dependent work should not have only one viable inference path.

A more resilient setup has a primary model, a fallback model or API route, and a documented degraded mode. But the degraded mode matters as much as the fallback model.

"Humans will do everything if AI goes down" is not necessarily degradation. If the workload was created around AI-assisted throughput, handing the same workload to the remaining human capacity does not create resilience.

A degraded mode reduces demand.

For example, keep incident response and acceptance-critical work moving while pausing new features, optional automation, and documentation that can safely wait.

Graceful degradation means matching the workload to the remaining capacity.

Inference cost becomes part of project cost

AI subscriptions are often discussed like ordinary SaaS expenses.

Once AI capacity affects delivery throughput, incident recovery, and required human effort, its economic role changes. Inference consumption and protected recovery capacity become production inputs.

A useful project cost view can include:

Human labor
Cloud infrastructure
External services
AI inference
Recovery headroom

Even when a contract does not expose precise per-task cost, you can still record operational events: capacity exhaustion, fallback-provider use, manual fallback, and recovery-mode execution.

That evidence is more useful for the next capacity plan than an apparently successful project whose missing AI capacity was silently paid for by an individual.

A first-pass business continuity table

Once AI behaves like production infrastructure, the team needs decisions that are operational, not just conceptual.

You do not need a perfect business continuity program before starting. Fill out this table first.

Failure surface	Decide now	Passing condition
Primary AI is unavailable	Fallback model or API route	Critical recovery work can start through another approved path
Inference capacity is constrained	What Recovery Reserve is protected for	Investigation, correction, and verification can run without consuming normal-production capacity first
Work must move to another model	Migration context	Goal, canonical source, current diff, acceptance criteria, verified evidence, and unresolved items can be reconstructed without copying secret values
Capacity is reduced	Work stop order	Incident and acceptance-critical work stays; deferrable work is explicitly paused
Emergency capacity must be purchased	Decision authority and recording rule	The authorized person and project-cost destination are known before the incident
AI cost must be attributed	Usage events to record	Normal production and recovery activity can be distinguished later at project level
A bad assumption could fan out	Checkpoint before high-propagation work	Assumption, acceptance criteria, and verification method are rechecked before broad generation or automation

Migration context deserves special attention. It does not mean saving an entire chat transcript.

The useful handoff is the smallest package that lets another model reconstruct the task safely: what outcome is required, which artifact is canonical, what changed, what defines completion, what has been verified, and what remains unresolved. Secret values stay in the secret store; the handoff contains references to where authorized tools should retrieve them.

After the table is filled out, rehearse an important recovery task without the primary AI path. Wherever the exercise stops is the next missing piece of Recovery Reserve, documentation, authority, or context.

AI operations begins where AI productivity needs resilience

For a long time, most AI adoption work focused on acceleration: better prompts, better models, agents, automation, and parallel execution.

Those still matter.

But once AI is deeply integrated into real delivery, another layer becomes necessary:

control
rollback
capacity limits
provider failure
model substitution
blast-radius reduction
recovery

The important question is no longer only:

How much faster can AI make us?

It is also:

How do we safely stop and recover work that has been accelerated by AI?

AI adoption starts with productivity.

AI operations starts when that productivity itself has to become resilient.

FAQ

How large should the Recovery Reserve be?

There is no universal percentage. Start from a concrete recovery scenario, such as the primary model being unavailable or a bad assumption having propagated, list the investigation, decisions, edits and verification on that path, and protect enough capacity to run it. The reserve is adequate when that path stays executable.

Is switching everything to human work a valid degraded mode?

Only if demand is reduced at the same time. Handing an AI-sized workload to the remaining human capacity cuts processing capacity without cutting the work. A degraded mode keeps incident response and acceptance-critical work moving and explicitly pauses deferrable work.

What belongs in a migration context for another model?

The smallest package that lets a different model reconstruct the task safely: the required outcome, the canonical artifact, what changed, the acceptance criteria, what has been verified and what is unresolved. Secret values stay in the secret store; the handoff carries references, not a chat transcript.

Your AI Video Passed QA—and Still Wasn't Publishable

Sho Naka — Tue, 15 Sep 2026 13:40:58 +0000

I defined this acceptance model after rejecting technically valid AI-generated videos in my own production pipeline. AI helped reorganize the Japanese framework for DEV readers and draft the examples. I am not claiming a measured quality increase or a completed three-video stability proof. #ABotWroteThis

The MP4 rendered.

It had an audio track. It had subtitles. ffprobe returned the expected codec, resolution, streams, and duration. The timeline checks passed.

I still rejected the video.

The background did not support the explanation. Sentence boundaries and visual changes disagreed. Speaker turns had no believable pause. The characters moved, but the result felt like two avatars standing still and opening their mouths.

The pipeline had succeeded technically. The artifact had failed as media.

That distinction changed how I define "done" for AI video generation.

I no longer use one completion state. I use four:

moves
usable
accepted
stable

Automated QA can prove the first. It cannot prove the other three.

Automated QA proves validity, not publishability

Automated checks still matter. They should catch deterministic failures before a person spends time watching the result.

For example:

the video file opens;
video and audio streams exist;
duration, codec, and resolution are within contract;
subtitles are present and their timestamps are ordered;
the timeline has no empty or overlapping segments;
every required artifact and source trace exists.

Those checks should not be weakened. They should be made stricter.

The mistake is using their PASS result as a proxy for audience acceptance.

A valid audio stream does not prove that pronunciation and pauses feel natural. A subtitle file does not prove that captions and visuals are not fighting for attention. One image per segment does not prove that the background, framing, or character movement carries meaning.

Technical validity is necessary. It is not the release decision.

Separate four release states

State	What it proves	Exit condition
`moves`	The pipeline produced a technically valid artifact	MP4, audio, subtitles, provenance, and automated QA pass
`usable`	A first-time viewer can follow it without production-side explanation	No major first-watch disruption; any required fixes are localizable
`accepted`	The release owner and target viewer accept the actual artifact	`publishable_as_is` or narrowly defined `publishable_with_minor_edits`
`stable`	The same production method works beyond one lucky example	Three different topics are consecutively Accepted under one series-template hash

This is a state machine, not a score.

moves does not gradually become usable because the codec score is high. accepted is not usable + 10 points. Each transition needs different evidence.

To move from moves to usable, somebody has to experience the artifact at normal speed.

To move from usable to accepted, the production team cannot be the only judge.

To move from accepted to stable, one successful video is insufficient.

Review the same artifact three different ways

A normal first watch mixes script, pronunciation, pacing, framing, captions, and visual continuity. When the result feels wrong, that integrated impression is real—but it is often too entangled to tell the pipeline where to return.

I use three passes in a fixed order.

Pass 1: audio only

Close the video or turn the display away. Listen without captions or visuals.

Check:

pronunciation of names and technical terms;
phrase boundaries, emphasis, and speed;
pauses between statements;
pauses when the speaker changes;
the gap between a question and its answer;
whether the argument and causal chain are understandable from audio alone;
whether the delivery fits the intended speaker or character.

If the explanation cannot be followed from audio, do not add more captions and images to conceal it. Return to the script, narration treatment, or affected audio segments.

Pass 2: silent visual

Mute the audio and hide the captions.

Check:

whether the background communicates topic, location, or state change;
whether characters do more than stand and lip-sync;
whether framing, expression, gaze, objects, and cuts carry information;
whether repeated background, placement, and motion create visual stagnation;
whether a viewer can recover at least the current topic from the screen.

If meaning disappears when the captions disappear, adding more text is not a visual fix. Return to the storyboard, visual treatment, or shot plan.

Pass 3: one integrated first watch

Now watch the complete artifact once, at normal speed.

Do not pause to explain what the production system intended.

Check:

whether audio, visuals, and captions compete for attention;
whether visual changes align with the same semantic units as the narration;
whether the background or character performance lowers the credibility of the explanation;
where the first urge to stop watching occurs;
which release verdict applies.

Use a small verdict set:

publishable_as_is
publishable_with_minor_edits
revise
redesign

The most important output is not an average score. It is the first material blocker and the layer that owns it.

Record the first blocker, not an average

Suppose pronunciation scores 8/10, visuals 7/10, and captions 9/10. An average of 8/10 looks healthy.

It can still be unpublishable.

If a product name is mispronounced in the first few seconds, later quality does not cancel the error. If the background implies the opposite of the narration, a polished art style does not make the claim trustworthy.

I store a review result like this:

review:
  pass: integrated_first_watch
  verdict: revise
  first_blocker:
    category: pacing
    segment_id: s07
    evidence: "The question ends and the answer starts without a perceptible turn-taking gap."
  return_to: audio_treatment
  allowed_fix_scope:
    - pause_before_s08
    - regenerate_audio_s07_s08
  material_revision: false

This record answers the operational questions that a score cannot:

What failed first?
Where did it fail?
What evidence supports the rejection?
Which production layer owns the repair?
What may change in the next run?
What must remain fixed?

Make the release boundary machine-readable

A human may supply the observations, but the decision boundary can still be explicit.

from dataclasses import dataclass
from enum import Enum


class Verdict(str, Enum):
    PUBLISHABLE = "publishable_as_is"
    MINOR = "publishable_with_minor_edits"
    REVISE = "revise"
    REDESIGN = "redesign"


@dataclass
class PassResult:
    name: str
    verdict: Verdict
    first_blocker: str | None = None
    return_to: str | None = None


def release_decision(results: list[PassResult]) -> dict:
    for result in results:
        if result.verdict in {Verdict.REVISE, Verdict.REDESIGN}:
            return {
                "accepted": False,
                "failed_pass": result.name,
                "first_blocker": result.first_blocker,
                "return_to": result.return_to,
            }

    return {
        "accepted": True,
        "minor_edits_only": any(
            result.verdict == Verdict.MINOR for result in results
        ),
    }

This is not a claim that computer vision or an LLM can reliably infer every perceptual defect.

The purpose of the code is narrower: once observations exist, do not let a scoring average erase an observable release blocker.

Define "minor edits" narrowly

publishable_with_minor_edits easily becomes a polite name for "not actually accepted."

I limit minor edits to changes such as:

a typo;
a local volume adjustment;
one segment's pronunciation;
one caption timing correction;
a short replacement that does not change the argument or structure.

I treat these as material revisions:

restructuring the script;
regenerating a substantial part of the narration;
changing the background or overall screen design;
changing the character's role or personality;
redesigning pacing across several segments.

If a material revision is required, that run does not count as Accepted. The repaired artifact becomes a new run and goes through all three passes again.

This rule prevents acceptance metrics from improving only because every rejected artifact was renamed "accepted with edits."

After the same rejection twice, stop micro-tuning

Small parameter changes are useful for a local defect:

add a little more pause;
reduce speech speed;
adjust one transition;
change one crop.

But if the same rejection category appears twice, the problem may no longer be a parameter.

I use this return rule:

same_rejection_category >= 2
    -> stop micro-tuning
    -> return to audio treatment or visual treatment
    -> create a new run

Repeating the same ambiguous prompt with slightly different numbers produces more artifacts without improving diagnosis.

Moving up one design layer makes the next hypothesis testable.

One Accepted video is not a stable pipeline

A template may work for one topic because the script happens to fit its timing and visual rhythm.

Change the topic and the pipeline can fail immediately.

I separate a single success from operational stability by keeping a consecutive-acceptance streak tied to one series-template hash.

def update_stability_streak(
    streak: int,
    accepted: bool,
    same_template: bool,
    material_revision: bool,
) -> int:
    if not accepted or not same_template or material_revision:
        return 0
    return streak + 1

My current release rule calls the template Stable only after three different topics are consecutively Accepted.

Three is not a universal statistical threshold. It is an operating rule for separating "one successful sample" from "a template I am willing to reuse."

The important part is the reset behavior:

revise resets the streak;
redesign resets the streak;
a material revision resets the streak;
changing the series-template hash starts a new streak.

Do not publish three successes selected after the fact and ignore the failures between them. The sequence is the evidence.

Keep machine QA and human QA in different jobs

The answer is not to replace automation with subjective review.

It is to assign each type of evidence to the right owner.

Check	Machine responsibility	Human responsibility
Codec, resolution, required streams	Primary	None
Duration, timeline gaps, subtitle ordering	Primary	Exception review
Required artifacts and source trace	Primary	Meaning and rights judgment
Natural pronunciation and conversational pauses	Assistance	Primary
Whether backgrounds and shots carry meaning	Assistance	Primary
Attention conflict across audio, visuals, and captions	Limited	Primary
Whether the first-watch artifact should be published	None	Primary
Repeatability across different topics	Record and calculate	Interpret and approve

With this split:

automated QA proves moves;
first-watch review proves usable;
owner and target-viewer decisions prove accepted;
run history proves stable.

Each claim has evidence appropriate to it.

A checklist for the next AI video

Before generation:

[ ] Decide which state this run is trying to prove.
[ ] Fix the series-template hash.
[ ] Define the boundary between minor and material revision.
[ ] Name the target viewer.
[ ] Prevent reviewers from receiving production-side explanations.

After generation:

[ ] Prove moves with automated QA.
[ ] Review audio only.
[ ] Review silent visual.
[ ] Perform one integrated first watch at normal speed.
[ ] Record one first blocker.
[ ] Map the blocker to a return layer and allowed repair scope.
[ ] After the same rejection twice, return to treatment design.
[ ] Reset the stability streak after anything other than Accepted under the same template.

What this model does not guarantee

This framework does not automatically create a good video.

It does not guarantee:

that every viewer will like the result;
that one review catches every quality problem;
that three accepted videos will prevent all future regressions;
that AI can replace the release owner or target viewer;
that one model or video provider is superior;
that production time or cost will decrease.

Its purpose is to stop treating technical generation as finished media.

A video that moves is an engineering result.

A video that somebody can publish without explanation is a different result.

A pipeline that can repeat that outcome across topics is different again.

Your AI Adoption Is Waiting on Approval, Not Better Prompts

Sho Naka — Sat, 12 Sep 2026 02:01:15 +0000

The author developed this approval model while incorporating AI into real work. AI recomposed the verified Japanese primary for DEV readers and generated the diagram under the standing delegation for English-market publication. #ABotWroteThis

AI can shorten drafting while leaving the organization waiting on approval.

When that happens, teams often reach for the familiar fixes: improve the prompt, buy a stronger model, or train people to use the tool better. Those fixes improve generation. They do not answer the question an approver actually receives:

What exactly am I accepting, what could go wrong, and who can stop it?

If those answers arrive only after the artifact is finished, every request forces the approver to reconstruct the operating boundary from scratch. The work moves quickly until it reaches the approval queue, then slows down in clarification, reassignment, and re-review.

The useful unit is not another AI policy document. It is a small approval interface attached to each AI-assisted request.

My minimum interface has five fields:

information class;
delegated task;
human checkpoint;
explanation duty;
stop authority.

Define those fields before asking for approval, then route low- and high-risk uses differently. That shifts approval from a vague judgment about “AI” to a concrete decision about one bounded use.

TL;DR

Faster generation can move elapsed time into approval rather than remove it.
An approver needs the input boundary, delegated task, human check, explanation duty, and stop authority.
Put those five fields on the work request before the artifact reaches review.
Use information sensitivity and decision depth to choose a light or high-assurance route.
Do not add a new committee by default. Make the existing decision legible and test whether the route catches the failures it claims to protect against.

The queue is asking for a missing interface

Consider a team that uses AI to prepare three items:

a summary of public documentation for internal use;
a draft customer proposal based on confidential notes;
a recommendation that changes a production configuration.

All three may be called “AI-assisted work,” but they do not ask the organization to accept the same exposure.

The first item uses public information and produces a reversible internal aid. The second introduces confidential information and external communication. The third delegates part of an operational decision and can affect a live system.

If every item reaches the same review queue with only a title and generated artifact, the approver must discover the differences manually. A cautious approver asks more questions. An overloaded approver may apply one heavy process to all three. An impatient team may work around the process.

None of those outcomes is primarily a prompting problem. The request omitted the fields needed to route the decision.

The five-field approval interface

The interface should be short enough to travel with the work, but precise enough that “approved” has a stable meaning.

1. Information: what entered the system?

Classify the input before debating the output.

A useful starting set is:

public;
internal;
confidential or customer-related;
regulated or otherwise restricted.

This field should also name the relevant handling boundary. “Internal” alone is weak if the reviewer still has to ask whether the chosen service may receive it, retain it, or use it for training.

The goal is not to publish secret infrastructure details on every ticket. The goal is to make the admissible information class observable at the decision point.

2. Task: what judgment was delegated?

“Used AI” does not describe the delegation.

These are materially different tasks:

formatting or summarizing supplied material;
generating a draft;
comparing options;
recommending a decision;
executing a change.

Approval becomes harder as the system moves from transforming provided material toward making or executing a consequential choice. State the deepest delegated action, not the safest-sounding label.

3. Human check: who verifies what before use?

Naming a reviewer is not enough. Define the pass condition.

For a technical change, the human check might require:

the proposed diff matches the request;
tests cover the changed behavior;
secrets and private identifiers are absent;
the exact deployment target is pinned;
rollback remains available.

For customer-facing prose, the check might focus on factual support, scope, confidentiality, and authority to make commitments.

The checkpoint should answer “what evidence lets this move forward?” rather than “who is eventually blamed?”

4. Explanation: what must remain reconstructable?

Not every AI-assisted output needs a model-level explanation. Every consequential use needs enough evidence to explain the decision after the fact.

That may include:

the sources used;
the material assumptions;
the reviewer and acceptance evidence;
the difference between observed facts and generated suggestions;
the destination and public or customer-facing claim.

The required depth should match the consequence. A public-document summary does not need the same record as a production migration. But “we cannot tell what was reviewed” is not an acceptable default for either.

5. Stop authority: who can halt or reverse the use?

A rule without an operator is decorative.

The request should name:

the signal that triggers a stop;
the person or role allowed to stop it;
the action that prevents further impact;
the safe state after the stop.

For a draft, stopping may mean discarding the artifact. For an automated production action, it may mean disabling a job, revoking a capability, or restoring a verified previous state.

This field is often missing because teams discuss approval as permission to start. Reliable approval also defines permission to stop.

Route by two dimensions, not by the word “AI”

Once the five fields exist, route the request using two dimensions:

information sensitivity — how damaging misuse or disclosure could be;
decision depth — how far the system moves from assistance toward consequential judgment or execution.

That gives a practical matrix:

Information	Delegated task	Route
Public	Summarize or format	Light
Internal	Draft with a named human check	Light or standard
Confidential	Compare or recommend	High assurance
Restricted	Execute or communicate externally	High assurance, or prohibited by policy

The table is an example, not a universal legal classification. Each organization still has to bind it to its contracts, regulations, and systems.

The important design choice is asymmetry. A harmless internal summary should not wait behind the same evidence package as a customer commitment. A production-changing recommendation should not inherit the light route merely because the generated prose looks polished.

What the two routes should change

A light route may allow one named reviewer, a compact evidence record, and a reversible destination. It optimizes for low coordination cost.

A high-assurance route may require separate content and execution authority, stronger source evidence, an exact target binding, rollback proof, and live read-back after the change. It optimizes for preventing silent expansion of consequence.

The route should change the evidence and authority required—not the truthfulness standard. Low risk is not permission to invent facts, leak information, or claim success without verification.

Likewise, a high-assurance route should not become a ritual that repeats checks without knowing what failure each one protects against. Every additional gate should have a named risk, acceptance condition, and removal or redesign condition.

A request schema you can implement

The interface can live in a ticket template, pull request form, document header, or workflow payload.

ai_use:
  information:
    class: internal
    handling_boundary: approved_service_only
  task:
    delegated_action: draft_and_compare
    prohibited_action: final_customer_commitment
  human_check:
    owner: account_lead
    pass_condition:
      - claims_match_source
      - scope_has_no_new_commitment
  explanation:
    retain:
      - source_refs
      - reviewer_receipt
  stop_authority:
    owner: account_lead
    trigger: unsupported_claim_or_scope_change
    safe_state: keep_as_internal_draft

This is not a complete governance standard. It is the smallest useful contract between the person asking for work and the person accepting its consequences.

Test the approval system itself

Approval design can fail even when every request has the five fields.

Measure outcomes that reveal where the process is moving time and risk:

requests returned because a field was missing;
time spent waiting for an owner-only decision;
low-risk requests sent through the heavy route;
high-risk requests that escaped the heavy route;
approval decisions reopened because the artifact or target changed;
successful stops and restorations when a trigger fired.

Avoid one misleading metric: raw approval speed. A process can approve quickly because it stopped looking. The target is the shortest verified path to an accepted or safely rejected outcome, with no ambiguous authority and no unverified success claim.

Start with one repeated queue

Do not begin by writing a company-wide AI constitution.

Choose one queue where AI-assisted work repeatedly waits: customer drafts, internal analysis, code changes, or publication. Add the five fields to the request. Define one light route and one high-assurance route. Run real items through them and record where approvers still need to reconstruct context.

When the same missing question appears repeatedly, turn it into a field or deterministic preflight. When a field never changes a decision, remove or redesign it. When a low-risk request repeatedly enters the heavy route, repair the classifier rather than telling people to be less cautious.

Better prompts may improve the artifact. Better approval interfaces improve whether the organization can use it.

The distinction matters because generation speed is local. Approval latency is shared across the team. If you optimize only the first, the saved time does not disappear—it reappears in someone else's queue.

Cloudflare Blocked My Playwright Login. Attaching to a Chrome I Started Myself Walked Straight Through.

Sho Naka — Wed, 09 Sep 2026 07:06:42 +0000

The author hit this block on their own machine and confirmed it in a real window. An AI agent working in that environment ran the debugging session, and every error string below was copied from its run logs. #ABotWroteThis

Short answer: in this incident, the browser Playwright launched failed the login check while a Chrome started outside Playwright and attached over CDP succeeded. If you see the same pattern on a browser and account you own, test an attach path instead of endlessly changing launch flags. Do not treat that contrast as evidence of what Cloudflare detects internally.

I needed to read a deploy log on a dashboard I own. A publishing pipeline was reporting success while the published articles returned 404, and the platform's own deploy log was the one place that would say why. It sits behind a login. I have that login. Automating the read should have been fifteen minutes.

It was not fifteen minutes, and the reason turned out to be worth writing down.

The problem: Cloudflare Turnstile showed "verification failed" in the browser Playwright launched, and kept showing it across the launch-side changes I tried.
The observed contrast: a Chrome I started outside Playwright, then attached to over CDP, reached the login/dashboard path successfully in the same environment.
The limit: those runs do not tell me whether Cloudflare reacted to launch arguments, process state, profile state, startup sequence, or some other signal correlated with the launch path.

The symptom

The script opened the dashboard URL and landed on the login page, which is fair enough. The interesting part was the login form itself. The Turnstile widget rendered, spun, and settled on a small red banner reading "verification failed" with a troubleshooting link. Not a challenge to solve. Not a timeout. A refusal.

The same URL, opened by hand in a normal window on the same machine, logged in without showing a challenge at all.

What did not help

Every one of these was tried and produced the same refusal:

launch_persistent_context(channel="chrome"), so the binary was the installed Chrome rather than bundled Chromium.
--disable-blink-features=AutomationControlled, a flag commonly suggested for automation-detection problems.
A freshly created browser profile.
Clicking the alternative sign-in button, which only moved the failure to a different provider's page.

All of those experiments stayed inside the Playwright-launch path. None tested the more important contrast: what happens if the browser process exists before Playwright connects to it.

The observed discriminator was the launch path, not the binary

The useful observation was narrow: changing the Chrome binary choice and launch-side flags did not help, while starting Chrome outside Playwright and attaching afterwards did. Those paths differ in several variables, so this does not identify Cloudflare's signal.

CDP attach tests the operational boundary directly: the browser process already exists before Playwright connects.

flowchart TD
    A[Need to drive a logged-in browser] --> B{Connection path tested}
    B -->|Playwright launches Chrome| C[Observed: Turnstile verification failed]
    B -->|Start Chrome externally| D[Attach afterwards over CDP]
    D --> E[Observed: login/dashboard path succeeded]
    C --> F[Cloudflare's internal discriminator: unknown]
    E --> F

In practice that means starting Chrome with a debugging port and a dedicated profile directory:

"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" \
  --remote-debugging-port=9222 \
  --user-data-dir="$HOME/.config/chrome-cdp"

Log in once, by hand, in that window. From then on the session lives in that profile and your scripts can attach to it. Keep the debugging endpoint local and treat that profile as privileged: anything allowed to drive it can act with that session's authority.

Three issues I hit right after switching

1. In this setup, zero page targets break `connect_over_cdp`

This one cost me the most time because the error text points at downloads while the state change that fixed it was opening a page target.

BrowserType.connect_over_cdp: Protocol error (Browser.setDownloadBehavior):
Browser context management is not supported.

In the environment below, connect_over_cdp throws when the running Chrome has no page target open. You cannot inspect browser.contexts afterwards and recover because there is no browser yet. Opening a tab first restores the connection.

Update, 2026-09-12: a reader (Skillselion) asked whether this was a fresh-start effect or something that happened only after closing the last tab in an already-running browser. I re-ran it on a throwaway profile with macOS 26.6.2, Chrome 149.0.7827.53, and Playwright 1.60.0. The exact incident-time Chrome/Playwright versions were not pinned and cannot be recovered.

Case	Chrome state	`connect_over_cdp`
Fresh process, 1 page open	1 page	succeeds
Same process, that page closed via `/json/close`	0 pages	fails, exact error above
Same process, `PUT /json/new?about:blank` after that	1 page	succeeds again
Chrome killed and restarted, then its default page closed immediately	0 pages	fails, same error

In this matrix, fresh versus previously running did not distinguish the result; success tracked whether GET /json/list contained a type: page target. That is scoped to this Chrome/Playwright pair: the same error can have other causes elsewhere.

import json, urllib.request
from playwright.sync_api import sync_playwright

EP = "http://127.0.0.1:9222"

with urllib.request.urlopen(f"{EP}/json/list", timeout=3) as r:
    targets = json.load(r)

if not any(t.get("type") == "page" for t in targets):
    urllib.request.urlopen(
        urllib.request.Request(f"{EP}/json/new?about:blank", method="PUT"),
        timeout=5,
    )

with sync_playwright() as p:
    browser = p.chromium.connect_over_cdp(EP)
    ctx = browser.contexts[0]
    page = ctx.new_page()
    page.goto("https://example.invalid/dashboard", wait_until="domcontentloaded")
    print(page.url, page.title())

Note the PUT. Recent Chrome versions reject GET /json/new.

2. Chrome refuses remote debugging on your default profile

DevTools remote debugging requires a non-default data directory

Chrome's remote-debugging change documents that Chrome 136 and later ignore the remote-debugging switches for the default Chrome data directory. This is a deliberate protection, not a bug to route around. Use a separate --user-data-dir and treat it as a purpose-built automation profile.

3. Teardown is an ownership problem

The reader comment exposed a flaw in my original framing: authentication state is not ownership. On Playwright 1.60.0 I measured:

Launch path (p.chromium.launch()): Playwright started the browser process. browser.close() killed the Chrome-for-Testing process group in my test.
Attach path (connect_over_cdp()): browser.close() disconnected the Playwright side while the external Chrome PID, CDP endpoint, and existing tab remained alive.
ctx.close() on browser.contexts[0] also left the external process and tabs alone in this test.

The Playwright Python Browser.close API makes the same launched-versus-connected distinction.

So the durable model is not "logged in versus not logged in." It is at least:

browser_owner = agent | external
page_owner    = agent | external
connection    = launched | attached

Page ownership still matters: an attached agent can navigate or close a human-owned tab. A safer teardown policy is:

def teardown(browser, browser_owner: str, agent_pages: list) -> None:
    if browser_owner == "external":
        for page in agent_pages:
            if not page.is_closed():
                page.close()          # close only pages this run created
        browser.close()               # tested here as connection teardown
        return

    browser.close()                   # owned launch: close the owned process

For externally owned browsers, keep human-owned/default pages out of agent_pages. If ownership is unknown, fail safe: do not close pages merely because they are visible through the attached context.

The reader also pointed to Microsoft's separate playwright-cli, not python -m playwright. Its documented attach --cdp=... / detach pair makes external-browser lifecycle explicit and leaves the browser running on detach. I use that as a design precedent, not as proof of Python connect_over_cdp() internals.

The profile copy that did not work here

Copying the authenticated Chrome profile retained account hints but not the authenticated session; the login page still asked for a password. I did not isolate why. The supported claim is only that this profile copy did not preserve login in this incident. I kept the session in the dedicated profile where it was established and attached to that running browser.

Whether it was worth it

The dashboard finally opened, and its deploy log showed the real problem: the newest article had not been deployed because the account had hit a posting-rate limit. I had spent about forty minutes comparing metadata and branches before reading the platform's own answer. When a service reports success but the public result disagrees, read the service's log before diffing your side.

Checklist

Compare the same URL in a normal browser and in the automation launch path; treat the difference as an observation, not a diagnosis of the bot detector.
If you own the target browser/account and need the already-authenticated session, start a dedicated Chrome with --remote-debugging-port and a non-default --user-data-dir.
Log in normally in that dedicated window. Do not automate a CAPTCHA, OTP, or access challenge.
In the Chrome/Playwright combination tested here, check /json/list for a page target and PUT /json/new?about:blank if there is none.
connect_over_cdp, then use the existing context from browser.contexts[0].
Track browser and page ownership explicitly. On teardown, close only agent-owned pages; browser.close() on the tested CDP attach path disconnects without killing the external browser.
Keep remote debugging bound to local/trusted use and keep a fallback to the normal launch path where no debug endpoint is available.

FAQ

Does this defeat bot detection? No. It reuses a session legitimately established on a dashboard the operator owns; it does not forge credentials or automate a challenge.

Will attaching always pass? Unknown. It passed in this environment; Cloudflare's internal decision process was not measured.

Puppeteer or Selenium? They can connect to running browsers, but I did not reproduce this Cloudflare result with them.

Why not headless? A human authenticated this incident's session in a visible browser. That is not a universal headed-mode requirement.

What I did not verify

I did not identify Cloudflare's internal detection signal. The original incident covered two launch paths, one environment and one site. Zero-page-target and teardown behavior were re-tested on 2026-09-12 with macOS 26.6.2 / Chrome 149.0.7827.53 / Playwright 1.60.0; the incident-time versions were not pinned. Microsoft's playwright-cli is cited only as a lifecycle design precedent, while Python behavior is supported separately by the local experiment and API documentation.

The sun does not rise in my world, and my AI keeps forgetting

Sho Naka — Tue, 08 Sep 2026 14:05:18 +0000

I defined the world and decided what its rules forbid. AI wrote the prose being checked and built the checker. #ABotWroteThis

The world I am writing has a fire god who governs the sun. The moment I settled on that, I could no longer let an AI write "the sun rises."

The sun does not rise there. The fire god wakes.

When the setting itself forbids words

Most content rules come from outside the work: don't write slurs, don't write this or that. This one comes from inside it.

There are no clocks and no calendars in that world. So "yesterday" does not exist. Neither does "a week later" or "three hours." Write one of them and the world now contains a measuring instrument it was never given.

The replacements come out of the setting too.

Cannot write	Can write
日が昇る (the sun rises)	光が灯る (light kindles) / 火神が目覚める (the fire god wakes) / 明けが来る (daybreak comes) / 白む (it whitens)

"The fire god wakes" works as a substitute for "the sun rises" only because the fire god governs the sun. Change the setting and the substitutes change with it. The list of forbidden words grows out of the worldbuilding.

The scariest entry was 商人 — merchant. The setting says there is no such profession in this world. When the AI casually writes "the merchant said," the world now has an economy. Unlike the time expressions, it reads perfectly naturally. That is why it slips through.

I counted, and it was not where I thought

The run after I redefined the scan scope (2026-09-06) came back exit 1 with 102 violations.

Before counting, I believed the violations were concentrated in dialogue. I had the impression that the model reverts to modern Japanese the moment it speaks as a character. Lines like these do stand out:

「昨日おっしゃっていた温度管理って、これのことですか」
  "Is this the temperature control you mentioned yesterday?"
「あの……昨日、色々教えていただいたので」
  "Um... you taught me a lot yesterday, so..."

But when I classified all 102 mechanically, only 31 lines contained quotation marks. The other 71 were narration. Roughly 30/70.

That is where I had to stop and check something. 30/70 does not mean "skewed toward narration." If the manuscripts are mostly narration to begin with, the ratio says nothing at all. So I counted the base rate: across the 132 files and 16,483 non-empty lines the checker actually reads, 4,938 lines (30.0%) contain quotation marks and 11,545 (70.0%) do not.

The violations' 30.4/69.6 is almost exactly the base rate of 30.0/70.0. In aggregate, then, the violations are concentrated in neither dialogue nor narration; they simply track the distribution of lines. Per term, though, the picture changes. 昨日 ("yesterday"), the most frequent at 34 hits, splits 17 dialogue to 17 narration — 50% against a base rate of 30.0%, twenty points high. That one term really is tilted toward dialogue. The aggregate stays flat because the remaining terms swing the other way and cancel it. My prediction of a concentration in dialogue was right about the most frequent word and wrong about the whole.

And the largest cluster was somewhere I had not been looking.

In descending order, all of them. Each line is assigned to exactly one term, so the column sums to the line count, 102.

Term	Lines
昨日 yesterday	34
夕 evening (夕日 sunset / 夕餉 supper / 夕暮 dusk)	24
日が傾く the sun tilts	7
今日 today	7
朝 morning (朝日 morning sun)	7
商人 merchant	5
陽光 sunlight	4
冬 winter	4
夏 summer	3
月 moon / month	3
夜 night	2
半日ほど about half a day	1
日が高く the sun high	1
Total	102

Morning and evening are, as I explain below, deliberately not matched as bare words; only compounds are. So every one of the 7 and 24 hits listed here is a compound.

All 24 hits on 夕 were narration: 10 夕日 (sunset), 5 夕餉 (supper), 5 夕暮 (dusk), 3 夕闘, 1 夕方 (evening). Most of them are lines like "the setting sun dyed the village orange," the kind of sentence that opens a scene. The 夕闘 group is different: 夕闘 is not a word at all, but a typo for 夕陽 ("setting sun"). All three hits are the same sentence, 夕闘が沈み、最初の星が東の空に瞬いた, duplicated into three places. (I set narration off without 「」 here, because in this article 「」 is what marks a line as dialogue.) The checker found it because 夕闘 is listed alongside the real words.

I am writing a world where the sun does not rise, and the sun was setting in it. All 7 hits on morning were 朝日 — and there the sun was coming up.

One note on how these are counted. 102 is a count of lines, not of words. A single line can carry more than one forbidden word, and counting every word occurrence gives 114. The breakdown: 93 lines carry one word, 8 carry two, and 1 carries five. The surplus is 8×1 + 1×4 = 12, and 102 + 12 = 114.

Here is one such line.

今日出会った商隊の青年のことを思い出していた。

It carries two: 今日 ("today") and 商人 ("merchant", as 商隊 "caravan"). But it is still one line. Every count in this article assigns each line to exactly one word. A number in any table row is not "lines containing that word" but "lines assigned to that word," and it can be smaller than the former. So the two-word line quoted just above is counted only under 今日.

The top five terms account for 79 lines, but the eight below them say the same things: the sun is moving, or this world is measuring time in calendar units it does not have. One entry was a different animal — 来月 ("next month"), counted under 月, which here is not a celestial body but a unit of calendar. A merchant saying "see you next month" in a world with no calendar is a different kind of violation from the sun moving.

5 of those 102 lines are not prose at all — they are the author's own review notes. One of them reads, in my own hand, that 日が高くなっていた is not on the explicit list of forbidden phrases but arguably violates its intent. That note was itself flagged as a violation.

This matters for the distribution too. All 5 contain quotation marks — they quote the forbidden phrases — so 5 of that earlier 31 sat on the dialogue side. Restricted to lines the AI actually wrote, it is 97 hits: 26 dialogue, 71 narration, 26.8% to 73.2%. That is below the 30.0% base rate. Excluding the notes, the violations lean slightly toward narration. In aggregate, with the notes or without them, this is not "concentrated in dialogue."

The 102-violation run above already uses the redefined scope, and these 5 survived it. The definition I settled on is "body text a reader might read," and that is a boundary drawn per path. But these 5 are annotations the author wrote inside a manuscript file. As long as prose and notes live in the same file, a path-level rule cannot separate them. The definition is not wrong; it just does not reach that far at path granularity. A line-level rule would clear them, and that is the next boundary — one I have not drawn yet.

If the counts are about even, why did I think dialogue dominated? Salience, I think. Modern speech inside quotation marks snags as you read, so it stays in memory. A sunset inside a landscape description reads too naturally to register as a setting violation at all. The same number of hits, but only one kind accumulates in memory. Reading with my eyes, I was never going to find the other.

Why they do not skew is not something this observation can tell me. I have not compared models, and I have not compared prompts. One work, one model, one run, and the distribution came out matching the base rate.

What I can say is the other thing: I should not have trusted my own expectation without counting it. It held for exactly one word — the most frequent one — and failed for the whole.

Listing the forbidden words in the prompt did not stop this. I never ran an A/B comparison, so I cannot claim prompts always leak — but in the way I write, the longer the passage, the more leaked through. Writing first and counting afterwards was the more reliable option for me.

I got the scan scope wrong twice

The checker is close to grep. Collect the lines that hit a forbidden pattern, exit 1 if there is at least one. The hard part was never the detection. It was deciding where to look.

The first run flagged 18 lines. I went to fix them and found that every single one was a false positive.

They were all in rejected drafts I had already set aside, or in non-prose files: foreshadowing trackers, notes about how a chapter came together. Writing "decided this yesterday" in a production note is correct. The author lives in the real world; the in-world rules do not apply to their memos.

The same look revealed that the directory holding my in-progress drafts was not being scanned at all. I was checking the places that did not matter and skipping the one that did.

So I added the drafts — and the count jumped to 450.

The cause was a folder of old material from when this was designed as an RPG. My own documentation says it is incompatible with the current line and kept isolated for reference. It was never something to fix.

Opposite mistakes, same root cause: I was choosing scan scope by directory name without ever defining what counts as prose.

Here is the definition I settled on. Scan only what a reader could end up reading. Anything I wrote standing in the real world is out. Anything quarantined as reference because it belongs to an incompatible earlier design is out. Strip those two away and what remains is the finished prose of the current line plus the drafts in progress.

So I stopped deriving scope from directory names and started listing the paths that satisfy that definition explicitly. Now a new directory can neither slip in nor slip out on its own. I have to decide which side it belongs to, at the moment I create it.

A checker whose numbers you cannot read

Coverage right now is 35 of 46 forbidden terms. Eleven are not implemented.

The other day I wrote down, for each of those eleven, why it was left out. I did not add any rules. I only wrote the reasons that had never been written.

All eleven turned out to be time expressions (morning, midday, evening, night, hour, one day, half a day, two days, "N days later", "half a day's walk", "a day's walk"), and the reasons fell into two groups. The first five are far too common on their own. This is sharper in Japanese than it sounds in English: 時間 is a single word meaning both hour and time in general, so matching a phrase like 時間がかかる ("it takes time") would wipe out almost all of the narration.

What was left out is the bare word, not the concept. Compounds that pin down a time of day — 夕日 (sunset), 朝日 (morning sun) — are still matched. That is where the 7 and 24 hits in the table above came from; both counts are compounds only. 時間 works the same way: the bare word is dropped, and only the counted form 三時間 ("three hours") is caught, because that one is unambiguously a measurement off a clock the world does not have.

The day counts were different in kind. Weekday and month names are violations no matter how they appear, because the world has no calendar at all. But a count of days is something you could actually observe, by watching the sun rise and set that many times. It is a weaker violation. And matching it strictly would reopen rewrite decisions in prose I had already fixed. So rather than drop the group, I narrowed it: the bare counts are not matched, but the forms that unambiguously mean a quantity still are — the implemented pattern is 十日|半日ほど|一日[がだで], so 半日ほど ("about half a day") and 一日が/だ/で are caught while bare 一日 is not. 十日 ("ten days") stands there bare because it is not an everyday word: 一日 turns up constantly in set phrases like 一日中 ("all day") and 一日も早く ("as soon as possible"), whereas 十日 almost never does. Across these 102 lines, 十日 did not appear even once. That is why the table carries one 半日ほど even though the day counts sit on this unimplemented list.

Which means "unimplemented" here was a decision to narrow, not to drop: matching the bare words mechanically would break things. It was not work I had failed to get to. If I had not written that down, I would later have misread it as exactly that — a corner I never got around to.

Unimplemented items without a stated reason cause a specific problem later. When the count comes back low, you cannot tell whether it is low because there are words you are missing or because you narrowed the detection on purpose.

A checker that cannot tell you which one it is cannot tell you what its own numbers mean. Right now it sits at 102 and red, so there is nothing to misread. But as I work that down toward zero, the meaning of that zero stays undetermined.

The same pass turned up a comment pointing at the wrong source of truth. It said the rules were "synced with that YAML file." The actual source of truth was the worldbuilding document. Which is the same failure one level up: the checker could not tell me what its numbers meant, and its own comment could not tell me where its rules came from. Both exit 0 and complete. What is broken is the content of the output, which you cannot see by looking at the exit status.

The enforcement tool became an observation tool

The thing I did not expect: this checker turned into an instrument for watching where an LLM drifts out of a world.

Where the violations would land was not something I knew when I wrote the word list. Until I counted, I believed they concentrated in dialogue. Counting showed that 昨日, the most frequent term, really is tilted toward dialogue, while the aggregate concentrates nowhere and simply tracks the distribution of lines. The prediction was right for exactly one word. The tool I built to enforce the rules is the one that told me how far my prediction actually held.

Reading my own generated prose, I only ever find the conspicuous violations. Where the quiet ones are, and how many, does not come out until something counts them.

My Quiz Has an AI Rival. Its Scoring Doesn't Need an LLM.

Sho Naka — Mon, 07 Sep 2026 17:46:05 +0000

After using Kahoot in a training session, I wanted to replay the questions. What I missed was the rhythm: answer, watch the score move, try one more round. In a solo session, there was nobody to compete with.

Quick answer

I started a small quiz prototype with a CPU rival. The useful design decision was to separate three things:

Simulate the rival's correctness and response time.
Calculate both scores with the same fixed function.
Treat generated dialogue as a possible future feature, outside scoring.

This is a prototype, not evidence that competition improves learning. Kahoot already described study modes with AI players in 2020. My goal was to explore that experience with my own material and timing rules.

A simulated rival does not have to understand the question

My CPU samples correctness from a configured probability and picks a virtual response time. It does not ask a language model to solve the question.

The code has 60%, 78%, and 92% accuracy profiles. The current UI uses the 78% profile; a difficulty selector is not implemented. These are prototype settings, not validated learning targets.

This gives me an opponent to test before adding conversational behavior.

Random answers, fixed scoring

The player answers against an answer key. The CPU's correctness is sampled. Once correctness is known, both use the same scoring rule.

Here is that rule extracted into a standalone function. Inputs assume nonnegative elapsed time and streak, and a positive time limit.

function score({ correct, elapsedMs, limitMs, streak }) {
  if (!correct) return 0;
  const remaining = Math.max(0, Math.min(1, 1 - elapsedMs / limitMs));
  const speed = 0.5 + remaining * 0.5;
  const combo = 1 + Math.min(streak, 5) * 0.05;
  return Math.round(1000 * speed * combo);
}

At half the time limit, a correct answer with streak: 0 scores 750. With streak: 2, it scores 825. These are arithmetic examples, not learner measurements.

The rival's result varies; the scoring rule does not. I checked the extracted function with seven assertions covering these examples, incorrect answers, time boundaries, and the streak cap.

To repeat that check, save the function as score.mjs, add export { score };, and run:

node --input-type=module <<'JS'
import assert from 'node:assert/strict';
import { score } from './score.mjs';
const cases = [
  [true, 5000, 0, 750], [true, 5000, 2, 825],
  [false, 0, 0, 0], [true, 0, 0, 1000],
  [true, 10000, 0, 500], [true, 12000, 0, 500],
  [true, 0, 8, 1250],
];
for (const [correct, elapsedMs, streak, expected] of cases) {
  assert.equal(score({ correct, elapsedMs, limitMs: 10000, streak }), expected);
}
console.log('7 assertions passed');
JS

Observed output: 7 assertions passed. This checks the extracted formula, not the full game.

Reveal an answer state before revealing a result

If the CPU answers first, the player sees only answered. Its correctness and points stay hidden until the player's answer is locked.

Question starts
  CPU finishes -> show "answered", keep result hidden
  Player answers or times out -> lock player input
  Finish the rival's waiting display
  Reveal correctness and points
  Next question or replay

The boundary is information: a player who can still answer must not see the rival's result. Locking also gives the game one place to reject repeated submissions.

Virtual response time is not display waiting time

When the player finishes first, the prototype caps the remaining on-screen wait. It still scores the CPU using the virtual response time chosen for that question.

For example, suppose the CPU's planned time is eight seconds and the player answers after two. Shortening the waiting animation must not replace the CPU's eight-second scoring input with the shorter display duration.

That is an illustrative timeline. The general boundary is that presentation changes should not silently change an already chosen scoring input.

FAQ: what belongs outside the game loop?

Where would I use an LLM?

The current scoring path makes no external LLM calls. Rival dialogue, encouragement, and rephrased explanations are future ideas. If added, I want fixed messages to keep the game moving when generation fails.

Does fixed scoring make the quiz educationally correct?

Question authoring is another responsibility. An incorrect answer key can be scored consistently and still teach the wrong thing. The current prototype reads questions from a lesson page; support for a common import format is still being designed.

What should I check next?

Before adding dialogue, trace these cases in your own quiz: CPU first, player first, timeout, repeated clicks, and replay while an old timer is pending. The seven function assertions above do not establish that this full browser test matrix passes.

Then measure completion, replay, and learning separately. Speed bonuses may reward fast reading or guessing, and more replays do not prove better retention.

Start by naming one piece of information that must stay hidden before an answer, and one scoring input that an animation must never change.

Based on the author's training experience and prototype. AI assisted with implementation review, English recomposition, and the code explanation.

I Cut AI Agent Handoff Payloads by 87%—Then the Important Questions Disappeared

Sho Naka — Sun, 06 Sep 2026 16:08:39 +0000

This article is based on the author's production handoff hooks. The author designed the system, measured 240 real handoff records, and verified the delivery behavior. AI recomposed the approved Japanese source for DEV readers under the standing delegation for English-market publication. #ABotWroteThis

I had a multi-agent coordination problem that looked like a context-window problem.

Several coding sessions needed to exchange questions, progress, and completion notices. A hook injected those handoffs into the next conversation turn. The first implementation took the conservative route: include the full message every time so nothing gets lost.

That safety margin became a recurring tax. Long messages were injected again on later turns. A free-form request addressed to another session could even be interpreted as an instruction for the current one.

I replaced full-message delivery with short notifications and an on-demand inbox. Replaying 240 production handoffs reduced the average payload per handoff from 969 bytes to 123 bytes: an 87.3% reduction.

Then the first version failed in a more important way. Seven unanswered questions existed in the previous 24 hours, but the first notification batch contained none of them.

The lesson was not “compress harder.” It was this:

Push the existence and priority of a handoff. Pull its full content only when needed. Measure payload reduction and critical-message delivery as separate SLOs.

TL;DR

Do not inject full handoff messages on every turn.
Push a compact notification containing type, state, sender, summary, and retrieval path.
Keep the full message in a pull-based inbox.
Rank unanswered questions ahead of status and completion messages before applying count or byte limits.
Test both payload size and first-turn delivery. A smaller payload that hides the next required decision is a regression.

Quick answer: how do you reduce agent context without losing handoffs?

Separate notification from retrieval.

The always-on hook should answer only:

What kind of handoff exists?
Is it still actionable?
Who sent it?
What should the receiving agent do next?
Where can it fetch the full message?

The inbox keeps the background, investigation notes, and free-form request. The receiving session pulls those details only when the notification is relevant.

Before
hook -> [full handoff text] -> every conversation turn

After
hook -> [short prioritized notice] -> conversation context
                                      |
                                      +-> pull full text from inbox when needed

This is not deletion. It is a routing change.

The dangerous version of “include everything just in case”

Full-message injection feels safe because the receiving agent cannot complain that information was omitted. But the design quietly conflates four responsibilities:

storing the durable message;
notifying the recipient that it exists;
deciding whether it is still actionable;
choosing how much of it belongs in the current turn.

When one hook does all four, every message becomes recurring context. The system pays for old background even when it needs only a one-line signal.

It also widens the instruction boundary. A message like “restart the deployment after the owner approves” may be useful evidence for one session and an unsafe command for another. Injecting arbitrary prose directly into the active context asks the model to rediscover that distinction every time.

A notification is a safer boundary:

[question][unanswered] from=session-api: Need decision on retry policy — open inbox item 841

The notification reports state. It does not impersonate a new instruction.

What the 240-record replay measured

I replayed 240 handoffs that had occurred in the real workflow through the before and after formatters.

Measurement	Before	After	Change
Mean bytes per handoff	969 B	123 B	-87.3%
Maximum bytes per handoff	3,066 B	295 B	-90.4%
Auto-commit notice	281 B	82 B	-70.8%

All three rows are measurements from this 240-record replay. The per-flow table further down measures a different population — the raw 893-event snapshot — which is why its auto-commit mean is not the same number. The two are not comparable as a single measurement.

You can recheck the arithmetic on those three rows directly:

awk 'BEGIN{
  printf "mean:   %.1f%%\n", (969-123)/969*100
  printf "max:    %.1f%%\n", (3066-295)/3066*100
  printf "notice: %.1f%%\n", (281-82)/281*100
}'

mean:   87.3%
max:    90.4%
notice: 70.8%

The byte boundary is the one the hook actually emits, so it is measurable with wc -c rather than estimated. A single push notice looks like this:

printf '%s' '[question][unanswered] from=session-api: Need decision on retry policy — open inbox item 841' | wc -c

94 bytes carries the type, the actionable state, the sender, the decision at stake, and the retrieval path. Whatever the original message said, the receiving session can now decide whether to read it before paying for it.

After the priority fix, a complete delivery response—including the inbox hint and unanswered-count summary—measured between 883 and 1,034 bytes in the observed cases. The notification-line budget was 1,000 bytes, so the final response could slightly exceed it when fixed guidance was added.

These are UTF-8 byte measurements at the hook output. They are not token counts, model input billing, or a claim that total daily context fell by 87.3%. The comparison is intentionally narrow: the same handoff records, through two formatters, measured at the same boundary.

That boundary is still useful. It tells us whether the change reduced the flow we actually modified without pretending to measure the entire agent runtime.

The first optimization hid every unanswered question

The compact formatter initially selected the newest five messages.

That sounds reasonable until the inbox contains several kinds of events: progress, completion, claims, answers, and questions. Five recent status messages can consume the whole batch while an older unresolved question remains the item that actually blocks work.

In the failure case:

seven unanswered questions existed in the previous 24 hours;
the first delivery contained zero questions;
the second delivery exposed one;
the third delivery exposed two.

Nothing was deleted. The questions were still in the inbox. But “available after several turns” is not the same as “delivered when the recipient needed to act.”

The 87.3% reduction had passed while the operational outcome had failed.

This is the core measurement trap in context optimization:

smaller payload != better delivery

You need at least two independent signals:

Volume: How many bytes were injected?
Reachability: Did the most important actionable item appear in the first eligible turn?

Combining them into one score hides the trade-off. A formatter can look excellent on volume precisely because it removed the information the recipient needed.

Rank meaning before enforcing limits

I changed the candidate order to:

unanswered questions;
state changes;
answers;
accepted work;
progress and completion notices.

Answered questions are removed from the delivery candidates. If the byte or count limit cannot fit every unanswered question, the remaining items stay unread and the compact output continues to say how many remain.

After the change, the first batch carried at least one unanswered question instead of none.

The implementation used three delivery controls:

at most five detailed notices per turn;
at most three redeliveries of the same notice;
approximately 1,000 bytes of notification lines.

One exception matters: the highest-priority item is shown even if it alone exceeds the byte budget. A strict limit that can erase the only blocking question is not a safety control; it is a denial-of-information mechanism.

The ordering rule should therefore come before the size rule:

PRIORITY = {
    "unanswered_question": 0,
    "state_change": 1,
    "answer": 2,
    "work_accepted": 3,
    "progress": 4,
    "completed": 5,
}

def select_for_push(items, max_items=5, byte_budget=1000):
    candidates = [item for item in items if item.is_actionable]
    candidates.sort(key=lambda item: (PRIORITY[item.kind], item.created_at))

    selected = []
    used = 0
    for item in candidates:
        notice = item.compact_notice()
        size = len(notice.encode("utf-8"))

        if selected and (len(selected) >= max_items or used + size > byte_budget):
            break

        selected.append(notice)  # Always allow the highest-priority first item.
        used += size

    return selected

This example is deliberately incomplete. Production code also needs durable read state, stable identities, concurrency control, malformed-record handling, and a pull command that returns the exact original message. The important part is the sequence: semantic priority first, transport limits second.

Test delivery as a state machine

A “formatter output is shorter” test would have approved the broken version. The useful tests describe how messages move through states.

Scenario	Expected behavior
No notices	Inject nothing and exit successfully
Normal delivery	Show a compact notice and the full-message retrieval path
More than five candidates	Show five; leave the remainder for a later turn
Byte budget exceeded	Defer the next notice, except the highest-priority first item
Unanswered question exists	Deliver at least one in the first eligible batch
Question has an answer	Remove that question from redelivery candidates
Redelivery limit reached	Stop repeating the body, but keep an unresolved-count signal
Malformed JSONL row	Skip the row without breaking the whole hook

Keep the volume assertion and delivery assertion separate:

def test_compact_notice_reduces_bytes():
    assert mean_bytes(compact_records) < mean_bytes(full_records)

def test_first_batch_contains_unanswered_question():
    batch = deliver(inbox_with_unanswered_questions)
    assert any(item.kind == "unanswered_question" for item in batch)

The second test is not a quality bonus. It protects the purpose of the system.

Measure by flow, not only by global total

Before the optimization, the broader measurement snapshot contained 893 injection events totaling 1,131,429 bytes through 2026-08-08 05:26:38 UTC.

The largest flows were:

Source	Events	Total bytes	Mean	Maximum
Cross-session handoff delivery	159	617,058 B	3,880 B	9,407 B
Session list at startup	5	68,052 B	13,610 B	14,132 B
Auto-commit notices	164	66,030 B	402 B	755 B

These are raw snapshot events, not the 240-record replay above, which is why the auto-commit mean here (402 B) differs from the 281 B replay row.

The startup list had the largest mean, but handoff delivery dominated the total because it ran often and redelivered pending content.

That is why one aggregate number is not enough. Track at least:

event count;
total bytes;
mean bytes;
maximum bytes;
first-turn delivery of the highest-priority unresolved class.

Which flow you optimize depends on the outcome. Mean size points toward startup. Total recurring cost points toward handoff delivery. Reachability tells you whether either optimization broke the coordination loop.

Record each injection as one JSON Lines row carrying the source, the UTF-8 byte length, and a UTC timestamp. The table above is then reproducible in your own environment with jq:

METRICS_LOG=/path/to/context-emission.jsonl

jq -s '
  [.[]
    | select(
        .metric == "context_emission_size"
        and .ts >= "2026-08-08T00:00:00Z"
        and .ts <= "2026-08-08T05:26:38Z"
      )
  ] as $rows
  | {
      count: ($rows | length),
      bytes: ($rows | map(.byte_len) | add),
      by_source: (
        $rows
        | group_by(.hook)
        | map({
            source: .[0].hook,
            count: length,
            bytes: (map(.byte_len) | add),
            average: ((map(.byte_len) | add) / length | floor),
            maximum: (map(.byte_len) | max)
          })
        | sort_by(-.bytes)
      )
    }
' "$METRICS_LOG"

Bytes are used because they are cheap and reliable to capture immediately after a hook writes its output. This is not a substitute for token measurement; the goal is a comparable before/after series for each injection path.

A practical implementation sequence

If you are retrofitting an existing agent workflow, this order keeps the evidence interpretable.

1. Instrument before compressing

Log the source, timestamp, UTF-8 byte count, and delivery class for every injected payload. Do not begin with token estimates if the component you can directly observe emits bytes.

2. Find the expensive flow

Group by source and inspect count, total, mean, and maximum. A rare large payload and a small payload repeated hundreds of times need different fixes.

3. Split push from pull

Keep type, actionable state, summary, sender, and retrieval identity in push. Store the complete immutable message behind a pull path.

4. Define semantic priority

Write down what must appear before applying max_items or byte_budget. In a coordination system, unresolved questions usually outrank progress narration.

5. Test state transitions

Cover unanswered, answered, overflow, redelivery exhaustion, and corrupt storage rows independently.

6. Continue measuring both axes

Watch injected bytes and critical-message reachability. A regression in either one is a reason to repair the change, not average the metrics together.

Architecture boundary

The design becomes easier to reason about when notification generation, durable message storage, prioritization, and measurement remain separate.

┌────────────┐      ┌──────────────────────┐
│ producers  │ ---> │ durable inbox/details│
└────────────┘      └──────────┬───────────┘
                              │ pull
                              v
                    ┌──────────────────────┐
hook push --------> │ priority + short note│ ----> active context
                    └──────────┬───────────┘
                              v
                    metrics + delivery tests

This separation also gives failures a location. If the inbox contains the question but the notification omits it, the problem is prioritization. If the notice appears but pull cannot retrieve the original, the problem is identity or storage. If both work but the hook repeats it forever, the problem is acknowledgement state.

“The agent did not see the message” stops being one vague failure mode.

FAQ

Why measure bytes instead of tokens?

Because bytes are what the hook emits. Token counts depend on the tokenizer and the model, and they are an estimate at the point where I could take a direct measurement. Measuring bytes at the hook boundary keeps the claim narrow and reproducible: the same 240 records, two formatters, one measuring point. If you need a billing number, measure it at the API boundary instead — that is a different question with a different instrument.

Doesn't a pull-based inbox just move the cost to a later turn?

Only when the recipient actually needs the detail. The failure mode of full injection is paying for every message on every turn regardless of relevance. Pull moves that cost from unconditional to conditional. The measurement that keeps this honest is the second SLO: if the notice is so thin that the agent must pull on every turn anyway, reachability will not improve and you have not gained anything.

What if the highest-priority notice alone exceeds the byte budget?

Show it anyway. Truncation applies to everything below the top of the ordering, never to the top itself: every other item defers to the next turn, and the first-priority item does not.

How do I know my compression didn't break delivery?

Assert the two axes separately. test_compact_notice_reduces_bytes and test_first_batch_contains_unanswered_question must both exist and must not be averaged into one score. In my case the first test passed at -87.3% while the second would have failed at 0 of 7 unanswered questions delivered. A single combined metric would have reported success.

Does this apply outside multi-agent setups?

The push/pull split applies anywhere a recurring hook injects variable-length content: notification digests, CI status summaries, monitoring alerts routed to an assistant. The specific priority ordering does not transfer — "unanswered questions first" is a property of a coordination inbox. Write down your own ordering before you write the byte limit.

The rule I kept

I still want context injection to be small. I just no longer accept smallness as the outcome.

The durable rule is:

Push only enough context to identify the next relevant action. Preserve the full handoff behind pull. Prove both the byte reduction and the first-turn delivery of what matters.

An 87.3% reduction is useful evidence. Zero of seven unanswered questions in the first batch is stronger evidence that the first design was wrong.

Your MCP Tools Are Listed. Is the Integration Actually Working?

Sho Naka — Sun, 06 Sep 2026 00:28:48 +0000

The observations in this article come from one application's connector sessions on 2026-09-05, not from a survey of MCP implementations. Tool discovery here was through a host connector catalog, not a captured wire-level exchange. #ABotWroteThis

A connector session opened against my own publishing server. The runtime health call came back fresh=true. The tool catalog was populated. A live portfolio scan ran and returned a scan ID.

And the workflow was stuck.

The scan's own response said the next required step was a saved article impact plan. The catalog that session had been given did not contain an operation that saves one. Up to that point every call had come back a success — the health check, a lookup, and the scan itself. The runtime was healthy. There was no error to debug. The workflow simply had no callable next step, and nothing in those successful responses said so.

Here is the acceptance test I now run before I let an assistant do anything but read. Five gates: discovery, invocation, identity, readability, reachability. None of them authorizes a write. Three make a call — gate 1's discovery, a tool listing plus an optional request for the server's own instructions, then two invocations of tools whose own declarations say they write nothing — and the last two only read what is already in front of you. Each gate carries the line that matters most — what a pass still does not prove — and gate 5, the one that was missing above, asks whether every prerequisite you hold is either human-owned or has a callable transition at all. It does not find the ones nothing has named yet. What it does is turn the moment one of them surfaces into a failing test instead of a mystery.

One scope note first: that session is reported from what it recorded at the time, not re-run for this article, and the protocol statements below are pinned to the specification revision they cite. Nothing here is a measurement of how often this failure occurs.

"The tools are listed" is the weakest evidence you have

A populated tool catalog is a claim about what a server advertised to a client at some point. It is not a claim about the server's current capability set, and it is definitely not a claim about your workflow.

Two things in particular are worth separating:

A catalog is a snapshot, not a capability set. In the case above, the operation that session needed did not exist yet. It was added to the server later the same day by a separate repair session, and a later catalog was confirmed to expose it. A server's tool set can change between the moment a client was told what was available and the moment you act on it; the protocol carries a mechanism for a server to announce that its tool list has changed. For your test, that means "the tool isn't in my catalog" dates an observation. It does not establish what the server can do now, and it does not tell you whether you are looking at a permanent gap or a build you don't have.

Discovery is defined separately from invocation. The Model Context Protocol specification defines listing tools and calling tools — tools/list and tools/call — as distinct operations with distinct results (see the Tools section of revision 2026-07-28; pin whichever revision you actually read, and note that the revision you read is not necessarily the one your host negotiated at runtime). A listing does tell you how to call — it carries each tool's name and description, its input schema, its output schema when the server declares one (that field is optional), and its annotations, the optional block where a server states behaviour such as “this one only reads.” That whole set is what the acceptance matrix below asks you to read from the listing. The matrix asks for one thing more, which the listing does not carry: the server's own instructions, returned by a separate request your client may or may not make. And the annotations are where the read-only claim that gates 2 and 3 rest on comes from. What it does not tell you is what happens when you do: a successful listing is no guarantee that the call succeeds, that the result comes back without an error, that your credentials permit it, or that nothing is written.

So the question is not "are the tools there." It is: which specific claim does each successful response actually license?

The acceptance matrix

One caution before you run gates 2 and 3. Both call a tool that declares itself read-only, but that declaration is the server's claim, not a guarantee: the MCP specification is explicit that an MCP client must not treat tool annotations as trustworthy unless the server itself is trusted. So run those two gates against a server you already trust, or against an instance you can throw away.

Each gate below is four lines: Run, Read, Pass, Does not prove. The fourth is the one that gets silently assumed.

Gate 1 — Discovery

Run: List tools. Then ask the server to describe itself, if your client does that — server/discover in the revision this article pins. The server's own instructions come back from that call, not from tools/list, and making it is optional for the client.
Read: From the listing: each tool's name and description, its declared input schema, its declared output schema if it has one, and its annotations — the optional block where a server states behaviour such as "this one only reads." From the server's own description of itself: its instructions, an optional free-text field where the server states how it expects to be used.
Pass: The operation your workflow needs is present, with the arguments you intend to pass.
Does not prove: That the server can execute it, that your credentials permit it, or that the list is current. Nor that you have the instructions: the field is optional, and a client that never makes the second call never sees them.

Gate 2 — Invocation

Run: Call one tool whose gate 1 annotations say it only reads, with valid arguments.
Read: First the description and annotations gate 1 already listed, to confirm the tool claims to write nothing; then the result payload and the error/isError channel — including anything the payload names as a prerequisite the workflow must satisfy before it can continue (a state field reporting something missing or invalid, a required-field error, a detail line naming a file it wants).
Pass: The tool declares itself read-only and returns a non-error result — matching the output schema it declared in gate 1, or simply coming back without an error if it declared none. Whether that payload is usable is gate 4's question, not this one.
Does not prove: That the call wrote nothing — that is the server's own claim, not something this gate verifies. Nor that other tools work: each tool is its own integration.

Gate 3 — Identity

Run: Call a lookup — again, one whose gate 1 annotations say it only reads — for a resource you can independently identify.
Read: The returned identifier, compared against the one you already know — and, as in gate 2, any prerequisite the response names.
Pass: The returned resource is the one you asked for.
Does not prove: That the resource is complete, current, or the only match.

Gate 4 — Readability

Run: Read the content gates 2 and 3 returned, not just their status.
Read: Actual field values — length, encoding, whether required fields are populated, and whether any field is stating a required next step or an unmet precondition rather than returning data.
Pass: A human or an assistant can act on the content as returned.
Does not prove: That the content is correct, or that it matches what the source holds — only that what came back is readable and usable as returned.

Gate 5 — Reachability

Run: For each prerequisite you hold — one the catalog's stated order names, or one a response you already received names — find the call that satisfies it.
Read: The tool descriptions and server instructions that state required order, and the declared input schema of the tool that would perform each transition.
Pass: Each of those is either human-owned or has a supported callable transition.
Does not prove: That the catalog named every prerequisite. One it never mentions is invisible to this gate until something names it, and a pass says only that the steps you know about are not dead ends — not that the transition will succeed.

Gate	A pass means	A pass does not mean
1 — Discovery	The operation you need is listed, with the arguments you intend to pass	That the server can execute it, that your credentials permit it, that the list is current, or that you hold the instructions
2 — Invocation	A tool declaring itself read-only returned a non-error result	That the call wrote nothing, or that any other tool works
3 — Identity	The resource returned is the one you asked for	That it is complete, current, or the only match
4 — Readability	The content, as returned, is usable by a human or an assistant	That it is correct, or that it matches what the source holds
5 — Reachability	Every prerequisite you hold is human-owned or has a callable transition	That the catalog named every prerequisite, or that the transition will succeed

That third column is the matrix. The five gates are cheap to run; what they refuse to tell you is the part worth copying.

Where gate 5's prerequisites come from

On a first run, before anything has gone wrong, they come from gate 1 — and from both of its reads. Tool descriptions and a server's own instructions state required order in plain words — call this only after that, this is rejected without a recorded X — and gate 1 already had you read them. Which of the two carried it matters: the descriptions arrive with the listing, so anyone who ran gate 1 has them; the instructions arrive only if the client asked for them and the server filled the field in. A response naming one at runtime is the second source.

Either way, gate 5 does not need you to trigger the prerequisite yourself. It takes each one you have and asks whether the catalog you hold contains a call that satisfies it. That question is answerable from what gate 1 already read, which is why it stays inside a test that authorizes no write.

Worth being exact about the opening story here. The scan that named the missing prerequisite is not a read-only tool — its own annotations declare that it is not, and it writes a scan record — so that discovery was not one of these five gates. It was the incident the gates exist to get ahead of. Both halves of that requirement have since been declared — an operation that saves the plan exists, and its description states the order it belongs in — added the same day by a separate repair session.

So run gate 5 against that earlier catalog, before the scan, and it passes. That is the uncomfortable part, and it is why every block here carries a fourth line. The catalog did state an order — call the scan first, record only after reviewing its evidence — and both of those steps had a callable transition. What it never stated was the plan, and a prerequisite the catalog does not mention cannot be found by reading the catalog. What gate 5 changes is the moment one surfaces anyway. The scan's response named the plan; re-running gate 5 on it costs one reading of the catalog already in hand — no tool there accepts a plan, so the workflow is not reachable, and the failure has a name before anything further is attempted. That session had no such test. Up to that point it had three successful calls, a healthy runtime, and no error to debug.

Gate 5 is the one that was missing there, and the one I had never written down as a test before this. It is not gate 1 restated. Gate 1 asks whether the operations you went to the catalog looking for are present. Gate 5 asks about the ones you did not go looking for: the steps the catalog's own prose says have to happen in between. That prose is in the same descriptions and instructions gate 1 handed you — read for argument names, skipped for order. When it states an order, gate 5 checks each step in it before anything that writes is called.

Gate 5, in detail: a next step in prose is not a next step

Workflow-shaped tools name their own prerequisites in the response — the second source, and also where the trap is. The scan in the opening session returned three fields for its plan requirement: required: true, a state of missing_or_invalid, and a detail naming the local file it wanted.

Those three fields state a requirement. They are not a transition. The test is mechanical:

Take the prerequisite, from either source above.
Ask whether it is machine-owned (something the assistant is supposed to do) or human-owned (an approval, a credential, a business decision that must not be automated).
If it is machine-owned, find the tool whose declared input schema accepts what you'd need to pass.
If no such tool is in the catalog, the workflow is not reachable with the catalog you currently hold — and per the point above, that is a dated observation, so check the server's current state before concluding the operation does not exist.

A human-owned prerequisite does not fail this test. Step 2 is where it stops, and gate 5 counts it as satisfied — the gate is asking about machine transitions, and a human step is a legitimate answer, not a hole. The hole is a machine-owned prerequisite that reaches step 4 with no tool to match, and it shows up as "the agent got stuck and I don't know why," because every individual call was a success.

One thing this test is not

Do not confuse pagination inside an application's returned inventory with protocol-level pagination of the tool catalog.

In the opening session, a lookup returned 50 historical candidates with no cursor. That is the application choosing how much of its own inventory to hand back — a usability property of that tool, and a fair thing to file as a request. It is not a protocol violation, and reporting it as one sends whoever reads your bug report to entirely the wrong layer. Gate 3 covers this correctly: the identity check is what tells you whether you got the resource you asked for, regardless of how many neighbors came along with it.

Running it on your own integration

Pick a workflow you already trust and start from its catalog: gate 1's two reads carry the descriptions and, when the server offers them, its instructions, and gate 5 can be answered from whatever came back, before anything that writes.

Walk the five gates against that catalog. Write down, per gate, the exact field you read — and, for the three that make a call, the operation you called to get it. All five are answerable without authorizing a write.
Then run the workflow's own read-only steps, in the order the catalog states.
From there on, carry gate 5 into the runs that do write. Every response that states a requirement — a required flag, a state of missing or invalid, a detail naming something you have to supply — goes back through gate 5 against the catalog already in hand, before the next call. That is the moment the opening session had and let pass.

Do not expect the read-only walk to reproduce the opening case. There, the only thing that named the missing prerequisite was a call that writes, which is outside all five gates; a read-only walk would have finished clean. Expect the first pass over the catalog to come back clean too — the reconstruction above already showed why. Reading a catalog for the order it states costs one pass and rules out only the dead ends already visible in it. What step 3 adds is the habit, and it only pays off where the writes are: that session held the same catalog the whole time and never read it again once the scan answered.

From that walk you are looking for two outcomes:

A gate you cannot run at all, because there is no read-only operation that produces the evidence. That is a hole in the integration's testability, not in your test.
A gate where the pass criterion is met but the "does not prove" line is something you had been assuming all along. That assumption is the one worth writing down before it costs you something.

The protocol's own inspector tooling is a reasonable place to run gates 1 through 4 by hand before you wire anything up: https://modelcontextprotocol.io/docs/2026-07-28/tools/inspector

Gate 5 you have to reason about yourself, because it depends on which prerequisites belong to the machine and which belong to a person. That division is the actual design decision, and no health check will make it for you.

If you run this against something of your own, I'd like to hear which gate failed first.

Your AI Character Looks Right—Except Every Asymmetric Detail Flipped

Sho Naka — Sat, 05 Sep 2026 09:01:02 +0000

This article comes from the author's work maintaining Tsuduri, a recurring AI-illustrated character with asymmetric design features. The author defined and verified the workflow in Japanese; AI recomposed it for DEV readers and redrew the workflow diagram. #ABotWroteThis

Tsuduri's hair ornament belongs on her left side.

When she faces the viewer, that ornament appears on the right side of the image. When she faces away, it appears on the left. In a side view, it may move to the near side or disappear behind her head.

That sounds simple until an image model returns a polished illustration with the ornament on the wrong side. The face, colors, and mood can all look correct. The image may be attractive. It is still the wrong character state.

I stopped trying to solve this by loading a longer character profile into every generation. The workflow I settled on was smaller and more explicit:

keep a short invariant in the character's own coordinate system;
start with the single reference view closest to the requested composition;
classify the actual output view, then validate at least two asymmetric traits.

Keep the full turnaround sheet as source material. Do not make it the default payload for every image.

TL;DR

Write "the ornament is on the character's left," not only "put it on screen right."
Store the complete design canon, but begin each generation with one view-specific reference.
Judge the image that was actually produced, not only the view you requested.
Check at least one primary identity trait and one independent secondary trait.
Use N/A when a trait is genuinely occluded or outside the frame; do not turn missing evidence into PASS.
After a failed retry, change the reference or method instead of repeating the same ambiguous prompt.

Treat the bug as a coordinate-system bug

"Left" can refer to two different frames:

subject-relative left: the character's own left side;
screen-relative left: the viewer's left side of the finished image.

Those frames agree only in some poses.

Actual output view	Character's left side appears	Hair ornament expectation
Front	Screen right	Visible on screen right
Back	Screen left	Visible on screen left
Character faces screen left	Near side	Usually visible
Character faces screen right	Far side	Usually hidden
Three-quarter view	Depends on angle and occlusion	Evaluate from the actual pose

If the prompt stores only "screen right," it becomes wrong as soon as the camera or character turns. The stable rule is the subject-relative relation. The screen position is a view-dependent projection.

A usable instruction therefore has two layers:

Invariant: the hair ornament is attached to the character's left side.
Requested view: frontal; the ornament should appear on screen right.

The first line survives a camera change. The second line helps this specific generation.

A beautiful image can still fail spatially

Image quality and spatial correctness are separate checks.

The 2022 paper Benchmarking Spatial Relationships in Text-to-Image Generation evaluated whether generated images preserved requested relations between objects. It found that producing recognizable objects did not guarantee that the requested relation was correct.

A 2026 preprint, Can Text-to-Image Models Draw from the Right Frame of Reference?, separates viewer-relative instructions from object-relative ones. Across 1,200 prompt pairs and 22 models, the paper reports that object-relative accuracy averaged 41.8% lower than the corresponding viewer-relative accuracy; even the best model reached 44.3% on the object-relative task.

The same preprint also reports that rewriting prompts from visual feedback raised mean accuracy from 25.0% to 29.2% under an equal generation budget. That is an improvement, not a solution. The practical implication is that a closed loop—inspect the output, then select or correct it—is more realistic than expecting a stronger opening instruction to settle the question by itself.

Those numbers are not Tsuduri's success rate. The models, prompts, scoring, and target designs differ. They support a narrower point: switching between "the character's left" and "the image's right" is a real evaluation problem, not merely a wording typo.

That is why my review asks two questions independently:

Does the image look good?
Are the character-relative relationships correct for the pose that was actually rendered?

Passing the first question cannot answer the second.

Keep a thick canon and a light generation payload

The full design canon is still valuable. It can contain front, back, side, and three-quarter views; color rules; clothing connections; asymmetric hardware; and known exceptions.

The mistake is treating "valuable to retain" as "mandatory to attach every time."

Making "attach everything" the default is a workflow cost, not a measured accuracy effect. The cost follows from the shape of the request itself:

every generation starts by locating the same image set and attaching it in a fixed order;
angles that this pose does not need are in the input, and the request says nothing about which image should win;
the things you actually asked for this time, such as pose, scene, expression and style, sit next to a large block of reference description;
when a generation fails, the request gives you no way to say which reference or instruction was ignored;
every correction means rebuilding the whole bundle, and the routine stops being something you keep doing.

OpenAI Academy's image-generation guide recommends clear, specific prompts. The rest is my operating policy, not the guide's claim: keep the full canon as the thing you preserve, and attach only the minimum evidence this run needs. That policy does not prove one image is always more accurate. It keeps the two roles separate so a failure is easier to trace.

Use three layers:

Layer	Contains	Used when
Character canon	Turnaround views, colors, clothing connections, exceptions	Updating the design or resolving ambiguity
Generation reference	The closest view for the requested composition	Normal generation
Validation contract	Expected visible traits and PASS/FAIL/N/A rules	Reviewing the actual output

Build an asymmetry inventory before writing a longer prompt

Do not inventory the whole character. Inventory the features whose spatial relation carries identity.

asymmetric_traits:
  - name: hair_ornament
    subject_side: left
    attached_to: left_hair_mass
    identity_weight: primary
  - name: diagonal_strap
    starts_at: right_shoulder
    ends_at: left_waist
    identity_weight: secondary

Each trait needs enough information to answer:

What is the feature?
Which side or attachment point is invariant?
In which views should it be visible?
When is it legitimately N/A because of occlusion, crop, or scale?
Is it a primary identity signal or a secondary cross-check?

Checking only the hair ornament is weak. A flipped image can accidentally place one item correctly while reversing the strap, hair flow, or garment closure. Use at least two independent asymmetric signals when both are observable.

The three-stage loop

1. Freeze a short invariant

Keep the stable relation short enough to inspect:

The ornament remains on the character's left side.
The diagonal strap runs from the character's right shoulder to left waist.

Do not bury these lines inside biography, personality, lighting, and scene history.

2. Start with the closest single view

Choose the reference whose pose and camera angle most closely match the request. State its role: identity, orientation, clothing connection, or style.

The rule is "one view first," not "one view forever." If the output requires a back connection that the frontal reference cannot show, add a second reference specifically for that missing role.

3. Validate the actual output

The generated image may not match the requested view exactly. Classify what it actually shows before checking trait placement.

def review(output_view, observations):
    expected = expected_traits_for(output_view)
    verdicts = []

    for trait in expected:
        if observations[trait].is_occluded:
            verdicts.append((trait, "N/A"))
        elif observations[trait].matches_subject_relation:
            verdicts.append((trait, "PASS"))
        else:
            verdicts.append((trait, "FAIL"))

    visible = [v for _, v in verdicts if v != "N/A"]
    return "PASS" if len(visible) >= 2 and all(v == "PASS" for v in visible) else "FAIL"

This is review logic, not a claim that computer vision can infer every trait reliably. A person or a bounded visual reviewer still has to supply the observations. The code makes the decision boundary explicit: two visible independent traits, no observed mismatch, and no invented PASS for an occluded feature.

Escalate references by missing role, not by anxiety

When a result fails, use this order:

Reclassify the actual output view. The image may have turned more than expected.
Confirm the invariant uses the character's frame, not the screen's frame.
Replace the first reference if another stored view is closer.
Add one second reference with a named role, such as "back strap connection."
If the same defect survives a bounded retry, change the generation method, composition, or review strategy.

Repeatedly attaching more images without naming their roles makes the input larger without making the failure easier to explain.

What this workflow does not guarantee

This loop targets left/right and other asymmetric spatial consistency. It does not by itself guarantee:

anatomy or hand correctness;
exact facial identity;
color fidelity under different lighting;
style continuity;
that the model followed the requested pose;
that one reference is sufficient for every composition.

It also asks a different question from an identity-ablation experiment. An ablation test asks how much a design may change before recognition breaks. This workflow assumes the design should not change and asks whether the generated pose preserved its spatial invariants.

A checklist for the next image

Before generation:

[ ] Write one or two subject-relative invariants.
[ ] Select the closest single reference view.
[ ] Name the role of every attached reference.
[ ] Choose one primary and one independent secondary asymmetric trait.

After generation:

[ ] Classify the actual output view.
[ ] Translate subject-relative traits into expected screen positions for that view.
[ ] Mark each trait PASS, FAIL, or N/A.
[ ] Reject when an observable trait is flipped.
[ ] After a repeated failure, revise the setup instead of replaying the same prompt.

The durable design record can stay rich. The per-image payload can stay small. The review must be specific enough to catch the polished wrong answer.

Stop Rerunning All CI for Tiny Changes: Fingerprint Each Check's Inputs

Sho Naka — Tue, 01 Sep 2026 10:30:54 +0000

This article is based on a real failure in the author's AI-assisted publishing workflow. The author defined the problem and operating boundary; AI implemented and tested the mechanism, drafted the source, and recomposed this English version under a standing delegation for English-market publication. #ABotWroteThis

The trigger was a tag edit. The body and images were unchanged, but every automated review ran again. A reviewer's small wording fix then created another revision and restarted the set.

CI often behaves the same way: a README typo starts integration and E2E tests; a CSS change waits for backend jobs with identical inputs. The safe fix is not to call a diff "small." It is to fingerprint what each check actually uses and reuse a prior PASS only when its inputs, rules, and evidence still match.

TL;DR

Freeze related edits as one revision before evaluation. Saving a file should not automatically mean "start every check now."
Give every check an explicit dependency projection and fingerprint that projection, not the whole repository.
Reuse a prior PASS only when its input fingerprint, rule digest, and evidence still match. Unknown means execute.
Start with one slow job: write down exactly what must change before that job needs to run again.

Quick answer: how do you avoid rerunning every CI job?

Hash each job's verdict-changing inputs, rules, and tool version. Rebind a matching attested PASS with a new receipt; execute on mismatch or uncertainty. Decide once per frozen edit batch, not after every save.

The smell appears before the pipeline feels slow

The problem exists as soon as any of these are true:

Nobody can explain why a given job runs after a documentation-only change.
A single review comment causes unrelated jobs to restart.
The only invalidation rule is the repository commit SHA.

Parallelism does not remove irrelevant runs. File patterns also miss broad inputs such as policies, generated schemas, base images, fixtures, or prompts.

The question is not "Was the diff small?" It is:

Did any input that this specific check used to reach PASS change?

Separate editing from evaluation

People and agents make related edits in sequence. If each save starts evaluation, the pipeline judges incomplete intermediate states. Treat the edits as one revision batch:

Make all known edits.
Freeze the revision that will be judged.
Compute the checks required for that frozen revision.
Finish those checks before asking for the next human decision.

Saving records work in progress. Freezing identifies the exact candidate whose evidence matters.

Fingerprint each check's real dependency set

A repository-wide hash makes every character invalidate everything. Instead, define a projection: the smallest complete set of inputs that can affect one check's verdict.

Examples:

A Markdown style check may depend on documentation files and its lint configuration.
A frontend visual test may depend on UI code, styles, assets, browser version, fixtures, and baseline images.
A backend integration test may depend on code, schema, service configuration, fixtures, and runner version.
A security check may depend on source, lockfiles, policy, and scanner version.

For each check, choose current_pass when it already passed this revision, reuse_prior_pass when an attested PASS matches its projected inputs and rule digest, or execute otherwise. project() must include everything the check reads. If the set is uncertain, execute.

A minimal executable model

This executable model shows a documentation edit changing only the documentation projection:

from hashlib import sha256
import json

checks = {
    "docs": ["README.md", "docs/config.yml"],
    "backend": ["src/app.py", "requirements.lock"],
}

before = {
    "README.md": "old heading",
    "docs/config.yml": "strict=true",
    "src/app.py": "return 200",
    "requirements.lock": "framework==1.0",
}
after = before | {"README.md": "new heading"}

def fingerprint(files, state, rule="v1"):
    projected = {name: state[name] for name in sorted(files)}
    payload = {"inputs": projected, "rule": rule}
    return sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()

for name, files in checks.items():
    decision = (
        "reuse_prior_pass"
        if fingerprint(files, before) == fingerprint(files, after)
        else "execute"
    )
    print(f"{name}: {decision}")

Run it:

$ python3 change_aware_demo.py
docs: execute
backend: reuse_prior_pass

In production, use content-addressed files, generated inputs, environment data, and tool versions. Store the prior PASS as immutable evidence; this comparison alone is not authorization.

A result depends on rules, not just files

Matching files is insufficient. If a personal-data rule, scanner, browser, prompt, threshold, or target changes, yesterday's PASS does not prove today's verdict. Fingerprint both:

The projected content and environment inputs.
The rule digest that produced the verdict.

Incident-driven acceptance conditions belong in the rule digest too. Selective execution narrows invalidation; it must not hide it.

Reusing PASS is not the same as skipping

"We ran this before" is not evidence.

A reusable PASS binds:

check identity;
projected input fingerprint;
rule digest and tool version;
verdict and immutable evidence;
the original revision; and
a new receipt binding that evidence to the current revision.

"Job skipped" is vague. "PASS reused because inputs A, B, C and rule R are unchanged" is falsifiable.

What should happen for common changes?

Change	Expected behavior
README only	Run docs checks; reuse unrelated PASS only when projections and rules match.
CSS or UI asset	Run build, accessibility, and relevant visual checks; backend may be reusable.
Dependency or lockfile	Invalidate every consumer of the resolved graph.
Article tag/metadata	Keep body and image PASS when unchanged; read the remote metadata back.
Article body or image	Run checks whose projections include that body or image.
Unknown/legacy diff	Run everything.

What we tested in the real workflow

We tested the implementation with a real article and real image inputs, without granting publication authority. Before an answer, the PASS receipt count was zero; after the answer it was exactly one; replay was rejected; and the next review received the real article and image instead of placeholders.

This verifies the review-return and evidence path, not a universal speedup or a successful publication. No improvement percentage is claimed because it was not measured.

A practical migration path

Choose one expensive or frequently unnecessary check and answer:

What exact files, generated data, environment values, fixtures, policies, and tool versions can change its verdict?
Can you build a deterministic projection of those inputs?
What immutable evidence proves the previous PASS?
What rule changes must invalidate that evidence?
What unknown states should fall back to execution?

Log the decision before enabling reuse, and compare it with full-pipeline results. False reuse is a design failure; extra execution is only a missed optimization. Enable one check at a time.

Before you close this tab

Find one job that runs after unrelated changes. Write: "This check must run when..." List its actual inputs, rules, and tool version. If you cannot prove an input is unchanged, execute.

One honest dependency boundary is enough to start replacing "something changed, run everything" with explainable execution and explainable PASS reuse.

DEV Community: Sho Naka

My Audit Caught Every Known Bug. Ten of Its Next 18 Flags Were Wrong.

Quick answer

Seven known defects made a regression set, not a benchmark

Count the findings before naming the percentage

“Empty input” was too broad a defect definition

The second audit did not prove an improvement

A report that keeps unknowns visible

FAQ

Why is "7 out of 7 caught" not evidence that the checklist works?

What is the difference between precision and a false-positive rate here?

Did tightening the criteria reduce false positives?

I'm dropping spec-first AI development for smaller experiments

Quick answer

Keep the outcome stable. Let the implementation change.

Readability is the cost of the next correct change

Keep the next change affordable

Make prohibited operations impossible through the controlled path

Optimize the whole request, not the last response

Start the clock at the request

FAQ

Is this an argument against specification-driven development in general?

Why enforce prohibitions with a hook instead of instructions?

How will I know whether the shorter loop is actually faster?

When AI Becomes Production Infrastructure, You Need Recovery Capacity

Quick answer

"We'll just continue manually" is not always a fallback

AI is becoming production capacity, not just a tool

Personal spending can hide a capacity failure

AI can expand the blast radius of a bad assumption

Recovery Reserve: protect the recovery path, not a percentage

A fallback provider is an operational dependency

Inference cost becomes part of project cost

A first-pass business continuity table

AI operations begins where AI productivity needs resilience

FAQ

How large should the Recovery Reserve be?

Is switching everything to human work a valid degraded mode?

What belongs in a migration context for another model?

Your AI Video Passed QA—and Still Wasn't Publishable

Automated QA proves validity, not publishability

Separate four release states

Review the same artifact three different ways

Pass 1: audio only

Pass 2: silent visual

Pass 3: one integrated first watch

Record the first blocker, not an average

Make the release boundary machine-readable

Define "minor edits" narrowly

After the same rejection twice, stop micro-tuning

One Accepted video is not a stable pipeline

Keep machine QA and human QA in different jobs

A checklist for the next AI video

What this model does not guarantee

Your AI Adoption Is Waiting on Approval, Not Better Prompts

TL;DR

The queue is asking for a missing interface

The five-field approval interface

1. Information: what entered the system?

2. Task: what judgment was delegated?

3. Human check: who verifies what before use?

4. Explanation: what must remain reconstructable?

5. Stop authority: who can halt or reverse the use?

Route by two dimensions, not by the word “AI”

What the two routes should change

A request schema you can implement

Test the approval system itself

Start with one repeated queue

Cloudflare Blocked My Playwright Login. Attaching to a Chrome I Started Myself Walked Straight Through.

The symptom

What did not help

The observed discriminator was the launch path, not the binary

Three issues I hit right after switching

1. In this setup, zero page targets break connect_over_cdp

2. Chrome refuses remote debugging on your default profile

3. Teardown is an ownership problem

The profile copy that did not work here

Whether it was worth it

Checklist

FAQ

1. In this setup, zero page targets break `connect_over_cdp`