DEV Community: Emeka Adirije

The skin tone scale that scores 16.67% on an arm

Emeka Adirije — Fri, 23 Oct 2026 12:40:22 +0000

If you build anything that reasons about skin, you almost certainly sort people onto the Fitzpatrick scale. A research team tried to do that automatically last year and scored 16.67% on a photograph of an arm.

The same team, on the same images, scored 65.67% on those arms when they expressed the answer on a different scale.

What the study found

A group at TU Munich built an algorithm that estimates skin tone from the Individual Typology Angle, computed from CIELAB colour values using DensePose and OpenFace. They mapped the same values onto two scales: Fitzpatrick, and the Monk Skin Tone scale Google published in 2022.

Same images, same pipeline, same colour measurement underneath. The only variable was which scale the answer was expressed in.

Balanced accuracy on clinical photographs:

	Monk	Fitzpatrick
Face	67.59%	44.26%
Arm	65.67%	16.67%

Their wording on the Fitzpatrick results is blunter than mine: no clinically useful classification was possible.

Monk held steady across body regions and across light and dark tones. Fitzpatrick collapsed, and collapsed hardest on the arm, where there is no face shape to help the model guess.

Why it collapses

Fitzpatrick measures sunburn.

Thomas Fitzpatrick built it in 1975 to choose phototherapy doses. It works by asking a patient how their skin behaves in the sun. Do you burn, do you tan, how easily. It is a questionnaire about ultraviolet response with six possible answers, and for its original job it was a reasonable instrument.

It is now routinely used as though it described skin colour. The study above is itself evidence of that: it takes a colour measurement and maps it onto Fitzpatrick types, which is only a sensible thing to attempt if the field expects the two to correspond. The authors describe the scale as widely used and note that it is based on UV response, which is the tension they set out to test.

Those are different quantities. Two people with the same skin colour can honestly give different Fitzpatrick answers, and the scale has no mechanism to tell them apart, because nobody ever asked it to.

Monk was built to describe apparent skin tone, has ten points rather than six, and was designed knowing it would be used in computer vision. It is worth saying plainly that Monk is not evenly spaced either, and anyone treating its ten points as uniform steps will be wrong. What it has over Fitzpatrick is that it is measuring the right quantity.

What we tested

That study was an important input into what we built, so we evaluated before changing anything.

We used the Monk Skin Tone Examples set, the public dataset Google publishes alongside the scale, which carries ground-truth Monk values for every image.

The question was whether the paper's approach, computing tone from the colour measured in the image, survives contact with photographs taken on a phone.

The pixel route fails on phone photographs

Across 600 images, computing tone from pixels landed within one Monk step 32% of the time. In the light band it was right zero times. Almost everything collapsed into Monk 7 and 8 regardless of the person in the picture.

A phone camera automatically exposes for the face in frame. That is the entire job of the auto-exposure system, and it works: it pushes whatever the subject is toward a mid-tone so the photograph looks correct. A cheek belonging to someone at Monk 1 came back measuring around L* 52, which is the middle of the range.

So the camera is normalising away the exact quantity you are trying to measure, before your code ever sees the file. Any pixel-based skin tone pipeline running on consumer phone photographs is measuring the auto-exposure algorithm as much as the person.

Clinical studies use controlled lighting and calibrated capture. Someone holding their own phone in their own bathroom has neither, and that is the only condition that matters for a product used that way.

Going in, we expected the arithmetic to win. It is the auditable route and it is what the research uses. It lost in every band.

What changed in the product

Three things changed.

The measurement is Monk, and our analysis no longer emits a Fitzpatrick type at all. Where a Fitzpatrick value comes from a person, a trial participant or a clinician recording their own assessment, we keep it and label it with its source, because a self-reported answer to a self-reported scale is a legitimate thing to store. What we stopped doing is generating one.

We store the measurement, and derive everything else from it. The band and the friendly name a customer sees are computed at display time rather than written down as facts of their own, so any label can be traced back to the number that produced it.

Anywhere the system treats darker skin differently, it now asks one question in one place. Rules of that kind accumulate copies, and the copies disagree. There was a tone sitting right on the boundary that one path included and another missed.

What it changes for the person in front of the camera

Guidance that depends on skin tone now reaches everyone it should. Several things in skincare are handled differently for deeper skin, and those rules used to ask their question in more than one place. The answers did not always agree, so somebody sitting near the boundary could be covered by one path and missed by another. There is a single definition now, and it is applied everywhere the question comes up.

Advice that claims to be tone-specific now is. A six-point scale with a gap in the middle pushes a wide band of people onto the same value, and when most people share a value, personalising by skin tone produces the same answer for all of them. A ten-point scale measured rather than guessed gives the rest of the system something real to vary on.

The range now covers people it previously did not. The old scale had nothing above its lightest point and a sizeable gap in its middle, so anyone falling in either was moved to the nearest available label whether it described them or not.

What nobody should be claiming

If a product reports a Fitzpatrick type, it owes you three things: which scale, measured how, and how the method performs at the dark end specifically rather than on average.

A systematic review of 232 dermatology AI studies found that only 1.3% described the Fitzpatrick skin type of their subjects at all, and only 3.2% of the images were types IV to VI. The same literature keeps finding that AI detects skin conditions less accurately on darker skin. So the population most affected by the accuracy gap is also the one least represented in the work measuring it.

Until that changes, "works across all skin tones" tells you that somebody checked, and nothing whatsoever about what they found.

Study: Ulrich P, Zink A, Biedermann T, Sitaru S. Beyond Fitzpatrick: automated artificial intelligence-based skin tone analysis in dermatological patients. npj Digital Medicine, June 2025. Review of 232 studies: Jairath et al., Dermatologic Surgery, 2024, via this narrative review. Evaluation dataset: Monk Skin Tone Examples.

Spec-driven development has a verification hole

Emeka Adirije — Fri, 16 Oct 2026 21:42:42 +0000

In part one I argued that markdown won the specification layer without anyone holding a vote, and that we carried version control and code review across to it and left the rest of our discipline behind. This is the specific thing that absence lets through.

Spec-driven development is mostly right, and I say that as someone who has reorganised how my team works around it.

The argument is sound. When generating an implementation becomes cheap, the quality of the specification becomes the constraint. Sean Grove's framing in The New Code is the clearest statement of it: prompts are transient, while a versioned spec aligns humans and models over time. GitHub's Spec Kit, AWS Kiro and Claude Code have all built real tooling on that premise. At the radical end of the movement, humans only ever edit the spec and treat the code as a compiled artifact.

I think that direction is correct. I also think the formats the movement has standardised on, including the ones I use, are all missing the same section.

The thing that keeps going wrong downstream

The delivery numbers are strange in a specific way.

Sonar's analysis of LinearB's 2026 Engineering Benchmarks reports that AI-assisted pull requests merge at roughly half the rate of human-authored ones. They also run about 2.5 times larger and wait roughly five times longer for a reviewer.

Happy Bhati's paper on the agentic software development lifecycle puts the same thing structurally. Agents change the bottleneck in software delivery, and "review, integration, testing, security, deployment, and production operations remain constraining stages." Field-study gains, the paper says, "attenuate sharply between writing code and shipping reliable software."

So code gets produced faster and gets accepted at half the rate. The failure is happening between produced and accepted, which is the part of the pipeline the spec movement has spent the least time on.

Specifications point one way

A spec describes a destination. That is its whole job, and it is good at it.

Accepting a result requires something else. You have to compare what you have now against what you had before, and decide whether the difference is the one you asked for. That comparison needs an origin.

Almost no spec format carries the origin. The person who wrote the spec held it, in their head, at the moment they decided the work was worth doing. They knew what was broken, they had probably seen it happen, and then they wrote down what they wanted instead and threw the rest away.

Which means the author is the only person in the building who can tell whether the work is done.

That was survivable when a human engineer did the implementing, because a human reads a spec, notices the gaps, and silently fills them from context. They have seen the product. They know what the user was complaining about. They supply the missing half without anyone asking.

An agent reads it literally and builds exactly what the words describe. The missing half stays missing, and it surfaces at review, which is where the merge-rate numbers are coming from.

A spec is an assertion

I wrote recently about tests that have never been watched failing. A test written against code that already passes was true the moment it was written, by construction, and has never once distinguished a working system from a broken one. It raises coverage and verifies nothing.

A spec has the same shape one level up. It asserts that the product should behave a certain way. If nobody has observed the product failing to behave that way, and nobody wrote down what that failure looked like, then the assertion cannot be checked by anyone other than the person who made it.

Same disease, different artifact.

The missing section

This is not a claim about people writing lazy specs. It is structural, and you can check it yourself in about ten seconds.

Here is the specification template GitHub's Spec Kit generates, headings verbatim:

# Feature Specification: [FEATURE NAME]
  Feature Branch | Created | Status | Input

## User Scenarios & Testing (mandatory)
### User Story 1 - [Brief Title] (Priority: P1)
### User Story 2 - [Brief Title] (Priority: P2)
### User Story 3 - [Brief Title] (Priority: P3)
### Edge Cases

## Requirements (mandatory)
### Functional Requirements
### Key Entities

## Success Criteria (mandatory)
### Measurable Outcomes

## Assumptions

Twelve headings, three of them marked mandatory, on a toolkit with around 141,000 stars.

Read down that list and look for the place where you write what is happening today. Success Criteria describes the destination. Edge Cases describes situations the new behaviour should survive. "Why this priority", inside each user story, argues for sequencing. Assumptions records what you are taking on faith.

Nowhere does the format ask what a person currently experiences, how you came to know it, or how somebody else could go and see it for themselves.

Kiro's requirements phase produces the same shape, an introduction followed by user stories and EARS-style acceptance criteria. Forward, all of it.

So when work specified this way comes back implemented, the only person who can say whether it worked is the person who was holding the before-state in their head on the day they wrote it. Everyone else is reading a description of a destination and taking the author's word for the journey.

What I would add

One section, placed before the user stories.

## Current behaviour (mandatory)

Problem
What a person experiences today, in a sentence or two.

Evidence
How you know. Tickets, a number, a recording, a complaint.

Steps to reproduce
What someone else does to see it for themselves.

Filled in, that looks like this.

## Current behaviour

Problem
The weekly digest arrives with at most 20 items for every account,
regardless of how much activity there was that week.

Evidence
Four accounts raised it this month. The largest had 300 qualifying
events in the week and received a digest listing 20 of them, with no
indication that anything had been left out.

Steps to reproduce
1. An account with more than 20 qualifying events in the period
2. Trigger the weekly digest
3. Count the items in the delivered email. It is 20, and the job
   reported success.

Three sentences and a numbered list. Notice that none of it tells the agent how to build anything, which is deliberate. It specifies the destination by specifying the gap.

Three things become true that were not true before.

A reviewer who has never spoken to the author can verify the work, because the reproduction steps are sitting right there and they either still produce 20 items or they do not.

The agent can check its own output before handing it over, against a stated observable rather than against its own reading of what it was asked to do.

And the acceptance criterion becomes a comparison between two numbers that already exist in the product, which is considerably harder to satisfy by accident than "pages through the full result set".

The whole thing, assembled

One added section, in a complete spec, in Spec Kit's own structure.

# Feature Specification: Complete Weekly Digest

Feature Branch: 014-complete-weekly-digest
Created: 2026-10-15
Status: Draft
Input: "The weekly digest is cutting off activity for our bigger accounts."

## Current behaviour (mandatory)

Problem
The weekly digest arrives with at most 20 items for every account,
regardless of how much activity there was that week. Accounts with
more activity than that receive a digest that silently omits the rest.

Evidence
Four accounts raised it this month. The largest had 300 qualifying
events in the week and received a digest listing 20, with nothing in
the email indicating anything had been left out. The job reported
success in all four cases.

Steps to reproduce
1. Use an account with more than 20 qualifying events in the period
2. Trigger the weekly digest for that period
3. Count the items in the delivered email. It is 20, while the activity
   view for the same period shows 300.

## User Scenarios & Testing (mandatory)

### User Story 1 - See the full week of activity (Priority: P1)

An account owner opens the weekly digest and finds every qualifying
event from that week, so they can review what happened without logging
in to check whether the email left anything out.

Why this priority: the digest exists to be a complete weekly record,
and an account owner who has to verify it elsewhere has no reason to
open it at all.

Independent Test: send a digest to an account holding more qualifying
events than the current cap, and confirm the delivered email lists all
of them.

Acceptance Scenarios:
1. Given an account with 300 qualifying events in the period, When the
   weekly digest is generated, Then the delivered email lists 300 items.
2. Given an account with 3 qualifying events in the period, When the
   weekly digest is generated, Then the delivered email lists 3 items.

### User Story 2 - Know when a digest has been shortened (Priority: P2)

An account owner receiving a digest that could not carry every event
sees how many were left out, so they know to go and look.

Why this priority: it only matters once the cap is raised, since at
that point omission becomes rare rather than routine.

Independent Test: force a digest past the message size limit and
confirm the email states the number omitted.

Acceptance Scenarios:
1. Given a digest whose full contents exceed the message size limit,
   When it is delivered, Then it states how many events were omitted.

### Edge Cases
- An account with zero qualifying events in the period
- An account whose event count changes while the digest is being built
- A digest whose full contents exceed the provider's message size limit

## Requirements (mandatory)

### Functional Requirements
- FR-001: The digest MUST include every qualifying event in the period
  for the account it addresses.
- FR-002: The digest MUST preserve chronological ordering across the
  entire set.
- FR-003: Where a size limit prevents including every event, the digest
  MUST state how many were omitted.

### Key Entities
- Digest: a periodic summary addressed to one account, covering a
  fixed window, containing an ordered list of qualifying events.
- Qualifying event: an account activity eligible for inclusion,
  carrying a timestamp and a type.

## Success Criteria (mandatory)

- SC-001: The item count in a delivered digest equals the item count
  shown in the activity view, for the same account and period, across
  100% of digests sampled over four weeks.
- SC-002: Digest-related support contacts fall from four a month to
  zero.
- SC-003: No delivered digest omits an event without saying so.

## Assumptions
- The activity view is the authority on what qualifies for a period.
- Message size limits are a provider constraint rather than a product
  decision, so FR-003 handles them rather than removing them.

"But my spec is for a new feature"

This is the strongest objection to everything above, and it deserves a real answer rather than a dodge.

Nothing is broken when you are building something that does not exist. There is no bug report, no reproduction, no before.

There is, though, a current behaviour. Someone wants an outcome today and does something else to get it. They export to a spreadsheet and sort it by hand. They ask a colleague in chat. They open four tabs. They give up and the outcome never happens at all.

That is observable, reproducible, and writable in the same section:

Problem
There is no way to see which customers have lapsed. Account managers
currently export the full customer list and sort it in a spreadsheet
by last order date.

Evidence
Three of five account managers do this weekly. It takes about 40
minutes each and the result is stale by the time it is used.

Steps to reproduce
1. Open the customer list
2. Attempt to filter by last activity date
3. There is no such filter, and the list view offers no ordering by it

If you cannot write that section for a feature, that is useful information. It usually means you have not yet established that anyone does the thing you are about to build a better way of doing.

Why adding more implementation detail makes it worse

Over-specification is already a named failure mode in spec-driven development, alongside spec-rot and slow starts. I want to be more specific about the mechanism, because "do not over-specify" is advice nobody can act on.

The detail people add under pressure is forward detail. More bullets about what to build, which classes to touch, which approach to take. Every one of those lines does two things. It removes the agent's freedom to find a route you had not considered, and it adds nothing you can check when the work comes back.

You are spending your specification budget on the half of the document that was never the problem. The same ten lines spent on what is currently happening, how to see it, and what number should change would make the result verifiable by someone who has never met you.

What this changes on a team

The format my team uses has seven fields. Problem, Evidence, Steps to reproduce, Expected result, Scope, Acceptance criteria, and Out of scope. Six of them are ordinary. The load-bearing one is the observable before-state, because a ticket with no reproducible symptom cannot be signed off by anybody.

Three things fell out of that once it was enforced.

Work became portable. A spec with a reproduction can be picked up by whoever is free, or by an agent, without a conversation to recover the context that was never written down.

Review got cheaper. The reviewer is not reconstructing intent from a diff. They are running three steps and reading a number.

And a fair number of proposed tickets died at the writing stage, because the person writing them could not produce evidence that the problem existed. That is the part I did not expect, and it has saved more engineering time than anything else on this list.

The claim

We are blaming the models for something we built into the process. A specification that was never checkable produces work that cannot be accepted, and the whole industry is discovering this at once because the humans who used to quietly supply the missing half have been taken out of that step.

Specs are the new code. Fine. Then they need the thing code has always needed, which is a stated expectation that something other than their author can evaluate.

Spec-driven development: the rise of Markdown

Emeka Adirije — Thu, 15 Oct 2026 21:34:13 +0000

Open my editor on a normal working day and most of the tabs are .md files.

Five years ago that would have been an odd thing to admit. The work lived in .java, .ts, .php, etc. Markdown was for READMEs and the occasional doc nobody opened. Now the markdown is where the decisions get made, and the implementation file is increasingly something I read rather than something I type.

That shift has a name. Spec-driven development starts from the observation that when an agent can produce a working implementation from a clear enough description, the description becomes the thing worth version-controlling, reviewing and arguing over. You write a specification. The agent builds from it. When you want something different, you change the specification and regenerate, rather than hand-patching the output and letting the two drift apart.

GitHub puts it more directly than I would dare to. Their post on the subject is titled "Using Markdown as a programming language when building with AI".

How quickly it happened

Look at where the important text in a modern repository actually lives.

Agent instructions are markdown. AGENTS.md is now in more than 60,000 repositories, read natively by Codex, Cursor, Copilot, Gemini CLI, Aider, Windsurf, Zed and twenty-odd others, and the spec is stewarded by the Agentic AI Foundation under the Linux Foundation.

Specifications are markdown. GitHub's Spec Kit generates a spec-template.md. Kiro's requirements phase produces a requirements.md.

So are the skills an agent loads, the prompt files teams check in, the plans, the runbooks, the context documents. All of it, one format, in roughly eighteen months.

Nobody argued about the format

Here is the part I find genuinely strange.

There was a real standards fight, and it was about the filename. AGENTS.md against CLAUDE.md against .cursorrules and the rest. It went on long enough and mattered enough that a Linux Foundation project now stewards the outcome.

The extension was never on the agenda. Nobody proposed YAML, which every CI system in the world already parses. Nobody proposed JSON, which has schemas and validators and editor support that would tell you about a missing field before an agent ever read it. Nobody built a small DSL, which is what this industry has done roughly every other time it needed machines and humans to agree on a document.

Markdown won before anyone thought to hold a vote.

It won for a good reason, and it is worth being precise about what the reason is. Markdown is the only format a person reads comfortably in a diff and a model reads natively with no parser, no schema and no ceremony at all. You can put a table in it, or a paragraph, or a numbered list, or three sentences of hedging about an edge case nobody has resolved yet. That flexibility is exactly why it spread into every corner of the toolchain in a year and a half.

It is also the whole problem.

What we brought with us

Two things came across from source code, and both are good.

Specs live in the repository. They are versioned, they diff, and you can see who changed what and when. That alone puts spec-driven development ahead of every requirements process I worked under before this.

And specs get reviewed. They arrive in pull requests, people comment on them, and a bad one can be argued with before anything is built. Moving the argument earlier is most of the value the movement claims, and the claim is fair.

That is where the inheritance stops.

What we left behind

Everything else we demand of source code is absent.

There are no types. Nothing checks that a section called Success Criteria contains anything resembling a criterion. You can leave it empty, or fill it with a sentence that means nothing, and every tool downstream will accept the document and proceed.

There is no compiler. Nothing reads the spec before an agent acts on it and tells you that requirement four contradicts requirement nine, or that the acceptance criteria describe a behaviour the functional requirements never mention. In a .ts file, a contradiction that blunt is a red squiggle. In a .md file it is valid markdown, and the first thing that notices is an agent confidently building one of the two.

There are no tests. The nearest thing the ecosystem has produced is a lint-spec.prompt.md, which asks a model to read your specification and grade it for clarity and completeness. That is a non-deterministic check, run by the same class of system that will consume the document, on an input with no defined structure. It is better than nothing and it is not a test.

What tooling does exist mostly confirms the gap. markdownlint checks style, which is formatting rather than meaning. OpenSpec ships a validate command, and a 2026 comparison of these tools describes it plainly: it checks structure rather than behaviour, so the spec is a convention rather than an enforced contract. The same comparison notes that verification of code against the spec is the phase most tools treat as optional.

Convention rather than enforced contract. That is a fair description of the entire layer, and that layer is now where the decisions are made.

The obvious objection

Markdown's looseness is a feature, and rigidity is why nobody wrote specs before. Every attempt to make requirements machine-checkable in the last forty years produced a format so tedious that teams quietly stopped using it, which is the actual history of UML, of heavyweight RUP documents, of most of IEEE 830.

That is true and I would not trade it back.

A stricter format would miss the point. You can keep every bit of the prose and still have something that reads the document before an agent does and says that a mandatory section is empty, that two requirements cannot both hold, that an acceptance criterion names a behaviour nothing else in the file mentions. None of that requires giving up a single sentence of English.

The rigour is missing for a duller reason than the format. We moved our most important text into a new home in about eighteen months, and we have not yet got around to building any.

Why the stakes changed

When markdown was for READMEs, an incoherent document cost you a confused reader who went and asked someone. The blast radius was one person and ten minutes.

That same document is now the input to a system that reads it literally, fills every gap you left with its own judgement, and emits a few thousand lines in a couple of minutes. A sentence that could be read two ways no longer waits for someone to ask. It gets resolved, silently, by whichever reading the model found most plausible, and the result arrives looking like completed work.

We spent forty years building tools that refuse to let us commit incoherent source, and then moved the decisions that matter most into a format where anything you type is valid.

In part two I will take one specific thing that absence lets through, and it is a bigger one than it looks: a gap in every mainstream spec format that makes the result impossible for anyone except its author to verify.

Red Once: never trust a test you haven't watched fail

Emeka Adirije — Tue, 13 Oct 2026 13:53:48 +0000

My visual regression check reported zero pixels changed across 28 screenshots.

Every one of those pages had a bright red background injected into it.

28 screenshots compared
0 px changed
PASS

Three separate defects were queuing up behind that zero. The check had been green since the day it was written, which is exactly what made it worthless.

The thing that changed

Test-driven development asks you to write a failing test before the code. Most engineers have never worked that way, and the profession has largely made peace with that. Tests get written after, or alongside, the suite goes green, and life continues.

That peace held because writing a test by hand carried some verification as a side effect. You had to imagine the failure in order to type the assertion. You pictured the thing breaking, decided what breaking looked like, and wrote that down. Nobody called it a methodology, but it was doing real work.

An agent generating forty tests from an implementation that already passes does none of that. It reads what the code does and asserts that the code does it. Every assertion is true at the moment it is written, by construction, and none of them has ever distinguished a working system from a broken one.

The output is a suite that raises coverage while verifying nothing. It is harder to catch than a missing test, because it looks reviewed. Agents produce well-named, well-shaped, arrange-act-assert tests. They look like the work of someone careful.

What the red step was actually for

The ordering in TDD exists to deliver one guarantee: that a human being has personally watched this assertion distinguish broken from working. Writing the test first is simply the most convenient moment to collect that guarantee, because the code does not exist yet and red is free.

Which means the guarantee is separable from the ordering. You can write the code first, the way nearly everyone and every agent actually works, and still collect it. You just have to pay for it afterwards, deliberately, instead of getting it for free.

The pattern: Red Once

No assertion is trusted until you have watched it fail, for the reason you think it fails.

Performed at the moment a test is authored, not retroactively across an existing suite.

1. Break the code under test

One change, inside the code under test. Invert a condition. Return a constant. Delete the line that does the work. Rename the column the query selects.

Editing the test instead proves nothing, because a test can always be made to fail by breaking the test.

2. Run that one test and read the failure

It has to go red, and the message has to name the thing you broke.

$ pytest tests/test_balance.py::test_excludes_body_only -x

FAILED  assert 4200 == 3900
          where 4200 = invoice_total(session)
1 failed in 0.41s

3. Restore, re-run, and write down what you broke

Green again, and the mutation recorded next to the test. One line is enough.

def test_excludes_body_only(session):
    # red-once: returned the unfiltered total from
    # invoice_total() -> 4200 != 3900. 2026-10-13.
    assert invoice_total(session) == 3900

This is the step everybody skips, and it is the one that turns a private act of diligence into something the next person can repeat.

Where it sits

Red for the right reason

A test that fails is not automatically a test that works. It can go red because an import blew up, a fixture collapsed, a migration had not run, a network call timed out, or a mock went stale. All of those produce red, and none of them tells you anything about your assertion.

So step 2 has two conditions, and the second one is the real one: it failed, and the failure names the thing you broke. If you inverted a filter and the test reports a connection error, you have learned nothing and you are not finished.

This is the same error that produced the red page and the zero. The check passed for a reason that had nothing to do with the pages matching: the diff library painted unchanged pixels as faded grey, the test read colour channels for any value at all, so nearly every pixel registered as unstable and the mask built to ignore instability grew to cover the entire page. A correct answer arrived by a route that could never have produced a wrong one.

What was hiding behind the zero

The harness was wrong three ways at once, and all three were invisible to a green run.

The mask ate the page. The diff library paints unchanged pixels as faded greyscale rather than leaving them transparent. The comparison tested colour channels for any value, so almost every pixel counted as changed, and the instability mask covered everything. Reading the alpha channel instead fixed it.
A footer that was never photographed. Screenshots captured the viewport. One of the footers sat four thousand pixels down the page.
Two sides on different frames. The CSS animations were keyed to page load, so the comparison was never made at the same moment in the animation, and the drift was absorbed as noise.

There was a fourth problem in the threshold itself. It was expressed as a ratio of total pixels, which scales with page height, so the longer the page the more a real regression could hide inside an acceptable percentage. Absolute pixel counts, set just above a measured noise floor, do not have that property.

All four surfaced within an hour of injecting a regression the check was supposed to catch and refusing to accept green.

Make the agent do it

This is the part that makes the discipline survive contact with how software is now written. An agent that can write a test can also break the code, run the test, and show you the failure. It is a strictly mechanical sequence, which is the kind of work agents are good at.

Add it to your agent instructions, and require the evidence rather than the assurance:

After writing a test, prove it discriminates:
  1. make one change to the code under test that
     should break it
  2. run that test alone, capture the output
  3. confirm the failure names what you changed
  4. revert, re-run, confirm green
Paste both outputs. Do not describe them.

The captured failure is the artifact. It goes in the pull request next to the diff, and it changes what a reviewer is looking at. A recorded failure is a receipt, and a reviewer can check it in seconds.

The old review question	The question that survives agents
Is there a test for this?	Has this test ever failed?
Answerable by looking. Satisfied by anything shaped like a test.	Answerable only with evidence, and the evidence is cheap to produce and impossible to fake by accident.

What it costs, and where it stops

About a minute per test. Break, run one test, read, revert, note. One case, start to finish.
It is not retroactive. Nobody is going to go back through four thousand existing tests. Do it at authoring time, and do it for any test standing guard over something whose failure would be expensive or silent.
One mutant is enough. The goal is to establish that the assertion discriminates at all, not to prove it catches every possible defect.
It does not replace the tools. Mutation testing frameworks do this exhaustively and automatically. If you can run one, run it.

On the name and the idea

None of the mechanism here is new. Deliberately breaking code to confirm a test notices is the manual form of mutation testing, which has been studied since the 1970s and has mature tooling: Stryker, PIT, mutmut.

What changed is the cost of skipping it. For decades this was a nicety, because hand-writing a test did some of the work incidentally. Generated tests do none of that work, so the step everyone used to skip is now the one holding the suite up. It needed a short name and a place in the workflow.

Write the code first if you want, but never trust an assertion you have not watched fail for the right reason. That is all of Red Once.