TL;DR: A green report is not evidence until its command is rerun against the artifact on disk; here, that exposed two failures in 16 runs. The parent slice log records the result.
Your test report says stable. Is the suite stable, or did the report get there first? That question can save you from approving a defect you cannot see. A durability suite reported stable, received approval from two independent reviewers, and still failed when someone ran it again against the worktree on disk.
In this note
- The approval that did not survive contact with disk
- What your green report cannot tell you by itself
- Go hunt these shapes in your pipeline
- What the slice actually proves — and what it does not
- Your next approval needs an artifact
The failure rate was roughly 12%. It was not hidden behind a complicated attack. A fixture threw away the stderr that would have named the problem.
If you run a CI pipeline, review agent work, or sign off on a release, this is your problem too. A green report is a statement. The files and command result are the thing that statement must answer to.
The approval that did not survive contact with disk
The suite was unstable even after it was reported stable and approved twice. The instability appeared only when the supervisor reran every gate against the actual worktree, instead of reading the session that created it.
SLICE-011 was a disposable prototype for five durability claims. It shipped nothing. Its job was to measure unsafe behavior red first, prove a proposed control green, preserve an exit record, and make later production work refuse to proceed without that digest-bound record.
The fifth claim covered session ownership. First, the prototype had to show two processes could drain one session. Then it had to show the existing fence refused the second owner. That is the right shape: demonstrate the hole before celebrating the lock on the door.
The reported result looked fine. The ownership check was marked 5/0, then 20/20 stable after a fixture fix. But before that correction, the reruns measured two failures in 16 full-suite runs.
The fixture set PRAGMA busy_timeout after journal_mode. Two workers opening concurrently could hit database is locked outside the Effect catch. The failure signal existed. The fixture captured worker stderr and discarded it.
The evidence was the problem, not the reviewers. Two independent reviewers approved what they were shown. The artifact on disk said something else.
Neither failure was found by reading a report.
There was a second hit in the same slice. Its compiled gate was built incorrectly for the case it existed to cover, following a mistaken supervisor instruction. A reviewer caught that, and it was reproduced on disk before the fix. The gate had derived the record from the open slice when it needed to resolve the prototype record in both the open and completed slice locations.
That is the scar. The process that was supposed to keep a durability record honest contained a wrong gate and a flaky control. Neither yielded to polished prose.
What your green report cannot tell you by itself
A green report cannot establish that the command was rerun, that the fixture exposed its diagnostics, or that the gate tests its own refusal path. You need the artifact, the command, and an independent rerun.
A session summary is useful for navigation. It is not evidence. The SLICE-011 record says this plainly: the session’s own summary is discarded self-report.
This matters most when a test touches timing, concurrency, process teardown, retries, or shared state. Those are exactly the places where one clean run can convince you that a mechanism works while the next run proves the control was never stable enough to speak.
Here is the deal: your negative control must be stable too.
SLICE-011 had a missing written control: the negative control itself needed to be stable. A flaky control cannot tell you whether you found a race or changed the mechanism. It turns every result into an argument about noise.
Do not replace that requirement with more confidence in the review. Review can catch a bad instruction, as it did here. It cannot transform an unrerun artifact into a measurement.
Go hunt these shapes in your pipeline
Start with the checks that block a deploy, approve an agent change, or certify a safety control. You are looking for ways the pipeline can describe success without preserving the result that earned it.
- Find a check whose result is accepted from a worker, agent, or session summary. Rerun its exact command against the checked-out worktree.
- Find fixtures that capture stdout or stderr. Make a deliberate failure and confirm the diagnostic reaches the person judging it.
- Find concurrency or timing tests with a single successful run. Run the full suite repeatedly and record failures as failures, not as an inconvenience.
- Find every negative control. Ask whether it is stable enough to distinguish the old mechanism from a race.
- Find a gate that protects a record or workflow. Break the path it was created to refuse and verify that it refuses for the stated reason.
- Find instructions that changed a control’s lookup or input. Treat the instruction as a hypothesis, then test the behavior on disk.
Do this before you add another dashboard. You do not need a new reporting layer to learn whether a fixture swallowed the one line that mattered.
What the slice actually proves — and what it does not
The prototype proves its five durability claims red to green in a scratch harness worktree, with negative controls and a digest-bound record. It does not ship production durability.
That boundary matters. The record exists to gate later durability production slices. It is not a license to claim that durable retry, durable blockers, or session-ID fencing are built in production. The README calls those three remaining production claims unbuilt and says Ranex is pre-release.
The corrected ownership control reached 20/20 stable. That is a result for this prototype after the fixture fix, not proof that all concurrency checks in your repository are sound. I do not have evidence for that.
The useful proof is narrower and stronger: when the supervisor reran the artifact on disk, the suite’s self-description lost to measurement. When a reviewer challenged the compiled gate, the gate was reproduced and corrected on disk. The process improved because it made claims answerable to an artifact outside the session that made them.
That is also the point of the kernel: the verdict comes from executable checks and evidence, not from a worker’s confidence. Ranex is still pre-release, with a working verdict path and substantial designed work remaining. The slice records live in the repository under docs/slices/done/, including the parts that failed before they were fixed.
Questions people actually ask
These questions help you inspect flaky suites before you accept a green check.
Why is a test report not enough evidence?
A report can describe a run inaccurately or hide the artifact that produced it. Re-run the gate against the worktree on disk.
How was the flaky test suite detected?
A supervisor reran every gate against the worktree on disk and measured two failures in 16 full-suite runs.
What caused the SLICE-011 flakiness?
The fixture set PRAGMA busy_timeout after journal_mode, and it discarded worker stderr that contained the database-lock message.
Your next approval needs an artifact
Before you approve the next flaky-looking check, take the command out of the report and run it against disk. Force the failure path. Read stderr. Run the full suite again. Then ask whether the control that proves the old behavior is stable enough to mean anything.
Try it. Break it. Tell me what broke. If this helped you catch a green report that could not survive a rerun, star Ranex on GitHub and send the honest critique. The useful critique is the one that finds the next hole.
Disclosure: this post was drafted with AI assistance. Every factual claim traces to the repository’s README or slice records — the same fact gate the product enforces on code. It ships only after Anthony’s own review.
Top comments (0)