A Green Check Is Not Learning Until the Next Run Reads It

#ai #agents #evaluation #softwareengineering

A Green Check Is Not Learning Until the Next Run Reads It

A current PASS proves one input passed; it does not prove that a later run used a lesson from an earlier failure.
The minimum receipt chain is failed bytes, one bounded edit, a held-out result, the current hash, and a next-run read status.
A pinned SkillOpt-Sleep mock run moved held-out evaluation from 0.3333 to 1.0 and blocked a harmful edit.
That is evidence for a fixed mock acceptance contract, not a production-quality, revenue, or intelligence measurement.

The nightly log ends in PASS. The operator still has one unanswered question: Did yesterday's correction reach tomorrow's input?

That distinction is the whole job of an evidence chain. This guide is for the engineer who runs a nightly, repeatable investigation and cannot prove that the next morning's input read yesterday's correction.

Three states a green check cannot prove

Rewrite means a document or setting changed. It proves only that bytes changed.

Current PASS means the current input passed its check. It proves only that check's verdict for those bytes.

Auditable learning keeps the failure, edit, evaluation, hash, and next-run read receipt together. It can show that an accepted change reached the later execution.

Do not use the third label when only the first two receipts exist.

How to prove five receipts reach the next run

1. Preserve the failure bytes

Keep the failed input, question, score, or post-run measurement. Keep the document or configuration bytes that produced it. Add the failure classification and one run identity.

A score without its input cannot tell you what was corrected. Hash the input and output in the same receipt, and retain the original failure text instead of replacing it with a summary.

2. Make one bounded edit

Change one bounded part: an addition, deletion, or replacement. Do not change the prompt, evaluator, data, and router in the same acceptance attempt.

Microsoft's SkillOpt README calls a skill document “the trainable state of a frozen agent.” It describes acceptance as a candidate that “strictly improves a held-out validation score” and names a “rejected-edit buffer.” The useful operational rule is narrower than the marketing claim: a small edit leaves a trace that can be compared with the next result.

3. Re-evaluate on held-out work

Do not accept a candidate only on the examples that motivated it. Keep work out of the edit-making step and use that held-out work for the acceptance decision.

If the evaluator cannot return a verdict, or the difference cannot be separated from noise, record unknown. A rejected edit remains evidence about the boundary; it should not be silently deleted.

4. Bind the chain to hashes

Put these fields into linked receipts:

failure: original failure, classification, input hash
edit: changed bytes or setting, change hash
evaluation: initial and held-out result, task version
decision: accept, reject, or unknown, with the reason
consume: next-run identity, loaded identifier, current hash, read status

If the current bytes do not match the receipt hash, do not attach an old PASS to the new document. The hash is the boundary that says which bytes were evaluated.

5. Require the next-run read receipt

Writing the accepted version to a ledger is not enough. The next execution must report what it loaded.

The read receipt needs at least four values: next-run identity, stable identifier of the loaded file or setting, current hash at load time, and machine-readable read status. If the evaluator reads path B while the accepted version lives at path A, the evaluation can pass while the lesson never reaches the loop.

The pinned mock receipt

I ran the documented command from commit 7da46ae693ee0329b80225c0128a37d65db10e9e:

python3 -m skillopt_sleep.experiments.run_experiment --persona researcher --assert-improves

The fresh repository run returned exit code 0:

tasks: 12   tokens(approx): 0
baseline held-out : 0.3333
after  held-out   : 1.0   (lift +0.6667)
gate blocks harmful edit: True

PASS: nightly consolidation improves held-out score AND gate blocks regressions.

This experiment result supports one narrow statement: in this fixed mock, the held-out score rose and the harmful edit was rejected. It is not a five-field next-run receipt, and it does not show a production agent became 0.6667 smarter, improved revenue, or generalized to every task.

What the evidence chain costs

The practical cost is not only hash computation. It is keeping records coherent, protecting the held-out boundary, and checking that the next run reads the accepted version.

Records: retain four small receipt types: failure, edit, evaluation, and consume. Keep the original failure and held-out data.
Hashes: compute hashes for the bytes under evaluation and the accepted version. This recovery did not benchmark CPU time or storage size.
Evaluation: replay held-out work and judge the result. A real backend spends provider budget for replay, judging, or reflection; the fixed mock does not.
Operations: choose retention, handle schema changes, and detect missing records. These checks are recurring maintenance, not a one-time setup task.

The pinned SkillOpt-Sleep documentation uses the phrases “writes a local evidence.jsonl,” “The mock backend makes no provider calls,” and “dry-run still incurs provider calls and spend.” It also asks operators to set a retention policy. So tokens(approx): 0 is a property of this mock result, not a price for production operation; provider budget is an operating cost to measure for the real backend, not a number this article invents.

For a small loop, the minimum is a schema, a held-out set, retention rules, and a read check. The exact API bill, storage footprint, and wall-clock time remain deployment measurements. Do not fill them in from the mock result.

Where this fits for a nightly investigation

Use the chain for recurring tests, routine investigations, or stable data transformations where a held-out set and a defensible correctness signal exist.

Return unknown when the holdout is contaminated, the evaluator fails, the task changes too much between runs, or the next-run read receipt is missing. A subjective edit may still be useful, but it does not earn this learning claim without a repeatable acceptance boundary.

If you operate a recurring agent, use the Anicca diagnostic entry point to verify five fields in its measurable receipt: failure record, bounded edit, held-out result, current hash, and next-run read status. If one field is absent, record unknown; this article's mock receipt and any revenue result are not the diagnostic result.