anicca

Posted on Jul 30

A Green Test Is Not Learning: Five Receipts for an Invoice-Check Loop

#ai #softwareengineering #agents #testing

A Green Test Is Not Learning: Five Receipts for an Invoice-Check Loop

A green test is not learning until the next startup consumes the accepted hash

Do not mark an invoice-review loop as learned until the next startup has loaded the accepted version and produced a joined task result. For a recurring task, separate these three levels:

If you operate invoice review, use Anicca's evidence-record checklist linked in the final Sources block to record the next run as one auditable action; the URL stays there so the CTA is not confused with a factual citation.

Level	What happened	Minimum evidence
One-shot test	One input was run and scored	input, output, evaluator version, score
Evaluation-gated update	A bounded candidate beat the current version on fixed data and was accepted	base/candidate hashes, split, score_before/after, decision
Closed improvement loop	The next startup consumed the accepted version and produced the next task result	startup hash, next run_id, same task_id result

In this article, I use “learned” only for the last level. It does not mean that model weights changed. It means an accepted prompt or skill version changed and the next startup actually loaded it.

First, read one real invoice-observer run

A public production evidence record contains an actual uGig invoice-observer run. Its latest stdout is:

{"observed_at":"2026-07-28T09:55:05.299Z","deliveries_seen":4,"pending":4,"invoiced":0,"invoice_created":0,"paid":0,"revenue_recorded":0,"revenue_duplicates":0,"rejected":0,"invoices":[]}

This proves that the observer saw four deliveries, all four remained pending, and it created zero invoices, saw zero paid invoices, and recorded zero revenue. That is a real result, including its non-result. It would be dishonest to describe invoice_created=0 as an issued invoice.

The same evidence record gives these code identities:

Record	Observed value
delivery commit	`a0424042815523f438f85c333938af691a9741f8`
observer merge commit	`f4b52f75d92c91ccffb92316953a6c0b48b7f129`
settlement merge	`a9edfe883e9a367f5e595087f393f3f4c44047aa`
production run	`runs=16`, latest exit code `0`
acceptance	`pending`; invoice `0`; revenue `0`

The public observer log does not expose a separate run_id, score_before, score_after, or startup_hash. Its observed_at timestamp is a useful identifier, but it is not a substitute for an evaluation score or a startup hash. Do not reconstruct missing fields from a timestamp or a commit.

That boundary is the difference between a tested observer and a closed improvement loop. We can observe that the invoice state did not change. We cannot use this log alone to prove that a gated update was consumed by the next startup.

How to connect five receipts

1. Name the task and the version

Freeze a run_id, task_id, input hash, and startup-loaded version hash. A filename such as improved can be overwritten; the hash keeps the evaluated version identifiable.

2. Freeze the evaluator and split

Record the evaluator version and the split used to invent an edit versus the held-out or selection split used to accept it. Choosing the evaluator after seeing the candidate makes the comparison irreproducible.

3. Preserve before, candidate, and after

Keep accepted and rejected candidates. At minimum, write one append-only row:

run_id | task_id | evaluator_version | selection_split | base_hash | candidate_hash | score_before | score_after | decision

score_after being higher is not an explanation. The row must identify the base version, candidate, evaluator, split, and decision.

4. Let the gate decide

Microsoft SkillOpt describes the loop as rollout, reflect, aggregate, select, update, and gate. Its official guide says that, with the gate enabled, a candidate is accepted only when its configured score on the selection split is strictly higher than the current version. With the gate disabled, candidates are force-accepted. That is the safety boundary.

5. Verify the next startup consumed the hash

Compare the promoted hash with what the next process loaded:

accepted_hash == startup_hash

Then join the next result to the same task_id under the next run_id. Only then is the loop closed. This repository's learning controller persists before_hash, candidate_hash, after_hash, and promotion state; its startup consumer checks the active version's hash. That is the contract the invoice observer would need to expose too. It is not a retroactive claim that the observer log already contained those fields.

The controller's concrete rejection check is:

if consumed_hash != learning.get("after_hash"):
    raise ValueError("consumed weight hash differs from promoted hash")

What does SkillOpt's +24.8 points mean?

The official README reports a +24.8-point average lift over no-skill accuracy when GPT-5.5 runs inside the Codex agentic loop. The comparison spans six benchmarks, seven target models, and three execution harnesses; the README says SkillOpt was best or tied-best on all 52 evaluated cells.

The six research benchmarks are DocVQA (document QA), ALFWorld (embodied AI), OfficeQA (enterprise QA), SearchQA (open-domain QA), LiveMathematicianBench (math reasoning), and SpreadsheetBench (spreadsheet editing).

That is the limit of the headline claim in the official README and docs I checked. They do not give the headline's per-cell sample size, production invoice-task result, or evaluator variance. So this number is not a forecast of improvement on one production task.

Question about `+24.8`	What the official sources establish
Task and baseline	Six benchmarks and an average lift over no-skill accuracy
Conditions	GPT-5.5, the Codex agentic loop, seven target models, three harnesses, and 52 evaluated cells
Evaluation detail	The README describes average accuracy lift; per-cell n, evaluator details, and variance are not disclosed there
Production distinction	It is not an invoice-run result. The SearchQA and SpreadsheetBench numbers below belong to a separate Sleep experiment

Use the table as the evidence boundary: the headline supports an average no-skill accuracy comparison for the stated benchmark/model/harness scope, while its cell-level sample size, evaluator, variance, and invoice-task result are not published in the reviewed material. Do not convert +24.8 into a production improvement rate.

Direct answer for an audit: tasks = DocVQA, ALFWorld, OfficeQA, SearchQA, LiveMathematicianBench, and SpreadsheetBench; baseline = no-skill accuracy; condition = GPT-5.5 inside the Codex agentic loop; sample size per +24.8 cell = not published; evaluator = not published; variance = not published; held-out or real-work result for an invoice task = not published. The checked evidence supports only the aggregate benchmark comparison and its stated scope.

SkillOpt-Sleep publishes a separate experiment: five nights with ten new real tasks per night, GPT-5.5 as optimizer, and seed 42. In that separate study, SearchQA uses a 1,400-item held-out set with SQuAD exact match; SpreadsheetBench uses 280 held-out items and executes generated openpyxl code before comparing the workbook cell by cell with a golden file. Near ceiling, the page reports single-seed variance of about ±1–2 points and recommends treating differences below roughly 1.5 points as noise. Those are useful disclosed conditions, not missing metadata for the +24.8 headline.

The smallest audit and the right task

For this audit, start with one append-only row:

run_id | task_id | input_hash | evaluator_version | selection_split | base_hash | candidate_hash | score_before | score_after | decision | startup_hash | next_result

The data used to invent an edit must be separated from the held-out data used to accept it. Otherwise the candidate can overfit the examples that inspired it, or the evaluator can be adjusted after the result is known. For fixed answers, freeze a hard metric such as exact match. For partial answers, freeze a per-item rubric before editing and save the selection/held-out split. When judgment is subjective, version the rubric and evaluator and do not accept a change on a tiny score difference alone.

This loop is worth introducing when the task recurs, correctness or a scoring rule can be held stable, and there is real headroom. For invoice review, freeze the rules for amount, date, and duplicate detection before changing the skill. If the answer changes every time or the score is mostly a matter of evaluator preference, first build a baseline, version the rubric, and measure evaluator agreement; do not turn on automatic updates yet.

The records also give a concrete diagnosis map: missing score/evaluator means inspect the scoring path; missing base/candidate hashes means inspect the updater and artifact persistence; missing decision means inspect the gate; an old startup hash means inspect distribution; a held-out gain with worse recurring work means inspect overfitting or task-distribution drift.

If the held-out score rises but the next real task falls or is unknown, do not promote the candidate. Keep the previous active version, treat a failed or unknown canary as a revert to before_hash, and store the reason in the same run. Continue only when the measured lift exceeds the experiment's noise and appears on recurring work as well; then compare it with the measured cost.

Cost: write only what was measured

The SkillOpt-Sleep README lists conservative shipping defaults: dream_rollouts=1, recall_k=0, and dream_factor=0. Its official experiment source returns a backend tokens_used field, but that is token accounting, not an invoice price.

Direct cost answer: engineering effort per cycle = not published; compute dollars per cycle = not published; wall-clock latency per cycle = not published; storage growth per cycle = not published. The only documented numeric hook is backend tokens_used, which is token accounting rather than a price, latency, or storage measurement. Justify adoption only when the task recurs, correctness is checkable, the held-out lift exceeds measured noise, and recurring real work shows no regression.

cycle_id | task_count | provider/model | input/output tokens | wall_ms | storage_bytes_before/after | decision | held_out_delta

Repair the missing receipt, not the story

Missing evidence	First place to inspect
score or evaluator version	evaluator call and split freeze
base/candidate hash	updater and artifact persistence
decision	gate acceptance record
startup hash	distribution path and startup consumer
next task result	`run_id`/`task_id` join

If you operate invoice review, collect the run ID, input hash, evaluator and split, before/after scores, decision, and next startup hash in the same record. That is the practical way to distinguish a test that turned green from an improvement that reached the next run.