DEV Community: simon levy

Runbook Repair Lab: Fixing Fictional Runbooks with Next.js and Sanity

simon levy — Sat, 03 Oct 2026 14:33:29 +0000

This is a submission for the Sanity Challenge, Path Two: Vibe-Code Something Strange.

What I Built

The paper town starts with seven findings: missing prerequisites, incompatible tools, and downstream steps waiting for outputs that never appeared. A saved repair marks the prerequisites available and selects compatible releases. All three steps pass, and the original guide stays intact.

Runbook Repair Lab is a Next.js app for exploring those dependencies. Its three fictional projects are Build a paper town, Light a lantern garden, and Stitch a pocket atlas. Change their prerequisites, tool versions, or enabled steps to see what breaks and why. Every command is display text; nothing executes.

Sanity stores the guides and their relationships, plus repair proposals with explicit approval or rejection decisions. A custom Repair review Studio tool records those decisions. The public app can load a validated, approved repair into a visitor's local experiment.

OpenAI Codex designed and implemented the app. The checker is deterministic TypeScript, with no AI calls at runtime. This is a Fully Autonomous entry; the article was also AI-written from the source, build log, and verification evidence.

Demo

Open Runbook Repair Lab. No login required.

Try the persisted repair

Open Build a paper town and choose Reset experiment. You should see seven findings.
Choose Try owner-approved repair. It loads the saved Sanity proposal's conditions: Toy workspace and Paper-town blueprint available, Grove CLI v2, Loom Engine v2, and all three steps enabled. The result is three passing steps and zero findings.
Disable Prepare the town scene. The preview step loses its town-scene input, and the final step loses town-preview. Re-enable the step or reset to explore again.
Open Light a lantern garden. Its proposal was rejected, so it has no owner-approved repair action. Try compatible conditions still lets you explore a locally calculated repair.

How a proposal reaches that button

The custom Studio workflow is create a pending proposal → approve or reject → rebuild the public snapshot:

Propose. Studio calculates compatible conditions and stores a repairProposal linked to the guide. The document includes selected prerequisites, tool references and versions, enabled-step state, and a fingerprint of the complete modeled content.
Decide. Approval checks that the proposal still matches the guide and its referenced content, then verifies that every step passes. Source and proposal revision guards commit with the decision in one atomic transaction. Rejection records the rejected state without changing the guide.
Read back. Real Sanity records show paper town moving from pending to approved, and lantern garden moving from pending to rejected. The garden proposal had compatible conditions; its rejection demonstrates that compatibility alone does not put a repair in the approved set. The published evidence does not record a rejection rationale.
Build. The public build independently validates approved proposals. Pending, rejected, stale, and malformed proposals are excluded. The included paper-town repair becomes an optional button that only changes the visitor's local experiment.

Who performed the decisions: after I signed into Studio and authorized the work, the assistant created and decided both proposals using Windows UIAutomation confined to the authenticated Studio window. The UI label “owner-approved” refers to that approved record. This demonstration does not establish independent human review of the repair.

Workflow implementation and live verification · Exact proposal IDs and machine-readable evidence

These screenshots show the verified local export built from the actual reviewed Sanity records. The deployed app is also a static, build-time Sanity snapshot, with its snapshot time displayed in UTC. Sanity content edits and review decisions require rebuilding and redeploying before visitors see them.

Experiments are separate browser-local state. Each guide remembers its own conditions across refreshes; reset restores the published baseline. If storage is blocked, the app works for the visit and explains the limitation. Public visitors cannot approve proposals or write to Sanity.

Code

MIT-licensed GitHub repository · Build log · Verification record

The repository includes the Next.js/React/TypeScript frontend, separate Studio configuration, schemas, fictional seed content, deterministic checker, and browser tests. Codex is credited in the README.

The workflow is easiest to inspect in the Studio review tool, shared proposal validation and decision guards, and approved-repair build query.

My Build Process

The build used OpenAI Codex on October 3, 2026. This is an account of implementation decisions and corrections, rather than a reconstructed prompt transcript.

Make each finding traceable

The model uses ordered steps, declared inputs and outputs, and inclusive major-version ranges. An output becomes available only after an enabled step passes every check. Failed or skipped steps cannot supply outputs, and a later step cannot satisfy an earlier requirement.

For paper town, Prepare the town scene needs workspace and blueprint, plus Grove CLI v2–v3. It produces town-scene. The next step consumes that capability and produces town-preview; the final step consumes the preview. The seven initial findings follow from two missing starting capabilities, three version conflicts, and two unavailable downstream outputs.

Try compatible conditions supplies declared prerequisites, re-enables steps, and chooses published releases that satisfy every relevant interval. It cannot invent releases, resolve conflicting intervals, or create a capability with no producer. Those problems remain findings requiring an author correction.

Invalidate repairs when their dependencies change

Review found that checking only the guide revision missed edits to referenced tool and prerequisite documents. The correction was a fingerprint of the complete resolved content, with regression coverage. After a rebuild includes changed content, outdated saved experiments return to baseline. Invalid or corrupt stored experiments do too.

The persisted workflow uses that same modeled-content fingerprint. Approval also guards the current revisions of the proposal and its dependencies, so a concurrent edit cannot silently receive an outdated approval. The transaction's dependency writes update guard metadata, leaving guide instructions, defaults, and tool releases unchanged.

Two deliberately stale live transactions returned 409. Direct reads confirmed unchanged source and proposal revisions, including rollback when a later revision check failed. A valid anonymous same-value write attempt returned 403.

Studio uses its normal signed-in client; no server write key or anonymous mutation endpoint was added. Existing project permissions protect writes. Administrators and other members already granted write access can still use the API directly, so these client-side checks are not server-enforced, per-document owner-only permissions.

Make snapshot freshness explicit

The Next.js frontend exports statically. Review exposed a persistent fetch-cache risk, so each build now reads current published Sanity data through server-only native HTTPS, with an eight-second timeout and one-megabyte response limit. Invalid or unavailable data fails the build instead of silently substituting sample content.

This deployment needs no runtime token, AI key, SSR worker, or CORS change. Its deliberate tradeoff is a rebuild after content changes. Reduced cloud-build concurrency addressed resource usage without changing the checking rules.

Verify real data and browser behavior

Early checks used explicitly labeled synthetic fixtures. Final verification used the actual public dataset and its real document IDs. The completed workflow extension records:

20 passing deterministic unit tests, including version boundaries, skipped/failed producers, conflicting intervals, stale/corrupt storage, proposal validation, and revision guards
Passing frontend lint and TypeScript checks, Studio typechecking, and production builds
Baseline findings of 7, 7, and 6, reduced to 0, 0, and 0 with compatible conditions
Six passing real-data browser workflow groups in the original verification, covering repair, reset, reload, guide isolation, and blocked/corrupt storage; stricter approval assertions were added later
Responsive checks at 1400, 768, 390, and 320 pixels, zero browser page errors, and inspected desktop/mobile screenshots
Source and exported-browser-asset secret-pattern checks before publication

October 4 regression update: review found that the earlier six-group suite allowed a missing approved action and did not assert garden exclusion; a separate screenshot/check script had supplied the latter evidence. The strengthened suite now requires exactly one paper-town approved action that yields three passing steps and zero findings, with reset restoring findings, plus no approved action for the rejected lantern garden or proposal-free pocket atlas. A fresh local build and the existing hosted demo each passed seven groups with zero uncaught JavaScript page exceptions. Three deliberately wrong action-count expectations were correctly rejected on each target. All 20 unit tests, lint, a fresh Next.js build, and Studio typecheck passed. Production code and schemas were unchanged; no redeploy or new Studio decision was needed. Pinned regression evidence.

Mobile inspection led to larger touch targets, clearer explanations, and experiment controls appearing before steps on narrow screens.

The checker covers modeled capabilities and major-version intervals. It does not parse command syntax, infer semantic-version compatibility, or establish whether real-world instructions are safe or correct.

Sanity Project Details

Project: Runbook Repair Lab
Project ID: ipp6nys2
Dataset: production (public)
Verified content: 12 fictional documents: three tools, four prerequisites, three guides, and two reviewed repair proposals

The relationships carry the checking rules:

A tool lists published fictional major releases. Guide defaults and step requirements reference the same tool documents.
A prerequisite describes a starting capability. For example, the Toy workspace document supplies workspace; guides reference it rather than duplicating its definition.
A guide references prerequisites and tools, and contains ordered steps with needs, gives, inert command examples, and inclusive min/max tool versions.
A repairProposal references its guide and selected tools, stores prerequisite IDs and conditions, and carries a content fingerprint, status, and timestamps. It preserves the decision separately from the guide's published starting conditions.

A GROQ projection resolves the guide references into the checker's input and excludes draft guides. Build-time validation checks that response before export. Because a repair is tied to the resolved content, changing a referenced release list or prerequisite can invalidate it even when the guide document itself is unchanged.

This version uses Sanity schemas, references, GROQ, and a custom Studio tool. It does not use App SDK, native Sanity Workflows, Context MCP, or a runtime agent. The public frontend has no real-time content subscription.

Agent Session

No raw agent session is public. The sanitized build log, verification record, and source document implementation decisions, failures, corrections, and checks without publishing private conversations or credentials.

This revision incorporated review feedback from OpenAI Astra, Claude Opus 5.5 High, and Claude Sonnet 5.5 High. Implementation remains credited to OpenAI Codex. The Claude reviews did not provide live browser verification.

Rehearsal Mirror helps you find your own story before an interview

simon levy — Fri, 02 Oct 2026 21:43:58 +0000

Entry for Hacktoberfest Weekend Challenge: Build for a Friend.

What I Built

Someone close to me wants help organising real experiences into interview answers. I built Rehearsal Mirror around the step before writing an answer: finding an experience to draw on.

A question about handling a calendar conflict might connect to a small piece of work that is easy to overlook: checking availability, finding another slot and making sure everyone received the change. The experience belongs to the person practising. The app's job is to help them find it and put it into words themselves.

Rehearsal Mirror is a browser-local story bank and rehearsal space. You save experiences as Situation, Task, Action and Result (STAR) cards, choose a practice question, and ask a small local AI model to retrieve related cards. It shows an excerpt copied from your notes, with a link back to the full card. You write your answer and practise with a 60- or 90-second timer.

The pilot has 12 questions across personal-assistant, customer-support and administration roles. It has no answer generator, grammar grade or hiring score. Keeping the original experience visible is central to the design: a polished sentence would be unhelpful if it introduced something that never happened.

I have not yet handed it to its intended recipient. The results below come from technical checks with fictional data; I cannot yet say whether it helps them prepare for an interview.

Demo

Try Rehearsal Mirror.

The demo uses three fictional stories: a calendar collision, a delayed delivery and a duplicate invoice. They are labelled as examples throughout. No personal experience is needed to try the flow.

Select Load fictional demo. If you already have cards, export a backup first: loading the examples replaces those cards after confirmation.
Choose Personal assistant and the first question: “Two appointments overlap unexpectedly. How would you resolve the calendar conflict while keeping everyone informed?”
Select Download & load local AI. The first load needs internet and roughly 50 MB of model/runtime downloads, depending on compression. You can browse cards manually without loading the model.
Select Find relevant stories. In the recorded checks, Fictional · Calendar collision ranked first. Open Review full card to see the Situation, Task, Action and Result behind the excerpt.
Write a short practice answer in your own words. Choose 60 or 90 seconds, then start, pause, resume or restart the timer as needed.

Desktop capture from the local production-build test using the actual MiniLM model on October 3. The answer and all three stories are fictional. This is a local-build screenshot, not a hosted-site verification.

The calendar card's action is concrete: check attendees' availability, offer another time, update the calendar and send the revised details. Those are the notes the person can draw on. The model does not turn them into a script or add a more impressive result.

For two more checks, try the upset-customer question and the administration question about correcting a record. The delayed-delivery and duplicate-invoice cards respectively ranked first in the original verification run. These are deliberately distinct examples, so they demonstrate the flow rather than establish retrieval accuracy.

Code

Source code and local setup on GitHub.

The original application code is MIT-licensed. Model, library and runtime terms are recorded separately in the dependency notices. The repository includes source, tests, model provenance and licences; generated build output is excluded.

With Node.js 22.12+ or 24 and Python 3 installed:

npm ci
npm run build
npm test
python tools/serve.py

Then open http://127.0.0.1:4173 and leave the server running. For a browser-only Linux installation, the README explains how to skip the unused native CUDA download.

The built app serves static files. There is no application backend, account system or upload endpoint.

How I Built It

Keep the source experience visible

The retrieval model is the Xenova ONNX conversion of all-MiniLM-L6-v2, running through Transformers.js 3.8.1. The model is pinned to revision 751bff37182d3f1213fa05d7196b954e230abad9. Both the conversion and the upstream Sentence Transformers model declare Apache-2.0 licensing.

The quantized model runs in a dedicated browser worker through WebAssembly. Matching has two stages:

Embed the selected question and each card as normalized, 384-dimensional vectors. Rank the cards by cosine similarity.
For each of the top three cards, compare the question with its individual STAR fields. Display the closest field exactly as written.

That second step makes the result inspectable. Someone can read the excerpt, open the whole card and decide whether it belongs in their answer.

There is an important limit: the closest card can still be a poor fit. With a nonempty story bank, a ranking can return something even when none of the experiences answers the question. Relatedness is only a prompt to review the notes. It cannot establish that an experience happened or that an answer is strong.

The bug appeared when rehearsal began

The original October 2 verification recorded four passing unit checks and 18 passing browser checks. A later browser review found a problem that those checks had missed. The app could load the model and find the calendar story, but typing the first character of a practice answer made the matches disappear.

The cause was in the UI state handling. The answer field called the shared save function on every keystroke. That function also cleared retrieval results and invalidated in-flight requests. Clearing was useful when the underlying story changed; doing it while writing an answer removed the reference at the moment it was needed.

The follow-up source revision separates saving practice text from invalidating story matches. In the recorded local checks, typing an answer and changing the timer length preserve the retrieved cards. Changing the question, role or underlying stories clears matches that may be stale. Request and worker checks also prevent late replies from restoring an old result.

This revision was deployed to the linked demo on October 4. Deployment and verification are separate: the screenshots and actual-model test results here come from the local production builds described below.

The revision puts a fictional-demo shortcut in the empty matching area and explains that the closest card may still be unsuitable. A final recovery change restores manual mode and a usable download control even if clearing the model cache fails.

Local production-build capture from the same actual-model run, using a 390-pixel-wide mobile viewport. The answer and retrieved cards remain available in the stacked layout. This predates the final cache-recovery change.

The useful test is the whole sequence: find a story, begin an answer, and keep using the story. A correct ranking alone does not show that the rehearsal flow works.

Check the privacy claim at the network boundary

Practice text is saved in this browser's local storage. Embeddings stay in worker memory. Model downloads contact Hugging Face and its file hosts, which receive ordinary connection metadata; stories and answers are not sent to a remote inference service.

The follow-up actual-model run on October 3 passed 20 checks in Chrome on Windows, including the three fictional ranking cases and answer/timer preservation. Its network checks covered two specific boundaries:

Text transmission: distinct synthetic markers were entered into a story and an answer. Neither appeared in captured page or worker request URLs or bodies during the recorded flows.
Loaded-model offline inference: a synthetic story was edited and matched with browser networking disabled after model loading. Matching succeeded without additional requests.

The offline check covers inference in an already loaded app. Startup after closing or reloading the page can still need internet.

After the final cache-recovery change, the production build, four unit checks and nine controlled browser regression groups passed. Those browser tests control worker replies to exercise preservation, invalidation, late replies, retry and mobile layout; they do not run inference. The 20-check actual-model suite was not rerun after that final change. The dated verification report records the commits and scope of both runs. Later local and hosted controlled-worker runs each passed 9/9 groups with zero page errors and no model inference or downloads; the hosted run recorded two blocked same-origin Cloudflare POST attempts.

There are other limits worth keeping visible. Browser storage and exported JSON backups are unencrypted, and clearing browser data can remove notes. Long text is truncated by the model, so concise cards work best. The three ranking examples are smoke tests. Other browsers, low-memory phones and full screen-reader accessibility have not been independently verified.

The report also records unresolved dependency advisories and the affected native-codec and development-server surfaces. This UI fix does not resolve those advisories.

Why Does Open Innovation Matter?

The open model makes the automatic matching possible. Without it, this build supports manual card browsing but cannot retrieve stories by meaning.

Running that model in the browser also fits the task. Personal practice notes can be compared with interview questions without sending those notes to a hosted inference endpoint. There is no per-query API charge, account or API key. The trade-off is an initial download and the memory and processing work on the person's device.

The pinned weights, runtime and application source can be inspected separately. Someone adapting the app can change the question set or try another embedding model, then rerun the tests. That openness is useful precisely because this small model has limits: subtle experiences may be hard to distinguish, and a nearest match may still be irrelevant.

The next useful test is with the intended recipient: can they add one truthful experience, find it for a relevant question and use it during a short rehearsal? That has not happened yet. It is the question this pilot is meant to answer.

AI assistance

Built using OpenAI Codex. OpenAI Codex and OpenAI writing tools provided substantial assistance with implementation, automated verification and this article. Claude Opus 5.5 High, Claude Sonnet 5.5 High and OpenAI Astra provided review feedback for this revision. Browser-verification claims come from the recorded browser checks described above; the Claude reviews did not run those checks. Technical claims are grounded in source code and recorded synthetic tests. No recipient testimonial or interview outcome has been generated or inferred.

A CSV benchmark with two zero scores and a useful diagnostic

simon levy — Fri, 02 Oct 2026 20:45:27 +0000

This is a submission for the Kaggle Benchmarking Challenge.

What I Benchmarked

Inventory reconciliation sounds straightforward: match a count to an exported product row and replace its new quantity. The difficult part is preserving everything else and recognizing when an update should be blocked.

CSV Contract Integrity tests this with synthetic, offline inputs. Each model receives an export CSV, a count CSV, an explicit column mapping, and a reconciliation contract. It must return one JSON object containing a candidate CSV, row-level issues, and any fatal file error.

Small differences matter. The SKU 0005 is a string identifier. Locations North and north are distinct. A blank new-quantity cell differs from 0. Quoted fields can contain commas or newlines. Unknown columns must survive unchanged. Duplicate keys, conflicting values, and incomplete product information can block an update.

The scorer allows harmless representation differences: reordered headers, CSV quote style, LF or CRLF line endings, and equivalent integer spellings in the editable quantity field. Every protected cell still requires exact string preservation. Candidate row order matters too.

The source contains 72 fixtures: 48 original cases and 24 compositions. Twelve original cases are reserved for development, leaving 36 original evaluation cases plus 24 compositions in each reported run. Those compositions come from six recipes with four related transformations each. The suite is small and deliberately inspectable, with correlated cases rather than 60 independent samples of inventory work.

Models Tested

The current comparison uses Gemini 3.7 Flash (google/gemini-3.7-flash) and Claude Haiku 4.5 (anthropic/claude-haiku-4-5@20251001) through Kaggle Model Proxy. Comparing two providers gives this narrow task more than one model response pattern to inspect.

Both completed the same 60 selected cases under the frozen contract and scorer. Each case received a fresh chat. The task requested temperature 0, seed 0, no tools, and a maximum of 8,000 output tokens per response. These were text-generation calls, without an enforced JSON output schema or generated-code execution.

An inspection of the installed SDK confirmed that its string-response path adds no formatting instruction and returns model text unchanged. The reported results therefore include the behavior of the configured model and provider interface, not an SDK operation that adds or removes code fences.

Findings

Both models returned 60 nonempty responses. Both scored 0/60 on the strict contract.

The primary score requires the raw response to satisfy the JSON interface and then match the allowed CSV semantics and issue report. A response wrapped in Markdown fails the raw JSON requirement even when its contents look correct to a reader.

A separate post hoc diagnostic removed exactly one complete outer json code fence. It accepted no surrounding prose, repaired no incomplete JSON, and made no other correction. It then applied the same scorer:

Gemini 3.7 Flash: 59/60, comprising 35/36 original cases and 24/24 compositions
Claude Haiku 4.5: 14/60, comprising 14/36 original cases and 0/24 compositions

Post hoc saved-response analysis, October 4, 2026. Following Arhan Canli’s request for a failure breakdown, I reanalyzed the saved responses from those same runs. No model was rerun, and the benchmark, adapter, and scorer were unchanged.

The diagnostic outcomes split into:

Outcome	Gemini	Haiku
Passed after one-fence removal	59	14
Parsed as JSON, then failed the contract	0	38
Eligible for the adapter, but JSON did not parse	0	0
Ineligible for the adapter	1	8
Total	60	60

All 52 adapter-eligible Haiku responses parsed as JSON. Its remaining 38 failures therefore concern the returned content, rather than another JSON parsing failure. The eight ineligible responses contained trailing explanation or multiple blocks; they stay in the denominator, and I do not extract a preferred block from them.

These numbers answer two questions. The strict result measures whether the specified interface can consume the response directly and obtain a correct result. The diagnostic measures correctness after one narrowly defined formatting adjustment.

Reporting only the zero scores would conceal the content differences that the diagnostic reveals. Presenting 59/60 or 14/60 as the official score would conceal the interface failures. Both views belong in the report.

Gemini’s remaining case, preservation-05, contains an unusually long identifier. Its response had incomplete JSON, an unclosed fence, and 448 visible characters, so it was ineligible for the adapter. Usage reported 7,996 output tokens against the requested 8,000 cap. That token total is not a visible-text length. The cause is unestablished; a larger budget is a follow-up experiment, not a demonstrated fix.

A warning that did not block the update

One small case makes the content gap concrete. In current_safety-05, export record 2 already has On hand (new) = 7; its matching count requests 4. The contract says any nonempty existing new-quantity cell blocks that row.

Expected: a header-only candidate CSV and NEW_VALUE_CONFLICT for export record 2.
Haiku after fence removal: the same, correct issue report, plus the blocked row with its new quantity changed from 7 to 4.
Gemini after fence removal: the expected header-only candidate and issue report.

Recognizing the conflict was insufficient: the candidate also had to obey it. This is why the scorer checks both the issue report and the emitted rows.

That pattern recurred. Among Haiku’s 38 parsed-but-wrong responses, 31 emitted at least one source row the contract prohibited; 33 had an incorrect issue report; 12 omitted at least one eligible row; and 7 changed at least one protected string value. These are overlapping descriptive flags, not a partition or a count of independent causes. A response can both emit a blocked row and omit an eligible one.

Two other cases show what those flags can hide:

Exact strings: in preservation-04, Haiku returned the right new quantity but changed the SKU from e\u0301 (U+0065 followed by U+0301) to \u00e9 (U+00E9). The glyphs can look alike; the identifier strings differ. Gemini preserved the original string.
Validation across locations: in composition-v3-04-01, one variant has packed dimensions at location L and blank dimensions at Other, which has no matching count. The contract still requires consistency across both locations and blocks the whole variant. Haiku reported PACKED_INCONSISTENT at Other but emitted the row at L; Gemini excluded both and updated only the unrelated eligible variant.

These examples describe the saved outputs. They do not establish what internal reasoning produced them.

Where the failures occurred

For the 36 original evaluation cases, each source-defined family contains six cases. These are case-family results under the one-fence diagnostic, not independent pass rates for every contract clause:

Original case family	Gemini passes	Haiku passes
Exact joins	6/6	1/6
Protected-cell preservation	5/6	4/6
Current quantity and update safety	6/6	1/6
Ambiguity and unmatched records	6/6	0/6
Quantity semantics	6/6	4/6
CSV format	6/6	4/6
Total	35/36	14/36

The 24 compositions combine rules in six recipes, with four related transformations per recipe:

Recipe and main interaction	Gemini passes	Haiku passes
1: exact keys, invalid counts, unmatched rows	4/4	0/4
2: duplicate export and count keys	4/4	0/4
3: existing values, blank quantities, zero	4/4	0/4
4: packed dimensions across locations	4/4	0/4
5: product-wide option completeness	4/4	0/4
6: duplicate variant/location and not-stocked rows	4/4	0/4

Recipe 3 has one adapter-ineligible Haiku response; all other composition failures parsed as JSON. Every one of those 23 parseable composition responses emitted a row the contract should have blocked. The pattern spans all six recipes, but the four variants within each recipe share ingredients. These results do not identify 24 independent failures or six independent causes.

The comparison is descriptive. It does not establish a general ranking of these model families. A different prompt, structured-output interface, token budget, provider configuration, or repeat run could produce different results.

What the benchmark cannot establish

This is conservative offline validation. No Shopify import or real inventory change was performed. A failed contract verdict does not demonstrate inventory damage.

The emitted-row mutation diagnostic also has a narrower meaning: it counts unexpected emitted rows relative to the expected candidate. Omitted rows fail overall correctness but are not counted as emitted-row mutations. For both models, all 60 primary responses failed JSON parsing and mutation remained unassessed. Unknown must never become “zero damage.”

Coverage is incomplete. Several error branches in the written contract have no evaluation fixture in this version. Expected outputs and grading code also need scrutiny: passing expected answers through the scorer checks consistency, not the independent correctness of every label.

What to measure next

A useful next experiment would predeclare separate interface conditions: the existing raw-text prompt, a one-fence adapter, and supported structured-output modes. Their scores should remain separate rather than retroactively changing this benchmark's primary metric.

Further work should vary the output budget for long preservation cases, add independently authored cases, fill the coverage gaps, and repeat runs. The current results are one completed run per model, without a formal statistical generalization claim.

My Benchmark

CSV Contract Integrity on Kaggle contains the CSV Inventory Contract Integrity task. The collection and task are now published publicly on Kaggle.

The benchmark's primary result is the mean of 60 strict binary verdicts. Source materials retain the contract, fixtures, expected outputs, scorer, request hashes, raw responses, usage, and separate diagnostics. These make the evaluation inspectable; requested seeds do not guarantee identical future provider output. There is no hidden-test claim or production-safety guarantee.

To inspect the examples above, open the saved Kaggle outputs for Gemini, script version 354731097 or Haiku, script version 354732397. Each includes primary-report.json with case IDs and raw responses, plus posthoc-diagnostics.json. The offline reanalysis verified all 120 saved request hashes and reproduced all saved primary and one-fence verdicts. This checks replay consistency, not independent correctness of the expected answers.

The pinned offline reproducibility supplement contains the saved evidence, analysis code, and tests. With Python 3.8 or later, run python3 -B analyze.py from the downloaded repository folder to verify hashes, replay the saved verdicts, and regenerate the breakdowns. Once downloaded, it needs no network access, provider credentials, or model inference. Replay checks consistency with the frozen scorer; it does not independently validate the expected answers.

AI disclosure: Fully Autonomous. AI assistance was used to create the prototype, synthetic fixtures, grading code, tests, documentation, and this article. The article was drafted primarily by an AI agent. This revision used Codex, with reviews from Astra, Claude Opus 5.5 High, and Claude Sonnet 5.5 High. The grader itself does not use an LLM judge.