DEV Community

Dakota Liu
Dakota Liu

Posted on

Require a Job Receipt. Apply Nothing the Schema Cannot Parse.

#ai #tutorial #python #git

A free remote model can draft a patch. It cannot be the process that mutates my tree. I treat every remote run as a proposal. The proposal is a job receipt. If the receipt fails the schema, the job is over.

No debate. No "just apply it this once."

Unconstrained generation is easy to confuse with engineering. This week's feed is full of that argument. Models write code. That part is boring. The part that still fails is inspection. I want a gate I can run in CI. A receipt is that gate.

The problem I keep hitting

I can keep the model off my shell. Then what? Someone still accepts a diff. If I accept it by eyeballing a chat window, I am the weakest parser in the loop. Can you grep a chat window in a pull request? I cannot.

So I stopped asking the agent for "the fix." I ask it for a receipt. The receipt is JSON. JSON I can validate, store, and reject in one function.

This is a proposed workflow. I am not publishing a benchmark. I am not claiming a quota, a GPU size, or a durability promise. Treat it as a contract you can run locally.

What you will build

Five pieces. That is the whole artifact.

  1. job_envelope.json — what the remote job may see and return.
  2. receipt.schema.json — the only shape of a passing proposal.
  3. fixtures/receipt.valid.json and fixtures/receipt.invalid.json — two frozen examples.
  4. verify_receipt.py — the local verifier.
  5. Makefile — one target per stage so each gate can fail on purpose.

Each stage has a verification step. If a stage does not fail when it should, stop. The rest is theater.

Stage 1 — Freeze the job envelope

Do not ship a working tree. Ship a contract. The envelope names the task, the allowlist, and the exact test command you will rerun at home. Who is the source of truth, the chat or this file? This file.

{
  "job_id": "2026-09-18-receipt-001",
  "task": "Make format_rate return 0 when the input is empty.",
  "language": "python",
  "allowlist": ["src/rates.py", "tests/test_rates.py"],
  "test_command": "python -m pytest tests/test_rates.py -q",
  "constraints": {
    "must_not_touch": [".env", "secrets/", "infra/"],
    "patch_format": "unified_diff",
    "result_must_be": "proposal"
  }
}
Enter fullscreen mode Exit fullscreen mode

Give the envelope a tiny subject so the later receipt is not abstract. Example code, not a war story:

# src/rates.py
def format_rate(value: str) -> int:
    if value is None:
        raise ValueError("missing")
    return int(value)
Enter fullscreen mode Exit fullscreen mode 
# tests/test_rates.py
from src.rates import format_rate

def test_empty_string_is_zero():
    assert format_rate("") == 0
Enter fullscreen mode Exit fullscreen mode

Verification for this stage is dull on purpose:

mkdir -p src tests fixtures
python -c "import json; json.load(open('job_envelope.json')); print('envelope: ok')"
test "$(python -c "import json; print(len(json.load(open('job_envelope.json'))['allowlist']))")" -ge 1
Enter fullscreen mode Exit fullscreen mode

Did that print envelope: ok? Good. If allowlist is empty, you just authorized the void. Fix it before you talk to any model.

Stage 2 — Write the receipt schema

The schema is the product. Everything else is glue. I reject extra fields. I reject a missing patch. I reject a status that is not proposal.

Why so rude? Because agents love to add "notes," "confidence," and a little surprise file. Surprise files are how .env dies.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "AgentJobReceipt",
  "type": "object",
  "additionalProperties": false,
  "required": ["job_id", "status", "files", "test_command", "patch", "notes"],
  "properties": {
    "job_id": { "type": "string", "minLength": 8 },
    "status": { "const": "proposal" },
    "files": {
      "type": "array",
      "minItems": 1,
      "maxItems": 16,
      "items": { "type": "string", "pattern": "^[a-zA-Z0-9_./-]+$" }
    },
    "test_command": { "type": "string", "minLength": 8 },
    "patch": { "type": "string", "minLength": 20 },
    "notes": { "type": "string", "maxLength": 500 }
  }
}
Enter fullscreen mode Exit fullscreen mode

Verification:

python -c "import json; json.load(open('receipt.schema.json')); print('schema: ok')"
Enter fullscreen mode Exit fullscreen mode

If that fails, your schema is not JSON. Do not debug the agent yet. Debug yourself.

Stage 3 — Two fixtures, one of them poisonous

I always keep a valid receipt and an invalid one in git. The invalid one is the unit test for my spine. What does "the model said it compiled" even mean if I cannot fail a fixture?

fixtures/receipt.valid.json (truncated patch is enough for the parser):

{
  "job_id": "2026-09-18-receipt-001",
  "status": "proposal",
  "files": ["src/rates.py"],
  "test_command": "python -m pytest tests/test_rates.py -q",
  "patch": "diff --git a/src/rates.py b/src/rates.py\n--- a/src/rates.py\n+++ b/src/rates.py\n@@ -1,4 +1,6 @@\n def format_rate(value: str) -> int:\n+    if value == \"\":\n+        return 0\n     if value is None:\n         raise ValueError(\"missing\")\n     return int(value)\n",
  "notes": "Empty string returns 0. None still raises."
}
Enter fullscreen mode Exit fullscreen mode

fixtures/receipt.invalid.json should fail for three reasons at once. Status is applied. Files include .env. Test command is a shrug.

{
  "job_id": "wrong-id",
  "status": "applied",
  "files": ["src/rates.py", ".env"],
  "test_command": "echo done",
  "patch": "not a diff",
  "notes": "looks good to me"
}
Enter fullscreen mode Exit fullscreen mode

Verification for this stage is only "these files parse":

python -c "import json; json.load(open('fixtures/receipt.valid.json')); json.load(open('fixtures/receipt.invalid.json')); print('fixtures: ok')"
Enter fullscreen mode Exit fullscreen mode

You are not proving the patch is correct yet. You are proving you can store a lie and a candidate in the same folder.

Stage 4 — The verifier

Install the one library the example script needs:

python -m pip install jsonschema
python -c "import jsonschema; print('jsonschema: ok')"
Enter fullscreen mode Exit fullscreen mode

The verifier does four checks, in order. Schema. Envelope match. Allowlist inclusion. Patch path extraction. If any check fails, exit 2. I reserve exit 1 for "you called me wrong."

Example script. Proposed workflow, not a vendor SDK:

#!/usr/bin/env python3
from __future__ import annotations

import json
import re
import sys
from pathlib import Path

import jsonschema

DIFF_PATH = re.compile(r"^\+\+\+ b/(.+)$", re.M)


def load(path: Path):
    with path.open(encoding="utf-8") as fh:
        return json.load(fh)


def main(argv: list[str]) -> int:
    if len(argv) != 4:
        sys.stderr.write("usage: verify_receipt.py ENVELOPE RECEIPT SCHEMA\n")
        return 1

    envelope = load(Path(argv[1]))
    receipt = load(Path(argv[2]))
    schema = load(Path(argv[3]))
    jsonschema.validate(instance=receipt, schema=schema)

    errors: list[str] = []
    if receipt["job_id"] != envelope["job_id"]:
        errors.append("job_id mismatch")
    if receipt["test_command"] != envelope["test_command"]:
        errors.append("test_command mismatch")

    allow = set(envelope["allowlist"])
    files = set(receipt["files"])
    if not files.issubset(allow):
        errors.append(f"files outside allowlist: {sorted(files - allow)}")

    paths = set(DIFF_PATH.findall(receipt["patch"]))
    if not paths:
        errors.append("patch has no +++ b/ paths")
    if not paths.issubset(allow):
        errors.append(f"patch paths outside allowlist: {sorted(paths - allow)}")
    if paths != files:
        errors.append(
            f"files field != patch paths: {sorted(files)} vs {sorted(paths)}"
        )

    if errors:
        sys.stderr.write("\n".join(errors) + "\n")
        return 2

    print("receipt: ok")
    return 0


if __name__ == "__main__":
    raise SystemExit(main(sys.argv))
Enter fullscreen mode Exit fullscreen mode

Verification — both directions, or the script is a decoration:

chmod +x verify_receipt.py
python verify_receipt.py job_envelope.json fixtures/receipt.valid.json receipt.schema.json
echo "valid exit: $?"
python verify_receipt.py job_envelope.json fixtures/receipt.invalid.json receipt.schema.json
echo "invalid exit: $?"
Enter fullscreen mode Exit fullscreen mode

You want 0 then 2 (or a jsonschema.ValidationError before your custom checks). If the poisonous fixture also prints receipt: ok, throw the script out. A verifier that cannot fail is a blog post.

Stage 5 — Point a free remote lane at the same contract

Once the verifier is mean enough, I need something to fill the receipt. I do not want that something on my laptop shell. Why would I let a remote box invent the test command? I would not. The envelope already wrote it down.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source coding-agent project. The operator-supplied bit I am using here is simple: free model access, plus a free server option, so the generation step can live off my working tree. That is the only product claim I am making. No model names. No token counters. No hardware story.

The remote lane's job is to return receipt.json. My job is to reject it. I prompt for this JSON shape, or I wrap whatever text comes back into the shape. The verifier does not care who typed the file. It cares whether the file lies.

A minimal glue step, labeled as a proposal, not a recorded integration:

# proposed: send job_envelope.json, write the model's JSON to receipt.json
test -s receipt.json
python verify_receipt.py job_envelope.json receipt.json receipt.schema.json
Enter fullscreen mode Exit fullscreen mode

If you cannot get a receipt file, you do not have an integration. You have a chat log.

Stage 6 — Local tests after the receipt passes

A valid receipt is not a green build. It is a ticket to run tests you already own.

python verify_receipt.py job_envelope.json receipt.json receipt.schema.json
# only then, and only locally:
python -m pytest tests/test_rates.py -q
Enter fullscreen mode Exit fullscreen mode

If tests fail, the receipt still "passed." That is correct. The schema never promised the patch was right. It promised the patch was inspectable. Want a second check? Extract the patch and refuse unknown paths before pytest:

python - <<'PY'
from pathlib import Path
import re, json, sys
patch = json.loads(Path("receipt.json").read_text())["patch"]
paths = re.findall(r"^\+\+\+ b/(.+)$", patch, re.M)
print("\n".join(paths))
PY
Enter fullscreen mode Exit fullscreen mode

Verification: pytest is red on the original format_rate(""), and only runs after receipt: ok. If you run tests first, you skipped the point of the receipt.

Stage 7 — Promote, or throw it away

I keep the receipt next to the patch in the branch. Future me can see what the agent claimed. Future CI can replay verify_receipt.py on that file. Chat scrollback is not an audit trail. JSON in git is.

Schema Envelope match Local tests Action
fail n/a n/a reject; do not open the diff
pass fail n/a reject; do not run tests
pass pass fail keep the receipt, drop the patch
pass pass pass promote with the receipt attached

A tiny Makefile so each stage has a name:

.PHONY: envelope schema fixtures verify-valid verify-invalid

envelope:
    python -c "import json; json.load(open('job_envelope.json')); print('envelope: ok')"

schema:
    python -c "import json; json.load(open('receipt.schema.json')); print('schema: ok')"

fixtures:
    python -c "import json; json.load(open('fixtures/receipt.valid.json')); json.load(open('fixtures/receipt.invalid.json')); print('fixtures: ok')"

verify-valid:
    python verify_receipt.py job_envelope.json fixtures/receipt.valid.json receipt.schema.json

verify-invalid:
    python verify_receipt.py job_envelope.json fixtures/receipt.invalid.json receipt.schema.json; test $$? -ne 0
Enter fullscreen mode Exit fullscreen mode

Run make envelope schema fixtures verify-valid verify-invalid in that order. If verify-invalid succeeds, your Makefile is lying. Fix the test $$? -ne 0 gate.

Limitations

This schema does not prove correctness. It proves shape. A polished wrong patch still validates.

The +++ b/ regex is naive. File deletes, renames, and diff --git copies without that hunk will slip or false-fail. If you need those operations, extend the parser. Do not pretend the current one is a patch library.

additionalProperties: false will break the first time you add a field. Version the schema. Keep old receipts readable.

A free remote lane is not a trust boundary. Network still moves your envelope. Do not put secrets in task. Do not put .env in allowlist. The receipt cannot save you from a prompt you should never have sent.

I am not claiming the free model or free server lane is always available, fast, or equivalent to a paid cluster. Availability is an operational fact you check the day you run the job.

Who should not use this

Skip this if you need a vendor SLA. A receipt workflow does not create one.

Skip this if you paste production credentials into prompts. The schema will not notice.

Skip this if you want the agent to run migrations, publish packages, or SSH anywhere. A proposal is not a deploy.

Skip this if you will "approve in the UI" and never run verify_receipt.py. Then you built a JSON souvenir.

If you already keep a verifier this strict, filling the receipt from a free remote lane instead of from a laptop shell is a reasonable next experiment. Keep the same make verify-valid. The lane is optional. The receipt is not.

Top comments (0)