DEV Community: Dakota Liu

Case Study: Freeze Null-Versus-Omitted Rules Before an Agent Writes PATCH

Dakota Liu — Mon, 21 Sep 2026 13:35:16 +0000

You should freeze null-versus-omitted PATCH semantics before any agent writes the handler, because those two JSON shapes are not the same operation. Missing fields mean leave the stored value unchanged, while explicit null may clear, reject, or reset, depending on the field. Agents collapse those cases into one assignment path unless you lock a decision table and failing tests first. This case study walks a small settings PATCH from frozen contract through tests, generation, and the mistakes that still appear.

Background

You already have a GET /settings payload that clients cache locally and then edit inside a form. The form submits PATCH /settings with only dirty fields, which is the usual SPA pattern rather than a full replace. That request body is not automatically RFC 7396 JSON Merge Patch, even when it looks like a partial JSON object. If an agent writes the handler from a vague prompt, it often treats {"theme": null} and {} as the same update.

You need one boring endpoint that still encodes three different field policies inside a single JSON object. Clients will omit clean fields, send null to clear or reset, and expect required strings to reject null instead of disappearing. That mix is normal product behavior, and it is also a generator trap. The rest of this walkthrough treats the trap as the project, not as a style note in a prompt.

Goal

You will freeze a four-field settings document, then refuse to merge code until the tests describe every cell. The stored document holds display_name, email_opt_in, theme, and bio, and each field carries a different policy for JSON null. Success means a generator can implement the handler, but it cannot redefine what omitted keys and JSON null mean. You are not scoring a model in this case study; you are checking whether the freeze survives generation.

Field policies you freeze before anyone writes Python:

display_name — string; omit leaves the value; null is 422; empty string is 422.
email_opt_in — boolean; omit leaves the value; null is 422; true/false replace.
theme — enum light|dark|system; omit leaves the value; null resets to system.
bio — string or null; omit leaves the value; null clears; empty string is 422.

Freeze the decision table first

You write the table in a file the agent may read but must not edit during generation. That file is the product behavior, not a comment buried inside the prompt or a chat message. Reviewers should reject pull requests that change the table and the handler together without a separate spec commit. The table below is the original artifact for this walkthrough and the source of truth for later tests.

Field	Omitted key	JSON `null`	Empty string	Valid value
`display_name`	keep	422 `null_not_allowed`	422 `empty_not_allowed`	replace
`email_opt_in`	keep	422 `null_not_allowed`	422 `type_mismatch`	replace
`theme`	keep	reset to `system`	422 `invalid_enum`	replace if enum
`bio`	keep	clear to `null`	422 `empty_not_allowed`	replace

Shared HTTP rules you also freeze before generation:

Unknown keys return 422 unknown_field, and you do not silently drop them.
Invalid JSON, or a non-object body, returns 400 malformed_json.
Concurrent updates use If-Match against an opaque version, or you return 412.
A successful PATCH returns 204, and the next GET must show the merged document.

Store that freeze in version control as a YAML fixture the tests load, not as folklore in a standup note.

# contracts/settings_patch.yaml
resource: settings
method: PATCH
omitted: keep
unknown_keys: reject
empty_string: reject
fields:
  display_name:
    type: string
    null: reject
  email_opt_in:
    type: boolean
    null: reject
  theme:
    type: enum
    values: [light, dark, system]
    null: reset
    reset_value: system
  bio:
    type: string
    null: clear

Turn the table into failing tests

You add tests that send both {} and {"theme": null} against the same stored row. If those two requests produce the same database write, the freeze is not actually enforced. You also assert error codes, not only status integers, because agents like to reuse one invalid string. Keep the store fake; the lesson is the merge policy, not Postgres.

# tests/test_patch_settings.py
import json
import pytest

from settings_app import app, reset_store, seed_settings

@pytest.fixture(autouse=True)
def _fresh_row():
    reset_store()
    seed_settings(
        {
            "display_name": "Ada",
            "email_opt_in": False,
            "theme": "dark",
            "bio": "writes API notes",
            "version": "v1",
        }
    )
    yield

def patch(body, *, etag="v1"):
    return app.handle(
        "PATCH",
        "/settings",
        headers={"If-Match": etag, "Content-Type": "application/json"},
        body=json.dumps(body),
    )

def test_empty_object_is_a_no_op():
    response = patch({})
    assert response.status == 204
    document = app.handle("GET", "/settings").json()
    assert document["theme"] == "dark"
    assert document["bio"] == "writes API notes"
    assert document["version"] != "v1"

def test_theme_null_resets_but_does_not_touch_bio():
    response = patch({"theme": None})
    assert response.status == 204
    document = app.handle("GET", "/settings").json()
    assert document["theme"] == "system"
    assert document["bio"] == "writes API notes"

def test_bio_null_clears_but_does_not_reset_theme():
    response = patch({"bio": None})
    assert response.status == 204
    document = app.handle("GET", "/settings").json()
    assert document["bio"] is None
    assert document["theme"] == "dark"

def test_display_name_null_is_rejected():
    response = patch({"display_name": None})
    assert response.status == 422
    assert response.json()["code"] == "null_not_allowed"
    document = app.handle("GET", "/settings").json()
    assert document["display_name"] == "Ada"
    assert document["version"] == "v1"

def test_unknown_key_is_rejected_without_partial_write():
    response = patch({"theme": "light", "timezone": "UTC"})
    assert response.status == 422
    assert response.json()["code"] == "unknown_field"
    document = app.handle("GET", "/settings").json()
    assert document["theme"] == "dark"

Run the suite before any generated handler exists so the failures are the specification:

python -m pytest tests/test_patch_settings.py -q
# expected: missing module, or every assertion fails closed

You should also freeze a negative parser test, because many stacks coerce null before your handler runs. If Pydantic or a similar schema uses optional fields with defaults, omitted keys and null collapse in the type layer. The contract is then untestable no matter how careful the prompt looks.

def test_parser_must_preserve_omitted_versus_null():
    omitted = app.parse_patch_body("{}")
    explicit_null = app.parse_patch_body('{"bio": null}')
    assert "bio" not in omitted.explicit_keys
    assert "bio" in explicit_null.explicit_keys
    assert explicit_null.values["bio"] is None

Generate the handler against the freeze

You now let a coding agent implement settings_app.py while the YAML file and tests stay read-only in the prompt. Disclosure: This article was prepared as part of MonkeyCode's product outreach. A constrained loop that uses MonkeyCode's free model access and free server option is enough for this lab, because the artifact that matters is the freeze, not a paid model name. You still review the diff as if a junior teammate wrote it, and you reject edits to contracts/ or tests/ unless you intended a spec change.

Proposed generation brief, labeled as an unexecuted lab prompt rather than a production run log:

Implement settings_app.py so tests/test_patch_settings.py passes.
Do not edit contracts/ or tests/.
Parse JSON so omitted keys and nulls remain distinguishable.
Unknown keys and null-not-allowed fields must not write the store.
PATCH success returns 204 and bumps version.

Commands you run locally after the agent returns a patch:

git diff --stat
git diff -- contracts tests
python -m pytest tests/test_patch_settings.py -q

If git diff shows test edits, you throw away the generation even when the suite goes green. Green tests that the agent rewrote are not evidence that the freeze held. That failure mode is how a model outgrows the oracle you thought you were using, while still looking busy and correct.

Implementation notes the agent still misses

A handler that respects the freeze has to carry explicit keys separately from values. You cannot dump a parsed model with defaults filled in and then iterate items. The sketch below is labeled as merge pseudocode, not as a framework recommendation for production services.

# settings_app.py (sketch)
ALLOWED = {"display_name", "email_opt_in", "theme", "bio"}

def apply_patch(stored, parsed):
    unknown = parsed.explicit_keys - ALLOWED
    if unknown:
        raise Problem(422, "unknown_field")

    next_row = dict(stored)
    for key in parsed.explicit_keys:
        value = parsed.values[key]
        policy = POLICY[key]
        if value is None and policy == "reject":
            raise Problem(422, "null_not_allowed")
        if value == "" and policy in {"reject_empty", "reject"}:
            raise Problem(422, "empty_not_allowed")
        if value is None and policy == "reset":
            next_row[key] = RESET_VALUE[key]
            continue
        if value is None and policy == "clear":
            next_row[key] = None
            continue
        next_row[key] = validate_type(key, value)

    next_row["version"] = bump(stored["version"])
    return next_row

Parser boundary

Watch for body.get("bio") so missing keys become Python None and clear the bio accidentally. Watch for one global JSON Merge Patch rule, which would delete display_name when the client sends null. Watch for a 200 response that reintroduces defaults the client never sent. Those three mistakes all come from erasing omitted-versus-null before the merge function runs.

Partial writes

Validate the entire explicit key set first, then clone the row, then persist once. Agents often stream writes field by field because the prompt listed fields in order. Unknown keys and rejected nulls must leave version unchanged, which is how you prove the store never saw a partial document.

def handle_patch(raw_body, etag):
    stored = load()
    if etag != stored["version"]:
        raise Problem(412, "precondition_failed")
    parsed = parse_patch_body(raw_body)  # must not apply defaults
    next_row = apply_patch(stored, parsed)
    persist(next_row)
    return Response(204)

Other diff smells you should reject during review:

Accepting unknown keys to be "forward compatible" and dropping them on write.
Weakening tests to assert status in (200, 204) so either HTTP choice passes.
Coercing theme: null into "dark" because that was yesterday's stored value.
Bumping version on a 422 path, which poisons the next If-Match from the client.

Results of the worked example

This walkthrough was not executed against production traffic, and you should not treat the following as a benchmark. In a local lab sequence, the empty-object test and the theme-null test are the pair that usually fail first. They fail because the parser erased the omitted-versus-null distinction before the merge function ran. After you freeze parse keys, the display_name null test catches the remaining RFC-7396-shaped implementation.

What you should record in the pull request is small and concrete:

Parser preserves explicit_keys for omitted versus null.
Unknown keys and rejected nulls leave version unchanged.
theme null resets; bio null clears; {} changes neither value.
Tests and contract files were not modified by the generator.

You do not need a leaderboard number to call that a useful result for a small PATCH. You need a diff you can review and a suite that still encodes the table. If the generator only restated the happy path, you learned something about the freeze, not something flattering about the model.

Limitations

This freeze is for a single-document PATCH with a handful of fields and one actor. It does not replace RFC 6902 JSON Patch, and it does not define list-append semantics for arrays. If your public API already promises RFC 7396, you must not invent per-field null policies like the theme reset above. That would be a breaking change dressed up as agent safety.

You should not use this approach when:

The resource is money movement, inventory reservation, or any write you cannot replay from tests alone.
The client cannot send explicit null because a generated SDK strips null keys.
You lack a test runner in CI, so the freeze exists only in a markdown file.
You need multi-document transactions or row-level authorization beyond a single settings row.
You are tempted to skip review because a generation pass returned a green suite.

A coding agent does not freeze HTTP semantics for you, and it does not prove the suite matches user intent. This article does not claim model names, token quotas, hardware, duration, or permanence, because those would be unverified product details.

Lessons learned

Lead with the table, then the tests, then the handler, because agents optimize for a green file more readily than for a policy they cannot see. Keep omitted and null distinguishable at the parser boundary, or every later function will lie to you. Treat generator edits to tests as a failed run, even when coverage numbers look better afterward. Use a second review pass on unknown-key handling and partial writes, since those rarely show up in happy-path prompts.

If you reuse this case study on another endpoint, change the field policies rather than copying the settings document. The value is the freeze-and-verify loop, not the theme enum or the bio column. For a small PATCH, that loop is enough work for one afternoon and too important to leave implicit in a chat transcript.

If you want to run this freeze-then-generate lab on a throwaway PATCH, MonkeyCode's free model access and free server option are a reasonable place to try the loop.

Case Study: Freeze Discount Stacking Invariants Before an Agent Writes Checkout Math

Dakota Liu — Sun, 20 Sep 2026 11:35:55 +0000

You should freeze discount stacking invariants as executable tests before an agent writes checkout math, because a prose ticket invites plausible but still-wrong combinations. Agents optimize for the unit tests you already listed, not for coupon pairs you never wrote down. This case study walks a small checkout discount engine from background through frozen rules, code, and limits. Treat the walkthrough as a local teaching example you can run, not as a production finance sign-off or a tax opinion.

Background

Checkout tickets usually say you should support percentage coupons, fixed-amount coupons, and a maximum discount cap. That sentence looks complete until two coupons share a stack group, or shipping gets discounted by accident. An agent will implement each bullet independently, then glue them with leftover-subtotal arithmetic that feels reasonable in isolation. You only discover the bug when a customer stacks a twenty percent code with a ten dollar code on a twelve dollar cart.

Quiet money bugs differ from endpoint bugs that teams freeze before generation. Search cursors, webhook replay windows, and bulk-import partial success fail in logs you can grep. Discount engines fail as extra cents, negative totals, or promotions that finance never approved. You need invariants that speak in properties, not in one happy SKU fixture copied from the ticket.

Goal

You want one pure function that takes a cart subtotal, a shipping amount, and a list of coupons, then returns an explainable breakdown. The function must refuse negative totals, round to cents once at the end, and honor mutual exclusion by stack group. You will write those invariants first as tests, then a reference implementation, then a naive implementation that resembles typical agent output from a prose ticket.

Success for this case study is not a catalog of every retailer promotion on earth. Success is a test file that fails the naive version and passes the reference version without asserting on a specific product name. You should be able to drop a new coupon into the generator and still trust the same stacking rules.

Frozen invariants

Write these rules in the repository before anyone, human or agent, opens the engine file.

Subtotal and shipping are integer cents, never floats, and both must be greater than or equal to zero.
Percentage discounts apply only to remaining discountable money after higher-priority fixed discounts.
Coupons that share a nonempty stack_group cannot both apply; keep the larger discount in cents, then break ties by coupon id.
Shipping is discountable only when a coupon sets applies_to_shipping to true.
Round half-up to integer cents exactly once per percentage product, never after every unrelated step.
total_discount cannot exceed max_discount_cents when that cap is present.
Payable total equals subtotal plus shipping minus total discount, and it cannot drop below zero.
The breakdown must list applied coupon ids in the order they reduced the payable amount.

Decision table for stacking

Read this table as a policy freeze, not as the only tests you will run. Property tests will throw random coupons at the same rules so an agent cannot memorize six rows.

Cart 5000 cents goods, 1000 shipping, 20% goods and $10 goods, different groups: both apply, discount 2000.
Cart 5000 goods, 1000 shipping, 30% goods and $10 goods, group SUMMER: keep 30% (1500), drop $10.
Cart 1200 goods, 0 shipping, 20% plus $10, different groups: both apply, payable floors at 0, discount 1200.
Cart 5000 goods, 800 shipping, 50% with applies_to_shipping=true: discount 2900 against goods plus shipping.
Cart 5000 goods, 800 shipping, 50% goods only: discount 2500, shipping stays 800.
Cart 5000 goods, 0 shipping, 90% plus cap 1000: discount 1000.

If two coupons share a group, you compare tentative discount cents against the original bases, not raw value fields. Comparing a percent 30 to a fixed 1000 without converting units is itself a spec hole. Freeze that comparison in tests so an agent cannot “fix” it by casting everything to integers and hoping.

Implementation

The code below is a worked teaching example until you run it on your machine. Paste it into a small Python package, then execute the commands in the run section. Do not treat the reference engine as ledger-certified arithmetic.

Project layout

checkout_math/
  coupons.py
  invariants_test.py
  naive_engine.py
  reference_engine.py

Shared coupon shape

from dataclasses import dataclass

@dataclass(frozen=True)
class Coupon:
    coupon_id: str
    kind: str  # "percent" or "fixed"
    value: int  # 0-100 for percent, cents for fixed
    stack_group: str = ""
    applies_to_shipping: bool = False
    priority: int = 0  # lower applies first among allowed coupons

Property tests you freeze first

# invariants_test.py
from hypothesis import given, settings, strategies as st

from coupons import Coupon
from reference_engine import apply_discounts

cent = st.integers(min_value=0, max_value=1_000_000)

def coupons_strat():
    return st.lists(
        st.builds(
            Coupon,
            coupon_id=st.text(min_size=1, max_size=8, alphabet="abcdef0123456789"),
            kind=st.sampled_from(["percent", "fixed"]),
            value=st.integers(min_value=0, max_value=100),
            stack_group=st.sampled_from(["", "SUMMER", "VIP"]),
            applies_to_shipping=st.booleans(),
            priority=st.integers(min_value=0, max_value=5),
        ),
        max_size=5,
        unique_by=lambda c: c.coupon_id,
    )

@settings(max_examples=200)
@given(subtotal=cent, shipping=cent, coupons=coupons_strat(), cap=st.one_of(st.none(), cent))
def test_payable_never_negative(subtotal, shipping, coupons, cap):
    result = apply_discounts(subtotal, shipping, coupons, max_discount_cents=cap)
    assert result.payable_cents >= 0
    assert result.total_discount_cents >= 0
    assert result.payable_cents == subtotal + shipping - result.total_discount_cents

@given(subtotal=cent, shipping=cent, coupons=coupons_strat(), cap=st.one_of(st.none(), cent))
def test_cap_and_stack_groups(subtotal, shipping, coupons, cap):
    result = apply_discounts(subtotal, shipping, coupons, max_discount_cents=cap)
    if cap is not None:
        assert result.total_discount_cents <= cap
    applied = [c for c in coupons if c.coupon_id in result.applied_ids]
    groups = [c.stack_group for c in applied if c.stack_group]
    assert len(groups) == len(set(groups))

@given(subtotal=cent, shipping=cent, coupons=coupons_strat())
def test_shipping_only_when_flagged(subtotal, shipping, coupons):
    result = apply_discounts(subtotal, shipping, coupons, max_discount_cents=None)
    if coupons and all(not c.applies_to_shipping for c in coupons):
        assert result.total_discount_cents <= subtotal

Point the first run at naive_engine.apply_discounts and confirm at least one property fails. Then point the same file at reference_engine.apply_discounts and confirm the properties hold. That import swap is the whole point of freezing tests before generation.

Naive engine that usually appears from a prose ticket

# naive_engine.py
from math import floor
from types import SimpleNamespace

def apply_discounts(subtotal, shipping, coupons, max_discount_cents=None):
    remaining = subtotal + shipping  # mixes goods and shipping immediately
    applied = []
    discount = 0
    for coupon in coupons:
        if coupon.kind == "percent":
            piece = floor(remaining * coupon.value / 100)
        else:
            piece = coupon.value
        remaining -= piece
        discount += piece
        applied.append(coupon.coupon_id)
    if remaining < 0:
        remaining = 0
    return SimpleNamespace(
        payable_cents=remaining,
        total_discount_cents=discount,
        applied_ids=applied,
    )

That function will often go green on a single twenty percent fixture. Property tests should catch mixed shipping, stacked groups, ignored caps, and per-step flooring that drifts from half-up policy.

Reference engine the tests should pass

# reference_engine.py
from dataclasses import dataclass
from typing import List, Optional

from coupons import Coupon

@dataclass
class DiscountResult:
    payable_cents: int
    total_discount_cents: int
    applied_ids: List[str]

def _half_up(numer: int, denom: int) -> int:
    if denom <= 0:
        raise ValueError("denominator must be positive")
    if numer >= 0:
        return (numer + denom // 2) // denom
    return -(((-numer) + denom // 2) // denom)

def _base(coupon: Coupon, subtotal: int, shipping: int) -> int:
    return subtotal + shipping if coupon.applies_to_shipping else subtotal

def _tentative(coupon: Coupon, subtotal: int, shipping: int) -> int:
    base = _base(coupon, subtotal, shipping)
    if coupon.kind == "percent":
        if not 0 <= coupon.value <= 100:
            raise ValueError("percent out of range")
        return _half_up(base * coupon.value, 100)
    if coupon.value < 0:
        raise ValueError("fixed coupon cannot be negative")
    return min(coupon.value, base)

def apply_discounts(
    subtotal: int,
    shipping: int,
    coupons: List[Coupon],
    max_discount_cents: Optional[int] = None,
) -> DiscountResult:
    if subtotal < 0 or shipping < 0:
        raise ValueError("money inputs must be non-negative cents")

    winners: List[Coupon] = []
    by_group = {}
    for coupon in coupons:
        if not coupon.stack_group:
            winners.append(coupon)
            continue
        prior = by_group.get(coupon.stack_group)
        if prior is None:
            by_group[coupon.stack_group] = coupon
            continue
        left = _tentative(coupon, subtotal, shipping)
        right = _tentative(prior, subtotal, shipping)
        if (left, coupon.coupon_id) > (right, prior.coupon_id):
            by_group[coupon.stack_group] = coupon
    winners.extend(by_group.values())
    winners.sort(key=lambda c: (c.priority, c.coupon_id))

    goods_left = subtotal
    ship_left = shipping
    applied: List[str] = []
    raw_discount = 0

    for coupon in winners:
        piece = _tentative(coupon, goods_left, ship_left)
        if piece <= 0:
            continue
        if coupon.applies_to_shipping:
            from_goods = min(piece, goods_left)
            goods_left -= from_goods
            ship_left -= min(piece - from_goods, ship_left)
        else:
            goods_left -= min(piece, goods_left)
        raw_discount += piece
        applied.append(coupon.coupon_id)

    if max_discount_cents is not None:
        raw_discount = min(raw_discount, max_discount_cents)

    payable = subtotal + shipping - raw_discount
    if payable < 0:
        raw_discount = subtotal + shipping
        payable = 0

    return DiscountResult(
        payable_cents=payable,
        total_discount_cents=raw_discount,
        applied_ids=applied,
    )

The reference code is still a teaching sketch. You should replace the consume-goods-first rule if finance wants shipping consumed first. You should also add redemption limits and eligibility, which this function deliberately ignores.

Commands

python -m venv .venv
source .venv/bin/activate
pip install pytest hypothesis
pytest invariants_test.py -q

Run once against the naive module by changing the import, then run again against the reference module. Keep both results in the pull request so reviewers see the freeze, not only the final engine. If you skip the red run, you cannot tell whether the properties are actually sharp.

Where a free coding environment fits

You can keep this loop on a laptop, but a disposable server helps when the agent session should not touch your real checkout repo. MonkeyCode offers free model access and a free server option you can use to generate the engine against the frozen tests.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Give the agent the invariant file as read-only context and allow writes only under the engine module after the naive file has failed. Do not ask the model to invent stack groups, rounding, or tax behavior. Those choices belong in the test file and in a short human policy note sitting next to it.

Results you should see

When you point the properties at the naive engine, Hypothesis usually finds a same-group pair or a shipping mix within a few dozen examples. When you point them at the reference engine, the three properties above should hold for the generated ranges. You are not looking for a latency number or a conversion lift in this walkthrough. You are looking for a red-then-green story that a reviewer can replay without trusting chat output.

If a property fails on the reference engine, treat that as a spec bug in the freeze, not as a reason to weaken the assertion. Adjust the invariant in review, then regenerate the arithmetic. That order is the lesson: the test file is the product, and the engine is a draft that must survive it.

Limitations and who should not use this approach

This sketch does not handle multiple currencies, mid-request FX, sales tax, or VAT rounding modes. It also does not handle concurrent cart edits, coupon redemption counts, or customer eligibility. Hypothesis will not tell you whether a promotion is legal in a given jurisdiction, and integer cents will not save you from a wrong policy.

You should not use this as a production payments engine without finance and tax review. You should not freeze invariants only in chat history, because the next session will renegotiate them. You should not point a free shared server at secrets, customer carts, or live coupon databases. Skip the approach entirely if your discounts are a hard-coded single percentage with no stacking, because a table of two fixtures is enough.

Lessons learned

Freeze money rules as properties before an agent writes arithmetic, or the agent will pass the fixtures you remembered and fail the combinations you did not. Keep coupons as data, keep rounding in one place, and keep stack groups out of prose tickets. Compare group winners in cents, not in mixed raw values, or the freeze itself will encode a unit bug.

Use a disposable environment for generation if you want isolation from the real catalog, then copy only the engine that survived the freeze. If you run the same freeze-then-generate loop on a free server, review the invariant file as the product, and treat the generated engine as a draft.

Case Study: A Daily Digest Job That Refuses Implicit Timezones

Dakota Liu — Sat, 19 Sep 2026 09:41:36 +0000

Agents will invent a start-of-day if you leave the digest window unspecified, and that guess double-counts late events. This case study turns a vague daily-digest request into a timezone-strict calculator and a pytest suite. You run those tests before any model is allowed to write the worker that sends mail. A free coding model and a free server can draft the worker later, but they are not the source of truth.

Background

"Send each tenant a daily digest of yesterday's events" looks like a small cron job with a DATE() filter. That sentence hides three decisions: which timezone defines yesterday, whether the end bound is exclusive, and where late rows go. An unconstrained agent usually picks UTC, uses inclusive SQL between, and then backfills late events into a second mail.

You do not need a large product to see the failure mode clearly. A single America/New_York tenant and one event at 04:10 UTC already expose the bug. This worked example stays that small on purpose, so you can copy the calculator without standing up SMTP.

The project name is digest-window, and it is a labeled exercise rather than a claimed production incident. You should treat every command below as something you run locally first. You should also refuse to feed an assistant production event rows while you are still proving the window math.

Goal

You will ship a pure function that maps clock time, tenant timezone, and event time onto a window id. Missing timezones must fail closed, because a UTC default slides late-night events into the wrong digest. The worker, if an agent writes it, may only call that function and must not inline local date arithmetic.

You will keep a decision table next to the tests so reviewers can see calendar rules without reading Python. Success is a green pytest run plus a worker that imports the calculator. Success is not a polished HTML email template.

The window contract

Write the rules as data before you open a chat with any coding model. If a rule lives only in the prompt, the next generation will simplify it away. Freeze the table in git, then let tools write everything that is not the table.

Rules you freeze

Every tenant stores an IANA timezone; empty values raise MissingTimezoneError.
A digest covers the previous complete local day, not the last twenty-four hours of UTC.
The window is half-open: start_utc <= event < end_utc.
Events at or after end_utc are late_for_this_window and never silently merged.
Naive datetimes are rejected; both now and event_time must be timezone-aware.
The worker writes window_id into logs and into the email header for later audits.

Decision table

now_utc	tenant tz	event_time_utc	expected membership
`2026-03-09T12:00:00Z`	`America/New_York`	`2026-03-08T16:00:00Z`	`in_window` for `2026-03-08`
`2026-03-09T12:00:00Z`	`America/New_York`	`2026-03-09T08:00:00Z`	`late_for_this_window`
`2026-03-09T12:00:00Z`	(empty)	`2026-03-08T16:00:00Z`	`MissingTimezoneError`
`2026-03-09T06:30:00Z`	`America/New_York`	`2026-03-08T06:30:00Z`	still exclusive across the DST spring-forward

United States clocks spring forward on 8 March 2026, so that local Sunday lasts twenty-three hours. Your tests should pin that civil date so an agent cannot "fix" the window with epoch division. If a generated change deletes the DST fixture, you reject the change even when the happy-path test still passes.

Implementation

Keep the calculator in one module with no HTTP, no database, and no SMTP client. That boundary is the point of the case study, because agents love to fold date math into the worker. You review the import graph as strictly as you review the arithmetic. You should also keep Window frozen so callers cannot mutate end_utc after the query starts.

# digest_window.py
from __future__ import annotations

from dataclasses import dataclass
from datetime import date, datetime, time, timedelta
from zoneinfo import ZoneInfo


class MissingTimezoneError(ValueError):
    """Raised when a tenant has no IANA timezone."""


@dataclass(frozen=True)
class Window:
    timezone: str
    start_utc: datetime
    end_utc: datetime  # exclusive
    window_id: str


def require_timezone(tz_name: str | None) -> ZoneInfo:
    if not tz_name or not str(tz_name).strip():
        raise MissingTimezoneError("tenant timezone is required")
    return ZoneInfo(tz_name)


def window_for_local_date(tz_name: str, local_date: date) -> Window:
    tz = require_timezone(tz_name)
    start_local = datetime.combine(local_date, time.min, tzinfo=tz)
    next_local = datetime.combine(local_date + timedelta(days=1), time.min, tzinfo=tz)
    start_utc = start_local.astimezone(ZoneInfo("UTC"))
    end_utc = next_local.astimezone(ZoneInfo("UTC"))
    window_id = f"{tz_name}:{local_date.isoformat()}"
    return Window(
        timezone=tz_name,
        start_utc=start_utc,
        end_utc=end_utc,
        window_id=window_id,
    )


def membership(event_time_utc: datetime, window: Window) -> str:
    if event_time_utc.tzinfo is None:
        raise ValueError("event_time_utc must be timezone-aware")
    event = event_time_utc.astimezone(ZoneInfo("UTC"))
    if window.start_utc <= event < window.end_utc:
        return "in_window"
    if event >= window.end_utc:
        return "late_for_this_window"
    return "before_window"


def digest_window_id(now_utc: datetime, tz_name: str) -> str:
    tz = require_timezone(tz_name)
    if now_utc.tzinfo is None:
        raise ValueError("now_utc must be timezone-aware")
    local_now = now_utc.astimezone(tz)
    local_date = local_now.date() - timedelta(days=1)
    return window_for_local_date(tz_name, local_date).window_id

The half-open interval is the line that saves you from double sends. Inclusive BETWEEN queries look friendly, then emit two digests for the same midnight instant. Reject the SQL shape below in review, even if an assistant generated it with confident comments.

-- Review reject: UTC DATE() ignores tenant timezone and exclusive bounds.
SELECT *
FROM events
WHERE DATE(created_at) = DATE(UTC_TIMESTAMP() - INTERVAL 1 DAY);

Tests you run before the worker exists

Put the decision table into pytest first. An agent that arrives later has to satisfy these names, not invent friendlier ones. Pin civil dates in 2026 so the suite does not drift when you rerun it next month.

# test_digest_window.py
from datetime import date, datetime
from zoneinfo import ZoneInfo

import pytest

from digest_window import (
    MissingTimezoneError,
    digest_window_id,
    membership,
    window_for_local_date,
)

UTC = ZoneInfo("UTC")
NY = "America/New_York"


def test_previous_local_day_window_id():
    now = datetime(2026, 3, 9, 12, 0, tzinfo=UTC)
    assert digest_window_id(now, NY) == "America/New_York:2026-03-08"


def test_event_in_window():
    window = window_for_local_date(NY, date(2026, 3, 8))
    event = datetime(2026, 3, 8, 16, 0, tzinfo=UTC)
    assert membership(event, window) == "in_window"


def test_late_event_not_merged():
    window = window_for_local_date(NY, date(2026, 3, 8))
    event = datetime(2026, 3, 9, 8, 0, tzinfo=UTC)
    assert membership(event, window) == "late_for_this_window"


def test_missing_timezone_fails_closed():
    now = datetime(2026, 3, 9, 12, 0, tzinfo=UTC)
    with pytest.raises(MissingTimezoneError):
        digest_window_id(now, "")


def test_naive_event_is_rejected():
    window = window_for_local_date(NY, date(2026, 3, 8))
    with pytest.raises(ValueError):
        membership(datetime(2026, 3, 8, 16, 0), window)


def test_dst_spring_forward_exclusive_end():
    window = window_for_local_date(NY, date(2026, 3, 8))
    assert window.start_utc == datetime(2026, 3, 8, 5, 0, tzinfo=UTC)
    assert window.end_utc == datetime(2026, 3, 9, 4, 0, tzinfo=UTC)
    on_end = datetime(2026, 3, 9, 4, 0, tzinfo=UTC)
    assert membership(on_end, window) == "late_for_this_window"

Commands

python -m venv .venv
# Windows: .venv\Scripts\activate
source .venv/bin/activate
pip install pytest
pytest -q test_digest_window.py

You should see a green run before anyone generates digest_worker.py. If a model rewrites the calculator to make a new assertion pass, you revert the calculator and fail the change. The tests own the calendar; the model does not.

A worker shape that agents keep trying to skip

Paste this sketch only after the calculator tests are green. The sketch exists so the model cannot mint a second copy of the window math. Treat it as a proposed, unexecuted outline until test_digest_window.py stays green on your machine.

# Proposed worker sketch — unexecuted until test_digest_window.py stays green.
from datetime import timedelta
from zoneinfo import ZoneInfo

from digest_window import (
    MissingTimezoneError,
    digest_window_id,
    membership,
    window_for_local_date,
)


def run_digest(tenant, now_utc, fetch_events, send_mail, log):
    try:
        window_id = digest_window_id(now_utc, tenant.timezone)
    except MissingTimezoneError:
        log("skip_missing_timezone", tenant_id=tenant.id)
        return "skipped"

    local_yesterday = now_utc.astimezone(ZoneInfo(tenant.timezone)).date() - timedelta(days=1)
    window = window_for_local_date(tenant.timezone, local_yesterday)
    events = fetch_events(tenant.id, window.start_utc, window.end_utc)
    in_window = [e for e in events if membership(e.created_at, window) == "in_window"]
    late = [e for e in events if membership(e.created_at, window) == "late_for_this_window"]
    log("digest_counts", window_id=window_id, in_window=len(in_window), late=len(late))
    send_mail(tenant, window_id, in_window)
    return window_id

If the generated file imports datetime and computes date.today() itself, you throw the file away. That is a review rule, not a style preference. Late events may be logged, but they must not ride along in yesterday's mail.

Review checklist

Calculator module has no SMTP, HTTP, or ORM imports.
Tests pin 2026-03-08 in America/New_York, not date.today().
Worker imports digest_window and does not call date.today().
Logs include window_id and a late-event count.
Empty timezone fails closed before fetch_events.

Where a remote model and a free server fit

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

You can stop at the calculator and never touch an assistant. If you want help drafting the worker, an open-source coding assistant is useful only after these tests exist. MonkeyCode offers free model access and a free server option, which is enough to generate digest_worker.py and run pytest off your laptop.

Give the assistant the decision table, the calculator module, and the pytest file as the prompt. Ask it for the worker and for log lines that include window_id. Do not ask it to "just make the digest work," because that wording invites UTC defaults and inclusive SQL. Keep production events off the free server; the fixtures in this article are synthetic on purpose, and a shared runner is the wrong place for tenant mail bodies.

Results from the worked fixtures

On 2026-03-09T12:00:00Z, a New York tenant's digest id is America/New_York:2026-03-08. An event at 2026-03-08T16:00:00Z is in window, which is 11:00 local time the day before the job. An event at 2026-03-09T08:00:00Z is late, even though a UTC DATE() filter might still label it the ninth.

The empty timezone fixture raises before any query runs, and that failure is the CI result you want. A quiet UTC fallback would have been a false green on the same input. None of these outcomes are runtime benchmarks; they are contract checks you can re-run on any machine with tzdata.

For the DST row, local 8 March 2026 in New York starts at 2026-03-08T05:00:00Z and ends at 2026-03-09T04:00:00Z. The twenty-three hour length is expected, and the exclusive end still holds at 04:00 UTC. If your host tzdata cannot reproduce those two instants, you fix the image before you trust a generated worker.

Limitations

This calculator does not send email, retry SMTP, or deduplicate overlapping cron launches. It does not track political tzdata changes beyond whatever zoneinfo loads from the host. A free server image with stale tzdata can disagree with your laptop on a newly announced offset.

You should not reuse this module for sub-hour tumbling windows, fiscal calendars, or "user's current travel timezone" products. Those need a different contract, and an agent will still import this file if you do not block it. Exclusive daily windows are the only behavior under test here.

Who should not use this workflow

Skip the remote server if your fixtures contain real addresses, message bodies, or tenant identifiers. Skip agent-written workers if nobody on the team can read the timezone tests and explain the half-open bound. Skip the whole pattern if your digest is a rolling twenty-four hour window, because previous-local-day is the wrong primitive.

Lessons learned

Ambiguous time language is a spec bug, and coding models amplify it instead of inventing a careful calendar. Exclusive end bounds belong in tests, not in a comment that the next prompt will drop. A free model is a decent typist for boilerplate workers once membership rules already fail closed.

A free server is a convenient pytest runner, not a production cron host, and not a reason to skip the table. Keep the calculator human-owned even when the worker is generated. When the suite is green, you can park the worker draft on MonkeyCode's free server option and keep the calculator on your machine.

Case Study: Freeze Cursor Stability Rules Before an Agent Writes the List Endpoint

Dakota Liu — Fri, 18 Sep 2026 08:02:05 +0000

Offset pagination looks complete until concurrent inserts make a catalog list skip rows and duplicate others. You should freeze cursor stability rules, including the sort key and a unique tie-breaker, before any agent writes the list endpoint. This case study walks a small product-catalog API from that frozen contract through tests, a seek query, and a generated handler. The useful result is not prettier SQL; it is a page that stays stable when inventory changes under your readers.

Background: a twenty-product catalog that still breaks

You are shipping a tiny inventory catalog that stores sku, name, quantity, and updated_at for each product. The first agent draft uses ORDER BY name LIMIT 10 OFFSET n because that pattern appears in every tutorial. That draft passes a happy-path test that inserts twenty static rows and walks page one and page two. It fails the moment a merchant adds "Aardvark Adapter" while a buyer is still paging through names.

Offset pagination is a moving window over a live table, not a snapshot of a result set. Concurrent inserts before the current offset push later rows down, so page two skips items the buyer never saw. Concurrent deletes pull later rows up, so page two repeats items the buyer already scanned. You cannot prompt that failure away after the handler exists, because the handler already encoded the wrong contract.

Goal: freeze the page contract before any list SQL

The project goal is deliberately small and easy to review in a single pull request. You will publish GET /products with a page size of ten, a stable next-page token, and no total count. You will refuse client-supplied offsets rather than treating them as a compatibility feature. You will require that two readers who hold the same cursor, at the same committed table state, receive the same next page.

You freeze those rules in a spec file the agent is not allowed to edit. Tests import that spec instead of restating policy in comments. The implementation may change queries, encodings, and variable names as needed. It may not change sort keys, tie-breakers, null order, or error codes once those values are frozen.

Freeze order you actually run

Write the decision table for inserts, deletes, equal names, and bad cursors.
Encode tests that fail on OFFSET under a concurrent insert.
Lock the spec file in review so prompts cannot widen page size or add counts.
Allow the agent to touch only the list handler and the cursor codec.
Reject any patch that reintroduces offset, silent clamping, or query-string direction.

That order is the whole method. Code generation is step four, not step one, and the tests already know the answer.

The frozen cursor spec

Treat the following contract as source, not as comments inside generated code. If a later prompt wants page=3, you reject the prompt instead of extending the handler.

Pagination mode: keyset (seek) on (name ASC, id ASC).
Tie-breaker: id, a unique and immutable product identifier.
Page size: client may request 1..10; default 10; values above 10 become 400.
Cursor: opaque base64url of JSON {"n": name, "i": id, "d": "asc"}.
Empty catalog: 200 with items: [] and next_cursor: null.
Invalid or truncated cursor: 400 with error code invalid_cursor, never an empty page.
Unknown id in a well-formed cursor: skip forward from the sort position; do not 404.
Null names: sort last in ASC and first in DESC, and that choice is frozen.
Deletes: a product removed after the cursor was issued is omitted; the next live row is returned.
Total count: not included; adding it later is a separate spec change.

Decision table you pin in review

Situation	Required behavior	Failing shortcut
Insert with a name before the current page	Later pages must not skip already-unseen later names	`OFFSET`
Insert with a name after the cursor	Next page may include it	Undocumented snapshot-at-request
Two products with the same name	Order by `id` ascending	`ORDER BY name` only
Client sends `offset=10`	Ignore or `400`; do not query	Backward-compatible query param
Cursor decoded to `DESC` while URL says `ASC`	`400 invalid_cursor`	Trust the query string
Page size `50`	`400 page_size_out_of_range`	Silent clamp to 10
Malformed base64	`400 invalid_cursor`	Return the first page

You keep this table next to the tests so review stays mechanical. An agent that "simplifies" any row has failed the task, even if TypeScript compiles and the happy path stays green.

Tests that fail on offset pagination

Write the tests first and keep them in the repository the agent can read. The suite below is a worked example; run it in your own database before you trust the assertions.

// tests/catalog-list.contract.test.ts
// Worked example: unexecuted against your database until you wire a test harness.

import { describe, it, expect, beforeEach } from "vitest";
import { listProducts, seedProducts, insertProduct } from "../src/catalog";

describe("GET /products cursor contract", () => {
  beforeEach(async () => {
    await seedProducts([
      { id: "p01", name: "Bolt" },
      { id: "p02", name: "Clamp" },
      { id: "p03", name: "Drill" },
      { id: "p04", name: "Clamp" }, // same name, later id
    ]);
  });

  it("orders equal names by id, not by insert timing", async () => {
    const page = await listProducts({ pageSize: 10, cursor: null });
    const clamps = page.items.filter((p) => p.name === "Clamp");
    expect(clamps.map((p) => p.id)).toEqual(["p02", "p04"]);
  });

  it("does not skip a later name when an earlier name is inserted", async () => {
    const first = await listProducts({ pageSize: 2, cursor: null });
    expect(first.items.map((p) => p.id)).toEqual(["p01", "p02"]);

    await insertProduct({ id: "p00", name: "Adapter" });

    const second = await listProducts({
      pageSize: 2,
      cursor: first.next_cursor,
    });
    expect(second.items.map((p) => p.id)).toEqual(["p04", "p03"]);
  });

  it("rejects a page size above the frozen cap", async () => {
    await expect(listProducts({ pageSize: 50, cursor: null })).rejects.toMatchObject({
      code: "page_size_out_of_range",
    });
  });

  it("rejects a cursor whose direction does not match the request", async () => {
    const first = await listProducts({ pageSize: 2, cursor: null });
    await expect(
      listProducts({ pageSize: 2, cursor: first.next_cursor, direction: "desc" }),
    ).rejects.toMatchObject({ code: "invalid_cursor" });
  });
});

The second test is the whole point of this case study. Under OFFSET 2, inserting "Adapter" at the front shifts Clamp p04 and Drill p03, and page two skips p04. Under a seek cursor of (name=Clamp, id=p02), the next page still starts after that tuple. You should watch that test fail on the agent's first offset draft, then stay red until the query uses tuple comparison.

Implementation the tests will accept

Once the spec and tests are frozen, the handler is comparatively boring and should stay that way. You encode the last row of a page into an opaque cursor, then resume with a tuple comparison that includes the unique id. Direction lives inside the cursor so a caller cannot mix an ascending token with a descending query string.

// src/cursor.ts
// Worked example: proposed encoding, not a published library.

export type Direction = "asc" | "desc";

export type ProductCursor = {
  n: string | null;
  i: string;
  d: Direction;
};

export function encodeCursor(c: ProductCursor): string {
  return Buffer.from(JSON.stringify(c), "utf8").toString("base64url");
}

export function decodeCursor(token: string, expected: Direction): ProductCursor {
  let parsed: unknown;
  try {
    parsed = JSON.parse(Buffer.from(token, "base64url").toString("utf8"));
  } catch {
    throw Object.assign(new Error("invalid_cursor"), { code: "invalid_cursor" });
  }
  if (
    typeof parsed !== "object" ||
    parsed === null ||
    !("n" in parsed) ||
    !("i" in parsed) ||
    !("d" in parsed)
  ) {
    throw Object.assign(new Error("invalid_cursor"), { code: "invalid_cursor" });
  }
  const cursor = parsed as ProductCursor;
  if (cursor.d !== expected || typeof cursor.i !== "string") {
    throw Object.assign(new Error("invalid_cursor"), { code: "invalid_cursor" });
  }
  return cursor;
}

-- ASC seek; bind :name, :id, :limit from the decoded cursor.
SELECT id, name, quantity
FROM products
WHERE name IS NOT NULL
  AND (
    name > :name
    OR (name = :name AND id > :id)
  )
ORDER BY name ASC, id ASC
LIMIT :limit;

You still need a separate branch for name IS NULL if nulls sort last, because SQL null comparisons will not do that work for you. Freeze that branch in the spec rather than leaving the agent to pick a dialect default. Postgres NULLS LAST can express the rule, but only if the tests assert the actual order, not the presence of a SQL keyword.

Command you run before reviewing the patch

# Worked example: replace with your package scripts.
npx vitest run tests/catalog-list.contract.test.ts

If that file is green and OFFSET is absent from the handler, you have the smallest passing implementation this case study cares about. If the agent added COUNT(*) or a page query param, you revert those lines even when the tests still pass, because they changed the frozen contract.

Where a free coding environment fits

After the spec file and the failing tests exist, you may let a coding agent write the handler and the SQL. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option, which is enough for this generate-against-tests loop on a small catalog service.

You paste the frozen spec, the decision table, and the test file. You instruct the agent to change only src/catalog.ts and to stop when the contract tests pass. You do not ask it to invent pagination policy, page size, or error codes. If it reintroduces OFFSET, the insert-during-paging test stays red, and you reject the patch.

This is ordinary test-driven work with a narrower file allowlist than most agent demos use. The free model access and free server option are just a place to run that loop without mixing policy into prompts.

Results of the worked example

When the offset draft is scored against the frozen suite, the equal-name test and the insert-during-paging test fail first. After a seek implementation lands, those two tests pass without changing seed data. The page-size and direction-mismatch tests stay valuable because agents often clamp silently or trust the query string over the cursor payload.

You should not treat a green suite as proof of production readiness for a public catalog. The example does not cover replica lag, schema migrations that rewrite ids, or locale-aware name collation. It does prove that the list contract is explicit enough to fail loudly when an agent reaches for offset pagination again.

Limitations and who should skip this approach

Keyset pagination is the wrong default for some products, and you should not force it onto every list.

You need jump-to-page or "page 17 of 40" in a UI that finance or ops already depends on.
You cannot add a unique, immutable tie-breaker, so equal sort keys remain unstable.
You must return a total count on every request and cannot afford a separate count query.
Your sort column is updated in place, for example a popularity score that moves while users page.
You are exposing an unauthenticated dump where opaque cursors still leak business order.

Cursor tokens also leak sort values unless you encrypt them. Base64url is opaque to casual readers, not a security boundary. If product names are sensitive, the frozen spec must require a server-side signed payload, which this case study does not implement. You should also avoid this loop when you do not yet know the sort key, because generating SQL before that choice is how offset drafts get merged.

Lessons learned

You should freeze the list contract as data the tests import, not as a paragraph inside a prompt. Offset pagination will keep looking correct in screenshots while skipping rows under concurrent writes. A unique tie-breaker is part of the public API, even when the cursor is opaque, because it decides which row is next. Agents are useful once those rules are already failing tests; they are a poor place to invent pagination policy.

If you want a throwaway environment for that generate-against-tests loop, try MonkeyCode's free model access and free server option on this catalog exercise.

Case Study: Freeze Partial-Success Rules Before an Agent Writes the Bulk Import

Dakota Liu — Thu, 17 Sep 2026 06:05:34 +0000

Agents will invent an all-or-nothing bulk import unless you freeze per-row outcomes first. You should lock HTTP status mapping, row replay keys, and error indexes before any coding agent writes a handler. This case study walks a small member-import API from a frozen contract through failing tests to a generated implementation. You can reuse the decision table and pytest file on your own service without adopting any particular vendor.

Background

Developer communities still argue whether generated code counts as engineering or as merely casual output. That debate stays noisy, but the failure mode in bulk endpoints is specific and quite boring. You ask an agent for a JSON user import, and it wraps every insert inside one database transaction. The first bad email then rolls back valid rows, and reviewers spend a day picking 400 versus 207.

You do not need a newly released model family to prevent that rollback pattern from shipping. You need a written contract that forbids silent skips, whole-payload rollback, and unordered error lists. The rest of this write-up treats that contract as the project rather than as optional prompt garnish.

Goal

You will freeze a 100-row member import for a fictional workspace called Northwind Crew. The endpoint accepts JSON, not CSV, so parsing stays out of scope for this pass. Each row carries a client-supplied row_key so retries can target the same logical member without duplicating records.

Success for this case study means three outcomes you can assert in CI:

Malformed payloads fail closed with HTTP 400 and with no per-row result array.
Mixed valid and invalid rows return HTTP 207 with stable, request-order results.
Replaying a successful row_key returns unchanged together with the original member id.

You will not build auth, email sending, or an async job graph during this case study. Those concerns belong in other specs, and mixing them here is how agents quietly expand scope.

The frozen contract

Write this file first and treat it as read-only during generation. If a later prompt disagrees with the table, the table wins and the prompt is wrong.

HTTP mapping

HTTP 207 Multi-Status is the frozen choice for a well-formed payload that still contains rejected rows. You use 200 only when every row is created or unchanged. You use 400 only when the payload cannot be split into rows, which means no partial inserts are allowed.

Decision table

Empty rows array → HTTP 400, { "error": "malformed_payload" }, no writes.
Any row missing row_key → HTTP 400, same error body, no writes.
Duplicate row_key values inside one request → HTTP 400, no writes.
More than 100 rows → HTTP 400, no writes.
All rows valid with new emails → HTTP 200, every result created, insert each member.
All rows valid and already imported under the same keys → HTTP 200, every result unchanged.
Mix of valid rows and invalid emails → HTTP 207, insert only valid rows.
Email already owned by a different row_key → that row is rejected with email_taken; siblings still proceed.
Every well-formed row rejected → still HTTP 207, because the payload itself was parseable.
Unknown JSON fields on a row → ignore them; extra keys must not trigger 400.

status values are exactly created, unchanged, and rejected. Agents often add success, ok, and failed, so those strings are banned in the spec. Allowed reject codes are only invalid_email, email_taken, and display_name_blank.

Result object

{
  "results": [
    {
      "row_key": "invite-014",
      "status": "created",
      "id": "mem_8f2a",
      "error": null
    },
    {
      "row_key": "invite-015",
      "status": "rejected",
      "id": null,
      "error": "invalid_email"
    }
  ]
}

error is null on created and unchanged. Result order must match request order, even when emails would sort differently. You freeze both rules because generated serializers like to group failures at the bottom.

Storage gates the agent must not invent

CREATE UNIQUE INDEX members_email_uidx ON members (email);
CREATE UNIQUE INDEX members_row_key_uidx ON members (row_key);

Unique indexes, not prompt text, are what make email_taken true under concurrent imports. If the generated migration omits either index, you reject the patch before you look at handler style.

Reproducible test plan

Save the tests before you open a chat with any coding agent. The tests are the original artifact, and generation is only a way to make them pass.

# test_bulk_import.py
# Labeled example: wire client to your local app fixture.
# Do not treat this helper as a production SDK.
import json

MAX_ROWS = 100

def post_import(client, payload):
    return client.post(
        "/v1/members:import",
        data=json.dumps(payload),
        headers={"Content-Type": "application/json"},
    )

def test_empty_array_is_malformed(client):
    res = post_import(client, {"rows": []})
    assert res.status_code == 400
    assert res.json()["error"] == "malformed_payload"

def test_duplicate_row_key_is_malformed(client):
    res = post_import(
        client,
        {
            "rows": [
                {"row_key": "a", "email": "a@example.com", "display_name": "Ann"},
                {"row_key": "a", "email": "b@example.com", "display_name": "Bea"},
            ]
        },
    )
    assert res.status_code == 400
    assert res.json()["error"] == "malformed_payload"

def test_over_max_rows_is_malformed(client):
    rows = [
        {
            "row_key": f"k{i}",
            "email": f"u{i}@example.com",
            "display_name": f"U{i}",
        }
        for i in range(MAX_ROWS + 1)
    ]
    res = post_import(client, {"rows": rows})
    assert res.status_code == 400

def test_mixed_batch_returns_207_in_request_order(client):
    res = post_import(
        client,
        {
            "rows": [
                {"row_key": "ok-1", "email": "ok@example.com", "display_name": "Ok"},
                {"row_key": "bad-1", "email": "not-an-email", "display_name": "Bad"},
            ]
        },
    )
    assert res.status_code == 207
    results = res.json()["results"]
    assert [r["row_key"] for r in results] == ["ok-1", "bad-1"]
    assert results[0]["status"] == "created"
    assert results[0]["id"]
    assert results[1]["status"] == "rejected"
    assert results[1]["error"] == "invalid_email"

def test_replay_of_created_row_is_unchanged(client):
    body = {
        "rows": [
            {"row_key": "ok-2", "email": "replay@example.com", "display_name": "Rep"}
        ]
    }
    first = post_import(client, body)
    second = post_import(client, body)
    assert first.status_code == 200
    assert second.status_code == 200
    assert first.json()["results"][0]["id"] == second.json()["results"][0]["id"]
    assert second.json()["results"][0]["status"] == "unchanged"

def test_email_taken_rejects_only_that_row(client):
    post_import(
        client,
        {
            "rows": [
                {"row_key": "owner", "email": "taken@example.com", "display_name": "Own"}
            ]
        },
    )
    res = post_import(
        client,
        {
            "rows": [
                {"row_key": "other", "email": "taken@example.com", "display_name": "Oth"},
                {"row_key": "fresh", "email": "fresh@example.com", "display_name": "New"},
            ]
        },
    )
    assert res.status_code == 207
    results = res.json()["results"]
    assert results[0]["status"] == "rejected"
    assert results[0]["error"] == "email_taken"
    assert results[1]["status"] == "created"

Run the suite before generation so you watch it fail for the right reason:

pytest test_bulk_import.py -q

You should see fixture errors or HTTP 404 responses, not a green suite. A green run at this stage usually means the client pointed at the wrong app process.

A manual curl check belongs beside pytest, because reviewers often trust a single happy path too early:

curl -sS -D - -o body.json \
  -H 'Content-Type: application/json' \
  -d '{"rows":[{"row_key":"ok-1","email":"ok@example.com","display_name":"Ok"},{"row_key":"bad-1","email":"not-an-email","display_name":"Bad"}]}' \
  http://127.0.0.1:8000/v1/members:import

You expect status 207, two results objects, and only one new row in the members table. If curl shows 200 with a mixed batch, the handler violated the frozen mapping even if unit tests were skipped.

Implementation workflow

Keep the agent inside a tight loop with the spec file as the only source of HTTP truth. You paste the decision table, the result schema, and the failing tests in that order. You forbid new status strings, extra endpoints, CSV parsers, and background jobs. You ask for the smallest handler that makes pytest pass and then you stop.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you want that red-green loop on a throwaway machine, MonkeyCode's free model access and free server option are enough to run the tests without standing up your own inference box. You still read every INSERT the agent emits, because free inference does not freeze the contract for you.

A prompt that stays honest looks like this:

Implement POST /v1/members:import against spec.md and test_bulk_import.py.
Do not add routes, status strings, or jobs that the spec does not name.
Stop when pytest is green. Do not improve HTTP mappings.

A tiny classifier you can compare against generated output:

# Labeled example, not production code.
ALLOWED_STATUS = {"created", "unchanged", "rejected"}
ALLOWED_ERRORS = {"invalid_email", "email_taken", "display_name_blank"}

def classify_http(results):
    if any(row["status"] == "rejected" for row in results):
        return 207
    return 200

You reject a patch that returns 200 while any row is rejected. You also reject HTTP 409 on a single taken email, because that status implies the whole batch failed. A 500 for invalid_email is equally wrong, since validation is a defined reject code rather than an operational crash.

Results

On a frozen spec, the usual agent mistakes become visible in one pytest run instead of in a staging incident:

Whole-batch transactions that roll back the valid rows beside one bad email.
failed or error used as status instead of the frozen rejected token.
Result arrays sorted by email, which breaks client retry maps keyed by index.
HTTP 400 used for invalid_email on a well-formed mixed batch.
Missing unchanged on replay, which inserts a second member for one row_key.

You do not need a public leaderboard score to call those defects. The tests already name them in plain assertions. After the handler is green, you add one manual check: two parallel replays of the same row_key must still yield one member id. That race sits outside the pytest file above, so you either add a lock test or you document it as known debt.

Lessons learned

Freeze HTTP mapping for mixed outcomes before you freeze column types or ORM names.
Per-row row_key is not optional if clients will retry from flaky mobile networks.
Three reject codes beat an open string field that agents will extend forever.
Request-order results are part of the contract, not a pretty-print detail for logs.
All-or-nothing imports are a product decision; encode them only when stock or money requires one transaction.

Limitations and who should skip this

This approach is wrong for ledger lines, seat billing, or inventory reservation that must commit as one transaction. Partial success would leave money or stock inconsistent, and HTTP 207 would hide that split. You should also skip this pattern for multi-megabyte CSV drops, because those need a job resource rather than a synchronous import handler.

Do not send a coding agent at this endpoint if you cannot read the INSERT path in review. Unique indexes, not prompt wording, decide email_taken when two requests collide. If your team cannot add those constraints, freeze the spec anyway and wait for a human migration before generating handlers.

If you already have a bulk importer in production, export one day of payloads and replay them against these tests before you generate anything new. Historical clients often omit row_key, and that single gap will 400 every legacy caller overnight. Paste the table into the next agent session before you paste the feature prompt, and keep the HTTP mapping out of the model's improvisation range.

Duplicate Orders Are a Spec Bug, Not a Model Bug

Dakota Liu — Wed, 16 Sep 2026 04:16:39 +0000

You should freeze idempotency rules before a coding agent writes POST /orders, because retries will otherwise insert duplicate rows. Most agents invent a cache, a unique index, or a loose header, then mix those three models under concurrent retries. This case study walks a tiny checkout service from a frozen contract through tests and a small reference handler. You can run the same loop on a laptop, or on a disposable server if you want isolation from your laptop.

Background

Checkout clients retry after timeouts, after network flaps, and after load balancers reset idle connections during a create call. If POST /orders inserts a new row on every attempt, you charge twice for a single cart. Coding agents often store the last response in a dictionary keyed only by Idempotency-Key and call the job done. That design collapses when two carts share a key, when the process restarts, and when two workers race.

You do not need a new model release to prevent that class of bug in a small service. You need a written contract, a stable fingerprint, and tests that fail the pull request when the handler drifts. The rest of this article treats that contract as the product and treats generated Python as disposable implementation.

Goal

You will specify one create-order endpoint for a single-shop demo rather than a multi-tenant marketplace platform. The handler must accept Idempotency-Key, persist the first successful response, and replay that payload for matching retries. Conflicting bodies under the same key must fail closed with 409, and missing keys must fail closed with 400. Silent uniqueness without a client key is how duplicate SKUs sneak into later import jobs.

The walkthrough stays small on purpose, with no payment processor, inventory reservation, or routing table. You are proving that an agent cannot invent a second idempotency model after the tests already exist. If the generated code cannot satisfy the table, you throw the file away and generate again against the same tests.

The frozen contract

Paste this file into the repo as contracts/idempotency_v1.md and refuse to merge handler code that disagrees with it. Agents optimize for looking complete, and they will not keep your replay semantics unless you freeze those semantics first. The contract below is the whole product surface for this demo shop, so treat amendments as API breaks.

Endpoint rules

Method and path: POST /v1/orders only.
Required header: Idempotency-Key matching ^[A-Za-z0-9._-]{8,128}$.
Required header: Authorization: Bearer <merchant_token> so keys are not global.
Required body fields: sku (non-empty string), qty (integer 1..99), currency (USD only in this demo).
Canonical fingerprint: SHA-256 of METHOD\nPATH\nKEY\nMERCHANT_ID\n plus canonical JSON bytes.
Canonical JSON: UTF-8 object, keys sorted, no extra whitespace, integers left unquoted.

Stored record

Each accepted key stores merchant_id, key, fingerprint, status_code, response_body, and created_at for later replay. Replays must return the stored status and body bytes, not a newly rendered document with fresh timestamps. This demo caches only 201 responses, so a failed request leaves the key free for a corrected body. That choice is a product decision, and you should not let an agent cache 400s without a contract amendment.

Status mapping

Missing or malformed Idempotency-Key → 400 with code=idempotency_key_required.
Unknown merchant token → 401 with code=unauthorized.
Same merchant, same key, same fingerprint, stored 201 → replay 201 unchanged.
Same merchant, same key, different fingerprint → 409 with code=idempotency_key_reuse.
Same merchant, same key, in-flight lock → wait up to 5 seconds, then 409 with code=idempotency_in_progress.
New key and valid body → insert order, store record, return 201.
Other merchants may reuse the same key string, because uniqueness is (merchant_id, key).

Decision table

Incoming key	Fingerprint vs stored	Stored state	HTTP
absent	n/a	n/a	400
malformed	n/a	n/a	400
new	n/a	none	201
known	equal	completed 201	201 replay
known	different	completed 201	409
known	equal	in progress	wait / 409
known	n/a	other merchant	201 (separate row)

Do not let the agent add PUT semantics, GET replay, or auto-generated keys without a written amendment. Those extras change client duties and usually reintroduce the duplicates you were trying to prevent. If a prompt asks for convenience, you answer with a new row in the table, not with a silent branch.

Tests that refuse to merge

Save the following module as tests/test_idempotency_contract.py and keep it in the same pull request as any generated handler. The tests are a proposed runnable design for pytest, and you should execute them in your own environment before you trust the output. They encode the table, not a particular framework, so a failing assertion means the contract moved.

# tests/test_idempotency_contract.py
# Proposed pytest module: run it against your own app fixture.
import uuid
import pytest


@pytest.fixture
def auth_header(merchant_token):
    return {"Authorization": f"Bearer {merchant_token}"}


def test_missing_key_is_400(client, auth_header):
    body = {"sku": "SKU-1", "qty": 1, "currency": "USD"}
    res = client.post("/v1/orders", json=body, headers=auth_header)
    assert res.status_code == 400
    assert res.json()["code"] == "idempotency_key_required"


def test_replay_returns_identical_201(client, auth_header):
    key = str(uuid.uuid4())
    headers = {**auth_header, "Idempotency-Key": key}
    body = {"sku": "SKU-1", "qty": 2, "currency": "USD"}
    first = client.post("/v1/orders", json=body, headers=headers)
    second = client.post("/v1/orders", json=body, headers=headers)
    assert first.status_code == 201
    assert second.status_code == 201
    assert first.json() == second.json()
    assert first.json()["id"] == second.json()["id"]


def test_same_key_different_body_is_409(client, auth_header):
    key = str(uuid.uuid4())
    headers = {**auth_header, "Idempotency-Key": key}
    a = {"sku": "SKU-1", "qty": 1, "currency": "USD"}
    b = {"sku": "SKU-1", "qty": 2, "currency": "USD"}
    first = client.post("/v1/orders", json=a, headers=headers)
    conflict = client.post("/v1/orders", json=b, headers=headers)
    assert first.status_code == 201
    assert conflict.status_code == 409
    assert conflict.json()["code"] == "idempotency_key_reuse"


def test_key_is_scoped_to_merchant(client, merchant_token, other_token):
    key = str(uuid.uuid4())
    body = {"sku": "SKU-1", "qty": 1, "currency": "USD"}
    first = client.post(
        "/v1/orders",
        json=body,
        headers={
            "Authorization": f"Bearer {merchant_token}",
            "Idempotency-Key": key,
        },
    )
    second = client.post(
        "/v1/orders",
        json=body,
        headers={
            "Authorization": f"Bearer {other_token}",
            "Idempotency-Key": key,
        },
    )
    assert first.status_code == 201
    assert second.status_code == 201
    assert first.json()["id"] != second.json()["id"]

Run the suite before the agent writes production code, and keep the suite red until the handler matches the table.

python -m venv .venv
source .venv/bin/activate
pip install pytest httpx fastapi
pytest tests/test_idempotency_contract.py -q

You should treat a green suite as the merge gate, not as a vibe check after the agent dumps a file. If a later prompt simplifies the unique index, these tests are the thing that still yells. A rename-only generation must pass the same four assertions, or you did not freeze anything.

Implementation notes for the handler

The FastAPI sketch below is an example, not a measured production service with real traffic or latency numbers. It shows the lock, the fingerprint, and the replay path the tests demand from the handler. Persistence uses SQLite so you can read the rows during review without standing up another datastore. Return JSONResponse with status 201, because FastAPI treats a bare dict as 200 and will fail the replay test.

# example_app/orders.py
# Example handler aligned to contracts/idempotency_v1.md
import hashlib
import json
import re
import sqlite3
import time
import uuid

from fastapi import FastAPI, Header, HTTPException, Request
from fastapi.responses import JSONResponse

app = FastAPI()
DB = "orders_demo.sqlite3"
KEY_RE = re.compile(r"^[A-Za-z0-9._-]{8,128}$")


def canonical_json(obj: dict) -> bytes:
    return json.dumps(obj, sort_keys=True, separators=(",", ":")).encode("utf-8")


def fingerprint(method: str, path: str, key: str, merchant: str, body: dict) -> str:
    prefix = f"{method}\n{path}\n{key}\n{merchant}\n".encode("utf-8")
    return hashlib.sha256(prefix + canonical_json(body)).hexdigest()


def merchant_from(authorization: str | None) -> str:
    if not authorization or not authorization.startswith("Bearer "):
        raise HTTPException(401, {"code": "unauthorized"})
    token = authorization.removeprefix("Bearer ").strip()
    if not token:
        raise HTTPException(401, {"code": "unauthorized"})
    return token  # demo only: token identity equals merchant_id


def key_ok(key: str) -> bool:
    return bool(KEY_RE.fullmatch(key))


def validate_body(body: dict) -> None:
    sku = body.get("sku")
    qty = body.get("qty")
    if not isinstance(sku, str) or not sku:
        raise HTTPException(400, {"code": "invalid_body"})
    if not isinstance(qty, int) or isinstance(qty, bool) or qty < 1 or qty > 99:
        raise HTTPException(400, {"code": "invalid_body"})
    if body.get("currency") != "USD":
        raise HTTPException(400, {"code": "invalid_body"})


@app.on_event("startup")
def init_db() -> None:
    con = sqlite3.connect(DB)
    con.execute(
        """
        CREATE TABLE IF NOT EXISTS idempotency (
          merchant_id TEXT NOT NULL,
          key TEXT NOT NULL,
          fingerprint TEXT NOT NULL,
          status INTEGER,
          body TEXT,
          state TEXT NOT NULL,
          created_at REAL NOT NULL,
          PRIMARY KEY (merchant_id, key)
        )
        """
    )
    con.execute(
        """
        CREATE TABLE IF NOT EXISTS orders (
          id TEXT PRIMARY KEY,
          merchant_id TEXT NOT NULL,
          sku TEXT NOT NULL,
          qty INTEGER NOT NULL,
          currency TEXT NOT NULL
        )
        """
    )
    con.commit()
    con.close()


@app.post("/v1/orders")
async def create_order(
    request: Request,
    idempotency_key: str | None = Header(default=None, alias="Idempotency-Key"),
    authorization: str | None = Header(default=None),
):
    merchant = merchant_from(authorization)
    if not idempotency_key or not key_ok(idempotency_key):
        raise HTTPException(400, {"code": "idempotency_key_required"})
    body = await request.json()
    validate_body(body)
    digest = fingerprint("POST", "/v1/orders", idempotency_key, merchant, body)
    con = sqlite3.connect(DB, timeout=5, isolation_level="IMMEDIATE")
    try:
        row = con.execute(
            "SELECT fingerprint, status, body, state FROM idempotency WHERE merchant_id=? AND key=?",
            (merchant, idempotency_key),
        ).fetchone()
        if row:
            stored_fp, status, stored_body, state = row
            if stored_fp != digest:
                raise HTTPException(409, {"code": "idempotency_key_reuse"})
            if state == "completed":
                return JSONResponse(status_code=status, content=json.loads(stored_body))
            raise HTTPException(409, {"code": "idempotency_in_progress"})
        try:
            con.execute(
                "INSERT INTO idempotency(merchant_id, key, fingerprint, state, created_at) VALUES (?,?,?,?,?)",
                (merchant, idempotency_key, digest, "in_progress", time.time()),
            )
            con.commit()
        except sqlite3.IntegrityError:
            raise HTTPException(409, {"code": "idempotency_in_progress"})
        order_id = str(uuid.uuid4())
        payload = {
            "id": order_id,
            "sku": body["sku"],
            "qty": body["qty"],
            "currency": body["currency"],
        }
        con.execute(
            "INSERT INTO orders(id, merchant_id, sku, qty, currency) VALUES (?,?,?,?,?)",
            (order_id, merchant, body["sku"], body["qty"], body["currency"]),
        )
        con.execute(
            "UPDATE idempotency SET status=?, body=?, state=? WHERE merchant_id=? AND key=?",
            (201, json.dumps(payload), "completed", merchant, idempotency_key),
        )
        con.commit()
        return JSONResponse(status_code=201, content=payload)
    finally:
        con.close()

Keep the helpers boring, and resist the extra cache layer the model will try to insert on the second pass. Agents love Redis, bloom filters, and eventual uniqueness, but your contract does not ask for those tools. If the generator adds a background sweeper for key TTL, you amend the markdown file first and then accept the code.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Free model access and a free server option can host the agent and pytest runner on a machine you can throw away. You can point that server at this repository while you iterate on the handler. The contract, decision table, and tests remain the actual method even if you never use that sandbox.

Results from the walkthrough

This is a lab walkthrough, not a customer study, so there are no conversion rates or benchmark numbers to quote. When you generate a handler without the contract, you should expect four failure modes during the first review.

In-memory maps that die on process restart and duplicate the order after deploy.
Keys that are global across merchants, so two shops collide on order-1.
Replays that re-execute side effects because only the key was stored, not the response.
Same-key POST with a changed qty that silently updates the row instead of returning 409.

After the tests exist, those four paths fail the suite before you spend time reading the generated diff. That is the result that matters for this case: the agent can still write ugly code, but it cannot quietly change replay.

Limitations

Do not copy this exact contract onto every POST route in your service without reading the failure modes. Public POST /search endpoints should not require keys, and streaming uploads need byte-range rules this demo never states. SQLite IMMEDIATE transactions are a teaching lock, not a multi-region lease you can run across two continents. Five-second in-progress waits will hurt if order creation calls a slow payment network during the critical section.

You should not treat a cached 201 body as a substitute for a payment idempotency key at your processor. If a crash happens after the charge and before the row commit, you still need a processor key that matches yours. This article does not claim durability across disks, regions, or vendors, and you should not infer that claim.

Skip this approach when you already have an exactly-once outbox with a proven unique constraint and a client library. Also skip it when your orders are command messages on a broker that already has a documented dedupe window. Layering a second key model on top of that design usually creates split-brain replays that no dashboard will explain.

Lessons learned

Write the table first, because if you cannot fill the fingerprint column you are not ready for generated code. Keep the key namespace per merchant, store the response, and fail closed on reuse with a different body. Run the four tests on every generation attempt, including the attempt that only renames variables for style.

Your job is not to prompt until the handler looks elegant in a screenshot of the editor. Your job is to keep POST /orders boring under retry, process restart, and two merchants who both like the key checkout. Freeze that boredom in git, then let the agent fill in the functions against tests you already trust.

Case Study: Freeze the Error Taxonomy Before an Agent Writes the Error Mapper

Dakota Liu — Mon, 14 Sep 2026 21:49:07 +0000

Six slightly different 4xx answers for the same failure is exactly what you get when an agent writes your error mapper without a frozen taxonomy. Fixing it costs one contract file, one hash check, and a test that runs in under a second. This case study walks a small project end to end: four known failure modes, one generated mapper, and the freeze step that keeps the generator honest. Treat the listings below as a reference implementation you should run in your own repository before you adopt any of it.

Background: the mapper is downstream of decisions nobody wrote down

The service in this example sits behind a queue worker and answers three clients: a web app, a mobile app, and a partner integration. Each client decides independently whether to retry, which means each client needs a stable answer to two questions: is this failure retryable, and what should we show the user? Nobody had written those answers down, because they lived in the heads of the people who wrote the original catch blocks.

When an agent generates a mapper from that state, it produces something plausible and inconsistent. It picks 422 for one validation failure and 400 for its sibling, marks a rate-limit response non-retryable because the word "limit" reads like a permanent condition, and forwards err.message into the response body where the partner client can read a stack trace. Every one of those choices is defensible in isolation and wrong in aggregate.

The problem is not that the agent is careless. The problem is that you handed it a task with no acceptance criteria, so it optimized for readability instead of contract stability.

Goal: freeze the taxonomy, not the implementation

The freeze here is narrow and deliberate. You commit a machine-readable list of error codes with their HTTP status, retry semantics, message key, and log level; that file is the contract. You then let the agent write, rewrite, or throw away the mapper as often as it likes, because the mapper is now a pure function of a file it cannot edit.

Two properties make this work in practice.

Every consumer-visible field is present in the frozen file, so the mapper has no room to invent policy.
The frozen file is hash-checked in CI, so a helpful regeneration pass cannot quietly "improve" your taxonomy.

The artifact: `contract/error-cases.json`

Keep the file boring and dependency-free. JSON parses everywhere, diffs cleanly in review, and needs no YAML loader in your test runner.

{
  "version": 1,
  "frozen_at": "2026-09-15",
  "cases": [
    { "code": "AUTH_TOKEN_EXPIRED", "http": 401, "retryable": false, "message_key": "errors.auth.expired", "log_level": "info" },
    { "code": "RATE_LIMITED",       "http": 429, "retryable": true,  "message_key": "errors.rate_limited", "log_level": "warn" },
    { "code": "UPSTREAM_TIMEOUT",   "http": 504, "retryable": true,  "message_key": "errors.upstream.timeout", "log_level": "error" },
    { "code": "VALIDATION_FAILED",  "http": 422, "retryable": false, "message_key": "errors.validation", "log_level": "info" }
  ]
}

Store the digest next to it, and regenerate that digest only through a reviewed pull request:

sha256sum contract/error-cases.json | cut -d' ' -f1 > contract/error-cases.sha256

The contract test that does the actual freezing

This runs on Node 20 or newer with no third-party packages. It asserts conformance rather than implementation, so the agent stays free to restructure the mapper however it prefers.

// test/error-contract.test.ts
import { test } from "node:test";
import assert from "node:assert/strict";
import { readFileSync } from "node:fs";
import { createHash } from "node:crypto";

const contract = JSON.parse(readFileSync("contract/error-cases.json", "utf8"));
const { toProblem } = await import("../src/error-mapper.js");

test("contract hash is unchanged", () => {
  const digest = createHash("sha256")
    .update(readFileSync("contract/error-cases.json"))
    .digest("hex");
  assert.equal(digest, readFileSync("contract/error-cases.sha256", "utf8").trim());
});

test("every frozen code maps to its exact status and retry flag", () => {
  for (const c of contract.cases) {
    const p = toProblem({ code: c.code, message: c.code });
    assert.equal(p.type, c.code, `type drift for ${c.code}`);
    assert.equal(p.status, c.http, `status drift for ${c.code}`);
    assert.equal(p.retryable, c.retryable, `retryable drift for ${c.code}`);
    assert.equal(p.message_key, c.message_key, `message key drift for ${c.code}`);
  }
});

test("unknown codes never echo internal text to a client", () => {
  const p = toProblem({ code: "ECONNRESET", message: "connect ECONNRESET 10.0.0.7:5432" });
  assert.equal(p.type, "INTERNAL");
  assert.equal(p.status, 500);
  assert.equal(p.detail, undefined);
});

The mapper it constrains is short enough to review by eye, which is the second half of the trick.

// src/error-mapper.ts  (generated; safe to regenerate)
import { readFileSync } from "node:fs";

const contract = JSON.parse(readFileSync("contract/error-cases.json", "utf8"));
const byCode = new Map(contract.cases.map((c: any) => [c.code, c]));

export function toProblem(err: { code?: string; message?: string }) {
  const hit = byCode.get(err.code ?? "");
  if (!hit) {
    return { type: "INTERNAL", status: 500, retryable: false, log_level: "error" as const };
  }
  return {
    type: hit.code,
    status: hit.http,
    retryable: hit.retryable,
    message_key: hit.message_key,
    log_level: hit.log_level,
  };
}

Wiring the generation loop with free model access and a free server

For this case study I drafted the mapper with MonkeyCode's free model access and ran the throwaway harness on its free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Both availability claims are operator-supplied and worth re-checking against the project's current offer page: free model access, a free server option, and an operator-stated free allowance of 10,000,000 tokens as of this writing. Verify the live quota, model list, and terms yourself before you plan any workload around them, and expect those terms to move over time.

The prompt matters more than the model choice here. Keep it explicit about the boundary:

You may edit ONLY src/error-mapper.ts.
contract/error-cases.json and contract/error-cases.sha256 are frozen:
do not edit, reformat, or "fix" them.
Run: node --test test/error-contract.test.ts
Done means: all three tests pass, no new dependencies.

Then run this check before you accept the patch, because a generator that edits two files instead of one is the most common failure in this loop:

git diff --exit-code -- contract/ || { echo "contract touched; reject the patch"; exit 1; }

What the harness blocks (example run, illustrative)

The rows below describe the failure classes this test design is meant to intercept. They illustrate the design, not a benchmark of any particular model.

Failure mode	Symptom without the freeze	Caught by
Status drift	`VALIDATION_FAILED` returns 400 in one handler and 422 in another	per-case status assertion
Retry inversion	`RATE_LIMITED` marked non-retryable, so clients stop backing off	retryable assertion
Silent renames	`errors.auth.expired` becomes `errors.auth.expired_token`	message key assertion
Leak	raw socket error text reaches the partner client	unknown-code test
Taxonomy edits	a helpful agent adds a fifth code mid-task	hash assertion

Decision table: when freezing pays off

Situation	Freeze first?	Why
Three or more independent clients consume your errors	Yes	Retry policy has become a public API
One internal caller, error semantics still being redesigned	No	The contract would change every week
An agent generates code faster than you can review it	Yes	The freeze is your review budget
Error text is user-visible and localized	Yes	Message keys must stay stable for translators

Limitations and who should not use this

The hash check creates real friction. Adding a genuinely new error code now requires a pull request that touches the contract, regenerates the digest, and re-runs the suite; if your taxonomy is still in flux, that friction will push people to bypass the check instead of respecting it.

The tests also only prove conformance. A mapper can pass all three assertions and still log at the wrong level, drop correlation IDs, or leak data through a field the contract never names. Freeze the taxonomy first, then write separate tests for redaction and logging.

Do not use this approach if you cannot run a test suite in CI, if you are mid-redesign of your error semantics, or if your latency budget depends on hand-tuned serialization you would never regenerate. It is also a poor fit for one-off scripts where nothing retries and nobody parses your status codes.

Lessons learned

Freeze the smallest artifact that carries policy, and leave everything else to the generator.
Assertions on consumer-visible fields catch far more than snapshot tests of whole response bodies.
A hash check turns a silent edit into a loud CI failure, which is the entire point.
Keep the frozen file small enough to actually read during review; a 200-line taxonomy gets rubber-stamped.
Re-verify every free-tier claim before depending on it, including allowance, model access, and server terms.

If you want a zero-cost place to run this draft-and-test loop, MonkeyCode's free model access and free server option are what I used for this case study, and a four-case harness like this one is a reasonable first project to try there. Check the current offer page before you commit, then keep the workflow only if the freeze genuinely reduces your review churn.

Case Study: Freeze Webhook HMAC Replay Rules Before an Agent Writes the Receiver

Dakota Liu — Sun, 13 Sep 2026 20:29:23 +0000

You should freeze webhook HMAC replay rules in tests before any coding agent writes the HTTP receiver. Agents invent header names, skip timestamp windows, and accept unsigned bodies when no contract is already failing. This case study walks one small webhook project from a frozen test file through a generated handler. You can keep the same tests on your laptop even if you never use a hosted coding assistant.

Background

Webhook providers retry deliveries, replay delayed events, and sometimes send the same POST long after success. A receiver that only checks a shared secret still accepts captured requests outside the allowed time window. Coding agents trained on mixed tutorials often hash re-serialized JSON instead of the raw request body bytes. They also rename headers to Signature or X-Hub-Signature-256 because those names dominate public code samples.

You need a contract that names the headers, the canonical bytes, the hash algorithm, and allowed clock skew. Without that file, an agent writes something that looks secure in review and still fails under replay. This project stays small on purpose so you can read every failing assertion before a handler exists. The work is a teaching slice, not a billing platform and not a vendor SDK rewrite.

Goal

The goal is a tiny FastAPI receiver that verifies HMAC-SHA256 signatures and rejects stale or future timestamps. You will not ask an agent for handler code until pytest already encodes header names, canonical bytes, and skew. A public callback only helps provider sandboxes; it is not required for the contract tests themselves. Local pytest remains the source of truth even when a remote process later serves the same application module.

Success for this case study means four checks, and none of them is a generated README:

Missing or malformed signature headers fail closed with HTTP 401.
Timestamps older or newer than 300 seconds fail closed with HTTP 401.
Canonical bytes are the ascii timestamp, one dot, and the raw body, never re-serialized JSON.
A valid signature with a fresh timestamp returns HTTP 204 and does not parse the body twice.

The frozen contract

Treat the decision table as the only protocol the agent is allowed to implement in this exercise. If a generated handler invents extra headers or a different payload encoding, you discard that draft immediately. Use a fictional vendor prefix so you do not copy a live provider document into the repository. Pin max_skew_seconds to 300 inside the test module rather than burying the number inside a chat prompt.

Case	`X-Acme-Timestamp`	`X-Acme-Signature`	Body	Clock vs timestamp	Status
valid	`1700000000`	`sha256=` plus matching hex	`{"ok":true}`	0s	204
missing signature	`1700000000`	absent	`{"ok":true}`	0s	401
wrong prefix	`1700000000`	hex without `sha256=`	`{"ok":true}`	0s	401
replay	`1700000000`	valid for that timestamp	`{"ok":true}`	+301s	401
future	`1700000301`	valid for that timestamp	`{"ok":true}`	-301s	401
mutated body	`1700000000`	signature for the original body	`{"ok":false}`	0s	401

Canonical bytes

You must authenticate exactly one byte string: ascii timestamp, one ASCII dot, then the untouched raw body. Prefer hashing timestamp.encode("ascii") + b"." + body, and never call json.dumps before computing the digest. Agents pretty-print JSON by default, and pretty-printed JSON will never match what the provider actually signed. Compare digests with hmac.compare_digest so invalid signatures do not become cheap timing oracles.

Implementation

Write the tests before the route

The pytest module below is the original artifact for this case study, not a sketch you should paraphrase later. Save it as tests/test_webhook_replay.py and run it before app.py exists so collection failures stay honest. Add a tiny stub that raises NotImplementedError if you want collection to succeed while every assertion still fails. That stub is cheaper than asking an agent to invent both the tests and the implementation in one pass.

# tests/test_webhook_replay.py
# Proposed example: pin replay rules before any agent writes the handler.
from __future__ import annotations

import hashlib
import hmac

import pytest
from fastapi.testclient import TestClient

from webhook_app import MAX_SKEW_SECONDS, app

SECRET = b"test-secret-not-for-production"
BODY = b'{"ok":true}'
TS = "1700000000"


def sign(timestamp: str, body: bytes) -> str:
    mac = hmac.new(
        SECRET,
        timestamp.encode("ascii") + b"." + body,
        hashlib.sha256,
    ).hexdigest()
    return f"sha256={mac}"


@pytest.fixture
def client(monkeypatch: pytest.MonkeyPatch) -> TestClient:
    monkeypatch.setenv("WEBHOOK_SECRET", SECRET.decode("ascii"))
    return TestClient(app)


def test_valid_signature_returns_204(client: TestClient, monkeypatch: pytest.MonkeyPatch) -> None:
    monkeypatch.setattr("webhook_app.now_seconds", lambda: 1_700_000_000)
    response = client.post(
        "/webhooks/acme",
        content=BODY,
        headers={
            "Content-Type": "application/json",
            "X-Acme-Timestamp": TS,
            "X-Acme-Signature": sign(TS, BODY),
        },
    )
    assert response.status_code == 204
    assert response.content == b""


def test_missing_signature_header_fails_closed(client: TestClient, monkeypatch: pytest.MonkeyPatch) -> None:
    monkeypatch.setattr("webhook_app.now_seconds", lambda: 1_700_000_000)
    response = client.post(
        "/webhooks/acme",
        content=BODY,
        headers={"X-Acme-Timestamp": TS},
    )
    assert response.status_code == 401


@pytest.mark.parametrize(
    "skew", [MAX_SKEW_SECONDS + 1, -(MAX_SKEW_SECONDS + 1)]
)
def test_timestamp_outside_skew_fails_closed(
    client: TestClient, monkeypatch: pytest.MonkeyPatch, skew: int
) -> None:
    monkeypatch.setattr("webhook_app.now_seconds", lambda: 1_700_000_000 + skew)
    response = client.post(
        "/webhooks/acme",
        content=BODY,
        headers={
            "X-Acme-Timestamp": TS,
            "X-Acme-Signature": sign(TS, BODY),
        },
    )
    assert response.status_code == 401


def test_reserialized_body_does_not_match_signature(
    client: TestClient, monkeypatch: pytest.MonkeyPatch
) -> None:
    monkeypatch.setattr("webhook_app.now_seconds", lambda: 1_700_000_000)
    mutated = b'{"ok": false}'
    response = client.post(
        "/webhooks/acme",
        content=mutated,
        headers={
            "X-Acme-Timestamp": TS,
            "X-Acme-Signature": sign(TS, BODY),
        },
    )
    assert response.status_code == 401

Generate only against failing tests

When you want a model to draft webhook_app.py, keep the prompt narrow and point it at the failing tests. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode provides free model access and a free server option you can use after the contract file already exists. Free model access is enough to draft the verify path; the free server option only matters when a provider sandbox must reach a public callback. Do not paste the production secret into any prompt; inject WEBHOOK_SECRET from the environment at process start.

Proposed local commands, labeled as an unexecuted example:

python -m venv .venv
. .venv/bin/activate
pip install fastapi httpx pytest uvicorn
export WEBHOOK_SECRET='test-secret-not-for-production'
pytest -q tests/test_webhook_replay.py

If pytest cannot import webhook_app, add the stub below and keep every test red until the real module lands. You want import success with assertion failure, not a green suite that never executed HMAC code.

# webhook_app.py — stub so pytest can collect before generation
from fastapi import FastAPI

MAX_SKEW_SECONDS = 300
app = FastAPI()

def now_seconds() -> int:
    raise NotImplementedError

A handler that should pass the frozen table

The FastAPI module below is a proposed example, not production-hardened code. Review the compare_digest call, the raw body path, and the integer timestamp parse before you expose it. Reject non-digit timestamps before HMAC so the handler does not raise five-hundreds on garbage headers.

# webhook_app.py — proposed example that matches tests/test_webhook_replay.py
from __future__ import annotations

import hashlib
import hmac
import os
import time

from fastapi import FastAPI, Header, HTTPException, Request, Response

MAX_SKEW_SECONDS = 300
app = FastAPI()


def now_seconds() -> int:
    return int(time.time())


def _secret() -> bytes:
    value = os.environ.get("WEBHOOK_SECRET", "")
    if not value:
        raise RuntimeError("WEBHOOK_SECRET is missing")
    return value.encode("ascii")


@app.post("/webhooks/acme")
async def acme_webhook(
    request: Request,
    x_acme_timestamp: str | None = Header(default=None),
    x_acme_signature: str | None = Header(default=None),
) -> Response:
    if not x_acme_timestamp or not x_acme_timestamp.isdigit():
        raise HTTPException(status_code=401, detail="invalid timestamp")
    if not x_acme_signature or not x_acme_signature.startswith("sha256="):
        raise HTTPException(status_code=401, detail="invalid signature")

    skew = abs(now_seconds() - int(x_acme_timestamp))
    if skew > MAX_SKEW_SECONDS:
        raise HTTPException(status_code=401, detail="replay window")

    body = await request.body()
    expected = "sha256=" + hmac.new(
        _secret(),
        x_acme_timestamp.encode("ascii") + b"." + body,
        hashlib.sha256,
    ).hexdigest()
    if not hmac.compare_digest(expected, x_acme_signature):
        raise HTTPException(status_code=401, detail="bad mac")

    return Response(status_code=204)

Results

This section does not report production metrics, incident rates, or latency numbers from a live webhook fleet. It records the failure classes the frozen suite is designed to catch when an agent ignores the table. Run the proposed session on the example handler after generation, and treat any extra passing tests as untrusted. Extra tests the agent adds often restate the implementation instead of protecting the canonical byte string.

Expected outcomes when the contract is frozen first:

An agent that HMAC-signs json.dumps(payload) fails test_reserialized_body_does_not_match_signature.
An agent that reads X-Hub-Signature-256 fails test_missing_signature_header_fails_closed and the valid case together.
An agent that skips timestamp checks fails test_timestamp_outside_skew_fails_closed for both directions.
An agent that returns 200 with a JSON error body fails the 401 and 204 status assertions.

That is the practical result of freezing replay rules: you see the invented protocol in red pytest output instead of in a missed incident. Keep the first green run in CI so later refactors cannot quietly switch back to parsed JSON.

Limitations and who should not use this

This approach is a teaching contract for one HMAC header pair and a fixed skew window. It is not a full webhook platform, and it will not replace provider SDKs that already document a different canonical form. Clock skew tests assume one injected now_seconds fixture, so they will not catch a handler that samples the clock twice around a slow body read. Rotate secrets, cap body size, and put network authentication in front of any public callback before you handle money or personal data.

Do not use this pattern when any of the following is true:

You terminate TLS at a proxy that rewrites bodies or collapses duplicate headers.
Your provider signs a different canonical form, such as a hex timestamp or an ordered header list.
You still need idempotency keys, at-least-once deduplication, or ordered event processing.
You cannot keep the shared secret out of prompts, logs, and the git repository.

Lessons learned

Freeze the replay window in pytest before you generate a single route, including the stub that only exists so collection works. Header names, canonical bytes, and skew belong in assertions, not in chat history that an agent can quietly rewrite. A public callback is optional infrastructure after the contract exists, never a substitute for fail-closed tests. If you skip the test file, the agent will still write a handler that looks careful and still accepts a replayed POST.

Case Study: Freeze If-Match Version Tokens Before an Agent Writes the PATCH Client

Dakota Liu — Sat, 12 Sep 2026 18:29:04 +0000

You should freeze optimistic concurrency before an agent writes any PATCH client, because agents default to last-write-wins and quietly drop concurrent edits. A small notes API makes the failure cheap to reproduce: two writers, one version column, and a contract test that rejects a missing or stale If-Match. Once that contract is red on purpose, you can let an agent fill in handlers without inventing timestamps, updated_at comparisons, or silent overwrites. This case study walks one project from background through tests, implementation, results, and the limits of the pattern.

Background: a notes resource two people can edit

The project is a single-resource notes service with GET /notes/:id and PATCH /notes/:id. You store id, body, and an integer version that only the server increments. The product rule is simple and non-negotiable: a write with a stale version must fail with 412 Precondition Failed, and a write with no version token must fail with 428 Precondition Required. Agents rarely infer that pair from a prose ticket, so they ship last-write-wins JSON and call the ticket done.

You are not building a CRDT, an append-only event log, or a view-counter. Those domains want different merge rules, and freezing If-Match on them would be the wrong contract. You are building a document that two humans edit in the same minute, which is the case where lost updates actually hurt. Keep the surface tiny so the frozen tests stay readable and the agent cannot wander into unrelated endpoints.

Goal for this case study

The goal is not a clever client. The goal is a red test suite that already encodes the version window, then an implementation that is not allowed to change the suite. You pin four behaviors before any model writes production code:

GET returns the current body plus a strong ETag derived from version.
PATCH with a matching If-Match succeeds, increments version, and returns the new ETag.
PATCH with a stale If-Match returns 412 and leaves body unchanged.
PATCH without If-Match returns 428 and does not create a default version of 1.

If you skip step four, agents “helpfully” treat a missing header as “just write it,” which reintroduces lost updates for every retrying script. Freeze the 428 so the client is forced to GET first. That is the entire product decision; everything else is wiring.

The contract you freeze first

Save the suite under tests/test_note_if_match.py and run it against an empty app so every assertion fails for the right reason. Label this as a worked example you can copy, not a production dump from an unnamed company.

# tests/test_note_if_match.py
# Worked example: freeze optimistic concurrency before any PATCH client exists.
import json
from pathlib import Path

import pytest
from httpx import Client

BASE = "http://127.0.0.1:8088"
NOTE_ID = "note_42"


def etag_of(resp):
    value = resp.headers.get("etag")
    assert value, "GET/PATCH must send ETag"
    assert value[0] == '"' and value[-1] == '"', "use a strong ETag, not W/"
    return value


@pytest.fixture(scope="module")
def http():
    with Client(base_url=BASE, timeout=2.0) as client:
        yield client


def test_get_returns_strong_etag(http: Client):
    resp = http.get(f"/notes/{NOTE_ID}")
    assert resp.status_code == 200
    body = resp.json()
    assert body["id"] == NOTE_ID
    assert "body" in body and "version" in body
    assert etag_of(resp) == f'"{body["version"]}"'


def test_patch_requires_if_match(http: Client):
    resp = http.patch(
        f"/notes/{NOTE_ID}",
        json={"body": "agent invented a write without a token"},
    )
    assert resp.status_code == 428
    assert http.get(f"/notes/{NOTE_ID}").json()["body"] != (
        "agent invented a write without a token"
    )


def test_stale_if_match_returns_412_and_preserves_body(http: Client):
    current = http.get(f"/notes/{NOTE_ID}")
    original = current.json()["body"]
    stale = '"0"'
    resp = http.patch(
        f"/notes/{NOTE_ID}",
        headers={"If-Match": stale},
        json={"body": "should not land"},
    )
    assert resp.status_code == 412
    assert http.get(f"/notes/{NOTE_ID}").json()["body"] == original


def test_matching_if_match_bumps_version(http: Client):
    current = http.get(f"/notes/{NOTE_ID}")
    token = etag_of(current)
    version = current.json()["version"]
    resp = http.patch(
        f"/notes/{NOTE_ID}",
        headers={"If-Match": token},
        json={"body": "second writer won the race fairly"},
    )
    assert resp.status_code == 200
    assert resp.json()["version"] == version + 1
    assert etag_of(resp) == f'"{version + 1}"'


def test_second_writer_with_old_token_loses(http: Client):
    first = http.get(f"/notes/{NOTE_ID}")
    token = etag_of(first)
    ok = http.patch(
        f"/notes/{NOTE_ID}",
        headers={"If-Match": token},
        json={"body": "writer A"},
    )
    assert ok.status_code == 200
    lost = http.patch(
        f"/notes/{NOTE_ID}",
        headers={"If-Match": token},
        json={"body": "writer B should 412"},
    )
    assert lost.status_code == 412
    assert http.get(f"/notes/{NOTE_ID}").json()["body"] == "writer A"

Run the suite before any handler exists so you watch the intended failures, not a green suite that never executed the contract:

python -m pytest tests/test_note_if_match.py -q
# expected: connection errors or 404s, not a silent pass

What agents invent when the contract is missing

Give the same ticket to an unconstrained coding agent and you will usually get one of these substitutes. None of them preserve the two-writer rule you actually wanted.

Compare updated_at timestamps, which collide inside the same second and depend on clock skew.
Hash the body and treat equal hashes as “no write,” which hides concurrent edits that happened to agree.
Retry the same PATCH on 500, which applies the write twice if the first attempt actually succeeded.
Accept a missing If-Match and default version to 1, which turns the first retry client into last-write-wins.
Emit a weak ETag (W/"3") and then compare it with strong equality in the handler.

A decision table belongs in the repo next to the tests, because prose in a chat window is not a contract. Keep it short enough that a reviewer can reject a pull request against one row.

Incoming `If-Match`	Current `version`	Status	Body mutated?	Next `ETag`
missing	any	428	no	unchanged
`"3"`	3	200	yes	`"4"`
`"2"`	3	412	no	`"3"`
`W/"3"`	3	412	no	`"3"`
`*`	3	412	no	`"3"`

You freeze * as rejected on purpose. Wildcard success is a last-write-wins escape hatch, and agents love escape hatches. If you later need an admin overwrite, add a separate authenticated route instead of loosening PATCH.

Implementation after the suite is red

Only after the tests exist do you allow an agent to write app.py. Seed one note so GET is deterministic, store versions in process memory for the case study, and refuse every write that does not present a strong token. The snippet below is labeled example code for local reproduction, not a claim about a shipped service.

# app.py — example server that satisfies tests/test_note_if_match.py
from fastapi import FastAPI, Header, HTTPException, Response
from pydantic import BaseModel

app = FastAPI()
NOTES = {"note_42": {"id": "note_42", "body": "seed", "version": 1}}

class Patch(BaseModel):
    body: str

def strong_etag(version: int) -> str:
    return f'"{version}"'

@app.get("/notes/{note_id}")
def get_note(note_id: str, response: Response):
    note = NOTES.get(note_id)
    if note is None:
        raise HTTPException(status_code=404)
    response.headers["ETag"] = strong_etag(note["version"])
    return note

@app.patch("/notes/{note_id}")
def patch_note(
    note_id: str,
    payload: Patch,
    response: Response,
    if_match: str | None = Header(default=None, alias="If-Match"),
):
    note = NOTES.get(note_id)
    if note is None:
        raise HTTPException(status_code=404)
    if if_match is None:
        raise HTTPException(status_code=428, detail="If-Match required")
    expected = strong_etag(note["version"])
    if if_match != expected:
        raise HTTPException(status_code=412, detail="stale version")
    note["body"] = payload.body
    note["version"] += 1
    response.headers["ETag"] = strong_etag(note["version"])
    return note

Start the server in one terminal and re-run the suite in another. You want green from the frozen file, not from a rewritten test that lowered 428 to 200.

uvicorn app:app --port 8088 --log-level warning
python -m pytest tests/test_note_if_match.py -q

The client the agent writes later must GET, copy ETag into If-Match, and on 412 re-fetch instead of retrying the same token. Put that in a second test file only after the server suite is green. Do not let the agent invent a retry loop inside the handler; retries belong to the caller that can see a fresh representation.

Where a coding agent belongs in this workflow

The agent is allowed to write app.py and a PATCH client, and it is not allowed to edit tests/test_note_if_match.py. That split is the whole method. If you need a place to run that loop without standing up a local GPU, MonkeyCode’s free model access and free server option can host the agent against the frozen suite. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Keep the product mention in that one role: it is a runner for an already-specified contract, not a source of HTTP semantics. Do not ask the model to choose between 412 and 409, strong versus weak ETags, or whether missing headers should succeed. Those choices are already in the table. If the agent submits a diff that touches the test file, reject the diff even when the new tests are greener, because green tests that encode last-write-wins are how lost updates ship.

Results you should expect from the example

On the seed note, GET returns version: 1 and ETag: "1". A PATCH with If-Match: "1" returns version: 2. A second PATCH that still carries "1" returns 412 and the body stays as writer A left it. A PATCH with no header returns 428. Those four outcomes are the results that matter; they are pass/fail signals, not traffic numbers.

If you deliberately break the handler by commenting out the 428 branch, the require-header test fails first. That is the result you want in review: the contract catches the agent’s most common shortcut before a human has to notice it in staging. If you break only the increment, the bump test fails and the stale-token test may still pass, which tells you the assertions are not duplicates.

Limitations and who should skip this

This pattern does not serialize writes across two resources, so checkout-plus-inventory still needs a transaction or an outbox. Integer versions also do not survive naive client merges; a client that concatenates two bodies after a 412 is doing conflict resolution, which you have not frozen. Weak ETags, CDN filtering of If-Match, and intermediaries that strip unknown headers can still make a correct handler look flaky, so test at the origin, not only through a cache.

Skip this approach when last-write-wins is actually the product, such as analytics counters, presence heartbeats, or true CRDT documents. Skip it when your store cannot atomically compare-and-swap the version column, because a read-modify-write in application memory will lose the race the tests claim to prevent. Skip it when an agent is also allowed to rewrite tests, because then you do not have a freeze, you have a conversation.

Lessons learned

Freeze the version window in tests before any PATCH handler exists, or the agent will pick last-write-wins.
Prefer 428 on a missing token over a friendly default version; missing headers are how retry scripts bypass your rule.
Reject weak ETags and * on this resource so the contract has one comparison, not four.
Keep retries in the client after a fresh GET; do not retry the same If-Match inside the server.
Treat a test-file edit from the agent as a failed review, even when coverage numbers go up.

You now have a complete small project: a seed note, a red-then-green suite, a handler that only mutates on a matching strong token, and a clear list of domains that should not copy it. Pin the table, run the commands, and only then let an agent type the boilerplate that used to invent lost updates.

Case Study: Freeze the Job Status Graph Before an Agent Writes the Worker

Dakota Liu — Fri, 11 Sep 2026 16:01:18 +0000

If you let a coding agent invent job statuses, you will ship retry loops, duplicate exports, and dashboards that cannot explain a row. Freeze the status graph, the legal transitions, and the terminal set in a fixture before any worker code is generated. This case study walks through one small report-export project from that fixture to a failing CI gate. You can copy the full artifact into another repo even if you never touch an agent product.

Background

You are building a tiny async exporter that turns a saved query into a downloadable CSV file. Users click export, a job row appears in the list, and a worker later writes the object. The product surface only needs five statuses, namely queued, running, succeeded, failed, and canceled for operators. That constraint looks obvious until a coding agent starts writing the worker module from a loose prompt.

Coding agents like extra branches because those extra branches look like robustness during a long coding session. They add retrying, partial, uploading, and cleanup without asking whether your API can render those words. You then spend a week mapping invented states back onto a real machine that support already agreed on. Finance also expected one failed state, not a family of almost-failed labels that never reach the invoice.

Goal

The goal is not a clever worker. The goal is a worker that cannot name a status the fixture does not list. You also want every transition checked against an allowlist of edges, including the cancel path from users. Terminal states must stay terminal, so a succeeded job cannot move back to running after a late retry.

You will keep the graph in version control, fail generation when the patch introduces a new string, and only then allow file I/O. The walkthrough uses Node's built-in test runner so you can reproduce the gate without extra frameworks. No production traffic and no invented latency numbers appear below; failing outputs are labeled examples only.

The frozen status graph

Put the contract in contracts/job-status-graph.json so humans and agents read exactly the same lifecycle file. Keep comments out of the JSON document and put the rationale inside the pull request instead.

{
  "job_type": "report_export",
  "states": ["queued", "running", "succeeded", "failed", "canceled"],
  "initial": "queued",
  "terminal": ["succeeded", "failed", "canceled"],
  "forbidden_aliases": [
    "retrying",
    "partial",
    "uploading",
    "cleanup",
    "pending",
    "done",
    "error"
  ],
  "transitions": [
    { "from": "queued", "to": "running", "actor": "worker" },
    { "from": "queued", "to": "canceled", "actor": "user" },
    { "from": "running", "to": "succeeded", "actor": "worker" },
    { "from": "running", "to": "failed", "actor": "worker" },
    { "from": "running", "to": "canceled", "actor": "user" }
  ],
  "side_effects": {
    "succeeded": ["write_csv_object", "emit_done_webhook"],
    "failed": ["emit_failed_webhook"],
    "canceled": ["delete_temp_object"]
  }
}

That file is the entire product conversation about lifecycle for this exporter. If someone wants retrying, they edit the fixture in a separate change, not inside generated worker code. Side effects live on terminal states so the agent cannot attach a webhook to running during a claim.

Contract tests you run first

Create test/job-status-graph.test.mjs and load both the fixture and the worker module the agent is supposed to produce. The worker must export STATES, canTransition, and applyTransition for the suite. You fail the run when those exports drift from the JSON file.

import assert from "node:assert/strict";
import { readFileSync } from "node:fs";
import { test } from "node:test";
import {
  applyTransition,
  canTransition,
  STATES,
} from "../src/job-status.js";

const graph = JSON.parse(
  readFileSync(new URL("../contracts/job-status-graph.json", import.meta.url), "utf8")
);

const src = readFileSync(
  new URL("../src/job-status.js", import.meta.url),
  "utf8"
);

test("worker states equal the frozen graph", () => {
  assert.deepEqual([...STATES].sort(), [...graph.states].sort());
});

test("every declared edge is accepted", () => {
  for (const edge of graph.transitions) {
    assert.equal(canTransition(edge.from, edge.to), true, `${edge.from}->${edge.to}`);
  }
});

test("undeclared edges are rejected", () => {
  const allowed = new Set(graph.transitions.map((e) => `${e.from}->${e.to}`));
  for (const from of graph.states) {
    for (const to of graph.states) {
      if (from === to) continue;
      const key = `${from}->${to}`;
      if (!allowed.has(key)) {
        assert.equal(canTransition(from, to), false, key);
      }
    }
  }
});

test("terminal states do not leave the graph", () => {
  for (const term of graph.terminal) {
    for (const to of graph.states) {
      if (to === term) continue;
      assert.equal(canTransition(term, to), false, `${term}->${to}`);
    }
  }
});

test("applyTransition throws on illegal moves", () => {
  assert.throws(() =>
    applyTransition({ id: "job_1", status: "succeeded" }, "running")
  );
});

test("source does not mention forbidden status aliases", () => {
  for (const alias of graph.forbidden_aliases) {
    assert.equal(src.includes(`"${alias}"`) || src.includes(`'${alias}'`), false, alias);
  }
});

Scaffold the folders, then run the suite before any generation step:

mkdir -p contracts src test
node --test test/job-status-graph.test.mjs

Labeled example: if the agent adds "retrying" to STATES, the first test fails with a deepEqual mismatch on the two arrays. That failure is the entire point of the case. You do not negotiate with the patch; you send the agent back to the fixture.

A worker the tests will accept

Keep src/job-status.js boring on purpose so later I/O cannot hide new branches. The agent may add database calls after the graph is green, but the transition function stays a pure lookup against the fixture. The implementation below is proposed sample code that is complete enough to pass the suite.

import { readFileSync } from "node:fs";

const graph = JSON.parse(
  readFileSync(new URL("../contracts/job-status-graph.json", import.meta.url), "utf8")
);

export const STATES = Object.freeze([...graph.states]);

const ALLOWED = new Set(graph.transitions.map((e) => `${e.from}->${e.to}`));
const TERMINAL = new Set(graph.terminal);

export function canTransition(from, to) {
  if (!STATES.includes(from) || !STATES.includes(to)) return false;
  return ALLOWED.has(`${from}->${to}`);
}

export function applyTransition(job, to, actor = "worker") {
  if (TERMINAL.has(job.status)) {
    throw new Error(`terminal:${job.status}`);
  }
  if (!canTransition(job.status, to)) {
    throw new Error(`illegal:${job.status}->${to}`);
  }
  const edge = graph.transitions.find((e) => e.from === job.status && e.to === to);
  if (edge.actor !== actor) {
    throw new Error(`actor:${actor}`);
  }
  return { ...job, status: to };
}

Notice the actor check on every legal edge. Users cancel running jobs; workers do not cancel them as a retry tactic. If the agent later writes a loop that calls applyTransition(job, "canceled", "worker"), the function throws before any object store call. That is cheaper than discovering the bug in a support ticket after duplicate webhooks.

Decision table for the exporter

Use this table in the pull request so reviewers do not re-litigate names during code review. The table is part of the artifact, not decoration, and it should match the JSON file byte for byte in meaning.

Current status	Event	Next status	Actor	Side effect
queued	worker claims job	running	worker	none
queued	user clicks cancel	canceled	user	delete temp object
running	CSV written	succeeded	worker	write object, done webhook
running	query error	failed	worker	failed webhook
running	user clicks cancel	canceled	user	delete temp object
succeeded	any retry	reject	—	none
failed	any retry	reject	—	none
canceled	any retry	reject	—	none

If you need automatic retries, you add a new non-terminal state in the fixture first, with a numeric cap stored beside that state. Do not let the agent encode "try a few times" as an unnamed loop that still reports running. Operators cannot alert on a loop they cannot name.

Implementation sequence for the case

Work the project in this order so the agent never sees a blank worker as the source of truth.

Commit contracts/job-status-graph.json with the five states and the actor-tagged edges.
Commit the test file, and confirm it fails because src/job-status.js does not exist yet.
Prompt the agent to implement only STATES, canTransition, and applyTransition against that fixture.
Re-run node --test and reject any patch that widens STATES or mentions a forbidden alias.
Only after the graph is green, allow the agent to write the CSV upload and webhook callers.

A useful prompt fragment looks like this, and you should paste the fixture rather than summarizing it in prose:

Implement src/job-status.js so test/job-status-graph.test.mjs passes.
Do not add states, aliases, or transitions that are missing from
contracts/job-status-graph.json. Do not implement object storage yet.

You want the first coding pass to be smaller than your instinct. File I/O is where agents hide extra statuses inside retry helpers, timeout wrappers, and "just in case" cleanup functions. Keep those files out of scope until the graph is locked.

Where a shared free server fits

You can execute every command above on a laptop with Node 20 or later installed. When you want the agent and the tests in one shared workspace, MonkeyCode is one option for that loop. Disclosure: This article was prepared as part of MonkeyCode's product outreach. It is an open-source project with free model access and a free server option, which is enough to keep the fixture, the tests, and the worker patch in a single run without standing up your own GPU box.

Do not treat that server as a production job queue or as the system of record for export rows. Use it to generate src/job-status.js, run node --test, and reject patches that widen STATES. If the free option is unavailable in your environment, the same fixture still works in GitHub Actions with the same test file.

If you already have the contract folder, try the shared runner only after the tests exist, not from a blank worker prompt.

Results of this case

Labeled example, not a production benchmark: the first agent patch introduced retrying and partially_written, and the deepEqual test failed immediately on sorted arrays. The second patch kept five states but allowed failed -> running, and the terminal test failed on that illegal edge. The third patch matched the graph, after which you could permit object-store work without renaming the lifecycle.

You did not measure tokens, wall clock time, or model quality here, because those numbers would be theater without a logged run under a fixed harness. What you did measure is binary and local: either STATES equals the fixture or CI stays red. That is the only result this method claims, and it is enough to stop duplicate CSV writes.

A second labeled failure is worth keeping in the pull request. The agent added applyTransition(job, "canceled", "worker") inside an error path that was trying to be polite. The actor check threw actor:worker, which taught the reviewer more than a style comment would have taught. Illegal actors are statuses by another name.

Limitations and who should skip this

This approach assumes the lifecycle is small and already agreed among API, UI, and support. A multi-step saga with compensating transactions needs a richer model than five strings and a handful of edges. If your jobs are truly event sourced, freeze the event catalog instead of a status graph, or you will fight the fixture on every replay.

Skip this method when the worker is a one-off script with no public API and no operator dashboard. Also skip it when you cannot name the actor for each edge, because the actor check will become guesswork and then get deleted. Do not use a coding server as the database for job rows, and do not pretend a fixture replaces an enum column.

The graph does not replace database constraints on the status column. You still want a SQL check or a native enum so a hand-written UPDATE cannot invent retrying at two in the morning. Contract tests catch agent patches; they do not catch a tired human in a replica shell.

Lessons

Name statuses in a file the agent cannot casually extend during a coding pass.
Test illegal edges and forbidden aliases, not only the happy path from queued to succeeded.
Treat terminal states as a lock on the row, not as a suggestion the worker may reopen.
Put side effects on terminal edges so webhooks cannot fire twice from a running claim.
Change the fixture in its own commit before you ask an agent for implementation code.

If you already pin clients and list pagination, freeze this job graph next on the smallest exporter you own. The tests stay boring, the dashboard keeps five labels, and the agent stops inventing a career ladder of statuses that nobody can operate.

Case Study: Freeze Cursor Pagination Before an Agent Writes the List Client

Dakota Liu — Wed, 09 Sep 2026 16:17:10 +0000

You should freeze cursor pagination in a contract test before an agent writes any list client. Coding agents routinely invent page numbers, offset parameters, and decodeable cursors that your public API never documented. Those inventions compile, look tidy during review, and then break the first time a real cursor is opaque. A frozen fixture plus a failing contract test will catch that drift before the client file exists.

This case study walks one small invoice list end to end: background, goal, implementation, expected results, and limits. You can strip every product mention and still run the same workflow in your own repository. Treat the code as a labeled local walkthrough, not as a production benchmark or a claim about live traffic.

Background: one list endpoint, not a billing platform

The project is deliberately small. You already have GET /v1/invoices with opaque cursor pagination, a hard maximum limit of 100, and no total count. The response always includes items, next_cursor, and limit. The last page returns next_cursor: null. The cursor is an HMAC blob, not JSON, and clients must never parse it.

The failure mode is boring and expensive. An agent sees the word "list" and writes offset pagination because that is the template it remembers. It then adds a helper that JSON-decodes the cursor so it can "skip bad tokens." Your mocks accept that helper, staging rejects it, and reviewers miss the gap because the tests were generated against the agent's own invented fixture.

You are not redesigning invoices. You are stopping the agent from treating pagination as generic CRUD folklore. That constraint is the whole case, and it is enough work for one article.

Goal: make illegal paging unrepresentable

Before any client code is generated, you want three guarantees that a reviewer can check in minutes.

The response shape is frozen in a fixture the agent cannot rewrite without failing CI.
The client must treat next_cursor as an opaque string or null, never as missing, empty, or parsed JSON.
Query keys such as page, offset, per_page, and starting_after are absent from source, not merely unused at runtime.

If the agent cannot satisfy those rules, the patch is rejected. Green tests that required editing contracts/ do not count as success. The contract is an input to the agent, not an output it is allowed to negotiate.

The frozen artifact

Keep one contract file and one golden response in the repo. Do not let the agent own those paths. The JSON below is the source of truth for this walkthrough.

{
  "endpoint": "GET /v1/invoices",
  "query": {
    "limit": { "type": "integer", "minimum": 1, "maximum": 100, "default": 25 },
    "cursor": { "type": ["string", "null"] }
  },
  "forbidden_query_keys": ["page", "offset", "per_page", "starting_after"],
  "response": {
    "type": "object",
    "additionalProperties": false,
    "required": ["items", "next_cursor", "limit"],
    "properties": {
      "items": { "type": "array" },
      "next_cursor": { "type": ["string", "null"] },
      "limit": { "type": "integer", "minimum": 1, "maximum": 100 }
    }
  },
  "cursor_rules": {
    "opaque": true,
    "may_parse_json": false,
    "empty_string_means": "illegal",
    "missing_field_means": "illegal"
  }
}

Save that document as contracts/invoices_list.json. Then freeze a golden page the client must keep matching.

{
  "items": [
    { "id": "inv_1001", "cents": 4900, "status": "open" }
  ],
  "next_cursor": "opaque_token_not_json",
  "limit": 25
}

Notice that next_cursor is present on every page, including the last page where the value becomes null. Presence is part of the contract. If your real API omits the field instead, freeze omission explicitly and never let the agent invent a third shape during generation.

Implementation: tests the agent is not allowed to edit

Put the tests behind the same path discipline you already use for review. The example is local and unexecuted here; run it on your machine before you trust the output. You will need pytest, jsonschema, and httpx.

# tests/test_invoices_list_contract.py
from __future__ import annotations

import ast
import json
from pathlib import Path

import httpx
import jsonschema
import pytest

ROOT = Path(__file__).resolve().parents[1]
CONTRACT = json.loads((ROOT / "contracts/invoices_list.json").read_text())
GOLDEN = json.loads((ROOT / "fixtures/invoices_list_page.json").read_text())
CLIENT = ROOT / "src/invoices_client.py"


def test_golden_response_matches_frozen_schema() -> None:
    jsonschema.validate(instance=GOLDEN, schema=CONTRACT["response"])
    assert GOLDEN["next_cursor"] is None or isinstance(GOLDEN["next_cursor"], str)
    assert GOLDEN["next_cursor"] != ""
    assert 1 <= GOLDEN["limit"] <= 100


def test_client_source_does_not_invent_paging() -> None:
    source = CLIENT.read_text(encoding="utf-8")
    tree = ast.parse(source)
    banned = set(CONTRACT["forbidden_query_keys"])
    found: set[str] = set()

    for node in ast.walk(tree):
        if isinstance(node, ast.Constant) and isinstance(node.value, str):
            if node.value in banned:
                found.add(node.value)
        if isinstance(node, ast.keyword) and node.arg in banned:
            found.add(node.arg)

    assert not found, f"client invented paging keys: {sorted(found)}"


def test_client_does_not_decode_cursor_payload() -> None:
    source = CLIENT.read_text(encoding="utf-8").lower()
    needles = (
        "json.loads(cursor",
        "json.loads(next_cursor",
        "base64.b64decode(cursor",
        "bytes.fromhex(cursor",
    )
    for needle in needles:
        assert needle not in source, f"client parses an opaque cursor via {needle}"

Add one transport test so the client cannot "pass" by ignoring the network shape.

def test_list_page_rejects_missing_next_cursor() -> None:
    def handler(request: httpx.Request) -> httpx.Response:
        assert "page" not in str(request.url)
        assert "offset" not in str(request.url)
        return httpx.Response(200, json={"items": [], "limit": 25})

    transport = httpx.MockTransport(handler)
    from invoices_client import InvoiceListClient

    client = InvoiceListClient("https://api.example.test", transport=transport)
    with pytest.raises(ValueError, match="missing next_cursor"):
        client.list_page(limit=25)

Run the file before the agent writes anything, and keep the first failure loud.

python -m pip install pytest jsonschema httpx
python -m pytest tests/test_invoices_list_contract.py -q

You should see a missing-file failure first. That is useful. The agent now has a red test and a frozen contract, which is a narrower problem than "write a billing client." A minimal legal client can look like the following once those tests exist.

# src/invoices_client.py
from __future__ import annotations

from typing import Any, Mapping

import httpx

MAX_LIMIT = 100


class InvoiceListClient:
    def __init__(self, base_url: str, transport: httpx.BaseTransport | None = None) -> None:
        self._client = httpx.Client(base_url=base_url, transport=transport, timeout=10.0)

    def list_page(self, *, limit: int = 25, cursor: str | None = None) -> Mapping[str, Any]:
        if limit < 1 or limit > MAX_LIMIT:
            raise ValueError("limit must be between 1 and 100")
        params: dict[str, Any] = {"limit": limit}
        if cursor:
            params["cursor"] = cursor
        response = self._client.get("/v1/invoices", params=params)
        response.raise_for_status()
        payload = response.json()
        if "next_cursor" not in payload:
            raise ValueError("list response missing next_cursor")
        if payload["next_cursor"] == "":
            raise ValueError("empty cursor is illegal; use null on the last page")
        return payload

Decision table: when the agent plan fails

Use this table in review, or encode extra rows as assertions. The point is to fail the plan before the client grows helpers you did not ask for.

Agent change	Why it usually appears	Verdict
Adds `page` or `offset` query params	CRUD template memory	Fail the patch
Treats missing `next_cursor` as last page	Common REST folklore	Fail unless the fixture omits it
JSON-decodes or base64-decodes the cursor	Convenience, "validation"	Fail; cursors stay opaque
Introduces `total` or `has_more`	Dashboard habits	Fail unless the contract adds them
Raises `limit` above 100 in client only	"just for tests"	Fail; clamp must match the API
Rewrites `contracts/` to match the client	Makes tests green	Fail; contracts are input, not output
Replaces `null` last page with `""`	String-only thinking	Fail; empty string is illegal here

Where a bounded coding loop fits

Once the contract is frozen, you can let an agent propose only src/invoices_client.py against the red tests. You do not need a large workspace for that loop, because the task is one endpoint, one schema, and one client class.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server option, which can host that bounded generate-and-test loop while contracts/ stays in your own repository. You still review the diff. You still keep fixtures out of the agent's write set. The environment does not discover missing product rules; it only iterates inside the rules you already froze.

If you use that path, keep the prompt explicit and small: implement InvoiceListClient so tests/test_invoices_list_contract.py passes, and do not edit files under contracts/ or fixtures/.

Results you should expect from the walkthrough

This is a worked example, not a traffic study and not a claim about model quality. After the contract test exists, a typical agent draft fails in one of three ways that the assertions above are built to catch.

It emits page because the prompt contained the word pagination.
It omits next_cursor handling because the first mock returned only items.
It parses the cursor so it can skip tokens the server already signed.

Each of those failures is cheaper than a staging incident with real invoices. When the tests pass, you get a client that can only speak the frozen query keys. You do not get retries, auth, or a full SDK. That scope limit is the result you should want, not a weakness to hide in the write-up.

Re-run the same file after the client exists. You want a boring, repeatable command, not a demo script that only works in one chat transcript.

python -m pytest tests/test_invoices_list_contract.py -q --tb=short

Lessons learned

Freeze response presence, not only types, because null and missing are different last-page stories.
Ban illegal query keys in source, not in a comment, because agents treat comments as optional color.
Opaque cursors need an explicit parse ban, or the client will decode them "just in case."
Put contracts outside the agent's write paths, because a test the agent can edit is only a suggestion.
Keep the first agent task smaller than the domain; one list endpoint is enough to surface the folklore.

The pattern also scales sideways without copying this invoice example. The next frozen field is usually sort stability, then filter names, then the error envelope. Do not freeze all of them in one prompt. Freeze one contract, fail one plan, then stop.

Limitations and who should skip this

Do not use this approach if you do not have a real pagination rule yet. Freezing fiction trains the agent on fiction, and the tests will defend the wrong story. Do not treat AST string bans as a security boundary. They catch sloppy drafts, not adversarial patches, renamed variables, or generated code that builds query keys at runtime.

Skip this if your list API is still offset-based and you have no plan to change it. Skip it for payment-critical paths that need a human-written client and a formal review, not a generated first draft. Skip it if reviewers cannot keep contracts/ off the agent's write list, because the agent will "fix" the fixture instead of the client.

The walkthrough ignores authentication, idempotency, rate limits, and streaming. Those need their own frozen contracts and their own failing tests. A free model run will not invent those rules for you. It will only fit code to the rules you already wrote down, which is exactly why the fixture has to exist first.

Case Study: Freeze Outbound HTTP Policy Before an Agent Writes the Client

Dakota Liu — Tue, 08 Sep 2026 04:38:46 +0000

You should freeze outbound HTTP policy in a checked-in file before any agent writes a client. Agents fill gaps with retries, extra headers, and unbounded timeouts that your production path never approved. A small schema, a validator, and two failing tests catch those inventions before they merge. This case study walks a tiny supplier-lookup service from a frozen contract to rejection of a bad patch.

Background

You are adding a read-only lookup against a supplier catalog for live SKU availability checks. The service is small, and the temptation is to let an agent draft the entire HTTP client. That shortcut usually injects policy the rest of your stack never agreed to, especially around time and retries. You need a gate that fails the patch when the generated client disagrees with a human-owned YAML file.

The failure mode is boring and expensive at the same time for teams shipping generated clients. An agent often adds three retries with exponential backoff because popular HTTP tutorials do that. It may also set a sixty-second timeout or follow redirects across hosts without asking you. None of those choices belong in a latency-sensitive availability check without an explicit written contract.

Goal

You want a repository check that treats HTTP policy as data, not as comments in a prompt. The agent may implement the client, but it may not invent timeouts, retry counts, or extra headers. If the implementation drifts, continuous integration fails on the policy file rather than on production incidents.

Success looks like three concrete outcomes you can reproduce on a laptop without extra infrastructure:

A versioned http-policy.yml file that lists timeout, retries, allowed methods, and allowed headers.
A validator that reads the client module and compares literals against that file.
A pair of tests that fail a plausible agent patch and pass a policy-compliant client.

This is a worked example you can copy, not a claim about one production incident or a measured company rollout.

The frozen policy artifact

Keep the contract tiny so a reviewer can finish it in one sitting without a design review. You are not designing a service mesh or a full resilience framework in this repository. You are preventing an agent from guessing transport behavior while it writes a lookup helper. Unknown keys should fail, because models like adding circuit_breaker fields you never discussed.

# http-policy.yml
service: supplier_availability
protocol: https
allowed_methods:
  - GET
timeout_ms: 800
retries: 0
follow_redirects: false
max_response_bytes: 65536
allowed_request_headers:
  - Accept
  - User-Agent
  - X-Request-Id
denied_request_headers:
  - Authorization
  - Cookie
user_agent: inventory-lookup/1.0
idempotency_header: null

Pair the YAML with a JSON Schema so accidental edits fail before the agent session starts. You should reject extra properties, non-GET methods, and any retry value other than zero. The numeric bounds below are sample constraints for this lookup, not a universal SLA you should copy blindly.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "additionalProperties": false,
  "required": [
    "service",
    "protocol",
    "allowed_methods",
    "timeout_ms",
    "retries",
    "follow_redirects",
    "max_response_bytes",
    "allowed_request_headers",
    "denied_request_headers",
    "user_agent"
  ],
  "properties": {
    "service": { "type": "string", "pattern": "^[a-z_]+$" },
    "protocol": { "enum": ["https"] },
    "allowed_methods": {
      "type": "array",
      "items": { "enum": ["GET"] },
      "minItems": 1,
      "maxItems": 1
    },
    "timeout_ms": { "type": "integer", "minimum": 100, "maximum": 2000 },
    "retries": { "type": "integer", "minimum": 0, "maximum": 0 },
    "follow_redirects": { "type": "boolean", "const": false },
    "max_response_bytes": { "type": "integer", "maximum": 65536 },
    "allowed_request_headers": {
      "type": "array",
      "items": { "type": "string" }
    },
    "denied_request_headers": {
      "type": "array",
      "items": { "type": "string" }
    },
    "user_agent": { "type": "string" },
    "idempotency_header": { "type": ["string", "null"] }
  }
}

Set real timeouts from measured p99 latency of the supplier, then encode that number in YAML. Do not let the model pick timeout=60 because that value appears in generic HTTP snippets.

Implementation: parse the client, do not trust comments

Static comparison beats prompt reminders because comments are not executable and they rot across sessions. You can parse a small Python client with the ast module and look for timeout, headers, and retry helpers. The script below is a complete checker for a constrained client style that uses keyword arguments. It will not understand clever wrappers, which is a limitation you should keep visible.

Parsing calls with ast

# check_http_policy.py
from __future__ import annotations

import ast
import pathlib
import sys

import yaml

POLICY_PATH = pathlib.Path("http-policy.yml")
CLIENT_PATH = pathlib.Path("supplier_client.py")


class PolicyError(Exception):
    pass


def load_policy() -> dict:
    data = yaml.safe_load(POLICY_PATH.read_text())
    if data.get("retries") != 0:
        raise PolicyError("policy retries must stay at 0 for this service")
    return data


def literal_keywords(call: ast.Call) -> dict[str, object]:
    out: dict[str, object] = {}
    for kw in call.keywords:
        if kw.arg is None:
            continue
        if isinstance(kw.value, ast.Constant):
            out[kw.arg] = kw.value.value
        elif isinstance(kw.value, ast.Dict):
            headers = {}
            for k, v in zip(kw.value.keys, kw.value.values):
                if isinstance(k, ast.Constant) and isinstance(v, ast.Constant):
                    headers[str(k.value)] = v.value
            out[kw.arg] = headers
    return out


def check_client(tree: ast.AST, policy: dict) -> list[str]:
    errors: list[str] = []
    retry_names = {"retry", "with_retry", "tenacity", "backoff"}
    for node in ast.walk(tree):
        if isinstance(node, ast.Call) and isinstance(node.func, ast.Name):
            if node.func.id.lower() in retry_names:
                errors.append("client imports or calls a retry helper")
        if isinstance(node, ast.Call) and isinstance(node.func, ast.Attribute):
            if node.func.attr in {"get", "request", "urlopen"}:
                args = literal_keywords(node)
                timeout = args.get("timeout")
                if timeout is None:
                    errors.append("HTTP call is missing a timeout keyword")
                else:
                    timeout_ms = int(float(timeout) * 1000) if float(timeout) < 20 else int(timeout)
                    if timeout_ms != policy["timeout_ms"]:
                        errors.append(
                            f"timeout {timeout_ms}ms != policy {policy['timeout_ms']}ms"
                        )
                headers = args.get("headers") or {}
                for name in headers:
                    if name not in policy["allowed_request_headers"]:
                        errors.append(f"header {name} is not in the allow list")
                    if name in policy["denied_request_headers"]:
                        errors.append(f"header {name} is explicitly denied")
    return errors


def main() -> int:
    policy = load_policy()
    tree = ast.parse(CLIENT_PATH.read_text())
    problems = check_client(tree, policy)
    for item in problems:
        print(f"POLICY_FAIL: {item}")
    return 1 if problems else 0


if __name__ == "__main__":
    sys.exit(main())

A compliant client stays dull on purpose so the checker can see every keyword. You hard-code the timeout from the policy and you refuse to retry failed reads. SKU validation stays in your module so the agent cannot send arbitrary paths to the supplier host.

# supplier_client.py
from __future__ import annotations

import json
import urllib.request

TIMEOUT_S = 0.8
USER_AGENT = "inventory-lookup/1.0"
MAX_BYTES = 65536


def lookup_sku(base_url: str, sku: str, request_id: str) -> dict:
    if not sku.isalnum():
        raise ValueError("sku must be alphanumeric")
    req = urllib.request.Request(
        url=f"{base_url}/availability/{sku}",
        method="GET",
        headers={
            "Accept": "application/json",
            "User-Agent": USER_AGENT,
            "X-Request-Id": request_id,
        },
    )
    with urllib.request.urlopen(req, timeout=TIMEOUT_S) as resp:
        raw = resp.read(MAX_BYTES + 1)
    if len(raw) > MAX_BYTES:
        raise ValueError("response exceeded max_response_bytes")
    return json.loads(raw.decode("utf-8"))

urllib.request.urlopen takes timeout in seconds, so the checker converts that literal into milliseconds. If you later switch the client to httpx, you must extend the visitor instead of hoping the model notices the unit change.

Tests that fail an invented client

Do not review the agent patch by vibe when a fixture can encode the failure. Drop tests next to the checker: one golden client, and one snippet that looks like a typical generated retry wrapper. The invented example is labeled as a fixture, not as code you found in a live outage.

# test_http_policy.py
import ast
import pathlib
import textwrap

import check_http_policy as chk


def test_compliant_client_has_no_policy_errors():
    tree = ast.parse(pathlib.Path("supplier_client.py").read_text())
    policy = chk.load_policy()
    assert chk.check_client(tree, policy) == []


def test_retry_wrapper_is_rejected():
    invented = textwrap.dedent(
        """
        import time
        import urllib.request

        def lookup_sku(base_url, sku, request_id):
            last_error = None
            for _ in range(3):
                try:
                    req = urllib.request.Request(
                        url=f"{base_url}/availability/{sku}",
                        method="GET",
                        headers={"Authorization": "Bearer secret"},
                    )
                    return urllib.request.urlopen(req).read()
                except Exception as exc:
                    last_error = exc
                    time.sleep(0.2)
            raise last_error
        """
    )
    policy = chk.load_policy()
    errors = chk.check_client(ast.parse(invented), policy)
    assert any("timeout" in e for e in errors)
    assert any("Authorization" in e for e in errors)

Commands you run locally

Run the suite before you ask any model to touch supplier_client.py. After the agent returns a patch, run the same commands again and treat POLICY_FAIL as a merge blocker. Keep the policy file out of the writable path you give the agent, or the freeze is theater.

python -m pip install pyyaml pytest
python -m pytest test_http_policy.py -q
python check_http_policy.py

If you want schema validation on the YAML itself, add a separate step with a JSON Schema library you already trust. Do not ask the model to “just confirm the policy looks fine,” because that returns fluent agreement without checking additionalProperties.

Running the review loop on a free coding server

You can run this gate on a laptop, and you should, because the checker has no model in the loop. When you want a model to draft the client, keep the policy file read-only in the prompt. Paste the checker output back as the next message instead of arguing about style in prose.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option for this kind of tight loop. That is enough to iterate on this small lookup service without standing up your own GPU box. If this gate already matches how you review agent patches, you can run the same loop there.

A practical sequence looks like this, and you should keep it boring on purpose:

Commit http-policy.yml and the schema before the agent session starts.
Ask the model only to implement lookup_sku against that file.
Run pytest and check_http_policy.py on the returned tree.
If the checker prints POLICY_FAIL, send that stdout back and refuse to expand scope.

Do not ask the model to improve reliability as a vague side quest during the same session. That phrase is how retries and extra headers appear in otherwise tidy patches. Your policy already defined reliability as failing fast under 800ms with no retries and no extra headers.

Results

This is a worked example, not a production benchmark, and you should treat the outcomes as fixture results. The compliant client produces zero POLICY_FAIL lines when you run the checker against supplier_client.py. The invented retry wrapper fails on the missing timeout keyword and on the Authorization header. The checker does not prove JSON parsing, DNS failures, or TLS behavior, so you still need contract tests against a recorded supplier payload.

What you gain is a review handle that is cheaper than reading a long generated client line by line. A reviewer can open the YAML file, confirm retries stay at zero, and ignore stylistic churn in the Python module. That split matters when agents produce more lines than your team can read before the next standup. You are not measuring latency wins here; you are measuring whether invented transport policy can merge.

Limitations and who should skip this

The AST visitor is brittle on purpose, and you should assume it will miss disguised calls. It will miss timeouts hidden behind helper factories, config objects, or **kwargs unpacking. It will also miss retries implemented as raw loops unless you extend the walk beyond named helpers and missing timeouts. If your HTTP stack is generated from OpenAPI with shared middleware, freeze the middleware config instead of parsing call sites.

You should not use this approach when:

Your client already lives behind a shared SDK that owns retries centrally.
You need multi-origin failover that cannot be expressed as retries: 0.
The agent is allowed to edit YAML and Python in the same patch, which defeats the freeze.
You lack a recorded supplier response, so you would be testing only the policy file.
Your language runtime makes literal extraction from source unreliable without a real compiler API.

Keep secrets out of the client module even when the denied header list looks thorough. That list is a tripwire for generated code, not a threat model for credential handling. If a generated client needs credentials, inject them from the environment in a wrapper you own, and never let the agent print them into fixtures or logs.

Lessons learned

Freeze policy as data before you invite a model to write transport code for a small service. Prompts do not survive the next session, while a YAML file and a checker keep failing the same way. You should fail closed on unknown headers and on any retry helper, because those are the assumptions agents reach for first when a tutorial mentions robustness.

Split ownership along that boundary and write it into the pull request template. Humans edit http-policy.yml. Models may edit supplier_client.py. Continuous integration owns the comparison and prints POLICY_FAIL on drift. If you keep those three roles mixed, you will review invented reliability theater instead of SKU lookup behavior.

When the supplier later needs a longer timeout, change the policy file in its own pull request first. Then re-run the checker and update the client in a follow-up change that cites the new number. That order keeps the contract readable, and it stops an agent from quietly moving your latency budget while it claims to implement the client.