Case Study: Freeze Partial-Success Rules Before an Agent Writes the Bulk Import

Agents will invent an all-or-nothing bulk import unless you freeze per-row outcomes first. You should lock HTTP status mapping, row replay keys, and error indexes before any coding agent writes a handler. This case study walks a small member-import API from a frozen contract through failing tests to a generated implementation. You can reuse the decision table and pytest file on your own service without adopting any particular vendor.

Background

Developer communities still argue whether generated code counts as engineering or as merely casual output. That debate stays noisy, but the failure mode in bulk endpoints is specific and quite boring. You ask an agent for a JSON user import, and it wraps every insert inside one database transaction. The first bad email then rolls back valid rows, and reviewers spend a day picking 400 versus 207.

You do not need a newly released model family to prevent that rollback pattern from shipping. You need a written contract that forbids silent skips, whole-payload rollback, and unordered error lists. The rest of this write-up treats that contract as the project rather than as optional prompt garnish.

Goal

You will freeze a 100-row member import for a fictional workspace called Northwind Crew. The endpoint accepts JSON, not CSV, so parsing stays out of scope for this pass. Each row carries a client-supplied row_key so retries can target the same logical member without duplicating records.

Success for this case study means three outcomes you can assert in CI:

1. Malformed payloads fail closed with HTTP 400 and with no per-row result array.
2. Mixed valid and invalid rows return HTTP 207 with stable, request-order results.
3. Replaying a successful row_key returns unchanged together with the original member id.

You will not build auth, email sending, or an async job graph during this case study. Those concerns belong in other specs, and mixing them here is how agents quietly expand scope.

The frozen contract

Write this file first and treat it as read-only during generation. If a later prompt disagrees with the table, the table wins and the prompt is wrong.

HTTP mapping

HTTP 207 Multi-Status is the frozen choice for a well-formed payload that still contains rejected rows. You use 200 only when every row is created or unchanged. You use 400 only when the payload cannot be split into rows, which means no partial inserts are allowed.

Decision table

• Empty rows array → HTTP 400, { "error": "malformed_payload" }, no writes.
• Any row missing row_key → HTTP 400, same error body, no writes.
• Duplicate row_key values inside one request → HTTP 400, no writes.
• More than 100 rows → HTTP 400, no writes.
• All rows valid with new emails → HTTP 200, every result created, insert each member.
• All rows valid and already imported under the same keys → HTTP 200, every result unchanged.
• Mix of valid rows and invalid emails → HTTP 207, insert only valid rows.
• Email already owned by a different row_key → that row is rejected with email_taken; siblings still proceed.
• Every well-formed row rejected → still HTTP 207, because the payload itself was parseable.
• Unknown JSON fields on a row → ignore them; extra keys must not trigger 400.

status values are exactly created, unchanged, and rejected. Agents often add success, ok, and failed, so those strings are banned in the spec. Allowed reject codes are only invalid_email, email_taken, and display_name_blank.

Result object

{
  "results": [
    {
      "row_key": "invite-014",
      "status": "created",
      "id": "mem_8f2a",
      "error": null
    },
    {
      "row_key": "invite-015",
      "status": "rejected",
      "id": null,
      "error": "invalid_email"
    }
  ]
}

error is null on created and unchanged. Result order must match request order, even when emails would sort differently. You freeze both rules because generated serializers like to group failures at the bottom.

Storage gates the agent must not invent

CREATE UNIQUE INDEX members_email_uidx ON members (email);
CREATE UNIQUE INDEX members_row_key_uidx ON members (row_key);

Unique indexes, not prompt text, are what make email_taken true under concurrent imports. If the generated migration omits either index, you reject the patch before you look at handler style.

Reproducible test plan

Save the tests before you open a chat with any coding agent. The tests are the original artifact, and generation is only a way to make them pass.

# test_bulk_import.py
# Labeled example: wire client to your local app fixture.
# Do not treat this helper as a production SDK.
import json

MAX_ROWS = 100

def post_import(client, payload):
    return client.post(
        "/v1/members:import",
        data=json.dumps(payload),
        headers={"Content-Type": "application/json"},
    )

def test_empty_array_is_malformed(client):
    res = post_import(client, {"rows": []})
    assert res.status_code == 400
    assert res.json()["error"] == "malformed_payload"

def test_duplicate_row_key_is_malformed(client):
    res = post_import(
        client,
        {
            "rows": [
                {"row_key": "a", "email": "[e-mail]", "display_name": "Ann"},
                {"row_key": "a", "email": "[e-mail]", "display_name": "Bea"},
            ]
        },
    )
    assert res.status_code == 400
    assert res.json()["error"] == "malformed_payload"

def test_over_max_rows_is_malformed(client):
    rows = [
        {
            "row_key": f"k{i}",
            "email": f"u{i}@example.com",
            "display_name": f"U{i}",
        }
        for i in range(MAX_ROWS + 1)
    ]
    res = post_import(client, {"rows": rows})
    assert res.status_code == 400

def test_mixed_batch_returns_207_in_request_order(client):
    res = post_import(
        client,
        {
            "rows": [
                {"row_key": "ok-1", "email": "[e-mail]", "display_name": "Ok"},
                {"row_key": "bad-1", "email": "not-an-email", "display_name": "Bad"},
            ]
        },
    )
    assert res.status_code == 207
    results = res.json()["results"]
    assert [r["row_key"] for r in results] == ["ok-1", "bad-1"]
    assert results[0]["status"] == "created"
    assert results[0]["id"]
    assert results[1]["status"] == "rejected"
    assert results[1]["error"] == "invalid_email"

def test_replay_of_created_row_is_unchanged(client):
    body = {
        "rows": [
            {"row_key": "ok-2", "email": "[e-mail]", "display_name": "Rep"}
        ]
    }
    first = post_import(client, body)
    second = post_import(client, body)
    assert first.status_code == 200
    assert second.status_code == 200
    assert first.json()["results"][0]["id"] == second.json()["results"][0]["id"]
    assert second.json()["results"][0]["status"] == "unchanged"

def test_email_taken_rejects_only_that_row(client):
    post_import(
        client,
        {
            "rows": [
                {"row_key": "owner", "email": "[e-mail]", "display_name": "Own"}
            ]
        },
    )
    res = post_import(
        client,
        {
            "rows": [
                {"row_key": "other", "email": "[e-mail]", "display_name": "Oth"},
                {"row_key": "fresh", "email": "[e-mail]", "display_name": "New"},
            ]
        },
    )
    assert res.status_code == 207
    results = res.json()["results"]
    assert results[0]["status"] == "rejected"
    assert results[0]["error"] == "email_taken"
    assert results[1]["status"] == "created"

Run the suite before generation so you watch it fail for the right reason:

pytest test_bulk_import.py -q

You should see fixture errors or HTTP 404 responses, not a green suite. A green run at this stage usually means the client pointed at the wrong app process.

A manual curl check belongs beside pytest, because reviewers often trust a single happy path too early:

curl -sS -D - -o body.json \
  -H 'Content-Type: application/json' \
  -d '{"rows":[{"row_key":"ok-1","email":"[e-mail]","display_name":"Ok"},{"row_key":"bad-1","email":"not-an-email","display_name":"Bad"}]}' \
  http://127.0.0.1:8000/v1/members:import

You expect status 207, two results objects, and only one new row in the members table. If curl shows 200 with a mixed batch, the handler violated the frozen mapping even if unit tests were skipped.

Implementation workflow

Keep the agent inside a tight loop with the spec file as the only source of HTTP truth. You paste the decision table, the result schema, and the failing tests in that order. You forbid new status strings, extra endpoints, CSV parsers, and background jobs. You ask for the smallest handler that makes pytest pass and then you stop.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you want that red-green loop on a throwaway machine, MonkeyCode's free model access and free server option are enough to run the tests without standing up your own inference box. You still read every INSERT the agent emits, because free inference does not freeze the contract for you.

A prompt that stays honest looks like this:

Implement POST /v1/members:import against spec.md and test_bulk_import.py.
Do not add routes, status strings, or jobs that the spec does not name.
Stop when pytest is green. Do not improve HTTP mappings.

A tiny classifier you can compare against generated output:

# Labeled example, not production code.
ALLOWED_STATUS = {"created", "unchanged", "rejected"}
ALLOWED_ERRORS = {"invalid_email", "email_taken", "display_name_blank"}

def classify_http(results):
    if any(row["status"] == "rejected" for row in results):
        return 207
    return 200

You reject a patch that returns 200 while any row is rejected. You also reject HTTP 409 on a single taken email, because that status implies the whole batch failed. A 500 for invalid_email is equally wrong, since validation is a defined reject code rather than an operational crash.

Results

On a frozen spec, the usual agent mistakes become visible in one pytest run instead of in a staging incident:

• Whole-batch transactions that roll back the valid rows beside one bad email.
• failed or error used as status instead of the frozen rejected token.
• Result arrays sorted by email, which breaks client retry maps keyed by index.
• HTTP 400 used for invalid_email on a well-formed mixed batch.
• Missing unchanged on replay, which inserts a second member for one row_key.

You do not need a public leaderboard score to call those defects. The tests already name them in plain assertions. After the handler is green, you add one manual check: two parallel replays of the same row_key must still yield one member id. That race sits outside the pytest file above, so you either add a lock test or you document it as known debt.

Lessons learned

• Freeze HTTP mapping for mixed outcomes before you freeze column types or ORM names.
• Per-row row_key is not optional if clients will retry from flaky mobile networks.
• Three reject codes beat an open string field that agents will extend forever.
• Request-order results are part of the contract, not a pretty-print detail for logs.
• All-or-nothing imports are a product decision; encode them only when stock or money requires one transaction.

Limitations and who should skip this

This approach is wrong for ledger lines, seat billing, or inventory reservation that must commit as one transaction. Partial success would leave money or stock inconsistent, and HTTP 207 would hide that split. You should also skip this pattern for multi-megabyte CSV drops, because those need a job resource rather than a synchronous import handler.

Do not send a coding agent at this endpoint if you cannot read the INSERT path in review. Unique indexes, not prompt wording, decide email_taken when two requests collide. If your team cannot add those constraints, freeze the spec anyway and wait for a human migration before generating handlers.

If you already have a bulk importer in production, export one day of payloads and replay them against these tests before you generate anything new. Historical clients often omit row_key, and that single gap will 400 every legacy caller overnight. Paste the table into the next agent session before you paste the feature prompt, and keep the HTTP mapping out of the model's improvisation range.