DEV Community: CivicDataForge

Your Agent Saved the File. Who Checked the Result?

Bryan Williams — Sun, 27 Sep 2026 23:45:53 +0000

The tool call succeeds. The file exists. The agent says, “Done.”

Then someone opens it and finds the wrong total.

Nothing crashed. The system answered a smaller question than the customer asked.

“Did the file write finish?” and “Does the delivered report meet the request?” are different checks. This tutorial builds a small example you can run without a model, then shows how the same distinction survives an agent handoff and a process restart.

Start with something you can check yourself

Our fictional inventory task asks for revision 3 of one report:

{
  "report_id": "stock-handoff-001",
  "revision": 3,
  "rows": [
    {"item": "bolts", "units": 7},
    {"item": "washers", "units": 9}
  ]
}

The delivered file contains those rows, but says "total_units": 13.

The write operation can genuinely succeed. The report is still wrong: 7 + 9 is 16. Retrying the same file write successfully will not fix the arithmetic.

Here is the useful habit: define the completion test before letting the agent announce completion.

For this task, that means the exact report ID, revision, item quantities and total. The expected rows come from the trusted task input, not from the generated file asking to be trusted.

Check the delivered bytes, not the agent's description

This is a deliberately small report verifier. Save it as verify-report.mjs. It is an original teaching example, not an agent runtime or a security boundary for arbitrary uploads.

import { createHash } from 'node:crypto';
export function verifyReport(bytes, expected) {
  const sha256 = createHash('sha256').update(bytes).digest('hex');
  let report;
  try { report = JSON.parse(bytes.toString('utf8')); }
  catch { return { ok: false, reasons: ['invalid_json'], sha256 }; }
  const reasons = [];
  if (!report || typeof report !== 'object' || Array.isArray(report)) {
    return { ok: false, reasons: ['invalid_report'], sha256 };
  }
  if (report.report_id !== expected.report_id || report.revision !== expected.revision) reasons.push('wrong_scope');
  const rows = report.rows;
  if (!Array.isArray(rows) || rows.length > 100 || rows.some(row =>
    !row || typeof row.item !== 'string' || !Number.isSafeInteger(row.units) || row.units < 0)) {
    return { ok: false, reasons: [...reasons, 'invalid_rows'], sha256 };
  }
  const ids = rows.map(row => row.item);
  if (new Set(ids).size !== ids.length) reasons.push('duplicate_item');
  // expected comes from the task's trusted input, not the generated report.
  if (rows.length !== expected.rows.length || expected.rows.some(want =>
    !rows.some(row => row.item === want.item && row.units === want.units))) reasons.push('wrong_rows');
  const computed = rows.reduce((sum, row) => sum + row.units, 0);
  if (!Number.isSafeInteger(computed) || !Number.isSafeInteger(report.total_units) || report.total_units !== computed) reasons.push('wrong_total');
  return { ok: reasons.length === 0, reasons, sha256 };
}

The hash identifies the bytes that were checked. It does not make them correct. The comparisons do the checking; the hash lets a later step identify the same artifact.

Try it in a second file, demo.mjs:

import assert from 'node:assert/strict';
import { verifyReport } from './verify-report.mjs';

const expected = {
  report_id: 'stock-handoff-001', revision: 3,
  rows: [{item: 'bolts', units: 7}, {item: 'washers', units: 9}]
};
const bytes = total => Buffer.from(JSON.stringify({
  ...expected, total_units: total
}));

assert.deepEqual(verifyReport(bytes(13), expected).reasons, ['wrong_total']);
assert.equal(verifyReport(bytes(16), expected).ok, true);
console.log('Wrong result rejected; corrected result accepted.');

Run node demo.mjs. No API key, paid service or model is involved.

In a real file workflow, replace the generated buffer with readFileSync(theDeliveredPath). Do not verify one in-memory object and assume a different saved file contains it.

Now interrupt the agent

The dangerous handoff is a paragraph that says, “The export worked; finish up.” It loses the distinction between the successful write and the failed report.

A useful checkpoint carries unfinished work explicitly:

{
  "subject": "inventory-report:stock-handoff-001:revision-3",
  "completed": false,
  "last_verified_problem": "wrong_total",
  "next_action": "Re-read the task scope and delivered file; repair the total; verify the resulting bytes"
}

Also retain the exact expected input, artifact reference and verification receipt in private state. The next process must re-read the file; the checkpoint describes what was observed before interruption, not what must still be true now.

That is recovery: preserving what remains to be proved, not preserving confidence.

A packaged-runtime test of that handoff

I work on Living Stack. For this tutorial, I ran a new synthetic inventory task through the single-agent component of the exact Complete Local bundle currently identified by its commerce metadata. I used its private evaluation rights, not a new customer purchase, and made no payment or model request.

This was an actual local MCP client/server run, not a transcript written to resemble one:

Step	Observed result
Record a successful write with only a tool-response receipt	Completion claim remained `UNRESOLVED`
Read the file and record the verifier's wrong-total result	Failed verification did not support completion
Save a checkpoint, close the server, start a new server	Same checkpoint and scoped context were recovered
Repair the file and verify its saved bytes	Correct report passed the small verifier
Offer that receipt for a different report	Claim remained `UNRESOLVED`
Offer it for the exact verified subject	Claim returned `PASS`

The teaching verifier passed 12 local cases, including wrong revisions, duplicate items, different rows with the same total, malformed input and valid zero quantities. That is evidence for this bounded example, not a universal claim about agents.

Deeper engineering: what each layer is allowed to conclude

Protocol success is the start of interpretation

An MCP response and a tool's own result have different meanings. The MCP tools specification distinguishes protocol failures from tool-execution errors. Even a successful tool result still needs interpretation against the user's acceptance condition.

A file-write receipt supports “the write operation completed.” It does not support “the report is correct,” “the recipient received it,” or “the customer accepted it.” Each larger statement needs evidence for its additional boundary.

Bind evidence to identity, revision and bytes

A passing receipt for revision 2 cannot settle revision 3. A passing receipt for another report cannot settle this report. A hash for an earlier file cannot settle bytes changed after verification.

For mutable destinations, re-read at the consequential handoff or use a versioned immutable object. Where the remote service supports version preconditions, use them to prevent an intervening write from silently replacing the verified version. An old green check should never become permanent permission to say “done.”

A receipt store is not an oracle

Living Stack checks the recorded evidence structure and bindings. It does not independently open every referenced artifact or determine that the host told the truth. If a host labels an invented reference “verification,” the label alone is not trustworthy ground truth.

The host therefore needs a real verifier, a trusted expected input and an explicit freshness policy. For higher-risk work, use an independently controlled downstream readback. Signing a receipt establishes which key signed it; it does not establish that an external event happened.

Recovery does not renew authority

A recovered checkpoint should not restart payments, publication or account changes merely because yesterday's state mentions them. Reconcile present authorization, destination and scope before acting. Keep expensive or external actions outside this local illustration.

Keep the completion target small enough to be true

Our final claim is that specific local report bytes match revision 3's contract. It is not proof of email delivery, a customer workflow, Team activation or the correctness of all inventory data.

This precision makes the result stronger: another person can reproduce exactly what passed and what would make it fail.

Where to use this tomorrow

Choose one task where an agent currently reports success after a tool call. Write its acceptance condition in one sentence. Add a verifier at the actual destination. Then interrupt the workflow halfway through and check whether the next process knows what still needs verification.

You do not need Living Stack to adopt that method. If you want its local session, evidence and checkpoint machinery, the Complete Local product page describes the paid package and boundaries; hosted checkout is a separate buyer-authorized step. This article does not distribute the runtime or require a purchase to run its small example.

The question before “done” is not just whether the tool worked. It is whether the evidence reaches the thing the user actually asked for.

Disclosure: AI-researched and AI-written, with original example code. The example tests and packaged local handoff were executed for this article. The packaged demonstration was owner evaluation with synthetic input, not an external customer or paid activation.

The Screen Is Correct. Can the Purchase Still Happen Twice?

Bryan Williams — Sun, 27 Sep 2026 05:30:01 +0000

A practical guide to retries, quote IDs, and the boundary between creating an order and charging for it.

A customer clicks Buy.

Your server creates the order. Before the confirmation reaches the browser, the connection drops.

The customer sees an error and tries again.

Should your server create another order?

That is the problem behind idempotency. The word sounds more complicated than the question: Can we repeat this request without repeating its effect?

In my previous post, I used a revision counter to stop an old response from updating a newer screen. Mike Dabydeen raised the next boundary: what happens when that screen leads to a purchase?

He is right to separate the two. A correct screen does not prove a correct transaction.

Let's start with the practical model, then open up the database and payment-provider layers.

Start here: three questions, three protections

Question	Protection
Does this response still belong on this screen?	Client revision check
May this customer purchase this exact offer?	Server-side validation and authorization
Has this purchase attempt already created an order?	Durable idempotency record and an atomic database transaction

None replaces the others.

Disabling a button can reduce accidental double-clicks. Aborting a browser request can stop the browser waiting. Neither establishes that the server stopped working.

The browser controls what it displays. The server must control what it changes.

Give the offer and the attempt different names

Imagine buying one property report.

The server creates a quote containing the customer, the exact report scope, the price, the currency, and an expiry time. The browser receives its identifier: quote_123.

When the customer chooses to purchase, the browser creates a second identifier: an idempotency key for that logical purchase attempt.

The two identifiers answer different questions:

Quote ID: Which offer are you accepting?
Idempotency key: Is this another delivery of the same purchase attempt?

An illustrative request looks like this:

POST /purchases
Idempotency-Key: 92ba0d47-46e2-4cd3-988c-9a95f93cc6ee
Content-Type: application/json

{"quote_id":"quote_123"}

Notice what the request does not get to decide: the authenticated customer, the report scope, or the price.

The server obtains identity from authentication and looks up its own quote. Knowing a quote ID is not permission to use it. Server-side authorization must connect the acting customer to the permitted transaction, not simply accept a browser's claim. OWASP's transaction authorization guidance explains this boundary.

For this example, quotes are immutable: changing the offer creates a new quote.

Keep the key when retrying

Generate the attempt key once, before the first submission. Keep it for retries of that same attempt.

If you generate a fresh key every time you call fetch, the server sees a new attempt every time. You have given the retry a new identity.

A changed offer needs a new attempt. A timeout does not.

If the page reloads, the application needs a way to recover the pending attempt and check its status. An in-memory variable alone cannot survive that reload.

What the server should remember

The server needs durable evidence connecting:

authenticated customer + operation + attempt key
    -> original request + original result

For our deliberately small example, the only accepted request field is quote_id. That makes request comparison straightforward.

The rules are:

Same customer, same key, same quote: return the stored creation result.
Same customer, same key, different quote: reject the conflicting request.
New key: validate the quote before creating anything.
Another customer's quote: do not reveal or purchase it.

There is one additional business rule: one quote may create one order.

That matters because idempotency cannot know that two different keys were supposed to describe the same intent. A unique constraint on the order's quote ID prevents a new key from creating a second order for this single-use offer.

If your product supports repeat purchases, define the appropriate business identity instead. Do not remove the constraint without replacing the rule it represents.

The race hidden inside “check, then insert”

This looks reasonable:

if not receipt_exists(customer, key):
    order = create_order(quote)
    save_receipt(customer, key, order)

But two requests can both check before either saves anything. Both see “not found.” Both create an order.

There is another failure window: the process can stop after creating the order but before saving the receipt.

The order and its receipt must commit together, or neither should commit.

The runnable example uses Python's standard-library SQLite support. This keeps installation simple while giving us a real database rather than an in-memory dictionary.

Its transaction follows this sequence:

BEGIN IMMEDIATE
  Look for this customer's stored attempt.
  If found:
    Reject a different request, or return the stored result.
  Otherwise:
    Load the quote belonging to this customer.
    Check its expiry using server time.
    Reject an already-consumed quote.
    Insert the order using the server's quote terms.
    Insert the attempt receipt containing the result.
COMMIT

The key database constraints are:

-- In purchase_receipts:
PRIMARY KEY (actor_id, idem_key)

-- In orders:
quote_id TEXT NOT NULL UNIQUE REFERENCES quotes(id)

This receipt table is dedicated to purchases. A shared table handling purchases, refunds, and other mutations would also need an operation namespace.

SQLite allows one writer at a time. BEGIN IMMEDIATE obtains the write transaction before our lookup and insert sequence, so competing writers cannot both pass the empty check simultaneously. This is a database guarantee, not a JavaScript flag or a process-local mutex. A busy database can still reject or time out a writer; the service must handle that outcome explicitly. SQLite transaction documentation.

This example creates a local order, not a payment. Keep that distinction in mind for the next section.

An expired quote can still have a successful receipt

Suppose an order committed at 11:59, the quote expired at noon, and the customer retried at 12:01 after losing the response.

Rejecting that retry merely because the quote expired would hide an existing success.

The example checks for a matching receipt before checking expiry. It can return the original result without accepting a new expired offer.

Authentication still applies. In a system with changing memberships or permissions, current authorization to retrieve the result must apply too. A receipt is not an access-control bypass.

What the example actually tests

The accompanying local suite contains 20 tests. They check the stored database rows as well as the function results, including:

A repeated attempt returns the original order, with one order and one receipt.
Eight concurrent calls on independent connections converge on the same result.
Concurrent different keys cannot consume the same quote twice.
Different customers can use the same key without sharing results.
Unknown, cross-customer, expired, malformed, or price-tampered requests are rejected.
An exception between order creation and receipt creation rolls back both.
An abruptly terminated child process leaves no committed partial order.
A fresh process can replay a previously committed attempt.
Deleting the receipt does not bypass the quote's one-order constraint.

These are service-and-database tests. They do not establish that a browser flow, authentication system, or payment integration works. Those need their own tests.

Going deeper: a payment provider is outside your transaction

Here is the tempting next step:

BEGIN
  create local order
  charge customer at payment provider
  save payment result
COMMIT

The problem is that a local database rollback cannot roll back a remote charge.

The provider can succeed, your process can stop, and your database can lose the record of that success. Holding a database transaction open around the network call does not make the two systems one transaction.

A more resilient design separates intent, execution, and reconciliation:

In one local transaction, create a pending order and a durable payment job.
A worker sends that job using a stable provider idempotency key tied to the payment operation.
Store the provider's identifier and confirmed result.
If the result is uncertain, reconcile that same operation rather than starting a new charge blindly.

The durable job is often called an outbox. It closes the gap between “we committed an order” and “we remembered to schedule its payment.” It does not make the worker's delivery exactly-once. The receiver still needs duplicate protection.

One local order might have several legitimate payment operations over its lifetime. Give each operation a stable identity; do not reuse one key indiscriminately for a charge, refund, and replacement payment.

Unknown is a real state

After a timeout, these statements are not equivalent:

“The payment failed.”
“We have not confirmed the payment outcome.”

The second is sometimes the only truthful statement available.

Represent it explicitly, keep the operation's identity, and provide a status or reconciliation path. Do not invite another purchase while pretending the first definitely failed.

Stripe, for example, documents that some server-error outcomes are indeterminate and warns against blindly switching to a new idempotency key, because the original request may have produced side effects. Stripe's low-level error guidance.

Your provider's key has a lifetime

An idempotency key is not an everlasting promise.

Stripe documents that stored keys may be pruned after they are at least 24 hours old; a subsequent request using a pruned key is treated as new. It also compares request parameters when a retained key is reused. Your application's retry and reconciliation policy must fit the provider's actual contract. Stripe's idempotency documentation.

Keep your own durable business identity and provider reference. Once outside the provider's replay window, reconcile before sending another money-moving request.

Webhooks need the same discipline

A webhook is another delivery of information, not an instruction to repeat every downstream effect.

Verify its authenticity, record its identity, and apply an allowed state transition. Commit the event record and local state change together. If that transition should send an email or grant access, give that action its own durable, duplicate-safe job.

Do not assume arrival order. Stripe explicitly documents duplicate event deliveries and unordered delivery. Distinct events may also refer to the same underlying operation, so event-ID deduplication is only one layer. Stripe webhook guidance.

The contract you should be able to explain

Before shipping, answer these questions in plain language:

What identifies the offer, the purchase attempt, and the resulting order?
Where does the server obtain the customer, scope, price, and currency?
Which records must commit together?
What prevents two concurrent requests from both winning?
What can a retry retrieve after the quote expires?
How long are receipts retained, and what survives their deletion?
What happens when the provider may have succeeded but confirmation is missing?
How does a customer recover after closing the page?

The tutorial intentionally leaves authentication middleware, HTTP error mapping, CSRF protection where applicable, inventory, taxes, provider integration, and fulfillment outside its small local example. Those are additional boundaries to implement and test, not guarantees supplied by an idempotency key.

You do not need every system to have the same architecture. You need each guarantee to have a clear owner.

The revision check protects the screen.

Authorization protects the transaction's meaning and permitted actor.

Atomic state and idempotency protect the retry.

Reconciliation protects the gap between systems.

A reliable purchase flow does not require the network to behave perfectly. It requires the application to remember what it intended, what it committed, and what it still does not know.

Run the complete local example

Use Python 3.10 or newer. Save the two files below in the same folder, then run:

python -m unittest -v test_purchase

The tests create temporary databases and use fictional customers and quotes. They do not contact a payment provider or move money. The example uses only Python's standard library.

purchase.py — local order creation and durable replay

"""Teaching example: atomic local order creation, NOT a payment service."""
import json
import re
import sqlite3
import time
import uuid
from contextlib import closing

SCHEMA = """
CREATE TABLE quotes (
    id TEXT PRIMARY KEY, actor_id TEXT NOT NULL, scope TEXT NOT NULL,
    amount_minor INTEGER NOT NULL CHECK(amount_minor >= 0),
    currency TEXT NOT NULL, expires_at REAL NOT NULL
);
CREATE TABLE orders (
    id TEXT PRIMARY KEY, quote_id TEXT NOT NULL UNIQUE REFERENCES quotes(id),
    actor_id TEXT NOT NULL, scope TEXT NOT NULL,
    amount_minor INTEGER NOT NULL, currency TEXT NOT NULL,
    state TEXT NOT NULL CHECK(state = 'created')
);
CREATE TABLE purchase_receipts (
    actor_id TEXT NOT NULL, idem_key TEXT NOT NULL,
    quote_id TEXT NOT NULL REFERENCES quotes(id), response_json TEXT NOT NULL,
    PRIMARY KEY(actor_id, idem_key)
);
"""


class Rejected(Exception):
    def __init__(self, code):
        super().__init__(code)
        self.code = code


def initialize(path):
    with closing(sqlite3.connect(path)) as db, db:
        db.executescript(SCHEMA)


def purchase(path, actor, key, body, *, clock=time.time, _after_insert=None):
    """actor must come from trusted authentication middleware, not request JSON.

    Quotes are immutable server-issued offers. One quote may create one order.
    _after_insert is a fault-injection test hook, not a request parameter.
    """
    if not isinstance(actor, str) or not actor:
        raise Rejected('authentication_required')
    if not isinstance(key, str) or not re.fullmatch(r'[A-Za-z0-9_-]{8,128}', key):
        raise Rejected('invalid_key')
    if (not isinstance(body, dict) or set(body) != {'quote_id'}
            or not isinstance(body['quote_id'], str)
            or not 1 <= len(body['quote_id']) <= 128):
        raise Rejected('invalid_body')
    quote_id = body['quote_id']
    db = sqlite3.connect(path, timeout=5, isolation_level=None)
    db.row_factory = sqlite3.Row
    try:
        db.execute('PRAGMA foreign_keys = ON')
        db.execute('BEGIN IMMEDIATE')
        receipt = db.execute(
            'SELECT quote_id, response_json FROM purchase_receipts '
            'WHERE actor_id = ? AND idem_key = ?', (actor, key)).fetchone()
        if receipt:
            if receipt['quote_id'] != quote_id:
                raise Rejected('key_reused_with_different_request')
            result = json.loads(receipt['response_json'])
            db.execute('COMMIT')
            return result

        quote = db.execute('SELECT * FROM quotes WHERE id = ? AND actor_id = ?',
                           (quote_id, actor)).fetchone()
        if quote is None:
            raise Rejected('quote_not_found')
        # Evaluate server time after obtaining the write lock.
        if quote['expires_at'] <= clock():
            raise Rejected('quote_expired')
        if db.execute('SELECT 1 FROM orders WHERE quote_id = ?',
                      (quote_id,)).fetchone():
            raise Rejected('quote_already_used')
        result = {
            'order_id': str(uuid.uuid4()), 'quote_id': quote_id,
            'scope': quote['scope'], 'amount_minor': quote['amount_minor'],
            'currency': quote['currency'], 'state': 'created',
        }
        db.execute('INSERT INTO orders VALUES (?, ?, ?, ?, ?, ?, ?)',
                   (result['order_id'], quote_id, actor, result['scope'],
                    result['amount_minor'], result['currency'], result['state']))
        if _after_insert is not None:
            _after_insert()
        db.execute('INSERT INTO purchase_receipts VALUES (?, ?, ?, ?)',
                   (actor, key, quote_id, json.dumps(result, sort_keys=True)))
        db.execute('COMMIT')
        return result
    except BaseException:
        if db.in_transaction:
            db.execute('ROLLBACK')
        raise
    finally:
        db.close()

test_purchase.py — the complete 20-test suite

import concurrent.futures
import json
from contextlib import closing
from pathlib import Path
import sqlite3
import subprocess
import sys
import tempfile
import unittest

from purchase import Rejected, initialize, purchase


class PurchaseTests(unittest.TestCase):
    def setUp(self):
        self.temp = tempfile.TemporaryDirectory()
        self.addCleanup(self.temp.cleanup)
        self.path = str(Path(self.temp.name) / 'orders.sqlite')
        initialize(self.path)
        with closing(sqlite3.connect(self.path)) as db, db:
            db.executemany('INSERT INTO quotes VALUES (?, ?, ?, ?, ?, ?)', [
                ('q1', 'alice', 'report:address-1', 3000, 'USD', 200),
                ('q2', 'alice', 'report:address-2', 4500, 'USD', 200),
                ('q3', 'bob', 'report:address-3', 2000, 'USD', 200),
            ])

    def buy(self, actor='alice', key='attempt_123', quote='q1', **kwargs):
        return purchase(self.path, actor, key, {'quote_id': quote},
                        clock=kwargs.pop('clock', lambda: 100), **kwargs)

    def counts(self):
        with closing(sqlite3.connect(self.path)) as db:
            return tuple(db.execute('SELECT COUNT(*) FROM ' + table).fetchone()[0]
                         for table in ('orders', 'purchase_receipts'))

    def rejects(self, code, fn):
        with self.assertRaises(Rejected) as result:
            fn()
        self.assertEqual(result.exception.code, code)

    def test_server_terms_copied(self):
        result = self.buy()
        self.assertEqual((result['amount_minor'], result['currency'], result['scope'],
                          result['state']), (3000, 'USD', 'report:address-1', 'created'))
        self.assertEqual(self.counts(), (1, 1))

    def test_lost_response_replay(self):
        original = self.buy()
        self.assertEqual(original, self.buy())
        self.assertEqual(self.counts(), (1, 1))

    def test_replay_after_expiry(self):
        original = self.buy()
        self.assertEqual(original, self.buy(clock=lambda: 999))
        self.assertEqual(self.counts(), (1, 1))

    def test_expiry_inclusive_boundary(self):
        self.rejects('quote_expired', lambda: self.buy(clock=lambda: 200))
        self.assertEqual(self.counts(), (0, 0))

    def test_cross_owner_before_and_after_purchase(self):
        self.rejects('quote_not_found', lambda: self.buy(actor='bob'))
        self.buy()
        self.rejects('quote_not_found', lambda: self.buy(actor='bob'))
        self.assertEqual(self.counts(), (1, 1))

    def test_unknown_quote(self):
        self.rejects('quote_not_found', lambda: self.buy(quote='missing'))
        self.assertEqual(self.counts(), (0, 0))

    def test_sql_syntax_is_data(self):
        self.rejects('quote_not_found', lambda: self.buy(quote="q1' OR 1=1 --"))
        self.assertEqual(self.counts(), (0, 0))

    def test_database_uniqueness_not_only_application_check(self):
        self.buy()
        with closing(sqlite3.connect(self.path)) as db, db:
            with self.assertRaises(sqlite3.IntegrityError):
                db.execute("INSERT INTO orders SELECT 'different-id', quote_id, "
                           "actor_id, scope, amount_minor, currency, state FROM orders")
        self.assertEqual(self.counts(), (1, 1))

    def test_pruned_receipt_does_not_allow_second_order(self):
        self.buy()
        with closing(sqlite3.connect(self.path)) as db, db:
            db.execute('DELETE FROM purchase_receipts')
        self.rejects('quote_already_used', self.buy)
        self.assertEqual(self.counts(), (1, 0))

    def test_actor_required(self):
        self.rejects('authentication_required', lambda: self.buy(actor=None))

    def test_key_request_mismatch(self):
        self.buy()
        self.rejects('key_reused_with_different_request', lambda: self.buy(quote='q2'))
        self.assertEqual(self.counts(), (1, 1))

    def test_different_key_same_quote(self):
        self.buy()
        self.rejects('quote_already_used', lambda: self.buy(key='attempt_456'))
        self.assertEqual(self.counts(), (1, 1))

    def test_strict_body(self):
        for body in ({'quote_id': 'q1', 'amount_minor': 1}, {'quote_id': 1}, {},
                     {'quote_id': ''}, {'quote_id': 'q' * 129}, None):
            with self.subTest(body=body):
                self.rejects('invalid_body', lambda: purchase(
                    self.path, 'alice', 'attempt_123', body))
        self.assertEqual(self.counts(), (0, 0))

    def test_invalid_keys(self):
        for key in (None, '', 'short', 'a' * 129, 'injection;', '\u00e9' * 8):
            with self.subTest(key=key):
                self.rejects('invalid_key', lambda: self.buy(key=key))

    def test_actor_key_namespaces(self):
        a = self.buy()
        b = self.buy(actor='bob', quote='q3')
        self.assertNotEqual(a['order_id'], b['order_id'])
        self.assertEqual(self.counts(), (2, 2))

    def test_rollback_then_retry(self):
        def fail():
            raise RuntimeError('simulated crash before receipt')
        with self.assertRaises(RuntimeError):
            self.buy(_after_insert=fail)
        self.assertEqual(self.counts(), (0, 0))
        self.buy()
        self.assertEqual(self.counts(), (1, 1))

    def test_concurrent_connections_same_key(self):
        import threading
        barrier = threading.Barrier(8)
        def call(_):
            barrier.wait(timeout=5)
            return self.buy()
        with concurrent.futures.ThreadPoolExecutor(max_workers=8) as pool:
            results = list(pool.map(call, range(8)))
        self.assertTrue(all(r == results[0] for r in results))
        self.assertEqual(self.counts(), (1, 1))

    def test_concurrent_different_keys(self):
        import threading
        barrier = threading.Barrier(2)
        def call(key):
            barrier.wait(timeout=5)
            try:
                return self.buy(key=key)['state']
            except Rejected as error:
                return error.code
        with concurrent.futures.ThreadPoolExecutor(max_workers=2) as pool:
            results = list(pool.map(call, ['attempt_123', 'attempt_456']))
        self.assertCountEqual(results, ['created', 'quote_already_used'])
        self.assertEqual(self.counts(), (1, 1))

    def child(self, code):
        return subprocess.run([sys.executable, '-c', code, self.path],
                              cwd=Path(__file__).parent, capture_output=True,
                              text=True, timeout=5)

    def test_fresh_process_replay(self):
        code = ("import sys,json; from purchase import purchase; "
                "print(json.dumps(purchase(sys.argv[1], 'alice', 'attempt_123', "
                "{'quote_id':'q1'}, clock=lambda:100)))")
        first, second = self.child(code), self.child(code)
        self.assertEqual(first.returncode, 0, first.stderr)
        self.assertEqual(second.returncode, 0, second.stderr)
        self.assertEqual(json.loads(first.stdout), json.loads(second.stdout))
        self.assertEqual(self.counts(), (1, 1))

    def test_abrupt_process_exit_rolls_back(self):
        result = self.child("import os,sys; from purchase import purchase; "
                            "purchase(sys.argv[1], 'alice', 'attempt_123', "
                            "{'quote_id':'q1'}, clock=lambda:100, "
                            "_after_insert=lambda:os._exit(23))")
        self.assertEqual(result.returncode, 23, result.stderr)
        self.assertEqual(self.counts(), (0, 0))
        self.buy()
        self.assertEqual(self.counts(), (1, 1))


if __name__ == '__main__':
    unittest.main(verbosity=2)

A 200 OK can still belong to the wrong input

Bryan Williams — Fri, 25 Sep 2026 09:23:01 +0000

A request succeeds. The JSON is valid. The interface shows the wrong result anyway.

One way this happens has nothing to do with the server:

Someone submits input A.
While the request runs, they change the form to input B.
A's response arrives and renders underneath B.

The response is correct for the request. The screen is wrong for the current input. On a quote, eligibility check, or data lookup, that distinction matters.

Think of a shipping quote: the customer changes the destination from Boston to Seattle, but Boston's price arrives last. A green success message makes the mistake look trustworthy.

The rule: results belong to a revision

Give the form a revision number. Advance it whenever relevant input changes and whenever a new request starts. Each request remembers its own revision. Only the current revision may update the interface.

A revision is just a counter: 1, 2, 3. It answers “which version of the user's intent is this?” rather than “which response arrived most recently?”

Advancing only on submit misses an important case: someone can edit A to B without submitting B. A is already stale.

Here is a small framework-independent example. fetchResult performs the request; render receives a snapshot of display state.

function createLookup(fetchResult, render) {
  let input = '';
  let revision = 0;
  let controller;
  const state = { busy: false, result: null, error: null };
  const paint = () => render({ ...state });

  function invalidate() {
    revision += 1;
    controller?.abort();
    controller = undefined;
    Object.assign(state, { busy: false, result: null, error: null });
  }

  function edit(value) {
    input = value;
    invalidate();
    paint();
  }

  async function submit() {
    invalidate();
    const mine = revision;
    const submittedInput = input.trim();
    if (!submittedInput) {
      state.error = 'Enter a value';
      paint();
      return;
    }
    const active = new AbortController();
    controller = active;
    state.busy = true;
    paint();
    try {
      const result = await fetchResult(submittedInput, active.signal);
      if (mine !== revision) return;
      state.result = { input: submittedInput, data: result };
    } catch (error) {
      if (mine !== revision) return;
      state.error = 'Request failed. Please retry.';
    } finally {
      if (mine === revision) {
        state.busy = false;
        controller = undefined;
        paint();
      }
    }
  }

  return { edit, submit, dispose: invalidate };
}

Wire edit to every input that affects the request, including programmatic changes. Call dispose when removing the component. Keep rendering synchronous and separate from request initiation. This example assumes a string input; for a larger form, capture an immutable snapshot of all relevant fields.

For an HTTP adapter, pass the signal to fetch, check response.ok, then return the parsed body. A fetch promise can resolve even for an HTTP error such as 404; the adapter must reject errors it does not want rendered as successful results. MDN's fetch guide covers that distinction.

Cancellation helps; ownership decides

AbortController can cancel a fetch and response-body consumption. That saves unnecessary work. But the revision check is what prevents an obsolete operation from changing this interface—even when a custom request adapter ignores cancellation.

Guard all completion paths:

A stale success must not replace the result.
A stale error must not replace a newer success or pending state.
A stale finally must not turn off the newer request's loading indicator.

Also clear old results immediately when inputs change. Otherwise a previously valid result can remain visibly attached to new inputs even before another response arrives. A purchase or continue button derived from that result should become unavailable too.

The result includes its submitted input so the view can label what was actually checked. This is a client-side consistency mechanism, not authorization. A server handling a consequential action still needs to validate the authenticated user, submitted scope, and any quote or result identifier. Aborting a request does not undo a server-side action.

Test the order you hope never happens

Paste the function above into a modern browser console, then run this miniature reproduction. No network or account is needed. The fake transport deliberately ignores cancellation:

let finishOldRequest;
let screen;
const lookup = createLookup(
  () => new Promise(resolve => { finishOldRequest = resolve; }),
  next => { screen = next; }
);

lookup.edit('Boston');
const pending = lookup.submit();
lookup.edit('Seattle'); // No second submit: this is the easy case to miss.
finishOldRequest({ price: 12 });
await pending;

if (screen.result !== null || screen.busy !== false) {
  throw new Error('Stale Boston result escaped into the Seattle view');
}
console.log('PASS: the old result was discarded');

For this example, eleven local tests passed on September 25, 2026. The tests covered:

A normal success bound to the submitted input.
Editing during a request, without a second submit.
An old failure arriving while a new request is pending, including its finally.
Two submits, with the older response arriving last.
A current failure followed by a successful retry.
Blank input clearing the previous result without making a request.
Disposal before a late response, with no subsequent render.
A → B → A edits: the first A request stays obsolete, even though the text matches again.
A request adapter throwing synchronously.
Editing after a completed success, immediately clearing the old result.
A legitimate falsy result (0) remaining distinguishable from no result.

The details that matter in a larger app

One counter per result surface. Independent widgets should not invalidate each other. Inputs that jointly determine one result should share a revision.

Matching text is not enough. The A → B → A case is why comparing strings alone is weaker than comparing request ownership. The first A request belongs to an abandoned interaction, even if the user later types A again.

Debouncing solves a different problem. It reduces how often requests start. It does not establish which response may render. You can use both.

Revalidate consequential actions on the server. A client counter prevents stale presentation; it cannot prove a quote is still valid when someone purchases. Bind the action to the server's own result or quote identifier and validate its scope and expiry there. Use server-side idempotency for duplicate side effects.

In the rendered UI, test the complete transition too: edit during loading, discard the old response, keep the next request's loading state, and expose a working retry. Make the status perceivable to assistive technology, not just a color change.

The useful question is not just “Did the request succeed?” It is “Does this response still own the right to change this screen?”

AI authorship disclosure: An AI agent drafted this article and its standalone example. The exact example and reproduction were executed, including the eleven automated cases described above.

Missing Is Not False

Bryan Williams — Wed, 26 Aug 2026 09:51:34 +0000

Missing Is Not False

A while back I wrote about what breaks when you turn government open data into something machines can trust — the quiet ways a publisher reshapes what it sends you, so the same question comes back a different shape depending on how (or when) you ask.

Then a reader, Vinh (@vinhnguyenthanhdn), left a comment sharper than my whole post. He didn't argue. He handed me a cleaner example — hiding inside an API I use every day. This one. DEV's.

Here's the catch.

Ask DEV for a single article the public way:

GET /api/articles/{id}

and the response has no published field at all. It has published_at, published_timestamp, a readable date — but no boolean that says "this is published."

Now ask the owner endpoint for your own posts:

GET /api/articles/me/all

and every article carries a published boolean.

Same article. Two representations. In one the field exists; in the other it was never there. I checked before I believed it — he's exactly right.

Why does that matter? Because of what your code does next. Almost every parser reaches for the field like this:

is_live = article.get("published")   # -> None on the public response

On the public response published is missing, so you get None, which is falsy, so your pipeline quietly decides the article is not published — when it plainly is. The request succeeded. The status was 200. Nothing errored. You just silently recorded the opposite of the truth.

And here's the part Vinh made me see clearly: "the publisher stopped sending this field" and "the publisher sent the field as false" are different events. Ordinary deserialization flattens both into the same value. One means draft. The other means I wasn't looking at a representation that carries this fact at all. Collapsing them isn't a rounding error — it throws away the one thing you needed to know: whether you actually have an answer, or just an absence.

That's not a bug to file against DEV. It's the shape of the whole problem I care about. Public data sources reshape what they send — across endpoints, across time — and the danger was never the reshape. It's that naive code turns "I don't know" into a confident "no."

In the original post I wrote the rule we run CivicDataForge on: an API should not manufacture certainty because a consumer wants a Boolean. Vinh read that and went one further. He didn't just report the bug — he handed me the fix for the whole class: treat absence and null as a monitored dimension of their own, and record the observed key set per endpoint next to the record hash, so "the publisher stopped sending this field" stays diffable from "the publisher sent it empty."

That sentence changed the company. We carried it into CivicDataForge's evidence envelope as a shape receipt on the source response — the keys we observed, the keys we expected but didn't get, which came back explicitly null, which came back explicitly false, and a fingerprint of the whole shape. It's live in our first services today, and we're rolling it out across the rest of the catalog, so field presence becomes evidence in its own right — not a parsing detail the deserializer swallows. A reader's comment became a schema. That's the best trade I've made all month.

And because the example deserved to be honored in its own language, we also built the small thing: a free DEV article checker that runs on exactly those rules. It now lives in the CivicDataForge catalog beside the permit and registry evidence tools — the side quest that proved the main quest. Nothing to sign up for.

You give it a link to a DEV article. It answers with one of three states, never fewer:

PUBLISHED — seen live on the public endpoint (200). It tells you that's where the evidence came from.
DRAFT — the owner representation says published: false. Different evidence, different door.
UNKNOWN — the fact could not be observed. Not "no." Unknown. The state everyone else quietly throws away.

Every answer ships with a receipt — a hash of what was checked and when — so it's evidence, not vibes. And there's a version that runs on your own machine for checking your own drafts: your DEV key only ever talks to DEV. It never touches my server, never gets logged, never gets stored. I don't want your key. I want you to trust the answer.

Check any DEV article, free, no signup: https://civicdataforge.pages.dev/data/devto-publication-evidence

That's the whole idea, and it's Vinh's more than mine: missing is not false, and a system you can trust says so out loud. If it can't see something, it hands you an honest UNKNOWN instead of a confident wrong answer.

Thanks, Vinh. I asked which publishers silently reshape results — you answered, then showed me the gap in my own envelope. The company is better for it.

What Broke When We Turned Government Open Data Into Agent-Callable Evidence

Bryan Williams — Mon, 24 Aug 2026 05:25:11 +0000

Government open data looks deceptively easy from a distance.

Find an endpoint. Make a request. Normalize the JSON. Ship an API.

That is enough for a demo. It is not enough for a production system that has to answer a consequential question next month, after the publisher changes a field, caps a response, moves an endpoint, or returns an empty page that looks exactly like “there are no records.”

We have been building CivicDataForge around one question:

What has to exist between an official government publisher and a software system—or an AI agent—before the result deserves to be called evidence?

This post is the practical answer we have reached so far. It is not a claim that government data can be made perfectly clean. It is an account of the failure modes we hit, the architecture we adopted, and the boundaries we refuse to erase.

1. A successful HTTP response is not a complete dataset

The first serious failure mode is pagination.

Government-data platforms often impose a maximum record count. A response can be valid JSON, have a 200 status, and still represent only the first page.

ArcGIS feature services expose controls such as resultOffset and resultRecordCount, and may signal that a transfer limit was exceeded. Socrata datasets similarly support paging and query controls. If a collector ignores those contracts, “the source returned 1,000 rows” can quietly become “the jurisdiction has 1,000 records.”

Those are not the same statement.

Our collection rule became:

Discover the publisher's page limit.
Request every page in a stable order.
Track page count, observed row count, and source-reported totals where available.
Reject a monitoring baseline when the selected query is capped, partial, or structurally inconsistent.
Bind the completed scope into a receipt.

The important product decision is step four. A partial page is useful for exploration, but it cannot prove that a record disappeared between two runs.

Official references:

2. “No match” is not a negative legal conclusion

Suppose an address lookup returns no permit record.

It is tempting to return NOT_PERMITTED or ILLEGAL. That is usually stronger than the evidence.

The selected publisher may not cover the full legal regime. The address may be formatted differently. A local permit may exist in another system. The source may be stale. The query may be incomplete. Or there may simply be no published match.

We use an explicit, fail-closed decision vocabulary:

EVIDENCE_FOUND
NO_PUBLISHED_MATCH
REVIEW_REQUIRED
SOURCE_UNAVAILABLE
SCOPE_INCOMPLETE

This vocabulary is deliberately less dramatic than a binary yes/no result. That is the point.

An API should not manufacture certainty because a consumer wants a Boolean. If a downstream workflow requires a Boolean, the customer should own the policy that maps evidence states to an action—and should preserve the original state for review.

3. Provenance has to survive normalization

Normalization is necessary. It is also where evidence can lose its identity.

We normalize names, addresses, dates, identifiers, and source-specific statuses, but retain enough lineage to answer:

Which authority published this record?
Which endpoint, file, or dataset did it come from?
What query and scope produced it?
When was it retrieved?
What did the publisher call this field?
Which transformation produced the normalized value?
Can the record and decision be fingerprinted again?

A minimal evidence envelope looks roughly like this:

{
  "source": {
    "publisher": "issuing authority",
    "url": "https://official.example/dataset",
    "retrieved_at": "2026-08-24T12:00:00Z"
  },
  "scope": {
    "query": "bounded source query",
    "complete": true
  },
  "record": {
    "source_id": "publisher-owned-key",
    "normalized_identifier": "stable-identifier",
    "record_hash": "sha256:..."
  },
  "decision": {
    "state": "EVIDENCE_FOUND",
    "receipt_hash": "sha256:..."
  }
}

The exact fields vary by source. The invariant is that the normalized result does not sever the path back to the official record.

4. Source health and data freshness are different dimensions

A source can be reachable and stale.

It can have a stable schema and return incomplete data. It can publish a recent file whose internal dates are old. It can change content without changing a URL. It can change a schema without failing a request.

So one green check is not enough.

We monitor independent dimensions:

availability
schema compatibility
record-count behavior
source date integrity
freshness, when the publisher exposes a defensible freshness signal
content fingerprint
contract compatibility

When freshness cannot be measured, we say freshness_not_measured. We do not let an availability check imply that the data is current.

This distinction matters because a production monitor should be able to quarantine a bad update instead of replacing a known baseline with an empty or damaged response.

5. Change monitoring needs replay and correction, not just alerts

“Send a webhook when something changes” is only the visible edge of a reliable change system.

The harder questions are:

Was the previous snapshot complete?
Is the new snapshot complete?
Did the source change, or did the collector fail?
Can the event be delivered more than once?
Can the consumer replay it?
What happens after a human corrects an ambiguous match?

Apify webhooks retry failed deliveries. That is helpful, but it means consumers must be idempotent. Our recommended key is the Actor run ID plus event type, with the downstream handler acknowledging quickly and queueing the expensive work.

idempotency_key = actor_run_id + ":" + event_type

We treat a webhook as a notification about an evidence object, not as the evidence object itself. The durable dataset, record hashes, scope, and decision receipt remain the inspectable source of truth.

Official reference: Apify webhook actions and retry behavior.

6. AI agents need selection contracts, not a pile of tools

Publishing an MCP server does not automatically make a tool safe or useful.

An agent still needs to know:

the task the tool supports
the minimum input
the preferred identifier
the source and jurisdiction boundary
the possible output states
what the output does not prove

We expose a stable Streamable HTTP MCP endpoint and machine-readable catalogs, but the more important artifact is the task-to-tool selection contract.

For example:

{
  "task": "verify India company evidence",
  "preferred_identifier": "Corporate Identification Number",
  "minimum_input": "exact CIN preferred",
  "boundary": "research evidence, not automatic KYC or eligibility clearance"
}

The transport gets the tool into the agent. The selection contract helps the agent call the right tool and preserve the result's meaning.

The MCP specification recommends Streamable HTTP for remote servers and defines the transport behavior. Authentication, origin validation, and secret handling still belong to the implementation and client configuration—not to public examples with embedded credentials.

Official reference: MCP transports specification.

7. We stopped thinking of the product as “a scraper”

The architecture now has six explicit services:

Source feasibility and rights review
Normalization and identity mapping
Source health and change monitoring
Evidence delivery and integration
Evidence packets and correction paths
AI-agent tool integration

That framing changes what we build.

A new source does not enter production because we found an endpoint. It needs a source and rights passport, a completeness test, a normalization contract, health probes, a bounded decision contract, and an acceptance-tested delivery path.

Likewise, a connector is not “available” because we can imagine it. Public APIs, datasets, schedules, webhooks, file exports, and the remote MCP endpoint are live interfaces. A customer-specific adapter, SFTP route, or platform-native app remains scoped work until its real downstream path passes an acceptance test.

The service catalog and integration recipes publish that boundary in human- and machine-readable form.

Our mission

CivicDataForge exists to reduce the hours lost between “the government publishes this somewhere” and “a person or system can use it without pretending it says more than it does.”

Our intention is to build a reusable evidence layer across public records: official-source first, inspectable, correction-friendly, and callable by both software and AI agents.

Our want is straightforward: make source quality and uncertainty visible enough that developers do not have to choose between a brittle one-off script and an opaque vendor answer.

We are open to collaboration and grateful for serious support, correction, and feedback. We are not asking for permission to build; we are inviting developers, public servants, data owners, researchers, and AI teams to make the contracts stronger with us.

What we need most from the developer community is adversarial input:

Which government source breaks your workflow today?
Which identifier refuses to match cleanly?
Which publisher silently caps or reshapes results?
Which evidence state does your current API flatten into a misleading Boolean?
Which integration path is missing from the recipes?

Those questions are more valuable than generic feature requests because they expose the real contract the system has to survive. If you bring one, we will meet it with respect, evidence, and a willingness to change what the facts prove should change.

What we would do differently

If we started again, we would make three decisions earlier:

Define failure and uncertainty states before defining the happy-path schema.
Prove complete collection before building change monitoring.
Publish task-selection and evidence-boundary metadata alongside every agent tool.

The code that fetches rows is often the smallest part of the system. The defensible work is keeping a source-linked statement truthful after paging, normalization, retries, source drift, and downstream automation.

That is the layer we are building.