A lost HTTP response can make an integration send another request for work the server has already committed.
Give that transmission a fresh request ID, but retain the operation ID that identifies the intended business change. A retry test should count the committed effects as well as the responses it receives.
The fixture in this article closes a connection after a SQLite commit and before the HTTP response. Its deliberately broken client creates two reservations. Changing the operation argument passed to the transport function makes the same six tests pass, with one reservation and two separately identifiable attempts.
These are executed results from a local, synthetic inventory example, using Python 3.14.7 and SQLite 3.53.4.
They are not measurements from a customer system. The complete runtime and test module appear below so the claim can be checked without a remote account, API credential or third-party Python package.
This article was prepared with AI-generated research, prose and code. The local test results are recorded below.
Specify what each identifier is allowed to mean
For this fixture, a business operation means one request to reserve a stated quantity of a test SKU for one tenant. The caller supplies its operation ID before the retry loop starts. Each HTTP transmission gets a separate request ID. The receiver returns a reservation ID only after creating or retrieving the business result.
| Identifier | Scope in this fixture | Changes on a transport retry? |
|---|---|---|
| Operation ID | One reservation intent within a tenant | No |
| Request ID | One HTTP attempt | Yes |
| Reservation ID | The stored business result | No, when the operation replays |
| Tenant | The namespace containing the operation | No |
Stripe states in its Request IDs documentation:
Each API request has an associated request identifier.
That is a useful observability role. It does not make a request identifier the correct duplicate-admission key for every receiver. The names on an interface can also differ: AWS calls its caller-supplied idempotency token a client request identifier. Read the promised semantics before deciding whether two fields can share a value.
The client accepts an operation ID and a request-ID generator as different inputs. Tests inject a two-value generator so the expected transmissions are reproducible. Normal calls can use the default UUID generator for each attempt. This arrangement makes identifier ownership visible at the function boundary, rather than hiding it in an exception handler or a transport wrapper.
Do not create the operation ID by hashing the payload alone. Two independent orders can legitimately reserve the same SKU and quantity. They need different operation identities. Conversely, changing a request ID because a connection failed must not change which order the integration is fulfilling. These are separate decisions, and the tests exercise both.
The header names used here are fixture choices. X-Request-ID carries the attempt identifier, while Idempotency-Key carries the operation identifier. X-Test-Tenant supplies a test namespace. None of these headers authenticates a caller. A production receiver must obtain its authorized scope through its actual authentication and authorization boundary.
Put the operation record beside the protected effect
The receiver stores two kinds of rows in one SQLite file. A reservation row is the synthetic business effect. An operation row binds a tenant and operation ID to the canonical request and stored business response. A composite primary key prevents two records from claiming the same identity within that namespace.
Each handler opens its own database connection and explicitly starts BEGIN IMMEDIATE. It looks up the operation while holding the write transaction. An unseen operation creates a reservation and stores its response. A seen operation with matching arguments returns the stored response. Conflicting arguments produce HTTP 409 without creating another reservation.
The important relationship is the transaction containing both writes. If the operation marker could commit while the reservation failed, a later retry might incorrectly report completed work. If the reservation could commit without its marker, a later retry could create another result. A lookup performed before the transaction would not reserve the right to perform the mutation. AWS's idempotent API guidance describes the need to combine recording the token with the related mutations atomically. This fixture applies that principle to one local database. The principle does not turn a database transaction into a transaction with an independent warehouse, payment provider or message consumer.
Request comparison uses JSON with sorted keys and compact separators. This makes object-key order irrelevant for the valid bodies used in the tests.
It is not a general business-equivalence algorithm. Different units, omitted defaults or an API revision can change meaning even when two payloads look similar. Agree those rules separately before adopting this serializer for another interface.
Python's SQLite transaction documentation explains the legacy isolation_level control. Here, isolation_level=None disables implicit transaction opening and the fixture issues its own transaction statements. Each HTTP handler owns and closes its connection. The executed environment is stated above; compatibility with other Python releases was not part of this run.
The reservation is deliberately a row creation, not a complete stock-allocation system. There is no warehouse availability calculation or inventory balance to inspect. This keeps the observable effect small enough to count without deriving it from the returned status. If adapting the fixture to a balance mutation, replace that row-count oracle with the actual invariant for the protected store, and keep the operation record in the same tested transaction. The receiver is a test server accepting the valid request shapes below. It does not implement production input validation, credentials or an internet-facing API contract. Its effect count is a direct database query, so a successful response cannot conceal a second reservation from the test oracle.
Complete runnable file: retry_fixture.py
"""Bounded local HTTP/SQLite experiment. Synthetic data; tenant is NOT authentication."""
import http.client
import json
import socket
import sqlite3
import threading
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from uuid import uuid4
def canonical(value):
return json.dumps(value, sort_keys=True, separators=(",", ":"))
class Server:
def __init__(self, path):
self.path, self.trace, self.drop_once = str(path), [], False
self.lock = threading.Lock()
self.stopped = False
db = self.connect()
try:
db.executescript("""
BEGIN IMMEDIATE;
CREATE TABLE IF NOT EXISTS reservations (
id INTEGER PRIMARY KEY, sku TEXT NOT NULL, quantity INTEGER NOT NULL);
CREATE TABLE IF NOT EXISTS operations (
tenant TEXT NOT NULL, operation_id TEXT NOT NULL,
payload TEXT NOT NULL, response TEXT NOT NULL,
PRIMARY KEY (tenant, operation_id));
COMMIT;
""")
finally:
db.close()
owner = self
class Handler(BaseHTTPRequestHandler):
def log_message(self, *args):
pass
def do_POST(self):
body = json.loads(self.rfile.read(int(self.headers["Content-Length"])))
tenant, operation = self.headers["X-Test-Tenant"], self.headers["Idempotency-Key"]
request = self.headers["X-Request-ID"]
payload = canonical(body)
db = owner.connect() # Each handler owns its connection.
try:
db.execute("BEGIN IMMEDIATE")
previous = db.execute("SELECT payload, response FROM operations WHERE tenant=? AND operation_id=?",
(tenant, operation)).fetchone()
status, outcome = 201, "replay"
if previous:
response = previous[1]
if previous[0] != payload:
status, outcome = 409, "conflict"
response = canonical({"error": "payload_conflict"})
else:
cursor = db.execute("INSERT INTO reservations(sku, quantity) VALUES (?, ?)",
(body["sku"], body["quantity"]))
response = canonical({"reservation_id": cursor.lastrowid, **body})
db.execute("INSERT INTO operations VALUES (?, ?, ?, ?)",
(tenant, operation, payload, response))
outcome = "created"
db.execute("COMMIT")
except Exception:
if db.in_transaction:
db.execute("ROLLBACK")
raise
finally:
db.close()
with owner.lock:
drop = owner.drop_once and outcome == "created"
if drop:
owner.drop_once = False
owner.trace.append({"request_id": request, "operation_id": operation,
"outcome": outcome, "response_lost": drop})
if drop: # Deliberate transport loss strictly AFTER COMMIT.
self.close_connection = True
self.connection.shutdown(socket.SHUT_RDWR)
self.connection.close()
return
encoded = response.encode()
self.send_response(status)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(encoded)))
self.send_header("X-Request-ID", request) # Current attempt, never persisted.
self.end_headers()
self.wfile.write(encoded)
self.http = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
self.port = self.http.server_port
self.thread = threading.Thread(target=self.http.serve_forever, daemon=True)
def connect(self):
# Explicit SQL transactions; None disables sqlite3's implicit BEGIN.
return sqlite3.connect(self.path, isolation_level=None, timeout=5)
def start(self):
self.thread.start()
def stop(self):
if not self.stopped:
self.http.shutdown()
self.http.server_close()
self.thread.join(timeout=5)
self.stopped = True
def effects(self):
db = self.connect()
try:
return db.execute("SELECT count(*) FROM reservations").fetchone()[0]
finally:
db.close()
def attempt(port, tenant, operation_id, request_id, body):
connection = http.client.HTTPConnection("127.0.0.1", port, timeout=5)
try:
connection.request("POST", "/reservations", canonical(body),
{"Content-Type": "application/json", "X-Test-Tenant": tenant,
"Idempotency-Key": operation_id, "X-Request-ID": request_id})
response = connection.getresponse()
return response.status, dict(response.getheaders()), json.loads(response.read())
finally:
connection.close()
def reserve(port, tenant, operation_id, body, request_ids=lambda: str(uuid4())):
for number in range(2):
request_id = request_ids()
try:
return attempt(port, tenant, operation_id, request_id, body)
except (OSError, http.client.HTTPException):
if number == 1:
raise
Lose the response after the commit
A timeout before the receiver performs any work is an easy retry case. The more discriminating case is uncertainty after the work has happened. The test arms a one-shot fault, sends a valid reservation and lets the receiver commit. Only then does the handler shut down the socket before writing a response.
From the caller's side, this appears as a transport failure. It cannot infer from that failure whether a reservation exists. The fixture can inspect both sides because it owns the receiver and database, but a real integration usually has only its local observation until a receiver receipt or reconciliation result resolves the uncertainty.
There is no delay-based race in this fault. The socket close is placed after the explicit COMMIT statement.
An in-memory trace records the request ID, operation ID, result classification and whether the response was deliberately lost. That trace explains the experiment; it is not a durable audit log or a substitute for the database count.
The deliberately broken retry loop passes its fresh request ID in both identity positions:
return attempt(port, tenant, request_id, request_id, body)
The receiver accepts the identities it receives. On the first attempt it sees a new operation key and creates a reservation. On the second attempt it sees another new key and creates another reservation. Adding a uniqueness constraint to the operation table cannot help when the client changes the identity that the constraint protects. The executed demonstration used synthetic SKU TEST-BOLT, quantity 2. Its broken-client trace contains test-attempt-1 followed by test-attempt-2 in both the request and operation columns. The final response reports reservation 2, while the database contains two effects. Checking only that response would miss the duplicate.
The broken and corrected demonstrations start with different fresh database files. They do not repair or delete an extra reservation before counting results. That matters because a compensating cleanup could make the final quantity look right while concealing the duplicate that the client caused. The retained trace labels every value as test data and preserves the created-versus-replayed distinction alongside the stored-effect count.
Keep the fault and the oracle separate when adapting the example. A client-side exception alone does not establish that the server committed. A server log line alone does not establish how many business rows exist. The fixture deliberately observes the post-commit fault and queries the protected store so those two claims can be reviewed independently.
Executed local fixture: the first response is lost after COMMIT; the retry replays reservation 1 under the same operation ID.
Run the same assertions before changing the client
Save the runtime as retry_fixture.py and the test module as test_fixture.py in one directory. The suite starts loopback HTTP servers on ephemeral ports and creates temporary database files. It needs no persistent local service. The test cleanup stops each server and removes its temporary directory.
python3 -m unittest -v test_fixture.py
Before the client correction, the executed suite ran six tests: five passed and one failed.
The failing assertion expected the set of operation identities to contain op-1. Instead it found attempt-1 and attempt-2. That was an identity failure, not a missing dependency, invalid fixture or timing-based setup error.
The failing test first checks the observed attempt sequence, then the operation identity and result classifications. It also requires one stored effect, a successful final status, the second attempt's response header and a business result containing only reservation fields. Those assertions describe behavior at the HTTP and storage boundaries rather than requiring a particular private function call order.
An early assertion failure means later assertions in that test do not run.
The separate buggy-client demonstration therefore supplies the observed two-effect count; the RED test output alone is not presented as proof that every later assertion executed. Preserve this distinction when collecting a failure report for a larger harness.
The other five cases exercise the receiver directly. Their passing results isolate the defect: the supplied key can protect the local effect, but the retry client does not preserve it. A test that replaced the receiver with a mock would need an additional oracle to establish the actual database mutation and replay behavior.
Check which runtime module the test imports before recording RED. A scratch directory with similarly named files can run a corrected module while the reviewer inspects an older copy. Record the loaded runtime file's path and hash with the result, then preserve that exact file. A test name and a passing process exit do not identify the implementation that actually ran.
To reproduce RED from the complete fixed file, work in a scratch copy and replace only the final call inside reserve with the broken call shown above. Keep the receiver transaction, response-loss fault and test module unchanged. Restore the operation argument to reproduce GREEN. A different assertion failure needs investigation before it is treated as evidence of this defect; for example, an import failure has not exercised the HTTP boundary at all.
Freeze the test module while correcting the implementation. Here its SHA-256 remained unchanged between RED and GREEN. That makes the evidence stronger than a passing result produced after quietly weakening the expected operation identity or removing the database assertion.
Move operation identity outside the attempt boundary
The corrected client passes the caller's existing operation ID to every transmission, while obtaining a fresh request ID inside the loop:
-return attempt(port, tenant, request_id, request_id, body)
+return attempt(port, tenant, operation_id, request_id, body)
The operation ID is not regenerated in response to an exception. In the fixed demonstration, both transmissions carry test-operation-1. The first creates a reservation and loses its response. The second replays that stored result. Its response identifies test-attempt-2, and the stored business response still describes reservation 1.
This separation matters on the response path too. If the server cached the first attempt's entire transport envelope, a replay could advertise the wrong request ID. The fixture stores the business response, then adds the current request header when sending it.
The tests check that header independently from the stable reservation result.
The client allows at most two transmissions and retries transport exceptions. It returns ordinary HTTP responses, including the conflict response, without creating a fresh operation. This bounded loop exists to exercise identity preservation. It is not a recommended retry budget, delay schedule or error-classification policy for a production dependency.
The caller must still retain the operation across its own recovery boundary. This fixture receives that identity as an argument; it does not implement a durable job queue or prove that a restarted caller can recover its original intent. Moving ID creation outside one loop is necessary here, but a worker that invents a replacement ID after restarting could reintroduce the same failure elsewhere.
The successful replay deliberately returns HTTP 201, matching this fixture's original creation result. There is no claim that every API must use that status for a replay. If a provider instead returns another documented success status or an asynchronous acknowledgement, test the promised business identity and effect separately from the transport status.
A blanket assertion that every response must be identical can conceal a legitimate current-attempt header or reject a valid provider contract.
Keep resource identity separate from operation identity after success. A reservation can later be amended or released through a different authorized operation. Reusing its original creation key for every future mutation would collapse distinct business actions into one replay record. The field should describe the requested change, not merely whichever resource happens to be affected.
Read the six passing contracts precisely
After the correction and explicit connection cleanup, all six unchanged tests passed. The demonstration recorded one effect for the fixed client and two for the preserved buggy client. These counts describe this fixture and its synthetic inputs. They do not establish throughput, a production error rate or a guarantee across every possible interruption.
| Executed case | Observable result | What it distinguishes |
|---|---|---|
| Lost response after commit | Two request IDs, one operation ID, one effect | A repeated transmission from a new business action |
| Changed payload under the same key | HTTP 409, original result retained, one effect | Conflicting intent from a valid replay |
| Same operation ID in different tenants | Two independent reservation IDs | Scoped identity from accidental global suppression |
| Different operations with identical payloads | Two reservation IDs | Business intent from payload equality |
| Receiver restart using the same database | Original result returned, one effect | Persisted operation state from an in-memory cache |
| Two concurrent callers for one operation | Matching results, one effect, distinct request headers | Atomic admission from a preliminary existence check |
The restart case closes the receiver normally and starts another instance over the same file. It does not kill a process during COMMIT or simulate loss of storage power.
The concurrent case submits one pair of callers together; it is not a stress test or proof that every scheduler interleaving was explored.
The conflict test also reverses the order of JSON object keys on a later valid replay. The business result must remain the original one. That check demonstrates the fixture's chosen object-order normalization without claiming that arbitrary strings, numbers or schema versions are semantically interchangeable.
The concurrent assertions inspect both returned results and the stored count. Identical response bodies alone would be insufficient: a defective receiver could create two rows but accidentally return the same one to both callers. Counting rows alone would also be incomplete if one caller received an unrelated reservation. The combined oracle checks that this pair shares the intended result while retaining distinct attempt metadata.
A tenant is not part of every provider's idempotency scope in the same form. The test makes its chosen namespace explicit so reviewers can see what is protected. For another integration, replace it with the receiver's documented account, route and operation scope, derived from an authorized identity rather than a caller's unauthenticated assertion.
Complete runnable file: test_fixture.py
"""Six observable integration contracts; all identifiers and stock are test data."""
import concurrent.futures
import tempfile
import threading
import unittest
from pathlib import Path
from retry_fixture import Server, attempt, reserve
BODY = {"sku": "TEST-BOLT", "quantity": 2}
class Contracts(unittest.TestCase):
def setUp(self):
self.temp = tempfile.TemporaryDirectory()
self.addCleanup(self.temp.cleanup)
self.db = Path(self.temp.name) / "reservations.sqlite"
self.server = Server(self.db)
self.server.start()
self.addCleanup(lambda: self.server.stop())
def send(self, operation="op-1", body=None, tenant="test-tenant", request="attempt-1"):
return attempt(self.server.port, tenant, operation, request, BODY if body is None else body)
def test_lost_response_retries_one_operation(self):
self.server.drop_once = True
status, headers, result = reserve(self.server.port, "test-tenant", "op-1", BODY,
iter(["attempt-1", "attempt-2"]).__next__)
trace = self.server.trace
self.assertEqual([t["request_id"] for t in trace], ["attempt-1", "attempt-2"])
self.assertEqual({t["operation_id"] for t in trace}, {"op-1"})
self.assertEqual([t["outcome"] for t in trace], ["created", "replay"])
self.assertEqual(self.server.effects(), 1)
self.assertEqual(status, 201)
self.assertEqual(headers["X-Request-ID"], "attempt-2")
self.assertEqual(set(result), {"reservation_id", "sku", "quantity"})
def test_payload_conflict_preserves_first_result(self):
first = self.send()[2]
status, _, error = self.send(body={"quantity": 3, "sku": "TEST-BOLT"}, request="attempt-2")
self.assertEqual((status, error), (409, {"error": "payload_conflict"}))
replay = self.send(body={"quantity": 2, "sku": "TEST-BOLT"}, request="attempt-3")
self.assertEqual(replay[2], first)
self.assertEqual(replay[1]["X-Request-ID"], "attempt-3")
self.assertEqual(self.server.effects(), 1)
def test_tenants_have_independent_operation_scope(self):
a = self.send(tenant="test-A")[2]
b = self.send(tenant="test-B", request="attempt-2")[2]
self.assertNotEqual(a["reservation_id"], b["reservation_id"])
self.assertEqual(self.server.effects(), 2)
def test_distinct_operations_same_payload_create_two_effects(self):
a = self.send(operation="op-1")[2]
b = self.send(operation="op-2", request="attempt-2")[2]
self.assertNotEqual(a["reservation_id"], b["reservation_id"])
self.assertEqual(self.server.effects(), 2)
def test_replay_survives_restart(self):
first = self.send()[2]
self.server.stop()
self.server = Server(self.db)
self.server.start()
replay = self.send(request="after-restart")
self.assertEqual((replay[0], replay[2]), (201, first))
self.assertEqual(replay[1]["X-Request-ID"], "after-restart")
self.assertEqual(self.server.trace[0]["outcome"], "replay")
self.assertEqual(self.server.effects(), 1)
def test_concurrent_identical_operation_has_one_effect(self):
gate = threading.Barrier(2)
def caller(i):
gate.wait(timeout=5)
return self.send(request=f"concurrent-{i}")
with concurrent.futures.ThreadPoolExecutor(max_workers=2) as pool:
results = list(pool.map(caller, range(2)))
self.assertEqual([r[0] for r in results], [201, 201])
self.assertEqual(results[0][2], results[1][2])
self.assertEqual({r[1]["X-Request-ID"] for r in results}, {"concurrent-0", "concurrent-1"})
self.assertEqual(sorted(t["outcome"] for t in self.server.trace), ["created", "replay"])
self.assertEqual(self.server.effects(), 1)
if __name__ == "__main__":
unittest.main()
Adapt the contract at the next system boundary
When the protected effect moves to an independent service, ask what that receiver actually promises. Does it recognize the supplied key, compare changed arguments and retain an authoritative result? A header sent by the client has no deduplication effect if the receiving component ignores it or applies it in a different namespace.
The current Stripe idempotent-request reference distinguishes API v1's saved-response behavior from API v2's retry handling. It also explains that reuse after a v1 key is pruned produces a new request. Those are provider-specific conditions, not settings inherited by this inventory server or every API offering an idempotency key.
An expired replay record leaves an unresolved operation unresolved. Neither reusing the old key nor minting a replacement proves that the earlier action failed. Before retransmitting, obtain the evidence needed under the receiver's recovery contract or hold the operation for reconciliation. A retry counter reaching its limit also does not cancel work already admitted remotely.
Adding an outbox can make a local dispatch intention durable, but the external effect still has its own boundary. A receiver receipt might be lost before local storage. A callback might arrive more than once. Those cases need their own acceptance tests and identity mapping; this reservation fixture does not execute a distributed commit or a webhook workflow.
Observability should preserve that distinction. A trace can connect the attempts, while the operation record identifies what they were trying to achieve. Sampling, log loss or a regenerated trace ID must not silently change duplicate admission. The durable operation store is the authority for this fixture's replay decision, not whichever request happens to remain visible in a tracing interface.
Use opaque identifiers and controlled evidence access in a real service. An idempotency key should not contain a customer's private record or an API credential.
A support operator needs enough context to join the operation to authorized business evidence without exposing sensitive payloads through routine request logs.
Keep an executable acceptance packet
Retain the two runtime versions, unchanged tests and actual command results. Include the interpreter and SQLite versions, the synthetic payload and the precise location of the injected fault. Preserve the demo trace beside the count of rows read directly from the protected database. A screenshot of a green test summary alone leaves those assumptions invisible. Give the reviewer a clean directory and the two files, then have them run the command above. Ask which observed record proves that the first attempt committed and which proves the retry did not create another effect. They should also be able to identify the current response attempt without changing the stored reservation result.
For a real receiver, add its current operation-key contract and the unresolved-outcome procedure before treating the fixture as delivery acceptance. Name any simulated dependency and the later integration gate it still needs. Do not extend a local passing result into a claim about a remote system that the test never called.
A retry is another opportunity to observe or complete the same intent. The integration test earns that conclusion by preserving operation identity through the deliberate response loss and checking the resulting store. Keep the request IDs available for investigation, and keep the business identity stable for the component that decides whether another effect is allowed.
More insights to read
- AI Agent Integration Estimates: Price the Connector Failure Paths
- Retry Identity Fieldbook: Transport Attempts and Business Operations
- The Payment Recovery Fieldbook: Identity, Replay and Reconciliation
- Audit Four Replay Boundaries in Smart-Account Signatures
- When a PSP Integration Needs a Second Delivery Milestone
About the author
Dmytro Nasyrov. Photo supplied by the author.
Written by Dmytro Nasyrov PhD, software architect with 24 years of production experience. Dmytro is the founder and CTO of Pharos Production. He works on production software architecture for FinTech, AI, Web3 and blockchain systems.
Top comments (0)