The Slack thread looked calm at 9:12 a.m. A product manager wanted one customer health report. The chat UI still showed a blinking Generate control. The first click opened a streaming draft panel. The laptop then slept during the standup call. Chrome restored the same tab after the meeting. Generate then ran a second time without ceremony. Finance mailed the stale PDF to the account team.
That failure did not start inside the model. It started at POST /reports. The handler minted a fresh job on every click. Two workers then raced the same customer period. The later write won the object key. The earlier write still billed tokens. The chat pane still displayed a single tidy plan.
Treat the Generate control as a doorbell, not a lock. A doorbell can ring twice in three seconds. The house should not grow a second kitchen. Chat clients ring for many boring reasons. They reconnect after a proxy idle timeout. They retry after a 502 from the planner. They revive a frozen tab from bfcache. None of those events mean start another report.
A public library already solved this with checkout cards. The card names one borrower and one due stamp. Shouting the title twice does not mint copies. Many agent features skip that checkout card. They mint the second copy, then invoice both. The model looks busy and expensive. The storage layer looks quietly wrong.
The team needed a host that could lose the race. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The rehearsal used MonkeyCode's free model access and free server option. Planner and worker could collide on a disposable box. That box could disappear after lease tests turned green.
Debounce, prompts, and timestamps still forked the file
Debounce was the first design, and it failed. Four hundred milliseconds cannot outlive a tab refresh. A prompt instruction was the second design. The model still emitted a second start after reconnect. Timestamped filenames were the third design. Two workers still wrote two objects. Billing still attached the wrong object to the thread.
Client Idempotency-Key headers were the fourth near miss. They help when the same HTTP client retries. Chrome mints a new key after a full reload. Mobile clients mint a new key after process death. The human still asked for Acme's August health report. The server had to recognize that natural key. Two tabs asking for August are one job. Two tabs asking for August and September are two jobs. A lease that forgets the period becomes a blunt global lock.
A lease row that outlives the tab
The design that held was a lease row. The chat never inserts the PDF itself. The API never starts a second worker for one report_key. The worker may write only under a live lease. Streaming appends a cursor, not a customer file. Finalization copies that cursor once. The sketch below is a rehearsal contract. It is not a framework mandate.
# report_lease.py — one natural key, one live owner, cursor until commit
import os
from datetime import datetime, timedelta, timezone
from fastapi import FastAPI, Header, HTTPException
from pydantic import BaseModel, Field
from sqlalchemy import text
LEASE_SECONDS = 45
REPORT_TYPES = {"customer_health"}
class StartBody(BaseModel):
report_type: str
period: str = Field(pattern=r"^\d{4}-\d{2}$")
claimed_tenant_id: str | None = None # evidence, never authority
app = FastAPI()
# engine = create_engine(os.environ["DATABASE_URL"], pool_pre_ping=True)
def db_now(conn):
return conn.execute(text("select now() at time zone 'utc'")).scalar()
def parse_session(authorization: str | None):
if not authorization or not authorization.startswith("Bearer "):
raise HTTPException(status_code=401, detail="missing_session")
return verify_signed_session(authorization.removeprefix("Bearer "))
@app.post("/reports/start")
def start(body: StartBody, authorization: str | None = Header(default=None)):
session = parse_session(authorization)
if body.report_type not in REPORT_TYPES:
raise HTTPException(status_code=400, detail="unknown_report_type")
if body.claimed_tenant_id and body.claimed_tenant_id != session.tenant_id:
raise HTTPException(status_code=403, detail="tenant_mismatch")
report_key = f"{session.tenant_id}:{body.report_type}:{body.period}"
worker_id = os.environ["WORKER_ID"]
with engine.begin() as conn:
now = db_now(conn)
row = conn.execute(
text("""
select job_id, status, lease_owner, lease_expires_at, artifact_id
from report_jobs
where report_key = :k
for update
"""),
{"k": report_key},
).mappings().first()
if row and row["status"] == "complete":
return {"job_id": str(row["job_id"]), "status": "complete",
"artifact_id": row["artifact_id"]}
live = (
row
and row["status"] in {"queued", "running"}
and row["lease_expires_at"]
and row["lease_expires_at"] > now
)
if live:
return {"job_id": str(row["job_id"]), "status": row["status"],
"join": True}
expires = now + timedelta(seconds=LEASE_SECONDS)
if row is None:
job_id = conn.execute(
text("""
insert into report_jobs (
report_key, tenant_id, status, lease_owner,
lease_expires_at, cursor_text, created_at
) values (
:k, :t, 'queued', :w, :e, '', :n
) returning job_id
"""),
{"k": report_key, "t": session.tenant_id,
"w": worker_id, "e": expires, "n": now},
).scalar_one()
return {"job_id": str(job_id), "status": "queued", "created": True}
stolen = conn.execute(
text("""
update report_jobs
set status = 'queued',
lease_owner = :w,
lease_expires_at = :e,
cursor_text = '',
artifact_id = null
where report_key = :k
and lease_expires_at <= :n
returning job_id
"""),
{"k": report_key, "w": worker_id, "e": expires, "n": now},
).scalar_one_or_none()
if stolen is None:
raise HTTPException(status_code=409, detail="lease_lost")
return {"job_id": str(stolen), "status": "queued", "stolen": True}
Notice what the handler refuses to do. It does not trust the browser to debounce. It does not trust the model to remember a job. It does not let an expired worker finalize later. If two starts arrive, they share one job_id. Tenant comes from the session, not from the JSON body. A claimed tenant is only a mismatch check.
Status codes carry that contract across layers. 201 is unnecessary here if the body already flags created. 200 with join: true means attach to the spinner. 409 lease_lost means this caller does not own the write. Mapping those codes to a spinner is a frontend problem. Mapping them to a second worker is an incident. Traces should log job_id beside turn_id. A trace that only logs the model request cannot explain the duplicate PDF.
The worker that must steal nothing
Heartbeats matter more than prompt tone here. A worker that dies mid-stream must drop the lease. Another worker may steal only after expiry. Steal uses compare-and-set on lease_owner and lease_expires_at. A late finalize from the corpse must get 409. Otherwise the stale PDF returns from the dead.
Streaming belongs on a cursor column, not the final object. Tokens can append while the lease is live. Clients reconnect by reading the cursor. They must not call start again for reconnect. Finalization copies the cursor once into object storage. That copy runs only if the owner still holds the lease. Partial streams therefore cannot become customer files.
# report_worker.py — labeled rehearsal, not a hidden product SDK
from fastapi import HTTPException
from sqlalchemy import text
@app.post("/reports/{job_id}/heartbeat")
def heartbeat(job_id: str, authorization: str | None = Header(default=None)):
session = parse_session(authorization)
worker_id = os.environ["WORKER_ID"]
with engine.begin() as conn:
now = db_now(conn)
row = conn.execute(
text("""
update report_jobs
set lease_expires_at = :e, status = 'running'
where job_id = :j
and tenant_id = :t
and lease_owner = :w
and lease_expires_at > :n
returning job_id
"""),
{"j": job_id, "t": session.tenant_id, "w": worker_id,
"n": now, "e": now + timedelta(seconds=LEASE_SECONDS)},
).scalar_one_or_none()
if row is None:
raise HTTPException(status_code=409, detail="lease_lost")
return {"ok": True}
@app.post("/reports/{job_id}/append")
def append(job_id: str, chunk: str, authorization: str | None = Header(default=None)):
session = parse_session(authorization)
worker_id = os.environ["WORKER_ID"]
with engine.begin() as conn:
now = db_now(conn)
row = conn.execute(
text("""
update report_jobs
set cursor_text = cursor_text || :c
where job_id = :j
and tenant_id = :t
and lease_owner = :w
and status = 'running'
and lease_expires_at > :n
returning char_length(cursor_text)
"""),
{"j": job_id, "t": session.tenant_id, "w": worker_id,
"c": chunk, "n": now},
).scalar_one_or_none()
if row is None:
raise HTTPException(status_code=409, detail="lease_lost")
return {"cursor_bytes": row}
@app.post("/reports/{job_id}/finalize")
def finalize(job_id: str, authorization: str | None = Header(default=None)):
session = parse_session(authorization)
worker_id = os.environ["WORKER_ID"]
with engine.begin() as conn:
now = db_now(conn)
row = conn.execute(
text("""
select cursor_text, lease_owner, lease_expires_at, tenant_id
from report_jobs
where job_id = :j
for update
"""),
{"j": job_id},
).mappings().first()
if row is None:
raise HTTPException(status_code=404, detail="unknown_job")
if row["tenant_id"] != session.tenant_id:
raise HTTPException(status_code=403, detail="tenant_mismatch")
if row["lease_owner"] != worker_id or row["lease_expires_at"] <= now:
raise HTTPException(status_code=409, detail="not_owner")
artifact_id = put_object(row["cursor_text"], session.tenant_id)
conn.execute(
text("""
update report_jobs
set status = 'complete',
artifact_id = :a,
lease_owner = null,
lease_expires_at = null
where job_id = :j and lease_owner = :w
"""),
{"a": artifact_id, "j": job_id, "w": worker_id},
)
return {"artifact_id": artifact_id, "status": "complete"}
The decision matrix the rehearsal actually ran looks like this. Keep it next to the handler, not in a wiki diagram.
event existing row http / body
---------------------------- ------------------------ -------------------------
POST, no row none 200 created=true
POST, running live lease running, same report_key 200 join=true
POST, complete complete 200 artifact_id
POST, expired lease running, expired 200 stolen=true
POST, body tenant mismatch any 403 tenant_mismatch
heartbeat from other worker live foreign owner 409 lease_lost
append after expiry expired 409 lease_lost
finalize from dead owner expired or other owner 409 not_owner
GET artifact, other tenant complete 403 tenant_mismatch
Replay the double click on purpose
The working path is a double click, not a happy screenshot. The fixture uses one tenant, one period, and two start calls. Both clicks hit the same start path the browser will hit. A unit test that never leaves the worker will miss the second POST. Commands below are the rehearsal, labeled as such.
# schema once
psql "$DATABASE_URL" -c "
create table if not exists report_jobs (
job_id uuid primary key default gen_random_uuid(),
report_key text unique not null,
tenant_id text not null,
status text not null,
lease_owner text,
lease_expires_at timestamptz,
cursor_text text not null default '',
artifact_id text,
created_at timestamptz not null
);"
# first click creates the job
curl -sS -X POST "$HOST/reports/start" \
-H "authorization: Bearer $TENANT_A" \
-H 'content-type: application/json' \
-d '{"report_type":"customer_health","period":"2026-08"}'
# expect created=true and a job_id
# second click must join, not fork
curl -sS -X POST "$HOST/reports/start" \
-H "authorization: Bearer $TENANT_A" \
-H 'content-type: application/json' \
-d '{"report_type":"customer_health","period":"2026-08"}'
# expect join=true and the same job_id
# other tenant claiming Acme in the body
curl -sS -o /tmp/body -w "%{http_code}\n" \
-X POST "$HOST/reports/start" \
-H "authorization: Bearer $TENANT_B" \
-H 'content-type: application/json' \
-d '{"report_type":"customer_health","period":"2026-08","claimed_tenant_id":"tenant_a"}'
# expect 403 tenant_mismatch
# corpse finalize after killing WORKER_ID_1
curl -sS -o /tmp/body -w "%{http_code}\n" \
-X POST "$HOST/reports/$JOB/finalize" \
-H "authorization: Bearer $TENANT_A"
# expect 409 not_owner once the lease expired or owner changed
Pytest wraps those calls so a green demo cannot hide a fork. The test is the contract. The streaming panel is decoration.
# test_report_lease.py
from fastapi.testclient import TestClient
from report_lease import app
client = TestClient(app)
def test_second_start_joins_same_job(tenant_a_header):
a = client.post(
"/reports/start",
headers={"authorization": tenant_a_header},
json={"report_type": "customer_health", "period": "2026-08"},
)
b = client.post(
"/reports/start",
headers={"authorization": tenant_a_header},
json={"report_type": "customer_health", "period": "2026-08"},
)
assert a.status_code == 200 and b.status_code == 200
assert a.json()["job_id"] == b.json()["job_id"]
assert b.json().get("join") is True
def test_body_tenant_cannot_override_session(tenant_b_header):
res = client.post(
"/reports/start",
headers={"authorization": tenant_b_header},
json={
"report_type": "customer_health",
"period": "2026-08",
"claimed_tenant_id": "tenant_a",
},
)
assert res.status_code == 403
assert res.json()["detail"] == "tenant_mismatch"
def test_foreign_finalize_is_409(tenant_a_header, monkeypatch):
start = client.post(
"/reports/start",
headers={"authorization": tenant_a_header},
json={"report_type": "customer_health", "period": "2026-08"},
)
job_id = start.json()["job_id"]
monkeypatch.setenv("WORKER_ID", "worker_other")
res = client.post(
f"/reports/{job_id}/finalize",
headers={"authorization": tenant_a_header},
)
assert res.status_code == 409
assert res.json()["detail"] == "not_owner"
Reconnect tests belong on the cursor, not on start. A dropped SSE session should GET /reports/{job_id} and keep reading cursor_text. If reconnect calls start, the lease still protects storage. The UI will still look like a fork. Users will click again. That click must remain a join.
What still breaks after the lease is real
A lease is not a distributed lock service. Clock skew can make expiry lie. Heartbeats must use the database clock, not the worker laptop. select now() in the same transaction beats datetime.utcnow() in application code. report_key must include tenant, type, and period. A missing tenant turns the mutex into a cross-customer jam. Object-level checks still belong on download. A leased write can still leak if GET /artifacts/:id is public.
Retries from the planner remain expensive even with one job row. A worker that appends the same paragraph twice will poison the cursor. Idempotent append needs a stream sequence number. This article leaves that number out on purpose. Ship the lease first. Then number the chunks. Doing both in one weekend usually ships neither.
A free rehearsal host is not a control plane. Secrets, egress, and disks on that box still need a kill switch. Free model access does not make double jobs cheaper in production. It only makes the race cheap to reproduce. Persistence still needs the same backups as any job table. Lease rows are operational data. They are not chat logs.
Skip this approach for a single-user toy. Skip it when nothing durable is stored. Skip it when a second click should create a second draft on purpose. Skip it when the team cannot keep job rows as long as finance keeps PDFs. If nobody can name the 409 for a stolen lease, stop. The chat UI is not ready for generation.
The reusable check is short on purpose. Two starts with one key must return one job. A live lease must reject a foreign finalize with 409. An expired lease may be stolen once, never twice. A reconnect must read the cursor, not insert. Download must still check the session tenant. Replay from the browser path must match TestClient. If those six sentences fail, the model never got a chance to be the bug.
This article is not a claim that agents are fake. It is a claim that retries are the real user. The model will keep talking through both clicks. The job table has to pick one kitchen. If the doorbell can build a second house, vocabulary will not save the invoice.
Send the job row after a double click. Include status, lease_owner, and artifact_id. A chat screenshot cannot show the race.
Top comments (0)