A job lease expires. Worker B takes over. Worker A wakes up with a result it computed before the handoff.
If completion is just UPDATE jobs ... WHERE job_id = ?, A can still overwrite B's result. The timeout made takeover possible. It did not stop A's code.
This example reproduces that sequence in SQLite, then makes the database refuse the old worker's completion. It includes an unsafe control and eleven local tests, with a controlled clock instead of sleeps.
The previous outbox example dealt with repeated delivery of one event. This is a different question: which worker's current claim may commit a result?
Give each claim a generation
A lease is temporary ownership of a job. A heartbeat renews it while the worker is still entitled to hold it. Expiry lets another worker claim the job.
The failure sequence is short:
| Fixture time | Action | Stored ownership |
|---|---|---|
| 0 | A claims report-1
|
A, generation 1, expiry 10 |
| 10 | B claims the expired job | B, generation 2, expiry 20 |
| 10 | B commits its result | Done, generation 2 |
| 10 | A submits its delayed result | Refused by the guarded path |
All numbers are test values. The fixture does not wait ten real seconds or run a report service.
Each successful claim increments the job's stored generation. Renewal keeps the same generation. Completion checks the job ID, owner, generation, state and deadline in its write predicate:
UPDATE jobs
SET state = 'done', result = ?, owner = NULL, lease_until = NULL
WHERE job_id = ? AND owner = ? AND generation = ?
AND state = 'ready' AND lease_until > ?;
Zero matched rows means refusal. It is not a successful completion with an empty result.
The generation matters even if the owner string is reused. A restarted process called worker-a can acquire generation 2; that must not revive a delayed operation carrying worker-a, generation 1.
The database participates in the decision
Martin Kleppmann's explanation of fencing tokens describes the general requirement: the protected resource must reject an older token. A check performed only by the worker can become stale before the write reaches its destination.
AWS's discussion of leader election also calls out pauses between checking a lease and performing work. It discusses using separate correctness mechanisms when more than one process believes it is leader.
This fixture uses a narrower arrangement: the lease row, protected job result and completion record live in one SQLite database. BEGIN IMMEDIATE acquires a write transaction before sampling the fixture clock. The guarded result update and the completion record commit together.
SQLite permits one simultaneous write transaction. Another claim therefore cannot commit between that guarded update and its commit. No transaction remains open while a worker performs the modeled computation; only claim, renewal and completion touch the database.
That is not a generic implementation of a distributed lock service. It is a local demonstration of keeping the ownership check at the same storage boundary as the protected change.
Test the worker that comes back
A takeover test that only checks “B eventually ran” misses the dangerous part. Retain A's old lease and actually attempt its completion.
The tests below check both orders: A returns while B still owns the job, and A returns after B has committed. In both cases, refusal leaves the current row unchanged.
| Test | Required observation |
|---|---|
| Unsafe completion after takeover | A overwrites B's result, demonstrating the bug |
| Stale completion during B's lease | Refusal; B's ownership and job data stay unchanged |
| Stale completion after B commits | Refusal; B's result remains |
| Claim at the exact expiry boundary | Before 10, B cannot claim; at 10, it can |
| Expired owner, with no takeover | Completion is refused at the sampled deadline |
| Expired heartbeat | It cannot revive the claim |
| Same owner name, newer generation | The old generation is still refused |
| Altered job ID with another owner on that job | Neither job changes |
| Two simultaneous claimers | One gets a lease; the other gets none |
| Completion-record insertion fails | The job update also rolls back |
| Live heartbeat | It extends the current claim without incrementing its generation |
The simultaneous-claim test releases two threads through a barrier. Each opens its own SQLite connection. The takeover cases use an explicitly advanced test clock; they do not depend on a scheduler pause happening by chance.
Run the complete example
Save this as worker_fence.py and run python worker_fence.py with Python 3.12 or later. It uses the standard library, creates temporary databases under test-db beside the file, and removes each test's own temporary subdirectory after closing its connections.
All eleven tests passed locally with Python 3.12.14 and SQLite 3.53.1.
"""Local lease-generation fixture. Python 3.12+; no queue or external API.
Run: python worker_fence.py
"""
from concurrent.futures import ThreadPoolExecutor
from contextlib import closing, contextmanager
from dataclasses import dataclass, replace
from pathlib import Path
from threading import Barrier
import sqlite3
import tempfile
import unittest
@dataclass(frozen=True)
class Lease:
job_id: str
owner: str
generation: int
class LeaseLost(Exception):
pass
class Clock:
"""Trusted deterministic test clock, shared by the fixture's DB methods."""
def __init__(self, now=0):
self.now = now
def __call__(self):
return self.now
def connect(path):
db = sqlite3.connect(path, timeout=5, autocommit=True)
db.row_factory = sqlite3.Row
return db
def one(db, sql, args=()):
with closing(db.execute(sql, args)) as cursor:
return cursor.fetchone()
@contextmanager
def transaction(path):
with closing(connect(path)) as db:
try:
db.execute("BEGIN IMMEDIATE").close()
yield db
db.execute("COMMIT").close()
except BaseException:
if db.in_transaction:
db.execute("ROLLBACK").close()
raise
def initialize(path):
with closing(connect(path)) as db:
db.executescript("""
CREATE TABLE jobs (
job_id TEXT PRIMARY KEY,
generation INTEGER NOT NULL DEFAULT 0,
owner TEXT, lease_until INTEGER,
state TEXT NOT NULL DEFAULT 'ready'
CHECK (state IN ('ready', 'done')),
result TEXT
);
CREATE TABLE completions (
job_id TEXT PRIMARY KEY,
generation INTEGER NOT NULL,
result TEXT NOT NULL
);
INSERT INTO jobs (job_id) VALUES ('report-1'), ('report-2');
""").close()
def valid_ttl(ttl):
if type(ttl) is not int or ttl not in range(1, 3601):
raise ValueError("fixture TTL must be an integer in 1..3600")
def claim(path, job_id, owner, clock, ttl=10):
valid_ttl(ttl)
if not isinstance(owner, str) or not owner:
raise ValueError("owner must be non-empty")
with transaction(path) as db:
now = clock() # Sample after acquiring the DB write transaction.
with closing(db.execute("""UPDATE jobs
SET generation = generation + 1, owner = ?, lease_until = ?
WHERE job_id = ? AND state = 'ready'
AND (owner IS NULL OR lease_until <= ?)""",
(owner, now + ttl, job_id, now))) as cursor:
if cursor.rowcount != 1:
return None
generation = one(db, "SELECT generation FROM jobs WHERE job_id = ?",
(job_id,))[0]
return Lease(job_id, owner, generation)
def renew(path, lease, clock, ttl=10):
valid_ttl(ttl)
with transaction(path) as db:
now = clock()
with closing(db.execute("""UPDATE jobs SET lease_until = ?
WHERE job_id = ? AND owner = ? AND generation = ?
AND state = 'ready' AND lease_until > ?""",
(now + ttl, lease.job_id, lease.owner,
lease.generation, now))) as cursor:
if cursor.rowcount != 1:
raise LeaseLost("renewal refused")
def complete(path, lease, result, clock):
with transaction(path) as db:
now = clock()
with closing(db.execute("""UPDATE jobs
SET state = 'done', result = ?, owner = NULL, lease_until = NULL
WHERE job_id = ? AND owner = ? AND generation = ?
AND state = 'ready' AND lease_until > ?""",
(result, lease.job_id, lease.owner,
lease.generation, now))) as cursor:
if cursor.rowcount != 1:
raise LeaseLost("completion refused")
# Both records are local to this same database transaction.
db.execute("INSERT INTO completions VALUES (?, ?, ?)",
(lease.job_id, lease.generation, result)).close()
return result
def complete_without_fence(path, lease, result):
"""Deliberately unsafe control; do not use as a completion path."""
with transaction(path) as db:
db.execute("""UPDATE jobs SET state = 'done', result = ?,
owner = NULL, lease_until = NULL WHERE job_id = ?""",
(result, lease.job_id)).close()
def snapshot(path, job_id='report-1'):
with closing(connect(path)) as db:
return dict(one(db, "SELECT * FROM jobs WHERE job_id = ?", (job_id,)))
class FencingTests(unittest.TestCase):
def setUp(self):
base = Path(__file__).resolve().parent / 'test-db'
base.mkdir(exist_ok=True)
self.temp = tempfile.TemporaryDirectory(prefix='fence-', dir=base)
folder = Path(self.temp.name).resolve()
assert folder.is_relative_to(base.resolve())
self.addCleanup(self.temp.cleanup)
self.path = folder / 'jobs.sqlite'
self.clock = Clock()
initialize(self.path)
def claim(self, owner='worker-a', job_id='report-1'):
return claim(self.path, job_id, owner, self.clock)
def takeover(self, owner='worker-b'):
first = self.claim()
self.clock.now = 10
second = self.claim(owner)
self.assertEqual((first.generation, second.generation), (1, 2))
return first, second
def completions(self):
with closing(connect(self.path)) as db:
return one(db, 'SELECT count(*) FROM completions')[0]
def test_unsafe_old_worker_overwrites_new_result(self):
old, current = self.takeover()
complete(self.path, current, 'new result', self.clock)
complete_without_fence(self.path, old, 'stale result')
self.assertEqual(snapshot(self.path)['result'], 'stale result')
def test_stale_completion_cannot_disturb_new_lease(self):
old, current = self.takeover()
before = snapshot(self.path)
with self.assertRaises(LeaseLost):
complete(self.path, old, 'stale result', self.clock)
self.assertEqual(snapshot(self.path), before)
self.assertEqual(self.completions(), 0)
complete(self.path, current, 'new result', self.clock)
self.assertEqual(snapshot(self.path)['result'], 'new result')
def test_stale_completion_after_new_commit_keeps_new_result(self):
old, current = self.takeover()
complete(self.path, current, 'new result', self.clock)
before = snapshot(self.path)
with self.assertRaises(LeaseLost):
complete(self.path, old, 'stale result', self.clock)
self.assertEqual(snapshot(self.path), before)
self.assertEqual(self.completions(), 1)
def test_live_lease_blocks_claim_until_exact_expiry(self):
first = self.claim()
self.clock.now = 9
self.assertIsNone(self.claim('worker-b'))
self.clock.now = 10
second = self.claim('worker-b')
self.assertEqual(second.generation, first.generation + 1)
def test_expired_owner_cannot_complete_without_a_takeover(self):
lease = self.claim()
self.clock.now = 10
before = snapshot(self.path)
with self.assertRaises(LeaseLost):
complete(self.path, lease, 'late result', self.clock)
self.assertEqual(snapshot(self.path), before)
self.assertEqual(self.completions(), 0)
def test_expired_heartbeat_cannot_revive_lease(self):
lease = self.claim()
self.clock.now = 10
before = snapshot(self.path)
with self.assertRaises(LeaseLost):
renew(self.path, lease, self.clock)
self.assertEqual(snapshot(self.path), before)
def test_same_owner_name_does_not_revive_old_generation(self):
old, current = self.takeover('worker-a')
self.assertEqual(old.owner, current.owner)
before = snapshot(self.path)
with self.assertRaises(LeaseLost):
complete(self.path, old, 'stale result', self.clock)
self.assertEqual(snapshot(self.path), before)
complete(self.path, current, 'current result', self.clock)
def test_wrong_job_is_refused_and_both_rows_stay_unchanged(self):
first = self.claim()
second = self.claim('worker-b', 'report-2')
before = [snapshot(self.path, name) for name in ['report-1', 'report-2']]
with self.assertRaises(LeaseLost):
complete(self.path, replace(first, job_id=second.job_id),
'wrong job', self.clock)
self.assertEqual([snapshot(self.path, name)
for name in ['report-1', 'report-2']], before)
self.assertEqual(self.completions(), 0)
def test_two_simultaneous_claimers_get_one_lease(self):
barrier = Barrier(2, timeout=5)
def compete(owner):
barrier.wait()
return self.claim(owner)
with ThreadPoolExecutor(max_workers=2) as pool:
futures = [pool.submit(compete, owner) for owner in ['a', 'b']]
results = [future.result(timeout=10) for future in futures]
winners = [result for result in results if result is not None]
self.assertEqual(len(winners), 1)
self.assertEqual(winners[0].generation, 1)
self.assertEqual(snapshot(self.path)['owner'], winners[0].owner)
def test_audit_insert_failure_rolls_back_completion(self):
lease = self.claim()
before = snapshot(self.path)
with closing(connect(self.path)) as db:
db.execute("""CREATE TRIGGER reject_completion
BEFORE INSERT ON completions
BEGIN SELECT RAISE(ABORT, 'injected'); END""").close()
with self.assertRaises(sqlite3.IntegrityError):
complete(self.path, lease, 'result', self.clock)
self.assertEqual(snapshot(self.path), before)
self.assertEqual(self.completions(), 0)
def test_live_heartbeat_extends_same_generation(self):
lease = self.claim()
self.clock.now = 9
renew(self.path, lease, self.clock)
self.clock.now = 10
self.assertIsNone(self.claim('worker-b'))
self.assertEqual(snapshot(self.path)['generation'], lease.generation)
complete(self.path, lease, 'still current', self.clock)
self.assertEqual(snapshot(self.path)['result'], 'still current')
self.assertEqual(self.completions(), 1)
if __name__ == '__main__':
import sys
print(f'Python {sys.version.split()[0]}; SQLite {sqlite3.sqlite_version}',
flush=True)
unittest.main(verbosity=2)
What this does not stop
A can still burn CPU, finish a download, or send an external request after its lease expires. The guarded SQLite update does not interrupt that process.
If the effect is an email, payment or social post, its actual receiver needs an appropriate enforcement and reconciliation contract. Adding a generation field to a request achieves nothing if the receiver ignores it. The database transaction here cannot roll back an external API call.
Keep these boundaries in mind when adapting the fixture:
- The clock is a test seam. It is trusted code, sampled inside the write transaction, not a timestamp supplied by the worker. The example does not establish a production clock policy or coordinated time across machines. A process paused inside the transaction can also hold the write lock past the sampled deadline; it blocks takeover rather than proving a real-time completion guarantee.
- The generation must survive recovery. The job row retains it after completion. Reusing a job identity or restoring an older generation can invalidate the stale-worker argument. Plan recovery for the authoritative state as well as the workers.
-
This is coordination, not authentication.
Leaseis a plain dataclass. The fixture has no authenticated transport, signed token or tenant authorization layer. It assumes trusted workers use their issued claim; it does not defend against someone forging all its fields or bypassing the guarded function. - Completion is not replay recovery. A repeated completion after success is refused because the job is already done. If its response was lost, recover the outcome from durable state instead of treating refusal as proof that the first attempt failed. The earlier operation-receipt example covers that separate problem.
The file also omits a queue, worker supervision, retry policy, cancellation propagation and throughput measurements. Its five-second database busy timeout is a fixture setting. The tests exercise the local storage boundary, not crash recovery or a deployed distributed system.
A useful first check for a real worker is to find its final write. Does that write still accept an old claim after another worker has taken over? A healthy new worker and a stopped old worker are separate things to verify.
Which stale-worker case would you add next: a late heartbeat, a reused worker name, or a completion delayed until after takeover? Leave the sequence in the comments, including which component can actually refuse the final effect.
Top comments (1)
worker_fence.pyran as posted: Python 3.14.9, SQLite 3.53.1, 11 of 11 tests pass in 0.249 s. The generation check is the right shape, and the expiry boundary is consistent in both directions (claim useslease_until <= now, renew and complete uselease_until > now).You asked which case to add next. I tried all three: a late heartbeat after takeover is refused on
owner/generationand leaves the row untouched (renewal refused), so that one is safe, as are the reused worker name and the delayed completion you already test. What leaks is elsewhere, in two places, plus a third on the requeue path.A refusal is untraceable. After B took over, A's completion raised
LeaseLost("completion refused"), the row and thecompletionscount were byte-identical, andsqlite_mastershows the two tables below hold nothing else. There is nostale_attemptsrow, so after the fact "the worker was asleep for 95 s" and "the worker ran twice" look the same in the database.Re-reading the generation borrows authority. A worker that pauses before the handoff, then reads
owner/generationout of the row itself, completes as the current holder:The newer worker loses to the older one, and nothing distinguishes the two completions. Carrying the
Leaseyourclaim()returned is what makes the fence work; a codebase that storesgenerationas a column and passesjob.generationgets this bug with the fixture's SQL unchanged.Requeue collides with the
completionsprimary key. Flip the job back toready, claim it again (generation 3), complete:Because the insert shares the completion transaction, the rollback takes the guarded
UPDATEwith it, so the result is written nowhere. WithPRIMARY KEY (job_id, generation)the same sequence commits. If a retried or reissued job is out of scope anddoneis terminal, say so in the article; it is the first thing an operator hits after a bad run.That is the case I would add next: a stale completion arriving after the job was requeued under a new generation, asserted against the
completionstable rather than the job row.