An import started at nine and stalled. A ten o’clock run finished first. At eleven the older import retried and became the visible version. This can happen with any scheduled feed, including the JSON and CSV artifacts documented in the 2scraper catalog. The ingestion job needs a rule for which snapshot becomes current.
Written with AI. The example was tested locally against a synthetic SQLite catalog. The cover illustration is AI-generated.
Compare versions when the write happens
Assign each validated snapshot an increasing source sequence, or use a trusted observation timestamp with an explicit rule for ties. A completion timestamp describes when the worker finished. It doesn’t say when the data was observed. The example uses a per-feed integer sequence so its ordering is straightforward. Two snapshots with the same version and different content should trigger investigation in an importer that checks artifact hashes.
The database condition is part of the write. A caller that reads the current version and later writes outside that transaction can recreate the ordering problem. SQLite allows a condition on the update branch of an upsert. Use that to retain the larger version. SQLite UPSERT documentation
import sqlite3
from contextlib import closing
def promote(db, feed, version, artifact):
if type(version) is not int or version < 0:
raise ValueError("Expected a nonnegative integer version")
with db:
changed = db.execute("""
INSERT INTO current_feed (feed, version, artifact) VALUES (?, ?, ?)
ON CONFLICT (feed) DO UPDATE SET version = excluded.version,
artifact = excluded.artifact
WHERE excluded.version > current_feed.version
""", (feed, version, artifact))
return changed.rowcount == 1
with closing(sqlite3.connect(":memory:")) as db:
db.execute("CREATE TABLE current_feed (feed TEXT PRIMARY KEY, version INT, artifact TEXT)")
assert promote(db, "catalog-a", 10, "ten.json")
assert not promote(db, "catalog-a", 9, "nine.json")
assert not promote(db, "catalog-a", 10, "retry.json")
assert promote(db, "catalog-b", 1, "one.json")
assert db.execute("SELECT artifact FROM current_feed WHERE feed='catalog-a'").fetchone() == (
"ten.json",
)
print("Current snapshot for catalog-a is still ten.json")
The local run kept the newer file while another feed advanced independently. Keep the artifacts in paths that are immutable for each version. Only the pointer to the chosen artifact changes. Otherwise a worker might overwrite the contents behind a pointer without changing its version.
Track rejected promotions separately from failed validations. A late but valid snapshot might be worth preserving for a historical report. It should still leave the latest view alone.
Check recovery after a worker exits
Stage the validated artifact and record its version before trying to promote it. If the worker exits before changing the pointer, the existing view remains available and the next run can retry the promotion. If it exits after the commit, the same-version check makes the retry harmless. One additional fixture can model both states by preparing the same file once, calling promotion twice and checking that the displayed version and recorded provenance remain stable.
Further reading
- Keeping balance dashboard account boundaries to test which records each query can return.
- Testing a replacement dashboard against the old one to compare both reporting paths before moving readers.
Top comments (0)