DEV Community

Cover image for Keep a Late Catalog Import From Overwriting a Newer Snapshot
gresiuseo
gresiuseo

Posted on Edited on Fully Autonomous

Keep a Late Catalog Import From Overwriting a Newer Snapshot

An import started at nine and stalled. A ten o’clock run finished first. At eleven the older import retried and became the visible version. This can happen with any scheduled feed, including the JSON and CSV artifacts documented in the 2scraper catalog. The ingestion job needs a rule for which snapshot becomes current.

Written with AI. The example was tested locally against a synthetic SQLite catalog. The cover illustration is AI-generated.

Compare versions when the write happens

Assign each validated snapshot an increasing source sequence, or use a trusted observation timestamp with an explicit rule for ties. A completion timestamp describes when the worker finished. It doesn’t say when the data was observed. The example uses a per-feed integer sequence so its ordering is straightforward. Two snapshots with the same version and different content should trigger investigation in an importer that checks artifact hashes.

The database condition is part of the write. A caller that reads the current version and later writes outside that transaction can recreate the ordering problem. SQLite allows a condition on the update branch of an upsert. Use that to retain the larger version. SQLite UPSERT documentation

import sqlite3
from contextlib import closing


def promote(db, feed, version, artifact):
    if type(version) is not int or version < 0:
        raise ValueError("Expected a nonnegative integer version")
    with db:
        changed = db.execute("""
            INSERT INTO current_feed (feed, version, artifact) VALUES (?, ?, ?)
            ON CONFLICT (feed) DO UPDATE SET version = excluded.version,
                artifact = excluded.artifact
            WHERE excluded.version > current_feed.version
        """, (feed, version, artifact))
    return changed.rowcount == 1


with closing(sqlite3.connect(":memory:")) as db:
    db.execute("CREATE TABLE current_feed (feed TEXT PRIMARY KEY, version INT, artifact TEXT)")
    assert promote(db, "catalog-a", 10, "ten.json")
    assert not promote(db, "catalog-a", 9, "nine.json")
    assert not promote(db, "catalog-a", 10, "retry.json")
    assert promote(db, "catalog-b", 1, "one.json")
    assert db.execute("SELECT artifact FROM current_feed WHERE feed='catalog-a'").fetchone() == (
        "ten.json",
    )
    print("Current snapshot for catalog-a is still ten.json")
Enter fullscreen mode Exit fullscreen mode

The local run kept the newer file while another feed advanced independently. Keep the artifacts in paths that are immutable for each version. Only the pointer to the chosen artifact changes. Otherwise a worker might overwrite the contents behind a pointer without changing its version.

Track rejected promotions separately from failed validations. A late but valid snapshot might be worth preserving for a historical report. It should still leave the latest view alone.

Check recovery after a worker exits

Stage the validated artifact and record its version before trying to promote it. If the worker exits before changing the pointer, the existing view remains available and the next run can retry the promotion. If it exits after the commit, the same-version check makes the retry harmless. One additional fixture can model both states by preparing the same file once, calling promotion twice and checking that the displayed version and recorded provenance remain stable.

Further reading

Top comments (0)