Retry Is Not Resume: Building a Web-to-CSV Pipeline That Survives Interruptions

The first run is the easy part.

The real test starts after record 70 of 100 succeeds, the process stops, and you
run it again. If the pipeline fetches the first 70 pages again, appends duplicate
rows, or trusts a half-written state file, it is not resumable. It is only
restartable.

While building a small Python + Playwright source kit, I separated retry from
resume and gave the output data, progress state, and run evidence different
responsibilities. This post walks through the decisions that made the biggest
difference.

Retry and resume solve different failures

A retry handles a temporary failure inside one run:

• a request times out;
• a page returns incomplete markup;
• a required field is temporarily missing.

Resume handles a different event: the run itself ends and a later process must
continue from durable evidence.

Putting both concerns inside one retry loop hides the distinction. Increasing
the retry count does nothing after a terminal closes or a machine restarts.
Instead, I used an item-level retry loop and wrote a completed-record map only
after parsing succeeded.

The essential flow is small:

completed = read_state(config_fingerprint)

for item in input_rows:
    if item.record_id in completed:
        log_skip(item.record_id, reason="completed")
        continue

    parsed = retry(lambda: parse(fetch(item.url)), max_retries=2)
    write_result(item.record_id, item.url, parsed)
    completed[item.record_id] = hash_output(parsed)

write_state_atomically(completed, config_fingerprint)

Retry decides whether the current item gets another attempt. Resume decides
whether the item should be fetched at all in a future run.

The CSV is not the progress database

It is tempting to inspect the last CSV row and treat it as a cursor. That works
until input order changes, a blank line appears, or the output schema changes.
The data file then has two jobs: representing results and controlling execution.

I kept them separate:

• normalized.csv contains the current result rows;
• state.json stores completed record IDs and output hashes;
• run_log.jsonl records succeeded, failed, retrying, and skipped events;
• report.html gives a human-readable result.

The input contract requires record_id,url. A seen set catches duplicate IDs
inside the current input. The durable completed map catches IDs that succeeded
in an earlier run.

This also makes a repeated run understandable. A row is not silently absent; the
log says whether it was skipped because it was a duplicate or because it had
already completed.

Old state must not silently match a new configuration

Resume state becomes dangerous when extraction rules change.

Suppose the first run extracted title and price. Later, the configuration
adds availability. If the old completed IDs are accepted without question,
the pipeline skips pages that have never been processed with the new rule.

The kit hashes a canonical representation of the field configuration:

fingerprint = sha256(
    json.dumps(
        canonical_fields,
        sort_keys=True,
        separators=(",", ":"),
    ).encode("utf-8")
).hexdigest()

That fingerprint is written beside the completed map. When a later run points
to the same output directory with a different configuration, the pipeline
raises an explicit error instead of combining incompatible results.

The fix is intentionally boring: use a new output directory for a new
extraction contract. Quietly guessing would be more convenient and less safe.

Atomic writes protect the mechanism that enables recovery

A state file is useful only if it survives the interruption it is supposed to
help recover from.

Writing directly to state.json can leave truncated JSON if the process stops
at the wrong moment. The next run then fails before it can resume. The same risk
applies to the normalized CSV.

The pipeline writes a complete temporary file and replaces the destination:

temporary = path.with_suffix(path.suffix + ".tmp")
temporary.write_text(value, encoding="utf-8")
temporary.replace(path)

The implementation uses the same pattern for JSON, CSV, and the append-style
run log. It does not make every filesystem failure impossible, but it removes
the common partial-overwrite window.

Evidence should be useful without leaking page content

Debug logs often become accidental data stores. Dumping raw HTML, full query
strings, or local paths would make troubleshooting easier and distribution less
safe.

The public log records the record ID, status, attempt number, and exception type.
It does not record page HTML or local filesystem paths. The HTML report escapes
output values before rendering them.

That boundary matters for a reusable source kit: the person running it controls
the permitted target pages, while the library keeps operational evidence
focused on the run.

I tested the extracted delivery archive, not only the development folder

A passing development environment is not proof that a buyer's ZIP contains the
same working pieces.

I extracted the delivery archive into a clean directory and ran its included
test suite. The result was 10/10 tests passed. One regression test forces
KeyboardInterrupt immediately after the first record is durably checkpointed. A
second process then resumes and proves that the completed record is not fetched
again.

I also ran the offline fixture demo twice:

• 2 records succeeded;
• 0 records failed;
• 1 duplicate ID was skipped safely;
• the next run read the completed state instead of fetching finished records again.

The fixture demo uses local HTML and requires no browser, account, proxy, cloud
service, or paid API.

Where Playwright fits

The state, retry, parsing, reporting, and fixture paths do not import Playwright.
The browser dependency is loaded only for permitted http:// or https://
pages.

That keeps the offline demo deterministic and lets the same pipeline accept a
real browser fetcher when JavaScript-rendered pages are in scope.

This is not a universal scraper. Selectors are deliberately simple, and the
operator must define them for pages they own or are allowed to automate. The kit
does not bypass login, CAPTCHA, access controls, rate limits, robots rules, or
website terms.

The finished source kit

I packaged the implementation as the
Resumable Web-to-CSV Pipeline Kit.
It includes the full Python source, tests, configuration examples, offline
fixtures, and run documentation.

• Personal: $18 launch price (normally $24) for one user
• Team: $44.25 launch price (normally $59) for up to five users in one legal entity

The 25% launch discount is automatically applied through August 1, 20:13 KST for the first 20 uses.

The public
delivery repository
shows the actual product screens and verification evidence before purchase.
Custom selectors, custom development, remote setup, ongoing support, and future
updates are not included.

Watch the crash window in 38 seconds

The short proof reel shows the exact failure path: interrupt after useful work, rerun with retry-only logic, then resume from durable state without refetching completed records.

Watch the 38-second crash-proof Web-to-CSV demo

The reel uses the same fixed offline fixture documented above. It is evidence for this included example, not a promise about every website.

You can also
run the interruption walkthrough in your browser
without installing Python. Run 1 saves the first record and stops; after a real
page reload, Resume skips that completed record before reading its fixture and
finishes the second. This is a fixed offline walkthrough, not an arbitrary-site
benchmark.

Prefer the guided lab instead?

If you want the failure map and the interruption exercise without buying source
code, I also published the 20-page
Retry Is Not Resume field manual.
It includes a 30-minute offline run → stop → resume lab, checkpoint crash windows,
a four-file responsibility map, and a production checklist.

The launch price is $9, and the product page includes a free 6-page preview. The
v1.1 purchase ZIP contains the PDF and a self-contained lab that runs with
python lab/run_demo.py. It is a self-serve download: no custom setup, remote
installation, ongoing support, or future updates are included.

The larger lesson is independent of this kit: retries recover an attempt;
durable state recovers a workflow. Treating those as separate responsibilities
turns "run it again" from a gamble into a defined operation.