Replayable Runs > Faster Runs. Stop Optimising for the Wrong Number.

Most "we made it 3x faster" scraper posts miss the actual point. Speed is rarely the constraint. Replayability is.

If your scraper takes 4 hours to run and a single URL fails halfway through, can you re-run just that URL in 30 seconds? Or do you have to start the whole 4-hour job over?

If the answer is "start over," you don't have a scraper. You have a long-running prayer.

The 3-item checklist

A replayable run looks like this:

1. Inputs are explicit and persisted. Every URL/parameter the run is processing is written to a queue or dataset before it starts. You can re-read the input list later.
2. Outputs are addressable per input. You can ask "did URL X succeed?" and get a yes/no, not "well, the run finished, so probably."
3. Failures are first-class records. Failed inputs go to a separate dataset/queue with the error reason, ready to feed back into a retry run.

When all three hold, "rerun the failures" is a one-liner. When any of them is missing, recovery is manual archaeology.

The trick — input/output as separate datasets

Here's the shape:

from apify import Actor, Dataset
from apify.storages import RequestQueue

await Actor.init()
input_data = await Actor.get_input()

# 1. Push inputs into a queue. Idempotent — re-runs skip already-done items.
queue = await RequestQueue.open(name="podcast-urls-2026-06")
for url in input_data["urls"]:
    await queue.add_request({"url": url, "uniqueKey": url})

# 2. Process the queue, splitting outputs into success and failure datasets.
results = await Dataset.open(name="podcast-results-2026-06")
failures = await Dataset.open(name="podcast-failures-2026-06")

while (request := await queue.fetch_next_request()):
    try:
        record = await transcribe(request["url"])
        await results.push_data(record)
        await queue.mark_request_as_handled(request)
    except Exception as e:
        await failures.push_data({
            "url": request["url"],
            "error": str(e),
            "failed_at": datetime.utcnow().isoformat(),
        })
        await queue.mark_request_as_handled(request)  # don't retry blindly

Three storages: input queue, success dataset, failure dataset. The queue is keyed by URL, so adding the same URLs again is a no-op. The failure dataset is the input for the next retry run.

Quick case

The podcast transcription actor used to be a 6-hour batch job. When a single episode failed (audio download timeout, transcription model glitch, anything), the recovery story was: "find the failed URL in the logs, hand-craft a one-URL run, hope the second try works."

After moving to the queue + split-dataset pattern:

• Failed URLs are visible in a dedicated dataset, with the error and timestamp.
• "Retry yesterday's failures" is one button: open the failures dataset, push its rows into a new run as input.
• The original run's success dataset doesn't get re-processed — it just gets appended to.

What used to take 30 minutes of manual triage is now a 30-second action. Same scraper, same selectors, same model — different runtime structure.

The CTA you didn't ask for

The queue + success-dataset + failure-dataset pattern is the third thing every actor we ship gets, after request blocking and selector ladder — visible in the podcast transcription actor. (We have a starter template now. Same shape every time.)

So:

Open your scraper. If a single URL fails, what does recovery look like? If your answer takes more than one paragraph, drop it in the comments — I'll show you the smaller version.

Agree, disagree, or have a recovery story that doesn't need this? Reply.

---

Written by **Nova Chen, Automation Dev Advocate at SIÁN Agency. Find more from Nova on dev.to. For custom scraping or automation work, hire SIÁN Agency.