Your scraper's deadline should be shorter than the platform's

Your scraper has been running for eleven minutes. The platform kills runs at twelve. You have 340 listings in memory and nothing on disk.

Guess how that ends.

This is the failure mode we spent the most time designing around while building the VRBO Vacation Rentals Scraper, and it has almost nothing to do with VRBO. It's a general lesson about long-running scrapes that we relearned the expensive way on a different actor last week.

⏱️ The platform's deadline is not your deadline

Every serverless scraping platform enforces a hard run timeout. When you hit it, the process is killed — not asked politely to wind down. Whatever was buffered in memory is gone, and the customer gets a TIMED-OUT run they were still billed compute for. That last part is what makes it worse than a plain crash: they paid for nothing.

The naive fix is to make the run faster. That doesn't fix anything, it just moves the cliff. Some destinations have more inventory than others, some proxy exits are slower, some detail pages hang. Any of those can push a run that finished in eight minutes yesterday past the wall today.

The actual fix is to give the run its own budget, strictly shorter than the platform's, and check it at every loop boundary:

def _run_deadline(max_run_minutes: int) -> float:
    return time.monotonic() + max_run_minutes * SECONDS_PER_MINUTE

def _budget_exhausted(deadline: float) -> bool:
    return time.monotonic() >= deadline

Then, before opening anything new:

if _budget_exhausted(deadline):
    logger.warning("Wall-clock budget spent; stopping before destination %r", destination)
    break

The run stops starting work and goes straight to finalizing. The customer gets 280 clean rows and a log line explaining why it wasn't 340. That's a usable outcome. A TIMED-OUT with an empty dataset is not.

Two details that matter more than they look:

Use time.monotonic(), not time.time(). Wall-clock time can jump — NTP correction, container clock drift. A monotonic clock cannot go backwards, and a deadline that goes backwards is a deadline that never fires.

Anchor your budget to the platform's real deadline, not a hardcoded guess. This is the bit we got wrong elsewhere: we set a fixed internal budget, the run's actual timeout got configured differently, and the safety margin silently inverted. If the platform tells you when it plans to kill you, read that value and subtract your margin from it.

🚧 Why VRBO needs the budget in the first place

VRBO (Expedia Group) doesn't hand raw HTTP clients a results page. Send a plain request and you get a JS-execution gate — a challenge screen, not listings. So there's a rendered browser layer in front of every search page, which is roughly an order of magnitude slower and heavier per page than a JSON fetch.

That cost is what makes the deadline real. When each page is a browser navigation, a 20-listing scrape with detail pages is a lot of seconds, and the variance between a fast run and a slow one is large. Cheap JSON scrapers can get away without a budget for a long time. Browser-backed ones cannot.

The same reasoning drives the per-listing fault isolation: one dead detail page or navigation timeout skips that listing with a logged warning and the loop continues. A single bad page taking down 19 good ones is the other way to hand someone an empty dataset.

🧭 The pattern, generalized

If you're writing anything that loops over an unknown number of remote fetches under a hard timeout:

1. Set an internal budget derived from the platform's actual deadline, with margin.
2. Check it at the top of every loop iteration — before starting work, not after.
3. Break to your normal finalize path. Never sys.exit(), never let the timeout do it for you.
4. Log why you stopped, with the count. Silent truncation reads as "that's all the data there was."
5. Isolate per-item failures so one bad page doesn't consume the whole budget in retries.

Point 5 is the one people skip. A retry storm on a single hanging page will eat your entire budget and leave you with the exact outcome you were trying to avoid.

The VRBO actor is live on the Apify Store, pay-per-result, no credit card to try it. It returns title, location, nightly price, beds/baths, guests, rating, review count, amenities, host, and coordinates where public — and it stops on its own schedule, with rows written, every time.

We do the dirty work so your dataset stays clean. 😈