Quick answer: run Hy8dPuLBeAhGBhXKP asked our Bayut scraper for 100 listings. It delivered 0 — and still billed the customer $0.30. Not because Bayut fully blocked us. The run's two retry attempts were: attempt 1, a genuine Bayut anti-bot challenge (fair, that one's on the target); attempt 2, Page.content: Unable to retrieve content because the page is navigating and changing the content — Playwright's own wording for a client-side redirect racing our read, which is a bug in our code, not a block from theirs. The entire retry budget meant to survive Bayut got spent on us instead. Both defects are fixed as of build 0.1.8.
Here's the run, attempt by attempt:
| Attempt | What actually happened | Whose bug |
|---|---|---|
| 1 | Bayut served a genuine anti-bot challenge | Target |
| 2 | Playwright raced a client-side redirect mid-read and threw | Ours |
| Billing |
actor-start fired at boot, before either attempt ran |
Ours |
Why did the retry budget run out before Bayut's defenses did?
Because one of the two attempts we had wasn't spent fighting Bayut at all — it was spent losing a race against our own page-read timing. A retry loop that treats "target pushed back" and "our own client crashed mid-navigation" as the same kind of failure will happily burn its budget on the second one and never get a clean shot at the first. From the outside, both look identical: zero rows, a failed run.
A retry budget spent on your own bug looks exactly like a target that blocked you.
What does "the page is navigating and changing the content" actually mean?
It's Playwright's exact error text for calling page.content() while the browser is mid-way through a client-side redirect — the DOM the call is trying to read is being torn down and rebuilt underneath it. It has nothing to do with a WAF, a captcha, or a ban. It's a timing bug, and until this fix it was indistinguishable in our logs from an actual block.
src/browser.py's fetch_page now reads content through a dedicated retry path: it retries a bounded number of times specifically on that race marker, and re-raises immediately on any other error. A genuine block still fails loud. A redirect race no longer eats the attempt meant for the real fight.
Why did a zero-row run still cost the customer money?
Because actor-start used to charge at Actor boot — before input validation, before a browser even existed to attempt a fetch. Under Pay-Per-Event that's backwards: the fee is supposed to mean "we did the work," not "the container turned on." The fix moves the charge into the write path itself — it now fires only from the first batch of rows that actually reaches the dataset. A run that delivers nothing costs nothing, which is the promise this Actor makes everywhere else in its pricing and the one this bug quietly broke.
Today's proof, on build 0.1.8: a 50-row for-sale run and a separate 25-row rental run, both charging exactly what they delivered — no charge ahead of data, no charge for rows that never landed.
None of this touches the part of the stack that was already working. We still run every Bayut request through a real browser session behind a residential proxy pinned to the UAE, rotate the exit on every block, and retry with backoff — that half of the job was never the problem here. The bug was entirely in how we read a page we'd already fetched, and in when we decided we'd earned the fee.
Bayut UAE Real Estate Scraper turns Bayut's own for-sale and to-rent listings into structured rows — price, bed/bath count, floor area, geo-coordinates, agency and agent details, and TruCheck verification status — searchable by purpose, location, or a custom polygon. Pricing is Pay-Per-Event: a $0.20 start fee (charged only once real rows land) plus $0.008 per listing row, so 1,000 results run about $8.20. Apify gives every new account free trial credit, no card required.
👉 https://apify.com/DevilScrapes/bayut-uae-real-estate
The devil's in the retry logic. 😈
Top comments (0)