DEV Community: Joe McRell

Bypassing Headless Pagination and Modal Blocks on Airbnb with Playwright

Joe McRell — Fri, 11 Sep 2026 07:42:15 +0000

When scraping modern Single Page Applications (SPAs) like Airbnb, standard pagination strategies (page.click('a[aria-label="Next"]')) frequently collapse in headless Linux environments (Xvfb / Docker).

Even if your selectors look completely sound in Chrome DevTools, Playwright will throw errors like:

playwright._impl._errors.TimeoutError: element is outside of the viewport

or fail silently when unexpected modal backdrops (translation dialogs, privacy notices, currency popups) intercept user interaction events.

Here is the multi-layered defensive pattern we engineered to reliably paginate through dozens of search pages in production.

The Root Causes

Sticky Footers and Responsive Viewports: In headless browsers with constrained resolutions, the pagination bar at the bottom of the grid is often overlaid or pushed outside the actionability bounding box.
Translation & Currency Modals: Airbnb periodically pops a "Translate reviews and descriptions to English" or "Choose currency" overlay. In headless mode, this modal consumes all click events.
Synthetic Event Shielding: Clicking the <a> tag with standard browser synthetic events doesn't always trigger the Next.js router change if the viewport isn't focused.

Pattern 1: Active Modal Dismissal

Before attempting pagination or scrolling, run an aggressive dismissal loop that catches close buttons, cookie banners, and presses the Escape key:

async def dismiss_modals(page):
    selectors = [
        'button[aria-label="Close"]',
        'button[aria-label="Dismiss"]',
        '#onetrust-accept-btn-handler',
        'div[data-testid="modal-container"] button',
    ]
    for sel in selectors:
        try:
            btn = page.locator(sel).first
            if await btn.is_visible():
                await btn.click(timeout=1000)
                await page.wait_for_timeout(300)
        except Exception:
            pass
    try:
        await page.keyboard.press("Escape")
    except Exception:
        pass

Pattern 2: Multi-Tiered Pagination Fallback

Instead of relying solely on .click(), use a three-tier fallback mechanism:

from urllib.parse import urljoin

async def advance_to_next_page(page):
    await dismiss_modals(page)

    next_btn = page.locator('a[aria-label="Next"]').first
    if not await next_btn.is_visible():
        return False

    # Tier 1: Extract href for direct URL navigation fallback
    next_href = await next_btn.get_attribute("href")

    # Tier 2: Scroll into view and standard click
    try:
        await next_btn.scroll_into_view_if_needed(timeout=2000)
        await next_btn.click(timeout=3000)
        await page.wait_for_load_state("networkidle", timeout=8000)
        return True
    except Exception:
        pass

    # Tier 3: JavaScript evaluate click
    try:
        await next_btn.evaluate("el => el.click()")
        await page.wait_for_load_state("networkidle", timeout=8000)
        return True
    except Exception:
        pass

    # Tier 4: Direct URL navigation fallback (Immune to DOM/viewport obstacles)
    if next_href:
        target_url = urljoin(page.url, next_href)
        await page.goto(target_url, wait_until="networkidle", timeout=15000)
        return True

    return False

If modals intercept the click, or sticky footers keep the link outside the headless viewport, Tier 4 (direct URL navigation fallback) completely bypasses the DOM layer and forces the browser to load the paginated state directly.

Production Cloud Implementation

If you don't want to maintain headless Playwright clusters, rotating residential proxies, and DOM selector manifests yourself, this entire logic is packaged and maintained as a serverless actor on Apify:

Actor: captainhandsome/airbnb-listings-search
Open-Source Examples: jlucasmcrell/apify-scrapers on GitHub

You can run it programmatically via Python in 4 lines:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("captainhandsome/airbnb-listings-search").call(run_input={
    "location": "Austin, TX",
    "max_items": 50
})

listings = list(client.dataset(run["defaultDatasetId"]).iterate_items())
print(f"Extracted {len(listings)} listings reliably.")

Reverse-Engineering Glassdoor Estimated Job Posting Dates in Python

Joe McRell — Fri, 11 Sep 2026 07:41:28 +0000

One of the biggest headaches when building automated job aggregators or market compensation trackers is posting age.

Job portals like Glassdoor don't expose clean ISO timestamps (2026-09-10T12:00:00Z) in the listing feed HTML. Instead, they present relative, human-readable strings like:

"24h"
"3d"
"14d"
"30d+"

If you are dumping these raw strings into Postgres, BigQuery, or Pandas, running temporal queries (e.g. "Show only jobs posted within the last 7 days") becomes messy.

Here is how to cleanly normalize relative posting ages into deterministic ISO dates and precision flags in Python.

The Mathematical Parser

We can parse relative durations using regular expressions and calculate estimated timestamps relative to run execution (scraped_at), tagging the calculation with a precision enum (exact_day, estimated, or lower_bound):

import re
from datetime import datetime, timezone, timedelta

def parse_posting_age(age_str: str, reference_time: datetime = None):
    if not reference_time:
        reference_time = datetime.now(timezone.utc)

    if not age_str:
        return {
            "posting_age": None,
            "posting_date_estimated": None,
            "posting_date_precision": None
        }

    clean_str = age_str.strip().lower()

    # Matches "24h", "5h", "1h"
    hour_match = re.match(r"^(\d+)\s*h(?:ours?)?$", clean_str)
    if hour_match:
        hours = int(hour_match.group(1))
        est_date = reference_time - timedelta(hours=hours)
        return {
            "posting_age": clean_str,
            "posting_date_estimated": est_date.strftime("%Y-%m-%d"),
            "posting_date_precision": "estimated"
        }

    # Matches "30d+", "60d+" (Lower bound)
    plus_match = re.match(r"^(\d+)\s*d(?:ays?)?\+$", clean_str)
    if plus_match:
        days = int(plus_match.group(1))
        est_date = reference_time - timedelta(days=days)
        return {
            "posting_age": clean_str,
            "posting_date_estimated": est_date.strftime("%Y-%m-%d"),
            "posting_date_precision": "lower_bound"
        }

    # Matches "3d", "7d", "14 days"
    day_match = re.match(r"^(\d+)\s*d(?:ays?)?$", clean_str)
    if day_match:
        days = int(day_match.group(1))
        est_date = reference_time - timedelta(days=days)
        return {
            "posting_age": clean_str,
            "posting_date_estimated": est_date.strftime("%Y-%m-%d"),
            "posting_date_precision": "estimated"
        }

    # Fallback for unexpected formats
    return {
        "posting_age": clean_str,
        "posting_date_estimated": None,
        "posting_date_precision": "unknown"
    }

Canonical Link Extraction

Glassdoor URLs from search grids are loaded with dynamic tracking telemetry, session IDs, and redirection wrappers:

https://www.glassdoor.com/partner/jobListing.htm?pos=101&ao=1136043&s=58&guid=00000191...&jobListingId=100987654321

To prevent database duplicates, extract the pure numeric job_id and construct a clean canonical permalink:

from urllib.parse import urlparse, parse_qs

def canonicalize_glassdoor_url(raw_url: str):
    parsed = urlparse(raw_url)
    qs = parse_qs(parsed.query)

    # Try finding the jobListingId or jl parameter
    job_id = qs.get("jobListingId", [None])[0] or qs.get("jl", [None])[0]

    if not job_id:
        # Regex search for digits in path
        id_match = re.search(r"-jl_(\d+)\.htm", raw_url) or re.search(r"/(\d{8,12})\.htm", raw_url)
        if id_match:
            job_id = id_match.group(1)

    if job_id:
        canonical_url = f"https://www.glassdoor.com/job-listing/-jl_{job_id}.htm"
        return job_id, canonical_url

    return None, raw_url

Production Cloud Endpoint

If you need this pipeline running with zero server maintenance, rotating proxies, and automated retries, we run this continuous enrichment pipeline on Apify:

Actor: captainhandsome/glassdoor-jobs-scraper
GitHub Repository: jlucasmcrell/apify-scrapers

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("captainhandsome/glassdoor-jobs-scraper").call(run_input={
    "keyword": "Software Engineer",
    "location": "Austin, TX",
    "max_items": 30
})

jobs = list(client.dataset(run["defaultDatasetId"]).iterate_items())
for job in jobs[:3]:
    print(job["job_title"], "|", job["employer"], "|", job["posting_date_estimated"], f"({job['posting_age']})")