DEV Community: SIÁN Agency

Schema Drift Is the Silent Killer. Here's What to Log So You Actually Catch It.

SIÁN Agency — Tue, 02 Jun 2026 09:00:00 +0000

TL;DR — Most scraper "bugs" aren't bugs. They're the source site changing its data shape underneath you while your selectors and your code keep returning success. This is schema drift, and you cannot prevent it. You can only detect it. The detection has to be designed in. Here's how we do it.

I have a low opinion of any scraper that does not log a per-field availability rate. It's the single most useful number you can produce, and almost nobody produces it.

The premise: every record you scrape has a set of expected fields. After every run, you compute, for each field, the percentage of records that had a non-null value for it. You log that number. You alarm on it.

That's it. That's the whole technique.

Why this matters

A scraper has three failure modes you actually care about:

Total failure — the run errors out, you get a stack trace, you fix it.
Partial failure — some URLs fail, you log them, you retry.
Schema drift — every URL "succeeds," every record looks fine, but a field has silently gone from 98% present to 30% present.

The first two are loud. The third is silent. Schema drift is what produces "the dashboard looks weird" support tickets a week after the cause.

Real example, from our Sephora product info actor: in March, the site moved the "ingredients" field from a top-level dropdown into a tab inside a modal. Our existing selector still found something on the page — a placeholder div — and our code happily wrote ingredients="" to the dataset. No error, no alarm. The CSV had ingredient column. The values were empty for new products. Detected eight days later by a customer who tried to filter by allergen.

If we had been logging field availability, we would have seen the ingredient field drop from 96% present to 11% present in a single deploy and caught it inside an hour.

The teardown of why this gets missed

Most scrapers track:

Rows extracted per run.
Errors per run.
Run duration.

None of those move when schema drift happens. The row count is the same. The error rate is zero. The run duration is the same. You have to be looking at field-level data to see it.

The replacement pattern

After every run, compute and log this:

from collections import Counter

def field_availability(records, expected_fields):
    """Returns the % of records where each field is non-null."""
    counts = Counter()
    total = len(records)
    for record in records:
        for field in expected_fields:
            if record.get(field) not in (None, "", []):
                counts[field] += 1
    return {field: round(counts[field] / total * 100, 1) for field in expected_fields}

At the end of the run:

availability = field_availability(records, EXPECTED_FIELDS)
log.info("field_availability", extra=availability)

# Alarm on regression vs last run.
prev = await KeyValueStore.getValue("last_field_availability") or {}
for field, pct in availability.items():
    delta = pct - prev.get(field, pct)
    if delta < -10:  # 10-point drop is suspicious
        log.warning(f"availability regression: {field} {prev[field]}% → {pct}%")
await KeyValueStore.setValue("last_field_availability", availability)

Three log lines per run. Persistent state across runs. An alarm when any field drops more than 10 percentage points.

What to monitor specifically

Field availability is the one that catches the most. Two more I find pay for themselves:

Value distribution shift. For numeric fields (price, rating, count), log the median and p95. If price suddenly goes from "median ~$30" to "median 0.0" you have a parser bug, not just availability drift.
Selector hit count. When you fall back from primary to secondary selector, log it. If your fallback rate goes from 1% to 40%, the primary selector is on its way out — you have a week or so before it goes to zero.

These three together (availability, distribution, fallback rate) catch ~90% of schema drift before it produces customer-visible bugs.

Result

We added per-field availability logging across the Sephora actor portfolio in February. In the four months since:

6 schema-drift incidents caught and fixed within 48 hours of the source-site change.
Mean detection lag went from "a customer noticed" (~7 days) to "the alarm fired" (~12 hours, the gap being our run cadence).
One incident where the field availability dropped in a way that was expected (Sephora removed a field site-wide); we acknowledged and updated the schema. Net cost: 20 minutes, including writing the postmortem.

The cost: about 30 lines of code per actor, run-time overhead measured in milliseconds.

When this is wrong

Field availability is a poor signal when your input is inherently heterogeneous. If you're scraping listings where some products have ingredients and most don't, "30% have ingredients" might be normal. The technique still works — you just compare to the previous run, not to an absolute target. A 10-point drop is the alarm; the absolute number doesn't matter.

If you're scraping a homogeneous catalogue (every product has a title and a price), absolute thresholds work fine. Title <99% present? Something is wrong.

We packaged the field-availability + distribution + fallback-rate triple into a small middleware that sits at the end of every actor we ship — first deployed on the Sephora product info actor and rolled out portfolio-wide. Three lines to wire up, alarms in your inbox the day a source site decides to change their schema.

Which of the three signals is missing from your scraper right now? Drop it in the comments — I'll show you the smallest version that works.

Written by **Jonas Keller, Senior Automation Architect at SIÁN Agency. Find more from Jonas on dev.to. For custom scraping or automation work, hire SIÁN Agency.

One Playwright Selector Trick Nobody Talks About: getByRole

SIÁN Agency — Sun, 31 May 2026 08:30:00 +0000

Everyone reaches for page.locator(".some-class") first. They shouldn't.

getByRole is the most stable selector in Playwright and almost nobody uses it for scraping. They think it's a testing-library thing. It's not. It's a way of asking the page "what is this element semantically" instead of "what classname does the design system happen to use this week."

That distinction is what kept our Facebook video transcript actor running through three Facebook redesigns this past year.

The 3-item checklist

When does getByRole work? When the site is built by people who care about accessibility. Which is: more sites than you think, especially big ones with legal requirements (US government, EU compliance, large e-commerce).

Check before you skip it:

Open the accessibility tree in Chrome DevTools (Elements → Accessibility tab). If your target element shows a role and an accessible name, getByRole will find it.
Buttons and headings are nearly always tagged correctly. Even sloppy sites give you role="button" and proper heading levels because the design system enforced it.
Forms expose label even when the visual design hides it. getByLabel("Email") works on inputs that don't visibly show "Email" anywhere.

The trick

Compare:

// Class-name brittle
const followBtn = page.locator('._a9-_._a9-_2._a9-_8._a9-_z');

// getByRole — survives layout changes
const followBtn = page.getByRole('button', { name: /follow/i });

The first one breaks the day Facebook tweaks their CSS-in-JS hash. The second one keeps working until they remove the button entirely.

Same for headings:

// "Get the post title"
const title = page.getByRole('heading', { level: 1 });

That works on every site that uses <h1> correctly. Which is most of them, because Google penalises sites that don't.

Quick case

The Facebook transcript actor extracts video metadata from public posts. Facebook ships A/B tests constantly — class names change every couple of weeks. Selectors built on _a9-_8 chains broke regularly.

I rewrote the extractor to use getByRole for everything that had a meaningful role:

Author name → getByRole('link', { name: /^[\w. ]+$/ }) near the post header.
Post text → no role, but [data-ad-comet-preview="message"] (a data- attribute, also stable).
Video player → getByRole('article') containing a <video> element.

Before: ~8 selector breakages per quarter. After: 1 in the last 6 months, and that one was a real structural change (Facebook moved to a new post type), not a class rename.

The CTA you didn't ask for

getByRole is now the first thing every new actor we write tries — including the rebuild of the Facebook AI Transcript Extractor. CSS-class selectors are reserved for the cases where the site's accessibility story is genuinely broken (rare in 2026 — most sites have been audited at least once).

So:

Open your scraper. Run a search for page.locator( with a CSS class chain. How many can you replace with getByRole? Drop the count in the comments — I'll bet it's more than half.

Agree, disagree, or have a site where getByRole falls apart? Reply.

Written by **Nova Chen, Automation Dev Advocate at SIÁN Agency. Find more from Nova on dev.to. For custom scraping or automation work, hire SIÁN Agency.

Scraping Without Tests Is Gambling. And the House Always Wins.

SIÁN Agency — Fri, 29 May 2026 06:30:00 +0000

Nobody writes tests for scrapers. I get it. The site changes, your tests break, you feel like you spent Tuesday writing tests for the site you don't control. So you skip them.

Then the site changes again. Your scraper silently returns empty rows. The dashboard goes blank. Your client texts at 11pm. You discover, in the cold light of debug, that this exact failure was deterministic and could have been caught in 30 seconds by a single fixture-based test.

The house always wins.

The 3-item checklist

What scrapers actually need to test:

Extraction against a frozen HTML fixture. Save a copy of the page once. Run the parser against it. Assert the fields. This catches your bugs.
Schema validation against a live response. Periodically (daily, weekly), hit one real URL and validate the output shape. This catches their changes.
Smoke test the full pipeline against a known-good URL. End-to-end. One URL. Asserts that you get one row out, with the expected fields. This catches integration breakage.

You don't need a Jest config or a pytest empire. You need three test files.

The replacement: a fixture-first test in <10 lines

# tests/test_extractor.py
from pathlib import Path
from my_scraper.extract import extract_comment

def test_youtube_comment_extraction():
    html = Path("tests/fixtures/youtube_comment_2026-04-01.html").read_text()
    result = extract_comment(html)
    assert result["author"] == "@somecreator"
    assert result["likes"] == 1247
    assert "great video" in result["text"].lower()

Then your extract_comment(html) is a pure function — give it HTML, get a dict back. No browser, no network. Runs in milliseconds. Survives a CI minute budget. Catches every regression in your parsing code instantly.

Save the fixture by literally hitting the URL once and writing the response to disk:

# scripts/refresh_fixture.py
async with async_playwright() as p:
    browser = await p.chromium.launch()
    page = await browser.new_page()
    await page.goto("https://www.youtube.com/watch?v=...")
    Path("tests/fixtures/youtube_comment_2026-04-01.html").write_text(
        await page.content()
    )

Run it once a quarter. When the test starts failing, refresh the fixture, fix the extractor, commit both. That's the loop.

Quick case

On our YouTube comments scraper, fixture-based tests caught three parsing regressions before they ever reached production:

A field rename (likeCount → likeCount plus a thousand-separator format change).
A new "pinned" badge that broke our author-name selector.
A timestamp format change from "2 days ago" to "2d".

All three would have shipped silently. The cron would still run. The CSV would still write. The fields would just be wrong or empty. Instead, the test failed in CI on the PR that introduced the change, fifteen minutes after the fixture was last refreshed.

The cost of writing the test the first time: 20 minutes. The cost of the bugs it caught, if shipped: at minimum a refund and an apology each.

The CTA you didn't ask for

Every actor we ship now starts with three test files:

tests/test_extract.py — fixture-based unit tests for parsing.
tests/test_schema.py — Pydantic / Zod schema check on a live URL, run on a schedule.
tests/test_smoke.py — single-URL end-to-end check on every deploy.

It's the most boring testing pyramid you've ever seen and it has paid for itself an embarrassing number of times — the YouTube comments scraper is where it caught the most regressions in 2026.

So:

Open your scraper. Do you have a tests/ folder? Drop "yes" or "no" in the comments. If "no" — what's stopping you?

Agree, disagree, or have a fixture strategy that actually works? Reply.

Written by **Nova Chen, Automation Dev Advocate at SIÁN Agency. Find more from Nova on dev.to. For custom scraping or automation work, hire SIÁN Agency.

Why Your Requests + BeautifulSoup Stack Will Fail in Production

SIÁN Agency — Tue, 26 May 2026 08:30:00 +0000

TL;DR — requests plus BeautifulSoup is the right tool for tutorials, side projects, and one-off audits. It is the wrong tool for any scraper that has to run unsupervised, longer than a quarter, against a site that has even basic bot defenses. I've watched a dozen teams discover this the expensive way. Here's the diagnosis and the replacement.

I'm not anti-requests. The library is fast, predictable, and elegant. For 30% of scraping tasks it's still what I reach for first. The problem is that the rest of the scraping pipeline — JavaScript-rendered content, fingerprinting checks, modern auth flows, lazy loading — silently breaks the assumptions requests is built on.

Most teams discover this in stages. Here's the timeline.

Month 1 — "It works"

You write the first version. requests.get(url) returns 200, BeautifulSoup parses the response, you find your selectors, you ship. Tests pass against the small URL set you tested with. Lunch.

Month 2 — "Some pages return empty"

You notice maybe 5% of pages return rows where half the fields are None. You add a check, log the URL, retry. The retry sometimes works.

What's actually happening: those pages render their data in JavaScript after the initial response. requests got the HTML skeleton. The data was never in it. The retries that "work" are coincidence — sometimes the cached page has stale rendered data; sometimes a CDN ships a different variant.

Month 3 — "We're getting 403s"

The target site rolled out a fingerprinting check. requests sends a default User-Agent that screams python-requests/2.31.0. You add headers. It works for two days. They tightened the check — now they look at TLS fingerprint, not just User-Agent. requests uses the system OpenSSL TLS stack, which is different from any real browser's. The block returns.

Month 4 — "We need a session, but it's stateful"

Login flow now requires a CSRF token, which is rendered in JavaScript, which requests can't run. You spend two days reverse-engineering the login flow, find the API endpoint behind it, hit that directly. Works for six weeks. They rotate the auth scheme.

Month 5 — "Let's just use Playwright"

You finally migrate. Most of the team is annoyed because the rewrite took longer than they wanted. The team that does it later is annoyed for the same reason.

The teardown

The fundamental issue: requests is an HTTP client. Modern websites are browser applications. The thing you're scraping is the output of running JavaScript, not a static document. You can fight that for a while — by reverse-engineering APIs, faking TLS fingerprints, hand-rolling JS interpreters — but you're paying interest on a debt you took on the day you reached for requests instead of a real browser.

Specific failure modes you're going to hit:

JavaScript-rendered content. The HTML you fetch contains <div id="root"></div> and not much else.
TLS fingerprinting. requests looks like Python; real browsers look like Chrome/Firefox. Block lists distinguish them easily.
Lazy-loading. Data appears in the DOM only after scroll, click, or visibility events. Static fetch never triggers them.
Modern auth. OAuth, CSRF tokens injected via JS, cookie-based session validation that requires running scripts.
Anti-automation challenges. Cloudflare, PerimeterX, DataDome — all rely on running JavaScript to validate the client.

requests answers none of these. Playwright (or Puppeteer) answers all of them, because Playwright is a browser.

The replacement pattern

Skip the year of pain. Start with Playwright. Use requests only when you've measured that the data is in the static HTML and the site has no fingerprinting:

from playwright.async_api import async_playwright

async def scrape(url):
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        ctx = await browser.new_context(
            user_agent="Mozilla/5.0 (...)",
            viewport={"width": 1920, "height": 1080},
        )
        page = await ctx.new_page()

        # Block heavy resources for speed.
        await page.route("**/*.{png,jpg,jpeg,gif,svg,woff,woff2}",
                         lambda r: r.abort())

        await page.goto(url, wait_until="domcontentloaded")
        # Wait for the *data* to appear, not just the document.
        await page.wait_for_selector('[data-product-id]', timeout=15_000)

        return await extract_fields(page)

Five things requests can't give you that Playwright does for free:

JavaScript execution — your selectors target rendered DOM, not the source.
Realistic TLS fingerprint — Chromium does this for you.
Cookie/session handling that matches a real browser.
wait_for_selector — semantic waits instead of time.sleep.
Routing controls — block what you don't need, accelerate what you do.

When `requests` is still right

Static documentation sites. Open RSS/Atom feeds. JSON APIs that don't require login. PDFs and CSVs hosted on S3. Anything where you've actually fetched the URL, looked at the response body, and confirmed your data is in it.

That's a real category. Just don't assume the next site you scrape will fall into it.

Result

Across our actor portfolio, the migration ratio settled around 80/20 — Playwright for 80% of jobs, requests for the 20% where the data is genuinely static. The 80% includes our entire Sephora catalog pipeline, which spent its first version as a requests + BeautifulSoup script and never made it past month 2. The Playwright rewrite has been running unsupervised for 14 months.

If your scraper is currently 100% requests, your sample size isn't "this works fine." Your sample size is "the sites I've scraped so far happen to have static HTML."

Which of the five failure modes have you shipped to production? Drop the symptom in the comments — I'll point at the fix.

Written by **Jonas Keller, Senior Automation Architect at SIÁN Agency. Find more from Jonas on dev.to. For custom scraping or automation work, hire SIÁN Agency.

Stop Fighting the DOM. Selector-First Thinking Will Save Your Scraper.

SIÁN Agency — Sun, 24 May 2026 09:00:00 +0000

Most broken scrapers I see have the same shape: someone wrote the extraction logic first and the selectors second. The selectors were an afterthought — whatever worked in DevTools at 2am.

That's backwards. Selectors are the contract between your code and the page. Get them wrong and the rest of your scraper is irrelevant.

The mindset shift

Selector-first thinking means: before you write a single line of extraction code, you decide how the data is identified. Not "how do I get the price?" but "what does the page tell me, programmatically, that this thing is a price?"

Three answers, in order of preference:

Semantics — getByRole, getByLabel, getByText. These mirror what an accessibility tree exposes. They survive design changes.
Data attributes — data-testid, data-product-id, itemprop. Devs often add these for their own tests; you get to free-ride.
Structured data — JSON-LD, microdata, OpenGraph. The page is already telling Google what's a price; let it tell you too.

CSS classes are last resort. Class names are styling, not identity. They change when the design changes. They're the equivalent of asking for "the third button from the top" — works until someone rearranges the menu.

The 3-item checklist

Before you write a selector:

Open the accessibility tree in DevTools (Chrome: Elements → Accessibility tab). If the data has a role and an accessible name, use getByRole.
Search the page source for application/ld+json. If it's there and contains your fields, parse it directly. No DOM walking needed.
Look for data-* attributes near the data. Devs leave testing hooks everywhere. Use theirs.

If none of those work, then fall back to CSS or XPath. And when you do, anchor to something stable — a parent landmark, an aria-label, a data- attribute — not just a class chain.

The 10-line replacement

Here's the priority I use in every new actor:

async function extractPrice(page) {
  // 1. Structured data first.
  const ld = await page.locator('script[type="application/ld+json"]')
                       .first().textContent();
  const data = JSON.parse(ld ?? '{}');
  if (data?.offers?.price) return data.offers.price;

  // 2. Semantic selectors.
  const priceByLabel = page.getByLabel(/^price$/i);
  if (await priceByLabel.count()) return priceByLabel.textContent();

  // 3. Data attributes.
  const priceByData = page.locator('[data-testid="price"]');
  if (await priceByData.count()) return priceByData.textContent();

  // 4. Last resort: CSS class. Logged loudly so we know we're in fallback.
  console.warn('Falling back to CSS selector — selector audit needed.');
  return page.locator('.price-tag').textContent();
}

Notice the warn() in the fallback path. When that warning starts appearing in your logs, it means the site changed its higher-priority signals and you're one design refresh away from breakage. Fix it before it breaks, not after.

Quick case

On our Idealista actor, the priority order above turned a "fix the selector every 6 weeks" routine into a "fix the selector twice a year" routine. The JSON-LD path catches 95% of listings without ever touching the DOM. The accessibility-role fallback catches another 4%. The CSS fallback fires on edge-case property types and tells us when a new layout has shipped — usually a week before any of our other monitoring would have noticed.

The CTA you didn't ask for

This selector ladder is the second thing every actor we ship gets, right after the request blocking from last week's post — see it in action in the Idealista actor. It's so consistent we made it a util.

So:

Open your scraper's selector code right now. Count how many class-name chains you have versus semantic / structured-data lookups. Drop the ratio in the comments. Bonus points for the longest CSS chain — I bet someone has .product-grid > .item:nth-child(3) > .price > span > strong.

Agree, disagree, or have a site that genuinely needs CSS chains? Reply.

Written by **Nova Chen, Automation Dev Advocate at SIÁN Agency. Find more from Nova on dev.to. For custom scraping or automation work, hire SIÁN Agency.

A 10-Line Playwright Trick That Saved Me Hours on Every Sephora Run

SIÁN Agency — Fri, 22 May 2026 12:30:00 +0000

Most Playwright tutorials teach you to scrape a single page. Real scrapers need to scrape thousands. The thing that kills you isn't the selector — it's everything Playwright does before it touches the selector.

By default, Playwright loads a page like a human visiting a website. It downloads CSS, fonts, analytics scripts, A/B testing pixels, hero images, lazy-loaded carousels, and three different chat widgets. On a product catalog page, that's 4–6 MB of stuff you don't need. Times 10,000 pages, that's the difference between a 20-minute run and a 3-hour run.

Here's the 10-line route handler I drop into every actor:

const BLOCKED = ['image', 'media', 'font', 'stylesheet'];

await context.route('**/*', (route) => {
  const type = route.request().resourceType();
  const url  = route.request().url();
  if (BLOCKED.includes(type)) return route.abort();
  if (/google-analytics|doubleclick|hotjar|segment|gtm/.test(url)) {
    return route.abort();
  }
  route.continue();
});

That's it. Two lists: resource types you don't need, and tracking domains you definitely don't need.

The 3-item checklist before you ship this

Test that your data is still there. Some sites lazy-load product info into image data- attributes. Aborting images can sometimes break extraction. Run with and without the route handler and diff the output.
Don't block scripts. Modern sites build the DOM with JS. Aborting scripts will give you an empty page. (CSS and fonts are safe — Playwright doesn't need them to find selectors.)
Watch for sites that detect this. Some bot-detection scripts check whether you fetched the analytics pixel. If your success rate drops after enabling this, allow the analytics domains back through.

Quick case

On our Sephora product info actor, this single change cut average page load from 4.8s to 1.3s. Across a 5000-product catalog scrape, that's the difference between 6.5 hours and 1.8 hours. Same selectors, same data, same success rate. We just stopped downloading hero images of moisturizers we never look at.

It also dropped our Apify compute units per run by ~60%, which directly affects what we charge customers. Faster scraper, lower cost, same output. The route handler now ships with the Sephora product info actor and every new scraper after it.

The CTA you didn't ask for

This route handler ships with our starter actor template. New scrapers get it on day one. Old scrapers got it bolted on the first time we noticed runtime > 1 hour.

The pattern works on any browser-based scraper — Playwright, Puppeteer, Selenium with CDP. The shape is always: tell the browser what not to load, before you tell it what to find.

One quick note for the JS-heavy among you: the same pattern applies to Puppeteer's page.setRequestInterception(true) — same idea, slightly different API. Same wins.

Drop your slowest scraper's runtime in the comments. I'll guess what's eating your minutes. (Hint: it's probably hero images.)

Agree, disagree, or have a site where blocking images breaks something subtle? Reply.

Written by **Nova Chen, Automation Dev Advocate at SIÁN Agency. Find more from Nova on dev.to. For custom scraping or automation work, hire SIÁN Agency.

Stop Building Fragile Scrapers — Build Actors Instead

SIÁN Agency — Mon, 18 May 2026 13:30:00 +0000

TL;DR — A "scraper" is a script that ran once. An "actor" is a unit of work with an input contract, an output schema, observability, and a billing model. Same code, completely different operational surface. We migrated our Bayut property pipeline from the first to the second this quarter and the support load dropped 70%.

I get sent a lot of scraper repos to "review" — usually after they've broken in production. They look surprisingly similar:

One Python file, 300–600 lines.
A main() that loops over URLs.
requests.get() plus BeautifulSoup plus a try/except: pass that swallows everything.
Output written to a CSV called output.csv in the working directory.
A cron job that triggers it nightly. Sometimes a Slack webhook on failure that stopped working six months ago.

This is what I call a script that ran once. The fact that it ran in production doesn't make it production code.

The teardown is always the same.

The five failure modes you inherit when you ship a script

No input contract. The script reads URLs from a hardcoded list or a file path that only exists on your laptop. New requirement → edit the file → redeploy → hope.
No output schema. Whatever fields happened to be present this run get written. When the source site adds a column, the CSV silently widens. When the source site removes a column, downstream breaks at parse time, three hops away from the cause.
No observability. "Did it run last night?" is answered by SSH-ing to the box and ls -la output.csv. Run history is the file's mtime. Failure mode is "the file is older than expected."
No retries with backoff. A 503 from the target site at 02:14 kills the run. There is no second attempt. The next run is in 24 hours.
No billing surface. The cost of running it is your time and your server. There is no per-unit price, so there is no signal that the unit economics are bad until you check the AWS bill.

A script is fine for "I need this data once." It is not fine for "we need this data nightly for the next two years." But teams keep shipping #1 to fulfill #2.

What an actor is

Strip the marketing word and an actor is just: a containerised job with a declared input schema, a declared output schema, and a runtime that handles scheduling, retries, logs, persistent storage, and billing. Apify is one implementation — there are others. The shape matters more than the vendor.

When we rebuilt our Bayut property scraper as an actor, four things changed at the level of code:

// 1. Input is validated against a schema before main() runs.
//    Bad input fails fast with a useful error, not silent miss.
const input = await Actor.getInput(); // INPUT_SCHEMA.json enforces shape

// 2. Output goes to a typed dataset. New fields require a schema
//    change — not a silent CSV widening.
await Dataset.pushData({
  listingId, price, currency, address, lat, lng, scrapedAt
});

// 3. Failures retry with backoff at the platform level.
//    Our code throws; the runtime decides what to do.
throw new ScrapeFailure('listing-blocked', { url, status: 429 });

// 4. Logs are structured, queryable, and indexed by run.
log.warning('rate-limit', { url, retryAfter: 60 });

That's it. Same Playwright, same selectors, same scraping logic. The difference is that all the boring infrastructure — input validation, output typing, retries, logs, scheduling, billing — is no longer your problem.

Result

For Bayut specifically, three months after the migration:

Mean time to detect a breakage went from ~36 hours (next-day stakeholder complaint) to under 15 minutes (failed runs alert with the offending URL and HTTP status).
Support tickets dropped 70%. Most of the volume was "the data is missing" — invisible failures from the cron-script era. With per-run datasets, failed runs surface themselves.
Cost per 1000 listings went down, not up. Concurrency at the runtime level is cheaper than spinning up your own queue.

The migration itself took about a week. Most of the time was not the scraping logic — that was already there. It was deciding what the input schema should be, what the output schema should be, and which fields were "nice to have" vs "the dataset is broken without this."

The replacement pattern

If you're sitting on a script-shaped scraper right now, the migration order is:

Write the input schema. Force every run to declare what it's scraping.
Write the output schema. Force every row to validate before it gets persisted.
Move retries from try/except: pass to the runtime.
Replace print() with structured logs.
Containerise. Whatever runs in python main.py should run in docker run.
Pick a runtime — Apify, your own k8s cron, whatever. The schema work is portable.

You do steps 1–5 inside your existing repo. You haven't committed to a vendor yet. By the time you reach step 6, the actor exists — the runtime is just a deployment target.

We packaged this migration shape into a starter we use for every new client engagement — same six steps that produced the Bayut property scraper above. Same six steps, every time.

Which of the five failure modes is currently shipping in your stack? Drop it in the comments — I'll point at the smallest change that fixes it.

Written by **Jonas Keller, Senior Automation Architect at SIÁN Agency. Find more from Jonas on dev.to. For custom scraping or automation work, hire SIÁN Agency.

If Your Scraper Uses Regex on HTML, You're Already Broken

SIÁN Agency — Thu, 14 May 2026 14:30:00 +0000

If your "scraper" is a requests.get() followed by re.findall(r'<div class=\"price\">.*?</div>', html), I have bad news.

You don't have a scraper. You have a layout sensor. The first time the dev team renames the class, adds a wrapper <span>, or A/B tests a new pricing component, your pipeline goes silent. Not loud, not error-throwing — silent. Empty rows in the dataset. No alarm. You find out a week later when a stakeholder asks why the dashboard looks weird.

I rebuilt our Idealista scraper this quarter and the regex stage was the thing I deleted first.

The 3-item checklist

Before you write another re.findall against HTML, check:

Is there a stable accessibility role or label? (getByRole('heading', { name: /price/i }) — survives class renames.)
Is the data actually in the rendered page, or is it injected via JSON? (Often the JSON-LD <script> block has everything you need, no DOM walking.)
Can you assert the schema fails loud? (If a field is missing, throw — don't silently default to None.)

If the answer to all three is no, you're not scraping. You're guessing.

The 10-line replacement

Here's the pattern I keep copying into new actors:

from playwright.async_api import async_playwright
import json

async def extract_listing(page):
    # Pull JSON-LD first — it's the spec, not the styling.
    ld_json = await page.locator('script[type="application/ld+json"]').first.text_content()
    data = json.loads(ld_json)
    return {
        "price": data["offers"]["price"],
        "currency": data["offers"]["priceCurrency"],
        "address": data["address"]["streetAddress"],
        "url": page.url,
    }

Ten lines. No regex. No CSS class names. No BeautifulSoup chain that breaks when someone wraps the price in a new <div>.

Why this works: JSON-LD is what Idealista, Bayut, and most listing sites publish for Google. It's stable because it's a contract with search engines, not with your scraper. When the visual layout changes, the JSON-LD almost always doesn't.

Quick case

Our Idealista actor went from 4 selector-related breakages per month to zero in the quarter after I switched extraction to JSON-LD + accessibility selectors. The breakages we still see are real changes — new property types, new fields — and they fail loud now, with a schema validation error, instead of silently returning empty strings.

That's the bar: when the site changes, your scraper either keeps working or throws an error you can read. "Returns empty rows" is not acceptable behaviour.

The CTA you didn't ask for

This pattern is now the default starter for every actor we ship — visible in the Idealista actor. Faster runs, fewer 3am Slack messages from clients asking why their CSV is half-empty. We turned the JSON-LD-first extractor into a reusable module that drops into any new actor in about a minute.

So:

Open your scraper. Search for re.findall, re.search, or BeautifulSoup chained more than two .find() deep. Drop the worst offender in the comments — I'll show you the JSON-LD or selector replacement.

Agree, disagree, or got a site where this falls apart? Reply.

Written by **Nova Chen, Automation Dev Advocate at SIÁN Agency. Find more from Nova on dev.to. For custom scraping or automation work, hire SIÁN Agency.

Rate Limits Are a Feature, Not a Bug

SIÁN Agency — Thu, 07 May 2026 05:39:33 +0000

Most scraper "incidents" I'm pulled into start the same way: someone shows me a graph of 429 responses and asks how to make them go away. The honest answer — that nobody likes — is that the 429s are the well-behaved part of the system. The rest is what's broken.

I'm going to argue that rate limits are not your enemy. They're a contract. And scrapers that treat them like a contract — instead of an obstacle — are the only ones I trust to run unsupervised for more than a quarter.

The teardown

Three things teams typically do when they hit rate limits, in order of how bad they are:

Add proxies. "If they limit me, I'll just be more people." This works for about six weeks. Then the target site fingerprints your residential proxy pool and you're back to where you started, with a higher monthly bill.
Decrease delays. "If we go faster, we'll finish before they notice." Faster only matters if the request budget exists. Going faster against a hard limit just stacks failures earlier.
Retry harder. Add exponential backoff with a 30-minute cap. Now your "1-hour scraper" is a 4-hour scraper that completes when the throttle window expires.

All three are forms of the same denial: refusing to accept that the source site is telling you the rate at which they're willing to serve you data. They are. You should listen.

What rate limits actually are

A rate limit is the source-site engineer's way of saying: here is the contract under which my system stays healthy. They published the rate (often: in headers) because they've measured what their infrastructure can serve before things degrade. When you exceed it, you don't just hurt yourself — you contribute to the conditions that get scrapers blocked entirely.

There are three signals you should be reading from every response, not just the body:

Retry-After header. This is the source telling you, in seconds, when it'll talk to you again. Respect it literally.
X-RateLimit-Remaining (or equivalent). Some sites publish their budget. Use it. Slow down before you hit zero.
Status code distribution over time. If your 200 rate is dropping while 429 rises, you're approaching a soft limit you can't see. Back off proactively.

If you're not reading those, your scraper is operating blind against an opponent who is leaving the lights on for you.

The replacement pattern

Here's the rate-aware request loop I drop into every actor:

import asyncio, time
from collections import deque

class RateBudget:
    """Token bucket — refills at `rate` per second, max `burst` tokens."""
    def __init__(self, rate: float, burst: int):
        self.rate = rate
        self.burst = burst
        self.tokens = burst
        self.last_refill = time.monotonic()

    async def take(self):
        while self.tokens < 1:
            await asyncio.sleep((1 - self.tokens) / self.rate)
            now = time.monotonic()
            self.tokens = min(self.burst,
                              self.tokens + (now - self.last_refill) * self.rate)
            self.last_refill = now
        self.tokens -= 1

async def fetch(url, budget, session):
    await budget.take()
    resp = await session.get(url)
    if resp.status == 429:
        retry_after = int(resp.headers.get('Retry-After', '60'))
        await asyncio.sleep(retry_after)
        return await fetch(url, budget, session)
    return resp

Three things this does that "decrease the delay" doesn't:

Token bucket means the rate is global, not per-request. Concurrency works without exceeding the contract.
Retry-After is honoured literally. No exponential backoff guessing — the source already told you.
No proxy rotation. You don't need to be more people. You need to be one well-behaved person.

Result

On the two scrapers I migrated to this pattern this quarter:

Idealista. 429 rate dropped from 8% to 0.4%. Total run time went up by 11% (from 47min to 52min average) — because we stopped hammering. Per-run cost went down 38% — because we stopped paying for retries that were never going to succeed.
Sephora. 429 rate from 15% to <1%. Run time about the same. Block rate (full IP block requiring rotation) went from "monthly" to "zero in the last 90 days." This one's the real win — we used to burn a residential proxy pool subscription. Now we don't need it.

The pattern that emerges every time: respecting the rate makes you slower per-request, but more reliable per-run, and significantly cheaper per-result. The unit economics of a polite scraper beat the unit economics of an aggressive one. By a lot.

When it's wrong

This is wrong if the source site doesn't publish a contract — no Retry-After, no rate header, just blanket blocks. There you genuinely are guessing. But the guess should still bias toward "much slower than you think you need to be," not toward "more proxies." A token bucket at 1 req/sec is a fine starting point for an unknown site; you can ratchet up while watching error rates.

This is also wrong if you have explicit business permission to scrape at higher rates — a partnership, an API key, a contract. Those are different relationships. The advice here is for scrapers running against the public web, where 429 is the only contract you have.

Closing

Stop thinking of rate limits as the cost of doing business. Start thinking of them as a free service the target site is providing you: telling you exactly how to stay welcome. Most blocked scrapers I see were blocked not because they "got caught" — they were blocked because they ignored repeated, clearly-articulated signals that they were being rude.

We packaged the token bucket + Retry-After honour into a small middleware that sits in front of every actor we ship — visible across our Apify portfolio. About 30 lines of code. It's the most boring reliability win I've shipped this year, and the most consistent.

Which response header is your scraper currently ignoring? Drop it in the comments — I'll show you what to do with it.

Written by **Jonas Keller, Senior Automation Architect at SIÁN Agency. Find more from Jonas on dev.to. For custom scraping or automation work, hire SIÁN Agency.

Instagram Reel Transcripts in 5 Lines — and Word-Level Timestamps Are Free

SIÁN Agency — Sat, 02 May 2026 04:54:03 +0000

If you've ever priced Instagram transcription at scale, you already know the trap: per-video pricing on the SaaS tier, plus an upcharge for word-level timestamps. Run the math on 500 reels and you'll close the tab.

I'm not going to talk you out of building your own pipeline. I'm just going to show you the five lines I run when I don't want to.

The trap: per-URL pricing on transcript metadata

Most Instagram transcription APIs in 2026 charge:

A base rate per processed video.
Sometimes a separate rate per minute of audio.
An additional fee to expose word-level timestamps (the thing you actually need if you're building captions, search, or any kind of clip editor).

That works for a single creator's library. It does not work for an agency processing client A's 200 reels, then client B's 1,000.

The five-line replacement

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("sian.agency/instagram-ai-transcript-unlimited").call(run_input={
    "bulkUrls": ["https://www.instagram.com/reel/DG06PnPT9aT/"],
    "wordLevelTimestamps": True,
})
print(next(client.dataset(run["defaultDatasetId"]).iterate_items())["transcript"])

Three input fields you actually need:

instagramUrl (string) — single reel or video post. Pattern enforced; /reels/ auto-corrects to /reel/.
bulkUrls (array) — paste 1, paste 1,000. Bulk edit, .txt upload, manual list. Same input shape regardless of volume.
wordLevelTimestamps (boolean, default true) — get a per-word timestamp on every transcript. Free. You don't pay extra for it.

That third one is the point of this post. It's on by default. Most tools hide it behind a paywall. This one doesn't.

What you can't transcribe

Be honest about the constraints up front:

Image carousels — no audio, nothing to transcribe.
Music-only videos — no spoken audio, the transcript will be empty.
Private profiles — Instagram blocks scraping public-side, so the actor only handles public reels and posts.

If you're building a "scrape any Instagram URL" feature, you'll hit those edges. The actor returns a clear error per URL — handle it client-side and skip silently.

Why "unlimited" is a real claim, not marketing

The actor doesn't charge per validated URL. It charges for compute time per run. If you're processing 1,000 reels in one batch, that's one run. The pricing model rewards batching, which is what you want anyway — bulk is faster than serial because the runtime queue stays warm.

I migrated an agency client's Instagram audit workflow last week. Old setup: a per-video API at $0.05 + $0.02 word-timestamp upcharge — $35 for 500 reels per audit. New setup: one bulk run, predictable monthly compute. Roughly 1/4 the cost at their volume, and the dataset shape is identical.

What to do next

If you want to see what 30+ data points + word-level transcripts look like for your own client list, run it once: Instagram AI Transcript Unlimited.

Single-URL test costs less than a coffee. Bulk run is unlimited.

Tell me where this breaks. If you've found a public reel format the URL pattern misses, drop it in the comments. I'll get the maintainer to ship a fix in the next build.

Written by **Nova Chen, Automation Dev Advocate at SIÁN Agency. Find more from Nova on dev.to. For custom scraping or automation work, hire SIÁN Agency.

I Stopped Writing TikTok Scrapers. Five Lines of Python Replaced Them.

SIÁN Agency — Mon, 27 Apr 2026 13:34:57 +0000

If your TikTok scraper still uses Playwright + custom selectors, this post will annoy you. Good. Read it anyway.

I burned three weekends last quarter on a "minimal" TikTok scraper. Selector-first, headless, the works. Worked beautifully for nine days. Then TikTok shipped a layout change at 2am UTC and my fixtures became fiction.

The honest answer most devs avoid: for known platforms with stable APIs around them, you should not be writing the scraper. You should be calling someone's actor.

Stop owning the layer that breaks

Three things break a TikTok scraper, and none of them are about your code:

Layout drift. Selectors are a liability the second TikTok touches the DOM.
Auth + rate-limit games. Cloudflare, fingerprinting, the whole party.
Audio extraction + transcription. Even if you got the video, now you need Whisper, ffmpeg, a queue, and a dead body to bury when it OOMs.

You're not getting paid to maintain that. You're getting paid to ship the thing on top of it.

What replaced 800 lines of Python for me

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("sian.agency/best-tiktok-ai-transcript-extractor").call(
    run_input={"bulkUrls": ["https://www.tiktok.com/@user/video/7565659068153531669"]}
)
print(list(client.dataset(run["defaultDatasetId"]).iterate_items()))

That's the whole thing. Five lines. The actor's input schema has exactly two fields you need to know about:

tiktokUrl (string) — single video. Pass any URL format. Short links from vm.tiktok.com get resolved. Mobile share URLs work.
bulkUrls (array) — paste 5, 50, or 500. Bulk edit, file upload, line-separated, comma-separated. It doesn't care.

That's the entire input surface. Two keys. No proxy config, no captcha settings, no "headless or headful" debate.

What you get back

Per video, you get the AI transcript (99%+ accuracy claimed by the actor — empirically I see ~98% on English, lower on heavy slang) plus 45 metadata fields: views, likes, shares, creator stats, hashtags, music ID, location, content categories. The transcript ships with detected language and segment timing, so you can search inside videos like text.

I rewrote a competitor-monitoring pipeline last month using this. Old stack: Playwright cluster + Whisper container + Redis + a cron + a Slack channel where I apologized weekly. New stack: a 60-line Python script and the actor. Same dataset, less surface area, no apologies.

The objection I keep getting

"Why pay per run when I can self-host?"

Because your time isn't free, and you don't actually self-host — you self-rebuild every two weeks when something shifts. The actor charges per validated result. You only pay for the runs that gave you usable data. That's a different cost model than "compute hours your worker spent crashing."

If your volume is genuinely huge, sure, build it. But "huge" is an engineering decision, not a default.

Try it on your own URL

The free tier handles 5 videos per run, 8s delay between them. If you want to see the dataset shape for your own use case, drop a TikTok URL in and watch it run: TikTok AI Transcript Extractor on Apify.

Bulk mode is paid — unlimited per run, no delays, no per-video charges. Use it when you're past the experiment phase.

Disagree? Drop the snippet you're using to scrape TikTok in the comments. I'll tell you which line is going to break first. Be specific — "I use Puppeteer" is not a snippet.

Written by **Nova Chen, Automation Dev Advocate at SIÁN Agency. Find more from Nova on dev.to. For custom scraping or automation work, hire SIÁN Agency.

DEV Community: SIÁN Agency

Schema Drift Is the Silent Killer. Here's What to Log So You Actually Catch It.

Why this matters

The teardown of why this gets missed

The replacement pattern

What to monitor specifically

Result

When this is wrong

One Playwright Selector Trick Nobody Talks About: getByRole

The 3-item checklist

The trick

Quick case

The CTA you didn't ask for

Scraping Without Tests Is Gambling. And the House Always Wins.

The 3-item checklist

The replacement: a fixture-first test in <10 lines

Quick case

The CTA you didn't ask for

Why Your Requests + BeautifulSoup Stack Will Fail in Production

Month 1 — "It works"

Month 2 — "Some pages return empty"

Month 3 — "We're getting 403s"

Month 4 — "We need a session, but it's stateful"

Month 5 — "Let's just use Playwright"

The teardown

The replacement pattern

When requests is still right

Result

Stop Fighting the DOM. Selector-First Thinking Will Save Your Scraper.

The mindset shift

The 3-item checklist

The 10-line replacement

Quick case

The CTA you didn't ask for

A 10-Line Playwright Trick That Saved Me Hours on Every Sephora Run

The 3-item checklist before you ship this

Quick case

The CTA you didn't ask for

Stop Building Fragile Scrapers — Build Actors Instead

The five failure modes you inherit when you ship a script

What an actor is

Result

The replacement pattern

If Your Scraper Uses Regex on HTML, You're Already Broken

The 3-item checklist

The 10-line replacement

Quick case

The CTA you didn't ask for

Rate Limits Are a Feature, Not a Bug

The teardown

What rate limits actually are

The replacement pattern

Result

When it's wrong

Closing

Instagram Reel Transcripts in 5 Lines — and Word-Level Timestamps Are Free

The trap: per-URL pricing on transcript metadata

The five-line replacement

What you can't transcribe

Why "unlimited" is a real claim, not marketing

What to do next

I Stopped Writing TikTok Scrapers. Five Lines of Python Replaced Them.

Stop owning the layer that breaks

What replaced 800 lines of Python for me

What you get back

The objection I keep getting

Try it on your own URL

When `requests` is still right