DEV Community: Andrii Votiakov

When you actually need a headless browser (and when you're just wasting RAM)

Andrii Votiakov — Fri, 31 Jul 2026 09:05:56 +0000

Most of the store-locator scrapers I build never touch a browser. I open the Network tab, find the JSON endpoint the map calls, hit it with fetch, done. A browser is slow, hungry, and flaky, so I treat it as the last resort.

National Book Tokens is where the last resort actually kicked in. It's a UK book gift-card brand, and their "find a bookshop" page shows a Google-Maps-style board of pins for every shop that takes the card. I wanted the full geocoded list. So I did the usual thing: opened the page, opened DevTools, watched the Network tab reload.

Nothing. No /api/shops, no locations.json, no XHR carrying an array of bookshops. The map just... had pins. That absence is the whole story, and it's the thing worth learning, because it's the signal that tells you a browser is unavoidable instead of just habit.

How to tell a browser is actually required

Here's the test I run before I let myself spin up Chrome. It takes about thirty seconds.

Load the page normally in the browser. See the data on screen? Good.
View the raw HTML. Not the Elements panel, the raw document. Ctrl+U (view-source), or in DevTools open the Network tab, click the document request, look at the Response sub-tab. This is the exact bytes the server sent, before any JavaScript ran.
Ctrl+F in that raw response for something you can see on screen. A shop name, a postcode, a latitude.

If the data is in the raw HTML, you (almost) never need a browser. Server-rendered pages put everything in that first response, and a plain fetch plus an HTML parser gets you the lot. If the data is not in the raw response but is in the live Elements panel, that gap is your answer: the page arrived nearly empty and JavaScript filled it in after the fact. You cannot fetch your way to something that only exists after JS runs.

On the National Book Tokens page I searched the Elements panel for markers and found this:

<div id="markers" style="display:none">
  [[&quot;Foyles&quot;,&quot;51.5155&quot;,&quot;-0.1300&quot;,100,&quot;&lt;div&gt;...&lt;/div&gt;&quot;,...
</div>

A hidden div stuffed with the entire dataset, as an HTML-entity-escaped JSON array. The site's own map code reads this div back, unescapes it, and drops the pins. Then I looked at that same #markers div in view-source, and it was empty. Just <div id="markers"></div>. Client-side JS builds the contents on load.

That's the convict. Data present in the DOM, absent from the raw HTML, sitting in a div that some script populates. No JSON endpoint to call directly. A browser it is.

The browser has exactly one job

This is the part people get wrong. They reach for Puppeteer and then start driving it like a robot user, clicking around, scrolling, taking screenshots, scraping rendered text off the page. That's slow and brittle.

The browser's only real job here is to run the page's JavaScript and let the DOM settle. That's it. The moment #markers is populated, everything after that is plain string work: read the innerHTML, unescape it, JSON.parse, map the array. String work runs fine anywhere and it's fast. So I keep the browser part as small as I possibly can and get out.

I also don't launch Chrome, I connect to one. Running the actual scraper inside a Firebase Function, launching full Chromium is a pain (cold starts, binary size, memory). So I run a headless Chrome somewhere else, a browserless-style service, and attach over a WebSocket with puppeteer-core. puppeteer-core is the same library without the bundled Chromium download, which is exactly what you want when the browser lives elsewhere.

Here's the whole thing.

import puppeteer from "puppeteer-core";

const PAGE_URL = "https://www.nationalbooktokens.com/redeem/find-a-bookshop";

// A remote headless Chrome (browserless-style). Locally you can just use
// full `puppeteer` and puppeteer.launch({ headless: "new" }) instead.
const BROWSER_WS =
  process.env.BROWSER_WS_ENDPOINT; // ws://your-chrome-host/browser/chrome?token=...

// Undo the HTML-entity escaping the site applied before it stored the JSON.
function unEscapeHtml(s) {
  return s
    .replace(/&lt;/g, "<")
    .replace(/&gt;/g, ">")
    .replace(/&quot;/g, '"')
    .replace(/&#39;/g, "'")
    .replace(/&amp;/g, "&"); // do &amp; LAST or you double-decode
}

async function scrapeBookshops() {
  const browser = await puppeteer.connect({
    browserWSEndpoint: BROWSER_WS,
    ignoreHTTPSErrors: true,
  });

  try {
    const page = await browser.newPage();
    await page.goto(PAGE_URL, {
      waitUntil: "domcontentloaded",
      timeout: 30000,
    });

    // 1. the div must exist...
    await page.waitForSelector("#markers", { timeout: 30000 });
    // 2. ...AND be substantially filled. An empty #markers appears early,
    //    so waiting on presence alone parses a half-built div.
    await page.waitForFunction(
      () => document.getElementById("markers").innerHTML.length > 100,
      { timeout: 30000 }
    );

    // Pull the raw escaped blob out of the page. That's all the browser is for.
    const rawInner = await page.evaluate(
      () => document.getElementById("markers").innerHTML
    );
    return rawInner;
  } finally {
    await browser.disconnect(); // always let go of the remote browser
  }
}

Notice waitForFunction. waitForSelector("#markers") on its own is a trap: the empty div often exists in the DOM before the script fills it, so the selector resolves instantly and you scrape nothing. Waiting for innerHTML.length > 100 means "wait until the JSON is actually in there." Pick a threshold above whatever the empty/half state looks like.

Now the plain string work

Everything past this point runs in Node, no browser needed.

function parseMarkers(rawInner) {
  // SSR frameworks sprinkle empty comments into the DOM; they break JSON.parse.
  const cleaned = rawInner.replace(/<!--\s*-->/g, "");
  const json = unEscapeHtml(cleaned);
  const listings = JSON.parse(json); // array of positional arrays

  return listings
    .map((row, i) => {
      // Positional, not keyed: [ name, lat, lng, zIndex, popupHtml, ..., uniqueId ]
      const name = (row[0] || "Unknown").replace(/&amp;/g, "&");
      const latitude = parseFloat(row[1]) || 0;
      const longitude = parseFloat(row[2]) || 0;
      const uniqueId = row[row.length - 1] || i;

      // address lives inside the popup HTML at [4]: first <p>, <br> -> ", "
      let address = "";
      const p = /<p>(.*?)<\/p>/s.exec(row[4] || "");
      if (p) address = p[1].replace(/<br\s*\/?>/g, ", ").replace(/,\s*$/, "");

      return { uniqueId, name, address, latitude, longitude };
    })
    .filter((loc) => loc.latitude && loc.longitude); // drop 0/0 junk
}

One page load returns the entire dataset. There's no pagination, no radius sweep, no tiling. Every pin is already in that single array, which is the upside of the DOM-blob pattern: once you've paid for the browser, you get everything in one shot.

A few things that bit me and will bit you:

The double-escape. &amp; shows up when the source escaped an already-escaped string. Decode & last in your unescape chain, or Foyles & Co turns into a broken parse.
String coordinates. Lat and lng come as "51.5155", quoted. parseFloat them or your map math silently produces NaN.
Positional arrays. There are no keys. row[0] is the name today because that's the order the site emits. If they reorder, your mapping is wrong and nothing throws. Log a sample row and eyeball it before you trust the indices.
Half-filled div. Covered above, but it's the number-one reason "it worked in DevTools, empty in the script."

Wrap the connect-navigate-extract cycle in a small retry loop (three attempts is plenty) and always disconnect in a finally. Remote browsers drop connections, and a leaked session is a session you're paying for.

Why this generalizes

The National Book Tokens pattern (Google Maps reading pin data from a hidden element) is everywhere on old-school store-locator pages. But the real transferable move isn't Puppeteer. It's the raw-HTML-versus-live-DOM diff. That one check tells you which of three worlds you're in:

Data in the raw HTML: fetch plus a parser, no browser.
Data from a JSON XHR you can see in the Network tab: call that endpoint directly, still no browser.
Data absent from both, appearing only after JS runs: now, and only now, a headless browser, used purely to run the script and read the settled DOM.

Reach for Chrome last, and when you do, make it do the smallest possible thing. It runs the JavaScript, it hands you a string, and then you fire it and go back to code that doesn't eat 400MB of RAM.

Good-citizen note: these are public store-finder endpoints. Cache the result, don't re-run it every request, and don't hammer someone's map page for data that changes maybe monthly.

I build scrapers like this for a living. My published actors are at apify.com/native_emblem if you'd rather rent the output than maintain the Chrome.

Your scraper isn't broken. The site changed, and it didn't tell you.

Andrii Votiakov — Wed, 29 Jul 2026 08:50:28 +0000

The scraper had been green for eleven weeks. Every run exited 0, every run wrote rows to the warehouse, the dashboard showed a healthy line. Then someone in the pricing team asked why 40% of products suddenly had no price. They'd been shipping decisions on that data for most of a week.

Nothing had failed. That's the whole problem. The site had quietly moved the price into a different element during an A/B test, our selector returned null, and the pipeline did exactly what we told it to: it wrote null and moved on. A scrape can succeed structurally and fail semantically, and almost nobody monitors for the second kind.

Exit 0 is not the same as correct

I've written about scrapers that return zero results and still exit clean. Silent drift is the same disease at a different stage. The request went through. The HTML parsed. The loop ran. Every layer reported success because every layer only knows about its own job, and none of them know what a good record is supposed to look like.

Markup drift is the number one way scrapers rot, and it rarely announces itself with a crash. A crash would be a gift. A crash pages someone. What actually happens is subtler: a retailer ships a redesign, or rolls out a variant to 10% of traffic, or renames a CSS class in a build step, and one field goes dark while the other nine keep flowing. Your fill rate drops from 99% to 61% and the only thing that changed color is a number in a spreadsheet three teams away.

Validate the output, not the input

The instinct is to harden the parser. More selectors, more try/catch, more defensive HTML handling. That helps, but it's aiming at the wrong target. You cannot assert your way to correctness on input you don't control. What you can control is a contract on the output: after a run, a record is only allowed to exist if it meets a schema you defined.

The cheapest version is a canary check on a sample. Pull N records from the run, assert the fields that must never be null actually aren't, and fail loudly if too few pass.

function assertHealthy(records, { sample = 200, minFillRate = 0.9 } = {}) {
  const batch = records.slice(0, sample);
  const required = ["title", "price", "currency"];

  const fillRate = {};
  for (const field of required) {
    const filled = batch.filter((r) => r[field] != null && r[field] !== "").length;
    fillRate[field] = filled / batch.length;
  }

  const failed = required.filter((f) => fillRate[f] < minFillRate);
  if (failed.length) {
    throw new Error(
      `Fill-rate check failed: ${failed
        .map((f) => `${f}=${(fillRate[f] * 100).toFixed(1)}%`)
        .join(", ")}`
    );
  }
  return fillRate;
}

Now a run that produces 40% null prices doesn't exit 0. It throws, and throwing is a thing your alerting already understands. You've converted a silent semantic failure into a loud structural one, which is the only kind of failure your on-call rotation can actually see.

A static threshold is a starting point, not the answer

minFillRate: 0.9 is a guess, and guesses age badly. Some fields are legitimately sparse. Not every product has a discount, not every listing has a review count, and a hard 90% floor on an optional field will page you at 3am for nothing. The failure you actually care about is not "this field is low," it's "this field is lower than it was yesterday."

So track fill rate per field over time and compare each run to a rolling baseline. The alert fires on the delta, not the absolute.

function driftAlarm(field, current, history, { drop = 0.15 } = {}) {
  // history: last ~14 runs of fill rate for this field
  const baseline = history.reduce((a, b) => a + b, 0) / history.length;
  if (baseline - current > drop) {
    return `${field} fill-rate ${(current * 100).toFixed(1)}% vs baseline ${(
      baseline * 100
    ).toFixed(1)}%, likely markup drift`;
  }
  return null;
}

A field that has hovered at 12% for two weeks and is still at 12% is fine. A field that lived at 99% and dropped to 61% overnight is a selector that just died, and you'll know within one run instead of one quarter. Persist these numbers somewhere boring, a small table or even a JSON blob per run, and the baseline builds itself.

Don't let one dead selector take the record down

The last piece is structural. If a single field extraction is the only path to that field, then the day it breaks, the field is simply gone. The fix is to run several extraction strategies per field and take the first that produces a valid value. A machine-readable data block on the page, a couple of markup patterns, a fallback derived from a neighbor. When one strategy dies to a redesign, another usually still stands, and the record survives with its price intact.

This is the shape I lean on in production. Multiple independent ways to reach each field, a schema contract on the way out, and fill-rate tracking so I find out from my own monitoring before a downstream team finds out from a broken report. The strategies do the resilience. The output validation does the alerting. You need both, because resilient extraction that you never measure will still drift eventually, and it won't tell you either.

The mental model that fixes this: your scraper's job is not to finish. It's to produce records that pass a contract, and anything short of that is a failure even when the process exits 0.

I'm building Cartpie, an e-commerce product-data platform where this multi-strategy extraction and output validation is the whole point, and I publish scrapers on Apify that are built the same way.

Your BigQuery bill is a query-shape problem. We cut one 72% in three weeks.

Andrii Votiakov — Mon, 27 Jul 2026 08:35:43 +0000

BigQuery doesn't charge you for having data. It charges you for touching it. On-demand pricing bills per byte scanned, which means your bill isn't a data-size problem or a traffic problem. It's a query-shape problem, and query shape is fixable in an afternoon per table.

A recent client was a textbook case. Dashboards running SELECT * over an unpartitioned events table, so every refresh scanned the full history back to 2019. Plus a pile of scheduled queries feeding tables nobody read anymore. Three weeks of work cut the bill 72%, about £2K a month, and the queries got faster. That last part surprises people. It shouldn't. Bytes scanned is both the cost metric and the latency driver. Same lever.

Week 1: find out where the money goes

Don't guess. BigQuery logs every job with its billed bytes:

SELECT
  user_email,
  query,
  ROUND(total_bytes_billed / POW(1024, 4), 2) AS tib_billed,
  COUNT(*) AS runs,
  ROUND(SUM(total_bytes_billed) / POW(1024, 4), 2) AS tib_total
FROM `region-eu`.INFORMATION_SCHEMA.JOBS
WHERE creation_time > TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY)
  AND job_type = 'QUERY'
GROUP BY user_email, query
ORDER BY tib_total DESC
LIMIT 25;

Almost every time, a handful of queries dominate. Here it was five queries producing most of the spend, and four of them were dashboard refreshes running on a schedule. Recurring queries are where the money is, because a bad shape times 48 runs a day compounds while you sleep.

Week 2: fix the shape

Partition by the column people actually filter on (usually a date), cluster by the next most common filters:

CREATE TABLE analytics.events_v2
PARTITION BY DATE(event_timestamp)
CLUSTER BY customer_id, event_type
OPTIONS (require_partition_filter = TRUE)
AS SELECT * FROM analytics.events;

Two details matter more than the DDL:

require_partition_filter = TRUE is the line that keeps the fix fixed. Without it, someone writes one exploratory full scan in month two and you're back where you started. With it, BigQuery rejects the query instead of billing you for it. Make waste an error, not a habit.

Clustering only pays if queries filter or join on the clustered columns. One team's ad-hoc analysis here filtered on a column we hadn't clustered, and their queries saw no improvement at all. Cluster for the workload you measured in week 1, not for the schema you imagine.

Then rewrite the top offenders. SELECT * on a columnar store bills you for every column, including the 40 you never render. Listing columns explicitly cut one dashboard query's scan by an order of magnitude before partitioning even entered the picture.

Week 3: kill the zombie jobs

This is the unglamorous part and it was worth a big slice of the 72%. Scheduled queries outlive their consumers. The dashboard gets deprecated, the Slack report gets muted, and the pipeline underneath keeps materializing tables daily, forever, at full scan cost.

Cross-reference every scheduled query against actual reads of its output table (INFORMATION_SCHEMA again, look for jobs referencing it). No reads in 60 days? Pause it. We paused first rather than deleting, waited two weeks for screams, heard none, deleted. Nobody has asked about any of them since.

What I'd tell you to skip

Slot reservations. Flat-rate pricing is a real tool, but it's the tool you reach for after fixing query shape, not instead of it. Buying capacity to run wasteful queries is paying rent on the waste. Fix the shape first, then check whether your steady-state scan volume still justifies reservations. After this cleanup, it didn't come close.

The whole engagement was three weeks because none of this is hard. It just requires someone to actually look at INFORMATION_SCHEMA instead of the billing page, which tells you what you spent but never why.

I do this kind of work at reducecost.cloud on pay-for-savings terms: nothing upfront, I take a share of the reduction I deliver, and if I save you nothing you pay nothing.

Build a SHEIN price tracker in 20 minutes with n8n and an Apify actor

Andrii Votiakov — Fri, 24 Jul 2026 16:56:13 +0000

I wanted to know the moment a specific SHEIN dress dropped below a price I'd set, without checking the page myself like it was 2011. The whole thing took twenty minutes to wire up in n8n against one of my Apify actors, it runs on a schedule for pennies, and it pings me only when a price actually falls. No servers, no cron on a box I have to remember exists, no browser automation to babysit. Here's the exact build.

The shape is five nodes: a schedule trigger fires, an HTTP node runs the SHEIN actor and gets the results back in one call, a code node parses the products, another compares each price against what I stored last run, and a notify node fires only on a drop. Everything talks to the public Apify API, so there's nothing exotic to install.

1. Schedule trigger

Add a Schedule Trigger node. Once a day is plenty for retail pricing, more often than hourly is just noise and wasted runs. Set it to something like every day at 08:00. That's the whole node. n8n handles the cron for you.

2. Run the actor and get results in one shot

Add an HTTP Request node. The trick that keeps this simple is Apify's run-sync-get-dataset-items endpoint: it starts the actor, waits for it to finish, and returns the scraped dataset in the same response. No polling, no run-ID juggling, no second call to fetch results.

Method: POST
URL: https://api.apify.com/v2/acts/native_emblem~shein-product-scraper/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN
Body content type: JSON
Body: the actor's input, the keywords or product URLs you want priced

{
  "keywords": ["summer midi dress"],
  "maxItems": 25
}

Put your token in n8n credentials rather than pasting it into the URL in the clear, but the shape is exactly that: one POST, token as a query param, actor input as the JSON body. The response is a plain array of product objects. If you're curious what comes back before wiring the rest, the same call works from your terminal:

curl -X POST \
  "https://api.apify.com/v2/acts/native_emblem~shein-product-scraper/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"keywords":["summer midi dress"],"maxItems":25}'

One warning: run-sync holds the connection open until the actor finishes, so keep maxItems modest for a tracker. You're pricing a watchlist, not mirroring the catalog. Small inputs come back in a few seconds and never bump the sync timeout.

3. Parse the products

Add a Code node. The dataset comes back as items, and you just pull the fields you care about into a flat shape the next steps can diff. Product ID as the key, price as the value, name for the notification text.

// n8n Code node, runs once over all incoming items
return items.map(({ json }) => ({
  json: {
    id: json.productId,
    name: json.name,
    price: json.price,       // current price as a number
    url: json.url,
  },
}));

Field names depend on the actor's output schema, so glance at one real item and map accordingly. The point is you leave this node with one clean record per product: id, name, price, url.

4. Diff against last run

You need somewhere to remember yesterday's price. A Google Sheet is the no-fuss choice and the one I'd start with: one row per product ID, columns for name, last price, and URL. n8n's Google Sheets node reads and writes it directly. A tiny Postgres table or even n8n's own static data works too, but a sheet you can eyeball is the friendliest for a first build.

Read the sheet, join on product ID, and compare. Add another Code node, or an IF node if you prefer clicking to typing:

// keep only products whose price dropped vs the stored value
return items.filter(({ json }) =>
  json.lastPrice != null && json.price < json.lastPrice
).map((item) => ({
  json: {
    ...item.json,
    drop: item.json.lastPrice - item.json.price,
  },
}));

Then write the fresh prices back to the sheet so the next run compares against today. That write-back is the step people forget, and without it every run re-alerts on the same old drop forever.

5. Notify on a drop only

Whatever survives the filter is a genuine price drop, so hand it to a notification node. Slack, Telegram, Discord, email, n8n has a node for all of them. Message body straight from the fields you carried through:

"summer midi dress" dropped 4.20 to 11.79. {url}

If nothing dropped, the filter passes zero items, the notify node does nothing, and you hear nothing. Silence is the correct default for a price tracker. You want it to interrupt you exactly when it matters and stay quiet the other 364 mornings.

That's the whole thing

Five nodes, one scheduled trigger, one API call that both runs the scraper and returns the data, a sheet for memory, and a filter that only speaks up on a drop. Swap keywords for a list of exact product URLs and you're watching a specific watchlist instead of a search. Point it at any of the fields the actor returns, stock status, rating, whatever, and the same skeleton becomes a back-in-stock alert or a review watcher. The pattern outlives the example.

The SHEIN actor I wired here is one I publish on Apify: proxyless and clean-JSON, so parse steps like the one above stay this short. It takes a keyword and returns matching products, so you don't need product URLs to start, and it's priced per product actually returned ($5 per 1,000 search results). It's the same extraction engine behind Cartpie, the e-commerce product-data platform I'm building. Wire it into n8n once and you'll find a dozen more things worth watching.

Half the web already hands you clean product data. You just have to ask.

Andrii Votiakov — Fri, 24 Jul 2026 07:30:15 +0000

I once spent an afternoon writing selectors to pull a price off a product page. Wrote the CSS path, handled the currency symbol, dealt with the sale-price-versus-list-price mess, tested it across a dozen listings. Then I opened view-source, scrolled a bit, and found the same price sitting in a <script type="application/ld+json"> block, typed, labeled, and currency-tagged. The retailer had published it there on purpose. I'd done a morning's work to reconstruct data that was already handed to me clean.

That block wasn't there for scrapers. It was there for Google. And that's exactly why it's the best starting point you have.

Retailers publish machine-readable product data for search engines

A huge share of e-commerce pages carry a structured description of the product embedded right in the HTML, separate from the visual markup. It exists so that Google, Bing, Pinterest, and every other crawler can show rich results: the price, the star rating, the in-stock badge you see under a search listing. Retailers are strongly incentivized to keep it accurate and current, because getting it wrong means losing the rich result or getting penalized for mismatched data.

There are three common shapes. JSON-LD is the one you want most, a self-contained JSON object following the schema.org vocabulary. OpenGraph tags are <meta> elements in the head, coarser but nearly universal, great for name, image, and sometimes price. Microdata is the old inline style, attributes like itemprop sprinkled through the visible HTML. All three describe the same underlying thing. JSON-LD just does it in one clean block instead of scattered across the DOM.

Reading a JSON-LD Product block

The pattern is boring in the best way. Find the script tags, parse each as JSON, and look for the object whose @type is Product.

import * as cheerio from "cheerio";

function extractProduct(html) {
  const $ = cheerio.load(html);
  const blocks = $('script[type="application/ld+json"]')
    .map((_, el) => $(el).contents().text())
    .get();

  for (const raw of blocks) {
    let data;
    try {
      data = JSON.parse(raw);
    } catch {
      continue; // malformed block, skip it
    }

    // JSON-LD is often an array or wrapped in @graph
    const nodes = Array.isArray(data) ? data : data["@graph"] || [data];
    const product = nodes.find((n) => n && n["@type"] === "Product");
    if (!product) continue;

    const offer = [].concat(product.offers || [])[0] || {};
    return {
      name: product.name,
      sku: product.sku,
      image: product.image,
      price: offer.price,
      currency: offer.priceCurrency,
      availability: offer.availability, // e.g. schema.org/InStock
    };
  }
  return null;
}

No currency-symbol parsing. No sale-versus-list guesswork. The availability field is a plain URL you can map to a boolean. You get name, sku, image, price, and stock state from one parse, and it survives visual redesigns because the retailer maintains it independently of the layout their designers keep changing.

Beyond being easier, it's more polite. You're reading a static block that's already in the HTML the retailer chose to serve, not hammering internal endpoints or driving a headless browser through a render loop. Less load on them, less cost for you, more stability for both.

Rank your sources by reliability

Once you internalize that pages carry data at several levels of quality, extraction becomes a preference order rather than a single brittle path. Conceptually, structured data sits above hand-written selectors. A JSON-LD Product block is typed and maintained for Google; a CSS selector against rendered markup is your guess about someone else's HTML, and it breaks the day their build tool renames a class.

So the order I reason in: prefer the typed, purpose-published structured block first. Fall back to OpenGraph and microdata for the fields it covers. Reach for hand selectors last, for the fields nothing else exposes. The higher up the stack you can stay, the longer your scraper lives between repairs.

The catch: structured data lies sometimes

It's cleaner, not infallible. Structured blocks can be stale, cached from a build that ran before the last price change. They can be partial, a Product with a name and image but no offers. They can be templated wrong, every variant reporting the parent's price. Some sites inject the JSON-LD client-side, so it isn't in the initial HTML at all. Trust it as the best signal, not the only one.

Which means you validate what you pull and fall back when it fails the check. If the JSON-LD has no price, drop to OpenGraph. If that's empty too, try the selector. Run the fill-rate and required-field checks over the final merged record so a silently-empty offers array gets caught before it ships. The structured block gives you a fast, stable default; the fallbacks and the validation are what make it trustworthy.

That layered approach, several independent ways to reach each field with a clear preference order and validation across them, is the core of how I build extraction in production. I won't get into how the strategies are weighted or combined against any specific site, because that tuning is the actual work and it shifts constantly. But the principle is public and it's the right instinct: before you write a single selector, view-source and check whether the retailer already handed you the answer in a block they maintain for Google. Half the time, they did.

I'm building Cartpie, an e-commerce product-data platform built on exactly this multi-source approach, and I publish proxyless scrapers on Apify that start from structured data first.

My pipeline parsed every page twice, and I blamed the parser

Andrii Votiakov — Thu, 23 Jul 2026 13:20:23 +0000

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

This story fits in one diff: memorySize: 2048 became memorySize: 512. Getting there took a week of optimizing the wrong thing, 404 lines of it, and one evening of actually reading the code between the functions.

The system

A product-data pipeline I ran for a client. In: a URL from any of tens of thousands of retailer domains. Out: a clean record with title, price, currency, images, availability. Eight figures of product pages refreshed every month, all of it on AWS Lambda and SQS.

Downloads went through a proxy waterfall. Plain HTTP request first, because it's free. If that fails, escalate: datacenter proxy, then residential, then a real browser. Each tier costs more than the one before it, so the economics of the whole pipeline depend on the cheap tiers winning as often as possible.

Extraction ran four strategies against every downloaded page: JSON-LD, microdata, OpenGraph, and a per-domain selector config. Parse the HTML once with cheerio, let the four extractors share the tree, merge results field by field. Boring, layered. It works. I've written before about why I like this shape: when a retailer redesigns, one strategy dies and the other three carry the record.

This is a story about the word "once" in that paragraph.

The symptom

Fashion-retailer product pages are enormous. A server-rendered page with a hundred color-and-size variants ships HTML by the megabyte, and a megabyte of HTML costs a multiple of itself in heap the moment it becomes a DOM tree. The download function had 2 GB of memory and still ran hot on heavy pages: slow invocations, memory climbing toward the ceiling, and enough background worry that the code had logMemoryUsage() calls sprinkled through it from earlier rounds of suspicion.

The graphs said memory. The parser is what uses memory. So the parser was guilty. Case closed, investigation over, off to optimize.

The wrong week

The commit is still in the repo, and the message alone is a confession: feat: EXPERIMENTAL cheerio memory reduction. Three files, 404 lines added, 53 removed.

What those lines contained:

A pre-parse HTML stripper that removed scripts, styles, and comments before cheerio ever saw the document, complete with a proud log line, quoted verbatim from the diff: HTML optimized: ${html.length} -> ${optimizedHtml.length} characters.
A brand-new utils/memory.ts with a whole toolkit: estimateHtmlMemoryUsage(), isHtmlTooLarge(), optimizeCheerio(), and the crown jewel, forceGarbageCollection(). Running Node with --expose-gc in production is the software equivalent of hitting the TV.
A rewrite of the extractor runner from Promise.all to a sequential for-loop, on the theory that four extractors sharing one tree "held references too long". They share a reference to the same tree, so running them one at a time saves almost nothing. That detail should have tipped me off that I didn't understand where the memory was going. It didn't.

Here's the trap: it helped. Stripping megabytes of scripts before parsing genuinely shrinks the tree, and "it helped" feels like progress when it's actually anesthesia. Symptom optimization pays out just often enough to keep you at the table.

The read

Five days later I sat down and read the waterfall code end to end. Not the extractors. The plumbing between them.

A proxy waterfall needs to know whether a tier succeeded, and in scraping a 200 response means nothing. Bot walls return 200, empty client-side shells return 200, parked domains return 200 with a smile. So each tier's success check called a function named checkMinimumData: parse the response, run all four extraction strategies, and answer one question. Did we get at least a title and a price? Honest check, correct implementation.

Then, one function later, the success path took the response it had just validated... and parsed it. Ran all four strategies. Again. Same bytes, same cheerio load, same selector config, this time to produce the record we'd actually save.

Every page that succeeded anywhere in the waterfall was fully extracted exactly twice. Under the right timing, both DOM trees existed in memory at once. Which is how a 2 GB function chokes on a 3 MB page. The parser I'd spent a week optimizing was doing its job flawlessly. Twice per page.

Why nobody saw it

Both call sites were individually correct, which is exactly why the bug survived every review.

Validation genuinely needs extraction: checking that a page yields real product data is the only honest way to grade a scrape. Processing genuinely needs extraction: that's the product. Each function read sane on its own, each was tested on its own, and the duplication lived in the seam between them. Seams belong to nobody, so nobody was looking there.

It also explains why the optimization week "helped". Stripping scripts before parsing made both parses cheaper, so the numbers moved for a while.

The fix

Cache what validation computed, hand it to the processor:

// tier success check: run extraction ONCE, remember the result
const minimumDataResult = await checkMinimumData(
  parserConfig, extractor, response, pageDocument, url,
);
if (minimumDataResult.status === "Success") {
  validationResultCache = minimumDataResult;
}
return minimumDataResult.status === "Success";

// success path: extraction already happened, do not do it again
const minimumDataResult = (result as any).cachedValidationResult;

Twenty meaningful lines. And yes, as any - the cache rode on an object whose type didn't know about it, and I wanted the win in production the same day. The cast is still in the file. I checked this morning.

The receipts

The commit timestamps from that day are the most honest part of this story:

At 12:00, high on a week of optimization, I shipped this:

-    memorySize: 2048
+    memorySize: 512

At 18:33, the extraction cache landed. I'll let you reconstruct how the hours in between went, with a function that was still doing every parse twice on a quarter of its former memory.

The order was backwards, and it worked anyway, in the way that matters: quartering the allocation turned a hidden inefficiency into a visible emergency, and the emergency forced the read that the graphs never did.

Lambda bills memory times duration. Cutting allocation 4x cuts the price of every invocation by 75% before you even count duration, and duration dropped too, because half of all parsing work in the pipeline's hottest function simply stopped existing. No product change. No extraction-quality change. The savings came from deleting work, not from doing the work faster.

The 404-line optimization commit is still in the history. I left it as a monument. forceGarbageCollection is still exported from utils/memory.ts, still imported by the extractor builder. Never called from anywhere. I checked that this morning too. The most honest line of code in the repo.

What actually changed

Two habits came out of that week, and they've paid for themselves several times since.

First, any pipeline with a validate-then-process shape gets one specific review question now: what does validation compute that processing recomputes? The answer is almost never "nothing". Retry wrappers validate. Health checks validate. Queue consumers validate, then hand the raw input to a processor that starts from zero. The duplicated work hides between the two functions, in code no single reviewer owns.

Second, when the symptom is resource usage, the first question is "what work is happening", not "how do I make this work cheaper". A profiler answers the first. Optimizations answer the second. I had them in the wrong order for a full week while the answer sat in a function I'd read ten times without reading it.

Competitive programming taught me, years ago, to read the problem twice before typing. Production has the same rule. The difference is that in production, the problem statement is your own code.

Scraping millions of pages a day: what actually breaks

Andrii Votiakov — Thu, 23 Jul 2026 07:50:26 +0000

I ran a scraping platform that processed millions of pages a day at roughly 95% extraction success, around three seconds per page. The fetch-and-parse code, the part everyone thinks of as "the scraper", was a tiny slice of the whole thing. The years went into everything around it.

Here's what actually broke, more or less in the order it broke.

The queue with no manners

Crawling is bursty in a way that surprises you the first time. One category page fans out into a few hundred product URLs. A sitemap refresh dumps half a million at once. Meanwhile your parsers chew through pages at a steady rate that doesn't care about your ambitions.

Our first queue was effectively unbounded. It absorbed every burst happily, Redis memory climbed for two days, and then the whole thing fell over at once instead of slowing down gracefully. Lesson: if your queue can't say no, it's not a queue, it's a landfill. Bound it, and make producers block or shed when it's full. A crawler that pauses discovery for an hour is a non-event. A crawler that OOMs the broker is a weekend.

Retries that stampede

A target site starts throwing 500s for two minutes. Fine, that happens. Every failed page gets rescheduled with the same fixed backoff. Which means 40,000 pages land back on that site in the same ten-second window, it tips over again, and now their WAF has opinions about you. You caused the second outage yourself.

Two fixes, both mandatory: full jitter on backoff, and a per-domain circuit breaker so a struggling site stops receiving traffic entirely for a while.

import random

def backoff(attempt, base=2.0, cap=300.0):
    # full jitter: spread retries across the whole window
    return random.uniform(0, min(cap, base * 2 ** attempt))

That one-liner is the difference between "we retry politely" and "we DDoS people by accident".

Dedupe before fetch, not after

The same product reaches you through six URLs. Tracking params, color variant params, three different category paths, a mobile subdomain. If you dedupe after fetching, you paid for six fetches to keep one page. At millions of pages a day that's real money and real block-risk for zero data.

Canonicalize at enqueue time. Strip the junk params, pick one canonical form, check the seen-set before the URL ever enters the queue. And watch that seen-set: it grows forever unless you give it a TTL or accept a small false-positive rate with something bloom-filter-shaped.

Parser drift, the quiet one

Sites almost never break your parser loudly. They drip. A price moves from a DOM node into an embedded script blob. An image attribute gets renamed. Your extraction still "succeeds", the run is green, and one field quietly goes null on 30% of pages. Nobody notices until a downstream consumer asks why half the prices vanished last Tuesday.

Field-level fill rates saved us. For every field, track what fraction of pages yielded a value, keep a rolling baseline per site, alert on the delta. A green run only proves the code didn't crash.

The other thing that got us to 95%: never rely on a single extraction strategy. Embedded JSON, microdata, plain DOM selectors, run more than one and merge by confidence. When a site redesign kills one of them, the others carry it while you fix things, instead of the number going to zero overnight.

The CPU step hiding in an I/O system

Scraping feels like an I/O problem. Fetch, wait, fetch, wait. So you build it all
async and assume the network is the only thing you're waiting on. Then you profile a
slow day and find the bottleneck is HTML parsing.

Loading a big product page into an HTML parser and running extractors over it is
CPU-bound and synchronous. On a single-threaded runtime, every parse blocks the event
loop, which means it blocks the very downloads you thought were the slow part. Your
concurrency number looks healthy and your throughput doesn't match it, because workers
are queued behind each other's parsing, not behind the network.

The fix is to move the CPU-heavy parse off the main thread into a worker pool, and fall
back to inline parsing when the pool is disabled or a job errors. Downloads keep flowing
while parsing happens elsewhere. Obvious in hindsight, invisible until you measure it.
It ended up one of the bigger single throughput wins I got at scale.

You will need to debug the past

The nastiest questions arrive late. "Why did brand X prices go missing on the 14th?" It's the 19th. The pages have changed since. Your logs say everything was fine.

We got out of that hole with a typed per-page event stream plus archived raw HTML. Every page emitted structured events (fetched, parsed, extracted, with typed payloads) and the raw response got stored. So "what happened on the 14th" became a replay: pull the archived pages, re-run the current parser against them, diff the output. Debugging in the past tense. Without replay you're reconstructing incidents from vibes.

The uncomfortable summary

None of this is glamorous. There's no clever algorithm in this post, and that's the point. At small scale, scraping is a parsing problem. At millions of pages a day it's a distributed systems problem wearing a parsing costume, and the failure modes are the boring ones: backpressure, thundering herds, silent data decay, no audit trail.

All of this is what I'm productizing at Cartpie, e-commerce product data as an API so you don't have to own any of the machinery above. Free tier is live.

Anti-bot without melting your budget: the proxy waterfall.

Andrii Votiakov — Wed, 22 Jul 2026 09:05:44 +0000

Residential proxies are the caviar of scraping infra. They work when nothing else does, and they cost accordingly. The mistake I see over and over is teams routing every single request through residential IPs because a handful of pages needed it. You end up paying steak prices to fetch a robots.txt.

On one client I cut BrightData spend by 90% and ScrapingBee by 67% without touching success rates, and the whole change was structural. Stop treating all traffic as equally sensitive. Tier it, and only escalate when you have to.

The waterfall

The idea is a fall-through ladder. Cheap options first, expensive last, and you only move down a rung when the current one demonstrably fails.

Tier 0, no proxy. Plenty of endpoints don't care. Public JSON APIs, sitemaps, image CDNs, anything that isn't fingerprinting you. If it works direct, that request costs you nothing. On my own stack naked direct requests clear roughly 40% of targets by themselves.
Tier 0.5, still no proxy, but fix your TLS fingerprint. This is the rung everyone skips. A plain HTTP client has a dead-giveaway TLS/JA3 handshake that Cloudflare and similar defenses match before your request even reaches the app. Swap in a client that mimics a real Chrome fingerprint (cycletls and friends) and a slice of "blocked" sites start returning 200s. Still free, still no proxy. This one change bought me a double-digit percentage-point jump on Cloudflare-fronted pages that direct requests couldn't touch. Do it before you spend a cent on IPs.
Tier 1, datacenter / mobile proxies. Cheap, fast, plentiful. Good enough for the large majority of HTML and API traffic that just wants a non-suspicious IP with reasonable rate limiting. Rotating mobile IPs sit around here too and punch above datacenter for not much more.
Tier 2, residential / rendered. The expensive tier, reserved for flows that actually fingerprint hard: checkout, some search endpoints, sites that clearly block datacenter ranges, anything needing real JS execution.
Tier 3, managed anti-bot / unlocker service. The nuclear option. Slowest and priciest per request, handles CAPTCHAs, clears ~99%. Only when everything above bounces.

The point of the ladder is the distribution. If the top is free and clears 40%, a fingerprint fix pushes that higher for nothing, and the ~99% unlocker only ever sees the genuinely hard tail, your average cost per page collapses even though your worst-case tool is still expensive.

The two things that make this work and not just be a fancy retry loop: knowing when a tier failed, and remembering what worked.

Failure is not just a status code

A 403 is easy. The dangerous case is a 200 that's actually a soft block. You get back a page, it has the right content-type, your parser even runs, and it's a captcha interstitial or a stripped-down decoy. If you only check the HTTP status, you'll happily record garbage and never escalate.

So validation is content-level, not transport-level:

def is_good(resp, expect):
    if resp.status_code != 200:
        return False
    body = resp.text
    if len(body) < expect.get("min_bytes", 500):
        return False
    if any(m in body for m in BLOCK_MARKERS):  # captcha/challenge strings
        return False
    # the actual proof: did the data we came for show up?
    return expect["must_contain"] in body


def fetch(url, expect):
    for tier in TIERS:  # cheapest first
        resp = tier.get(url)
        if is_good(resp, expect):
            remember(url_pattern(url), tier)  # cache the winner
            return resp, tier
    raise AllTiersFailed(url)

must_contain is the honest check. A price marker, a known JSON key, a product-id pattern. If the thing you came for isn't in the body, you didn't get the page, whatever the status line claimed.

Remember what worked

The other half of the savings is remember. Once you learn that a given domain or URL shape needs Tier 2, cache that decision with a TTL and start future requests to that pattern at the right rung instead of walking the whole ladder every time. Walking the ladder on every request means you pay the datacenter attempt and the residential attempt for pages you already know need residential. Cache it and you pay for the tier that works, most of the time, and re-probe occasionally in case the site relaxed.

Flip side: expire those decisions. Sites change defenses. A domain that needed residential in March might be fine on datacenter by June, and a stale cache keeps you overpaying. A day or two TTL was the sweet spot for me.

Where the money actually went, once I measured it: the vast majority of requests were fine on Tier 0 or Tier 1, and a small slice of genuinely sensitive flows justified the residential spend. Before the waterfall, that small slice was setting the price for everything. That's the whole trick. Not a cheaper proxy vendor, a cheaper distribution of traffic across the vendors you already have.

One caveat worth stating: this adds moving parts. A tier ladder, a validation layer, a decision cache with expiry. If you're scraping a few thousand pages a month off one friendly site, skip all of it and use one proxy. The waterfall earns its complexity at volume, when a 90% cut is real money.

I do cloud and infra cost work under reducecost.cloud, where this kind of "same result, structured cheaper" thinking shows up constantly, and the scraping side of it feeds cartpie.com, an e-commerce product-data scraping platform with a free tier live now.

Your scraper probably doesn't need a browser. It needs the right request.

Andrii Votiakov — Tue, 21 Jul 2026 09:21:42 +0000

The default move when a page is JavaScript-heavy is to reach for headless Chrome. Spin up Puppeteer, wait for the DOM to settle, scrape the rendered result. It works. It also costs you a couple hundred megabytes of RAM per worker, a few seconds per page, and a proxy bill that scales with how much you hate money.

Most of the time you don't need it. The page you're looking at is JavaScript-heavy precisely because the data arrives separately, as JSON, over an API the site's own frontend calls. If you can call that API directly, you skip the browser, the render wait, and most of the cost. I've run a SHEIN scraper on Apify this way with no proxies at all, and it holds up at volume.

The data is already JSON, somewhere

Think about how a modern shop is built. The HTML shell loads, then the frontend fires XHR/fetch requests to some backend-for-frontend, gets back clean JSON, and paints the page. That JSON is the thing you want. It's already parsed, already structured, and it doesn't have a single CSS selector waiting to break on you.

Rendering the page to scrape the DOM is doing all that work twice: once by the browser, once by you. Going straight to the JSON cuts out the browser entirely.

How to actually find it

Open the site in a normal browser and open devtools. Then:

Go to the Network tab and filter to Fetch/XHR. This drops images, fonts, and scripts, and leaves the data calls. Clear it, then reload or click the thing that loads the products.
Watch what comes back. Click a request, look at the Preview tab. You're hunting for a response that looks like the page's data: a list of products, a price, an ID. When you find one, you've found your target.
Read the request that produced it. What's the URL shape, what query params or body does it take, what headers actually matter. Most headers are noise. A few are load-bearing. You find out which by removing them one at a time.
Check the mobile site. This is the tip people skip. The mobile web app or the app's own backend is often simpler, less defended, and returns flatter JSON than the desktop equivalent. If the desktop API is a pain, m. something or the mobile user-agent path frequently isn't.
Reproduce it outside the browser. Right-click the request, copy as cURL, paste into a terminal. If it still returns data, you have a scraper. If it 403s, something in the headers or params was doing authentication work, and now you know where to look.

Here's the shape of what you end up with once the browser is gone:

const res = await fetch(endpoint, {
  headers: {
    "user-agent": MOBILE_UA,
    "accept": "application/json",
    // the handful of headers that actually gate the response,
    // discovered by removing them until it breaks
  },
});

const json = await res.json();
const products = json.data.items.map((it) => ({
  id: it.goods_id,
  title: it.goods_name,
  price: it.retail_price.amount,
  currency: it.retail_price.currency,
}));

That's the whole thing. No render loop, no waitForSelector, no headless binary to keep patched.

The parts I won't hand you

I'm being deliberately vague about the exact endpoints, params, and signing for any specific site, including SHEIN. Figuring those out is the actual work, and it's a moving target that shifts when the site ships a new app version. What I'll give you is the method. What I won't is the finished recipe, partly because it's my edge and partly because it'd be stale by the time you read it.

Some genuinely need a browser, and I'll say so. Endpoints signed with a token generated by obfuscated client-side JS, or flows gated behind a real interaction, are sometimes not worth cracking. Rendering once to harvest a token, then firing cheap direct requests after, is a fair compromise. But reach for that after you've checked the Network tab, not before.

Rule of thumb I'd actually bet on: if a site has a mobile app, it has an API you can probably reach, and it's probably nicer than anything you'd scrape off the desktop DOM. Start there.

If you want a worked example that runs proxyless in production, my SHEIN product scraper on Apify is built exactly this way: apify.com/native_emblem/shein-product-scraper. And the broader platform version of this lives at cartpie.com, an e-commerce product-data scraping platform with a free tier live now.

We cut a Lambda bill 96.5% by fixing a retry storm.

Andrii Votiakov — Mon, 20 Jul 2026 16:51:44 +0000

The invocation graph is the first thing I open on any serverless engagement, and this one looked wrong from across the room. 29.3K Lambda invocations per hour. For a workload that justified a few hundred.

Nobody had noticed, because nothing was down. That's the ugly property of a retry storm. The system stays green while it burns money. Every retry eventually succeeds, lands in a dead-letter queue, or quietly expires. Dashboards show traffic, and the only symptom is a bill that grows faster than the product does.

What a retry storm actually is

AWS retries on your behalf in more places than most teams can list. Async Lambda invocations retry twice by default. An SQS event source leaves failed messages on the queue, and after the visibility timeout they become visible again and are redelivered. EventBridge has its own retry policy. Step Functions too. Each layer is individually reasonable. Stacked together, they multiply.

The failure mode that bit this client was surprisingly simple. The handler wrapped everything in try/catch, logged the error, and re-threw. Everything. Including validation errors on malformed messages. A malformed message can never succeed, but the code told SQS, "please try again."

Without a redrive policy, SQS keeps redelivering a failing message until the retention period expires. The default retention period is four days. One poison message became four days of Lambda invocations. There were thousands of poison messages, and the function that consumed their output failed too, creating its own retry storm downstream.

The fix

Four changes, in order of impact.

1. Classify errors before throwing

Retryable and terminal errors are different animals. A network timeout deserves another attempt. A schema violation belongs in a parking lot, not in an infinite retry loop.

export const handler = async (event) => {
  const batchItemFailures = [];

  for (const record of event.Records) {
    try {
      await process(JSON.parse(record.body));
    } catch (err) {
      if (err instanceof ValidationError) {
        await parkMessage(record, err);
        continue;
      }

      batchItemFailures.push({
        itemIdentifier: record.messageId,
      });
    }
  }

  return { batchItemFailures };
};

This uses SQS partial batch responses, so only the failed messages are retried. Without this feature, one failed record can cause the entire batch to be retried, including messages that were already processed successfully.

2. Configure a dead-letter queue with a sensible `maxReceiveCount`

Three attempts, then move the message to a DLQ and alert on queue depth.

{
  "RedrivePolicy": {
    "deadLetterTargetArn": "arn:aws:sqs:eu-west-1:123456789:orders-dlq",
    "maxReceiveCount": 3
  }
}

The DLQ is not a trash can. It is a work queue for humans. Someone should periodically inspect it, fix the underlying issue, and decide whether those messages should be replayed.

3. Add a circuit breaker around flaky downstream services

When a dependency is down, ten thousand Lambda invocations hammering it will not make it recover any faster. Fail fast, back off, and let SQS absorb the burst.

One caveat: circuit breakers are awkward in Lambda because execution environments are ephemeral. The breaker state has to live somewhere external. We stored ours in a DynamoDB item with a TTL. It worked well enough, even if it was not the prettiest implementation.

4. Tune timeouts to reality

The function timeout was set close to the Lambda maximum "to be safe." Safe for whom?

Long timeouts increase the maximum cost of every failed invocation and delay retries. We set the timeout to our p99 execution time plus some headroom. That reduced the cost of failures while still giving healthy requests enough time to complete.

The result

Lambda invocations dropped from 29.3K per hour to 669. That's a 97.7% reduction. The Lambda line item fell by 96.5%.

Same product. Same traffic. Same features.

The only difference was that we stopped paying to retry work that could never succeed.

Don't forget idempotency

Retries only work safely if your processing is idempotent.

If processing the same order twice sends two emails, ships two packages, or charges the customer twice, retries become a business problem instead of an infrastructure problem.

Whenever you rely on retries, make sure your handlers can safely process the same message multiple times.

How to catch this in your own account

Spend ten minutes checking these today.

Track the ratio of Errors / Invocations in CloudWatch. Alert when the error rate is consistently above your normal baseline.
For every SQS-triggered Lambda, verify the queue has a redrive policy. Without a DLQ, poison messages keep cycling until the retention period expires.
Compare Lambda invocations with real business events. If you process 10,000 orders per day but see 300,000 invocations, retries are probably inflating the numbers.
Review MaximumRetryAttempts for asynchronous Lambda invocations. The default is two retries, but for some idempotency-sensitive workloads you may want zero.

Retries are a good default for a world where failures are often transient. They are a terrible default for deterministic failures. Your error handling is what tells AWS which kind it is.

If your AWS bill feels wrong and you'd rather not gamble on another generic cloud consultancy, I do this at reducecost.cloud on a pay-for-savings basis. Zero upfront. You only pay a share of what I actually save you.

Stop writing a parser per site. Run five and let confidence decide.

Andrii Votiakov — Mon, 20 Jul 2026 09:40:12 +0000

For a long time I ran product extraction off a database of custom selector configs. Hundreds of retailers, each with its own set of CSS selectors mapped field by field: price here, title there, image over there. It worked. It was also a treadmill. Every retailer that redesigned its frontend silently broke its config, and the maintenance tax grew with every retailer I added. Hundreds of configs is hundreds of things that can rot without telling you, usually right before someone downstream asks why a brand's prices went null.

At some point I stopped feeding it. The scraping platform I ran at my last engagement fed a 20M+ product catalogue at millions of pages a day, and the thing that made that survivable wasn't better per-site config. It was leaning on almost no per-site config at all.

The pattern

Instead of one careful parser per site, run several cheap generic extractors on every page, in parallel:

JSON-LD. A huge share of e-commerce pages ship a schema.org/Product block because Google rewards it. It has name, price, currency, availability, images. It's the closest thing to a public API hiding in the HTML.
OpenGraph tags. og:title, og:image, product:price:amount. Lower quality than JSON-LD, but present on sites too lazy for structured data, because everyone wants pretty link previews.
Microdata / RDFa. Older sites, still surprisingly common outside the US.
Embedded state. __NEXT_DATA__, window.__INITIAL_STATE__ and friends. A JSON blob the site's own frontend hydrates from.
Heuristics. A price-shaped string near a currency symbol inside the main content region. The largest above-the-fold image. The h1. Dumb, and dumb works more often than you'd think.

None of these is reliable alone. OG price tags go stale, JSON-LD sometimes describes the wrong variant, heuristics grab the crossed-out "was" price. The trick is you don't pick an extractor. You pick per field, and you let agreement between sources carry the decision.

Confidence merge

Each extractor emits candidates with a prior based on how trustworthy that source usually is for that field. Then candidates that agree with each other get boosted, and the top score wins per field:

function merge(results, fields) {
  const out = {};
  for (const field of fields) {
    const candidates = results
      .filter((r) => r.data[field] != null)
      .map((r) => ({
        value: r.data[field],
        source: r.name,
        score: PRIORS[r.name][field] ?? 0.3,
      }));

    for (const c of candidates) {
      const agreeing = candidates.filter(
        (o) => o !== c && normalize(o.value) === normalize(c.value)
      ).length;
      c.score *= 1 + 0.4 * agreeing;
    }

    candidates.sort((a, b) => b.score - a.score);
    out[field] = candidates[0] ?? null;
  }
  return out;
}

normalize matters more than the scoring. "1 299,00 zł" and "1299.00 PLN" have to compare equal or your agreement boost never fires. I spent more time on value normalization than on everything else in this system combined.

Add a validation pass after the merge: does the price parse, is the currency a real ISO code, is the title longer than three characters. A merged record that fails validation is a signal, not a result. Log it, sample those pages weekly, and you'll find out which extractor is drifting before your customers do.

Why bother

Because the long tail is where per-site config dies. The top few hundred retailers can justify hand-tuned selectors, and I maintained exactly that for years. Site number 4,000 does not, and with an ensemble it doesn't need any. A new domain comes in, four of five extractors fire, three agree on price, and you have a usable record with zero engineering hours spent. Across that platform we held roughly 95% extraction success at about 3 seconds a page, and most of the domains involved had nobody who had ever looked at their HTML.

The honest limitation: variant-heavy product pages are still rough. When one URL carries twelve colorways at different prices, the ensemble happily returns a confident answer to the wrong question, because every source on the page describes the default variant. I handle that with a separate variant expansion step, and it's the least elegant code I own.

But for "give me name, price, image, availability from an arbitrary shop URL," the ensemble beat every hand-written parser I've replaced with it. Write extractors for formats, not for websites. Formats change maybe once a decade. Websites change on Tuesdays.

This approach is the core of what I'm building at Cartpie, an e-commerce product-data scraping platform, free tier live at cartpie.com if you want to throw some long-tail URLs at it.

Most company job boards are just a public JSON API you can GET

Andrii Votiakov — Sun, 19 Jul 2026 16:24:37 +0000

I spent an afternoon trying to scrape a careers page with a headless browser before I noticed the page itself was calling a JSON endpoint. The company runs Greenhouse. Greenhouse serves the whole board — every open role, full descriptions — over one unauthenticated GET. No Playwright, no proxy, no waiting for React to hydrate. I deleted the browser code and never looked back.

That's the thing nobody tells you when you're building a job aggregator or a "who's hiring" tracker. Stripe, Spotify, Ramp — thousands of companies post through four ATS platforms, and each one exposes a per-company feed that's already public. You just have to know the URL shape.

Here are the four I use, with real slugs so you can curl along.

The endpoints

Every platform keys off a slug — the token buried in the careers URL.

Greenhouse

https://boards-api.greenhouse.io/v1/boards/{slug}/jobs?content=true

?content=true gives you the full HTML description inline. Hit stripe and you get 500+ jobs in a single response — no pagination at all. The slug is whatever sits in boards.greenhouse.io/stripe.

const res = await gotScraping({
    url: 'https://boards-api.greenhouse.io/v1/boards/stripe/jobs?content=true',
    responseType: 'json',
});
// res.body.jobs -> [{ id, title, location: {name}, content, absolute_url, departments, offices, first_published, updated_at }]

One trap cost me twenty minutes: Greenhouse double-encodes the content field. You get <p> where you expected <p>. Run it through an HTML-entity decode once before you touch it, or you'll render literal tags in your UI and blame your own template.

Lever

https://api.lever.co/v0/postings/{slug}?mode=json

Try spotify or ro. You get a flat array back, and the useful fields (team, location, commitment) hide one level down under categories:

// [{ id, text, categories: { team, location, commitment }, description, descriptionPlain, hostedUrl, applyUrl, createdAt }]

A 404 here is not an error. It just means the company isn't on Lever, which is exactly what you want to hear when you're guessing which platform a slug belongs to.

Ashby

https://api.ashbyhq.com/posting-api/job-board/{slug}?includeCompensation=true

Try ramp. Ashby is the one I wish everyone used. Structured location, secondaryLocations, a real isRemote boolean, employmentType, and both descriptionHtml and descriptionPlain so you don't have to strip tags yourself. When the company fills in comp bands, you get those too.

SmartRecruiters

https://api.smartrecruiters.com/v1/companies/{slug}/postings?limit=100&offset=0

Try Visa. This is the odd one out — it paginates. You get a totalFound and you loop limit/offset until you've pulled everything. And here's the catch that bit me: the list endpoint returns metadata only. No description body. If you want the actual job text you have to call a separate per-posting detail endpoint, one request each. So the "free and fast" story falls apart a bit for SmartRecruiters — you're paying N+1 requests for full text.

Turning four shapes into one

The endpoints are the easy part. The value is a single record shape so the rest of your code never has to know or care which ATS a company runs. A thin per-provider normalizer does it:

function normalizeGreenhouse(job) {
    return {
        source: 'greenhouse',
        id: String(job.id),
        title: job.title,
        location: job.location?.name ?? null,
        department: (job.departments ?? []).map((d) => d.name).join(', ') || null,
        remote: /remote/i.test(job.location?.name ?? ''),
        url: job.absolute_url,
        descriptionHtml: decodeEntities(job.content),
        postedAt: job.first_published ?? null,
    };
}

function normalizeLever(job) {
    const c = job.categories ?? {};
    return {
        source: 'lever',
        id: String(job.id),
        title: job.text,
        location: c.location ?? null,
        department: c.team ?? c.department ?? null,
        remote: /remote/i.test(job.workplaceType ?? c.location ?? ''),
        url: job.hostedUrl,
        descriptionHtml: job.description,
        postedAt: job.createdAt ? new Date(job.createdAt).toISOString() : null,
    };
}
// ...ashby, smartRecruiters similarly

Notice remote is a regex guess on Greenhouse and Lever because neither gives you a clean flag. Ashby actually hands you isRemote, so use it there instead of guessing. That inconsistency is the whole reason the normalizer exists.

Guessing the platform when you only have a name

Say a user types "Ramp" and you have no idea which ATS they're on. You could look it up. Or you could throw the slug at all four at once and keep whatever answers:

async function findJobs(slug) {
    const results = await Promise.allSettled([
        fetchGreenhouse(slug),
        fetchLever(slug),
        fetchAshby(slug),
        fetchSmartRecruiters(slug),
    ]);
    return results
        .filter((r) => r.status === 'fulfilled' && r.value.length)
        .flatMap((r) => r.value);
}

The 404s from the three wrong platforms come back in a couple hundred milliseconds and cost nothing. In a real build you'd resolve the actual slug by parsing the careers-page embed — the ATS board is almost always iframed or linked from there — but bare-slug guessing covers more companies than you'd expect: company names and slugs match more often than not.

1,199 roles in one pass

To sanity-check that this actually holds up, I ran it across 8 real companies spanning all four platforms and pulled 1,199 open roles in a single sweep. Proxyless, roughly 200ms per company on the single-request platforms. The stable part is what sells it: these are the same feeds the companies' own job widgets call, so a marketing redesign doesn't touch them. Scrape the rendered page and you're one CSS class rename away from a broken parser.

Where it stops working

Not every ATS plays along, and I'd rather tell you now than have you burn an afternoon.

SmartRecruiters' list feed has no descriptions. Covered above — plan for the N+1 or ship jobs without body text. And log() the pagination totals loudly: a truncated limit/offset loop will quietly return page one and let you think you got the whole board.
Workable is auth-gated now. The old open feed wants a token. Same story with Recruitee. A year ago both were open; that window closed.

So this isn't a universal skeleton key. It's four platforms that happen to still be open, and they happen to cover a large slice of tech hiring. Good enough that I stopped writing careers-page scrapers entirely.

The ready-made version

I packed all four platforms, the unified schema, and the auto-detect into an Apify actor — ATS Job Feed Scraper. But the endpoints above really are most of it. Build your own; it genuinely is a fun afternoon, and you'll understand every field.

If you know another ATS that still serves an open feed, drop it in the comments and I'll add notes. I'm especially curious whether anyone has a live Workable path that doesn't need a token.

DEV Community: Andrii Votiakov

When you actually need a headless browser (and when you're just wasting RAM)

How to tell a browser is actually required

The browser has exactly one job

Now the plain string work

Why this generalizes

Your scraper isn't broken. The site changed, and it didn't tell you.

Exit 0 is not the same as correct

Validate the output, not the input

A static threshold is a starting point, not the answer

Don't let one dead selector take the record down

Your BigQuery bill is a query-shape problem. We cut one 72% in three weeks.

Week 1: find out where the money goes

Week 2: fix the shape

Week 3: kill the zombie jobs

What I'd tell you to skip

Build a SHEIN price tracker in 20 minutes with n8n and an Apify actor

1. Schedule trigger

2. Run the actor and get results in one shot

3. Parse the products

4. Diff against last run

5. Notify on a drop only

That's the whole thing

Half the web already hands you clean product data. You just have to ask.

Retailers publish machine-readable product data for search engines

Reading a JSON-LD Product block

Rank your sources by reliability

The catch: structured data lies sometimes

My pipeline parsed every page twice, and I blamed the parser

The system

The symptom

The wrong week

The read

Why nobody saw it

The fix

The receipts

What actually changed

Scraping millions of pages a day: what actually breaks

The queue with no manners

Retries that stampede

Dedupe before fetch, not after

Parser drift, the quiet one

The CPU step hiding in an I/O system

You will need to debug the past

The uncomfortable summary

Anti-bot without melting your budget: the proxy waterfall.

The waterfall

Failure is not just a status code

Remember what worked

Your scraper probably doesn't need a browser. It needs the right request.

The data is already JSON, somewhere

How to actually find it

The parts I won't hand you

We cut a Lambda bill 96.5% by fixing a retry storm.

What a retry storm actually is

The fix

1. Classify errors before throwing

2. Configure a dead-letter queue with a sensible maxReceiveCount

3. Add a circuit breaker around flaky downstream services

4. Tune timeouts to reality

The result

Don't forget idempotency

How to catch this in your own account

Stop writing a parser per site. Run five and let confidence decide.

The pattern

Confidence merge

Why bother

Most company job boards are just a public JSON API you can GET

The endpoints

Greenhouse

Lever

Ashby

SmartRecruiters

Turning four shapes into one

Guessing the platform when you only have a name

1,199 roles in one pass

Where it stops working

The ready-made version

2. Configure a dead-letter queue with a sensible `maxReceiveCount`