DEV Community: Anakin

Scraping API pricing: read the defaults, not just the headline rate

Anakin — Wed, 28 Oct 2026 10:00:03 +0000

A scraping API can look cheap in the pricing table and still cost several times more in production. The usual reason is not the base price. It is the default request mode: JavaScript rendering, residential proxies, geo targeting, retries, or parsing options that change how many credits one URL actually consumes.

If you only compare “requests per dollar,” you can miss the part that matters: what the API does when you call it with the simplest possible integration.

The default request is the real unit price

Most teams start with the shortest working request. Something like this:

curl "https://api.example-scraper.com/scrape?api_key=$API_KEY&url=https://example.com"

That request might mean “fetch plain HTML.” It might also mean “start a browser, render JavaScript, route through a premium proxy pool, and return parsed output.” Those are very different cost profiles.

Scrapingdog is a useful concrete example. Its pricing is low on paper, but its JavaScript rendering option is on by default according to its docs. A default request costs 5 credits. If you set dynamic=false, the same plain HTML request drops to 1 credit.

# Default: dynamic rendering enabled, billed as 5 credits
curl "https://api.scrapingdog.com/scrape?api_key=$API_KEY&url=https://example.com"

# Plain HTML: billed as 1 credit
curl "https://api.scrapingdog.com/scrape?api_key=$API_KEY&url=https://example.com&dynamic=false"

That does not make the product bad. It means the integration has a sharp edge. If your workload is mostly static pages and you remember to disable rendering, the low price can be real. If you forget, your credit burn is 5x before you have scraped anything interesting.

This is the first thing I check with any scraping API:

const urls = 100_000;

const headlineCreditsPerUrl = 1;
const defaultCreditsPerUrl = 5;

console.log({
  headlineCredits: urls * headlineCreditsPerUrl,
  defaultCredits: urls * defaultCreditsPerUrl,
  multiplier: defaultCreditsPerUrl / headlineCreditsPerUrl
});

// { headlineCredits: 100000, defaultCredits: 500000, multiplier: 5 }

A pricing page tells you the best case. Your first curl command tells you the likely case.

Rendering is not optional for many modern pages

Turning JavaScript off saves money, but it can also silently break extraction.

You usually notice it as missing data, not as a clean error. The HTTP request succeeds. The API returns HTML. Your parser runs. Then the fields are empty because the page shipped a shell like this:

<div id="root"></div>
<script src="/assets/app.js"></script>

For pages like that, dynamic=false gives you a successful response with the wrong content. That is worse than a failure in some pipelines because it can pollute downstream data.

A quick test is to compare the rendered and non-rendered response for the fields you need:

function hasProductPrice(html) {
  return /class="price"|data-testid="price"|itemprop="price"/.test(html);
}

const staticHtml = await fetchStatic(url);
const renderedHtml = await fetchRendered(url);

console.log({
  staticHasPrice: hasProductPrice(staticHtml),
  renderedHasPrice: hasProductPrice(renderedHtml)
});

If rendered output is required for most targets, treat rendering as part of the base request when you estimate cost. Some providers charge the same for rendered and non-rendered requests. Anakin’s URL scraper, for example, lists JavaScript rendering at the same 1-credit cost as a basic scrape. Others charge a multiplier.

Neither model is universally better. The cheaper static option works well when you control the target list and know the pages do not need a browser. Flat pricing is easier to budget when your target mix changes or product pages vary by region, device, or A/B test.

Proxy tiers can change the math again

The next multiplier is usually proxy type.

Datacenter proxies are cheaper, but some sites block them quickly. Residential proxies cost more, but they often get through sites that reject obvious automation. In Scrapingdog’s public docs, premium residential proxies cost 10 credits, or 25 credits when combined with rendering.

That matters because hard pages tend to need both. A site that blocks basic requests may also require JavaScript rendering. Your “1 credit per URL” estimate can become 25 credits per URL for the exact pages that your business actually cares about.

You can model this before committing:

const workload = {
  staticPages: 60_000,
  renderedPages: 30_000,
  premiumRenderedPages: 10_000
};

const credits =
  workload.staticPages * 1 +
  workload.renderedPages * 5 +
  workload.premiumRenderedPages * 25;

console.log(credits); // 460000
console.log(credits / 100_000); // 4.6 average credits per URL

The average request cost is what hits your bill, not the cheapest row in the table.

Failures, caching, and batches are billing features too

Two APIs can charge the same per successful scrape but behave differently around repeated work.

Check these details:

Do failed requests cost credits?
What counts as a failure: timeout, 403, CAPTCHA page, empty extraction?
Does the API cache recent fetches?
Can you submit batches, or do you pay request overhead one URL at a time?
Does async mode return a job ID you can retry safely?

A typical async scraping flow looks like this:

# Submit work
curl -X POST "https://api.example.com/scrape" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com","render":true}'

# Response
# { "jobId": "job_abc123" }

# Poll result
curl "https://api.example.com/scrape/jobs/job_abc123" \
  -H "Authorization: Bearer $API_KEY"

# { "status": "completed", "html": "..." }

This pattern is easier to operate than long blocking requests, but you still need idempotency. If your worker crashes after submitting the job and before storing the jobId, you may submit the same URL again. A 24-hour cache or your own URL/job table can prevent duplicate spend.

Authenticated scraping needs a credential plan

Cookie headers work until they do not. Sessions expire. Sites rotate CSRF tokens. Some flows require 2FA. If developers paste cookies into environment variables, those cookies eventually end up in logs, crash dumps, or shared config.

For authenticated extraction, decide where credentials live before you build the scraper. A minimal internal table might look like this:

create table scrape_identity (
  id uuid primary key,
  site text not null,
  username text not null,
  encrypted_secret bytea not null,
  created_at timestamptz not null default now(),
  rotated_at timestamptz
);

You also need rules for who can read, rotate, and use those identities. The scraper should receive short-lived session material, not raw passwords, when possible.

If you need managed identities for authenticated web actions rather than passing cookies through request headers, Wire treats credentials as named identities backed by a vault instead of returning secrets from an API.

That is not necessary for every scraper. Public pages do not need it. Small internal jobs may be fine with a locked-down secret manager and careful logging. But once multiple services or teams use the same authenticated targets, credential handling becomes part of the scraping architecture, not a detail.

A practical comparison checklist

Before choosing a scraping API, run this against your own URLs:

Pick 100 representative pages, including the annoying ones.
Fetch each page with rendering off and rendering on.
Validate the actual fields you need, not just HTTP 200.
Record credits used per mode.
Repeat a subset to see whether caching changes cost.
Test blocked pages and confirm whether they are billed.
Test one authenticated flow if your product needs it.

Put the result in a small table with url, mode, status, fields_found, credits_used, and error. That table will tell you more than any pricing page.

The next useful step: take your current scraper target list, sample 50 URLs, and calculate the average credits per valid extraction rather than the advertised credits per request.

How to evaluate scraping APIs before your crawler budget surprises you

Anakin — Tue, 27 Oct 2026 10:00:01 +0000

You usually do not learn whether a scraping API fits your workload from one successful request. You learn it after the 300th URL, when some pages need JavaScript, some return a CAPTCHA, some time out, and your cost model suddenly depends on details you did not measure up front.

Most scraping APIs can fetch a product page, job listing, or article. The harder questions are more operational: what does this call cost, what happens when it fails, how do you crawl more than one page, and does the workflow stop at reading HTML?

Price per request is not the same as price per workload

A lot of scraping pricing looks simple until you run mixed targets. Plain HTTP requests might be cheap. Browser-rendered requests cost more. Heavily protected sites may land in a higher tier. Some providers classify the request for you, which is convenient, but it also means you may not know the final cost until you test the actual URL.

That is not automatically bad. Zyte, for example, has a long track record in scraping infrastructure and Scrapy hosting. If your workload hits a known set of sites and you can run those URLs through an estimator first, tiered pricing can work fine.

The problem shows up in pipelines that discover URLs dynamically. A research agent, marketplace monitor, or lead enrichment job may hit hundreds of domains that nobody priced ahead of time. In that case, “per 1,000 requests” is not enough information. You need to know which requests were rendered, which tier they landed in, and whether failures were billed.

A simple evaluation harness helps more than a vendor comparison table:

import csv
import time
import requests

API_URL = "https://example-scraping-api.test/scrape"
API_KEY = "replace-me"

urls = [
    "https://example.com/product/123",
    "https://example.com/category/widgets",
    "https://example.org/jobs/backend-engineer",
]

with open("scrape_eval.csv", "w", newline="") as f:
    writer = csv.DictWriter(f, fieldnames=[
        "url", "status", "http_status", "duration_ms", "bytes", "error"
    ])
    writer.writeheader()

    for url in urls:
        started = time.time()
        try:
            r = requests.post(
                API_URL,
                headers={"Authorization": f"Bearer {API_KEY}"},
                json={"url": url, "render": "auto"},
                timeout=60,
            )
            duration_ms = int((time.time() - started) * 1000)
            body = r.text or ""

            writer.writerow({
                "url": url,
                "status": "ok" if r.ok else "failed",
                "http_status": r.status_code,
                "duration_ms": duration_ms,
                "bytes": len(body.encode("utf-8")),
                "error": "" if r.ok else body[:300],
            })
        except requests.Timeout:
            writer.writerow({
                "url": url,
                "status": "failed",
                "http_status": "",
                "duration_ms": 60000,
                "bytes": 0,
                "error": "client timeout after 60s",
            })

Run this against a representative sample, not three friendly URLs. Include pages behind Cloudflare, pages that require rendering, category pages with pagination, and pages you expect to fail. The output tells you what you actually need to price.

Track failures as first-class results

A failed scrape is not one thing. It can mean:

403 Forbidden: the target blocked the request.
429 Too Many Requests: your rate limit or proxy rotation failed.
Timeout: the page never completed or the browser hung.
Empty content: the HTML loaded, but the data came from an API you did not capture.
Selector miss: the page changed and your parser returned null.

These failures affect architecture. If the provider bills failed attempts, retries can double your cost. If failures return a generic error, you cannot decide whether to retry, lower concurrency, switch countries, or mark the URL as permanently unavailable.

Flat request pricing, like Anakin.io uses for URL scraping and crawling, makes this part easier to model because the unit cost is known before the job runs. That does not remove the need to measure success rate, but it separates scrape reliability from billing surprises.

Decide whether you need a crawler or just a fetcher

Fetching one URL is the smallest part of many scraping systems. Real jobs often start with “get every product page in this category” or “find all job listings under this company domain.”

If your API only fetches URLs, you still need to build URL discovery:

seed URL
  -> fetch sitemap.xml
  -> parse nested sitemaps
  -> filter URLs by pattern
  -> fetch category pages
  -> extract links
  -> dedupe
  -> enqueue scrape jobs
  -> retry failures

That is not hard once. It becomes annoying when every site has different pagination, canonical URLs, locale paths, and duplicate query parameters.

Scrapy is a good fit when you want full control and your team is comfortable maintaining spiders. Scrapy Cloud, Zyte’s hosted Scrapy platform, makes sense for teams that already have that code.

An API-level crawl endpoint is better when the crawl rules are simple: include /products/, exclude /cart, cap at 100 pages, return page content for each match. You give up some control, but you avoid hosting and deploying spider code for routine jobs.

Reading pages and taking actions are different products

Scraping tools mostly read public pages. Authenticated workflows are a different category.

For example, this is not just scraping:

log in
  -> open billing settings
  -> download latest invoice
  -> update account metadata
  -> submit form
  -> confirm success

A browser automation script can do it, but the failure modes change. Sessions expire. Login challenges appear. Buttons move. A slow page turns a 500 ms request into a 20 second browser run. If you need to perform actions across many accounts or sites, you should evaluate that separately from HTML extraction.

Wire is an example of exposing authenticated site actions as predefined operations, rather than asking every developer to maintain their own browser script for login, navigation, and submission.

That distinction matters when you compare tools. A read-only scraping API can be excellent at unblocking pages and still be the wrong fit for a workflow that eventually needs to click, submit, or update something.

A practical evaluation checklist

Before committing to a scraping provider, build a small test set and record these fields:

URL
target domain
expected data type
needs JavaScript or not
response status
provider error code
duration
extracted byte size
structured extraction success
billed units or estimated cost
retry count

Then group results by domain and page type. One hard site can dominate your cost and error budget. Average success rate hides that.

If pricing depends on tiers, run your real URLs through the estimator and store the returned class next to the test result. If pricing is flat, still measure failures and latency. Predictable cost does not guarantee usable data.

The next useful step is to take 50 to 200 real URLs from your backlog, run them through your top two options, and compare the CSV output instead of the marketing pages. You will learn more from the failed rows than the successful ones.

Choosing a structured extraction API without getting trapped by the JSON

Anakin — Mon, 26 Oct 2026 10:00:01 +0000

You point an extraction API at a product page and get clean JSON back. Great. Then the sixth site renders price in JavaScript, the seventh changes content by country, the eighth needs a login, and the ninth returns a valid response with the wrong fields. The hard part is not getting JSON from HTML once. The hard part is choosing an extraction model that keeps working when pages stop looking like your demo.

Start with the schema question

Most structured extraction tools fall into one of two buckets.

The first bucket gives you a fixed ontology. You send a URL, the system decides whether it is an article, product, person, organization, or something else, then returns fields from its own model. Diffbot is the clean example here. Its extraction APIs and Knowledge Graph are useful when your project fits those entity types and you want normalized public-web data.

The second bucket lets you define the fields you want per job. Anakin.io is an example of this approach: you scrape a URL and ask for structured JSON fields as part of that request.

Neither model is universally better. A fixed ontology helps when you want consistency across millions of pages. A custom schema helps when your data is specific:

{
  "productName": "string",
  "price": "number",
  "currency": "string",
  "shippingWindow": "string",
  "isMarketplaceSeller": "boolean",
  "warrantyText": "string"
}

That last field, warrantyText, is where fixed schemas often get awkward. If the extractor does not model the field, you either post-process raw text, store vendor-specific extensions, or accept missing data. Custom extraction flips the problem around: you get the field names you asked for, but you now own validation and quality checks.

Fetch behavior is part of extraction

Developers often evaluate extraction APIs by the response body. That misses half the integration.

Before an extractor can parse anything, it has to fetch the page correctly. That means handling JavaScript rendering, redirects, bot checks, cookies, language headers, and geography. A page from example-store.com might show one price from a US IP, another from Germany, and a cookie banner instead of content from a new session.

If the API gives you no control over those inputs, extraction quality becomes hard to debug. You see price: null, but the real failure happened earlier: the fetch got an interstitial, a login page, or a region-specific variant.

For authenticated scraping, Wire models login state as named identities with vault-backed credentials rather than making each scraper replay expiring cookies.

At minimum, check whether your provider exposes these fetch controls:

JavaScript rendering on or off
country or region selection
custom headers
cookie handling
proxy behavior
raw HTML or screenshot access for debugging

If you cannot inspect what the extractor actually saw, you will spend time guessing.

Watch how pricing behaves on failures

Monthly pricing looks simple until you add retries, failed fetches, and test runs. Diffbot's paid plans start at a higher monthly floor, which makes sense for teams buying into a Knowledge Graph or web-scale entity extraction. It makes less sense if you need structured data from a few hundred pages a day.

Per-call pricing has a different tradeoff. It usually starts cheaper and maps better to variable workloads, but you need to know exactly what counts as a billable request. Ask these questions before wiring it into a queue:

Does a failed fetch consume credits?
Does a timeout consume credits?
Does JavaScript rendering cost extra?
Does structured extraction cost extra?
Are cached responses free or discounted?
Are retries automatic, and are they billed separately?

A cheap request that fails 20 percent of the time may cost more than it looks. A more expensive request that only bills successful jobs may be easier to reason about. The only useful number is cost per valid row in your database.

Treat extraction as an async job

Many extraction APIs return a job ID first, then require polling. That is a good shape for real workloads because rendering and parsing can take longer than a normal HTTP request timeout.

Here is the pattern I usually want in application code: submit, poll with a deadline, validate the result, then store only typed data.

import { z } from "zod";

const Product = z.object({
  productName: z.string().min(1),
  price: z.number().nonnegative(),
  currency: z.string().length(3),
  shippingWindow: z.string().nullable(),
  isMarketplaceSeller: z.boolean().nullable(),
  warrantyText: z.string().nullable()
});

async function waitForExtraction(jobId, apiKey) {
  const deadline = Date.now() + 60_000;

  while (Date.now() < deadline) {
    const res = await fetch(`https://api.example.com/extract/${jobId}`, {
      headers: { Authorization: `Bearer ${apiKey}` }
    });

    if (!res.ok) {
      throw new Error(`poll failed: ${res.status} ${await res.text()}`);
    }

    const body = await res.json();

    if (body.status === "failed") {
      throw new Error(`extraction failed: ${body.error ?? "unknown error"}`);
    }

    if (body.status === "completed") {
      return Product.parse(body.data);
    }

    await new Promise(resolve => setTimeout(resolve, 1500));
  }

  throw new Error(`extraction timed out for job ${jobId}`);
}

The important part is not the endpoint name. It is the boundary. Do not let loosely structured AI output flow straight into your database. Validate it. Log the raw response somewhere short-lived. Track failures by domain so you can see whether one site is drifting.

Common failure modes are easy to miss:

HTTP 200 with extracted fields set to null
price returned as "$19.99" when your code expects a number
currency inferred incorrectly from locale
stale cached content after a product update
login page extracted as if it were the target page

Schema validation catches some of this. Sampling catches the rest.

When a knowledge graph is the right tool

A Knowledge Graph is not just a scraper with a bigger database. If you need entity resolution, public-web facts, company profiles, people, relationships, and query language support across billions of entities, a graph product can save years of work.

That is the case where Diffbot makes sense. You are not paying just to parse one page. You are paying for a maintained model of the public web.

If your job is narrower, such as pulling custom fields from specific URLs, monitoring a set of pages, or feeding an internal enrichment pipeline, a page-level extraction API may fit better. You keep control over the schema and avoid buying infrastructure you do not use.

A practical evaluation checklist

Before choosing a provider, run the same test set through each option. Use at least:

10 normal pages
5 JavaScript-heavy pages
5 pages with regional differences
5 pages with missing or weird fields
3 pages that should fail

Measure valid rows, not successful HTTP responses. Record fetch status, extraction status, validation errors, latency, and billed units. Then calculate cost per valid row.

That small test will tell you more than a feature table. It will also show whether you need a fixed ontology, custom fields, better fetch control, or a full Knowledge Graph.

Scrape E-commerce Sites at the Lowest Layer That Works

Anakin — Fri, 23 Oct 2026 10:00:01 +0000

You point a scraper at a product page, get a 200 response, and still extract nothing useful. The HTML has a root div, a few script tags, maybe a loading skeleton. Prices, variants, reviews, stock status, and seller data arrive later through JavaScript. If you retry with a headless browser, you may get the data, but you also inherit browser fingerprints, CAPTCHA risk, slower jobs, and more moving parts.

The useful question is not “which scraper should I use?” It is “which layer has the data?”

Start below the browser

Most e-commerce pages are not built from static HTML anymore. The frontend renders a page shell, then calls backend endpoints for search results, product details, reviews, availability, recommendations, and pricing.

Open DevTools, go to the Network tab, filter by Fetch/XHR, then reload a product page. You will usually see requests returning JSON. That is the first place to look.

A rough workflow:

Load the page manually in a browser.
Filter network requests to XHR or Fetch.
Search response bodies for a known product title, SKU, price, or review text.
Copy the request as cURL.
Remove headers one by one until you find the minimum request that still works.

For some sites, the request works directly:

curl 'https://example.com/api/products/12345' \
  -H 'accept: application/json' \
  -H 'user-agent: Mozilla/5.0'

For others, it fails with a 403, returns { "error": "Unauthorized" }, or sends an empty result unless you include a session cookie, geo header, store ID, or CSRF token. That tells you the browser is doing more than displaying the page. It is creating state that the API expects.

For marketplaces where those internal endpoints change often, Wire exposes catalogued network-layer actions for product search, product detail, reviews, and seller data without asking you to maintain DOM selectors.

Shopify is the easy case people forget

Before launching Playwright at a Shopify store, check /products.json.

Many Shopify stores expose their catalog as paginated JSON:

curl 'https://storename.myshopify.com/products.json?limit=250'

The response includes product titles, handles, variants, prices, image URLs, and inventory policy fields. You can paginate with page or cursor-style parameters depending on the storefront configuration.

A simple collector might look like this:

const base = 'https://storename.myshopify.com/products.json?limit=250';

async function fetchAllProducts() {
  let page = 1;
  const products = [];

  while (true) {
    const res = await fetch(`${base}&page=${page}`);

    if (!res.ok) {
      throw new Error(`Shopify request failed: ${res.status}`);
    }

    const data = await res.json();

    if (!data.products || data.products.length === 0) break;

    products.push(...data.products);
    page += 1;
  }

  return products;
}

This is faster and less fragile than parsing collection pages. It also avoids a common failure mode: your CSS selector works today, then the theme changes and your scraper silently starts returning null prices.

There are limits. Some stores block /products.json with Cloudflare. Some hide inventory quantities. Some large catalogs do not expose everything through the public endpoint. But you should still test this path first because it costs almost nothing to verify.

Use rendering when data exists only after JavaScript runs

Some sites do not expose a convenient public JSON endpoint, or they require JavaScript to build tokens before calling internal APIs. In that case, render the page and extract from the DOM or captured responses.

Playwright gives you both options:

import { chromium } from 'playwright';

const browser = await chromium.launch();
const page = await browser.newPage();

page.on('response', async (response) => {
  const url = response.url();

  if (url.includes('/api/') && response.headers()['content-type']?.includes('application/json')) {
    try {
      const json = await response.json();
      console.log('API response:', url, json);
    } catch {
      // Not all JSON-looking responses parse cleanly.
    }
  }
});

await page.goto('https://example.com/product/12345', {
  waitUntil: 'networkidle'
});

const price = await page.locator('[data-testid="price"]').textContent();
console.log({ price });

await browser.close();

This works, but it is slower. A direct JSON request might finish in a few hundred milliseconds. A browser render often takes several seconds, and at volume you must manage memory, retries, proxy routing, timeouts, and browser crashes.

Typical symptoms at this layer:

TimeoutError: page.goto: Timeout 30000ms exceeded
CAPTCHA pages returned with status 200
product grids that render only the first page because scrolling was not simulated
prices changing because the site inferred the wrong region
selectors returning null after a frontend deploy

Rendering is a fallback, not a default.

Use a full browser only for interactive state

Some workflows genuinely need a browser:

login-protected prices
wishlists
cart totals
shipping estimates
loyalty pricing
checkout availability
store-specific inventory after selecting a location

Trying to fake these with raw HTTP can turn into a pile of brittle cookie handling and hidden token extraction. If the workflow requires clicks, navigation, persisted sessions, or user-specific state, use a browser and make that cost explicit.

The tradeoff is simple: browser automation reaches more flows, but each job costs more time and infrastructure. You also need session storage so you do not log in on every run. Repeated logins are noisy and often trigger account protection.

A practical decision tree

I usually choose the layer in this order:

Public JSON endpoint, such as Shopify /products.json or a WooCommerce catalog API.
Internal XHR or Fetch endpoint discovered through DevTools.
Rendered HTML with a headless browser.
Full browser automation with saved sessions for login or cart flows.

On catalogued marketplaces like Amazon, eBay, Walmart, and Flipkart, Wire fits the second category: network-layer extraction where the useful data sits behind frontend API calls rather than stable public endpoints.

The important part is to avoid starting at layer four. It works in demos, then becomes expensive and flaky when you scale from ten URLs to ten thousand.

For your next scraper, pick one target page and spend ten minutes in the Network tab before writing extraction code. If you find a clean JSON response, build around that. If you do not, then move up one layer and document exactly why the lower layer failed.

Scraping API credits are not the same as requests

Anakin — Thu, 22 Oct 2026 10:00:02 +0000

You pick a scraping API plan that says 1,000,000 credits, connect it to a job queue, and assume you can fetch something close to 1,000,000 pages. Then the first real target needs JavaScript rendering, a better proxy pool, or retry logic, and your budget model stops matching production.

A credit is a billing unit, not a request

Most scraping APIs sell the same basic shape: send a URL, get back HTML or extracted data, let the vendor deal with proxies, browsers, and anti-bot systems.

That hides an important detail. The API call is not always the thing you pay for. You pay for whatever the provider defines as a billable unit, and that unit can change based on options.

A static HTML fetch might cost 1 credit. The same URL with browser rendering might cost 5. A premium proxy might cost 10 or 25. A stealth proxy can cost much more. AI extraction may add another fixed charge.

That is not inherently bad. Running browsers and higher-quality proxies costs more. The problem is when the pricing page makes the large credit number easy to see and the multipliers easy to miss.

Here is the budgeting code I usually write before trusting any scraping plan:

const credits = 1_000_000;

const scenarios = [
  { name: "static HTML", creditsPerRequest: 1 },
  { name: "JS rendering", creditsPerRequest: 5 },
  { name: "premium proxy + JS", creditsPerRequest: 25 },
  { name: "stealth proxy", creditsPerRequest: 75 },
  { name: "JS + AI extraction", creditsPerRequest: 10 }
];

for (const s of scenarios) {
  console.log(
    `${s.name}: ${Math.floor(credits / s.creditsPerRequest).toLocaleString()} requests`
  );
}

Output:

static HTML: 1,000,000 requests
JS rendering: 200,000 requests
premium proxy + JS: 40,000 requests
stealth proxy: 13,333 requests
JS + AI extraction: 100,000 requests

Same plan. Very different workload.

The hard pages are the expensive pages

The multiplier usually appears exactly where your scraper starts failing.

A static request works fine until you hit a React app and get an empty shell:

<div id="root"></div>
<script src="/assets/app.83f1c.js"></script>

Your HTTP request returned 200 OK, but your parser finds no product price, no listing rows, and no useful text. You turn on browser rendering.

Then another site returns a CAPTCHA page:

<title>Just a moment...</title>
<p>Checking if the site connection is secure</p>

Your API call still may return 200 OK, but the content is not the target page. You turn on a better proxy tier or stealth mode.

Then the site randomizes markup, so your CSS selector breaks:

const price = $(".product-price").text(); // ""

You add AI extraction or a more flexible parser.

Each fix can be reasonable. Together, they can turn a cheap scrape into a much smaller quota than you expected.

Measure cost per successful extraction

Request cost is the wrong metric. Successful extraction cost is the useful one.

If you send 10,000 URLs and get 8,000 usable records, you should divide credits by 8,000, not by 10,000. Failed requests matter, but so do silent failures where the HTTP response looks fine and the data is wrong.

A simple harness can make this visible:

async function scrapeAndValidate(client, url) {
  const result = await client.scrape(url, {
    renderJs: true,
    extract: {
      title: "string",
      price: "string"
    }
  });

  if (!result.title || !result.price) {
    return {
      ok: false,
      reason: "missing_required_fields",
      statusCode: result.statusCode,
      sample: result.text?.slice(0, 200)
    };
  }

  return { ok: true, data: result };
}

Log these counts per provider and per option set:

urls_submitted=10000
http_200=9420
valid_records=8110
captcha_pages=620
empty_spa_shells=410
timeouts=180
credits_spent=50000
credits_per_valid_record=6.16

That last number is the one you should compare.

Some providers also avoid charging failed jobs, or cache repeated fetches for a period such as 24 hours. Those details can matter more than the headline quota if your pipeline retries aggressively or revisits the same URLs often.

Authenticated scraping changes the problem

Public pages mostly fail because of rendering, blocking, geo restrictions, or bad selectors. Authenticated pages add credential handling.

The quick version often looks like this:

await page.goto("https://example.com/login");
await page.fill("input[name=email]", process.env.LOGIN_EMAIL);
await page.fill("input[name=password]", process.env.LOGIN_PASSWORD);
await page.click("button[type=submit]");
await page.waitForURL("https://example.com/dashboard");

That works for a prototype. In production, it raises less fun questions:

Where do the credentials live?
Can the scraping provider see them?
How do you rotate them?
What happens with 2FA?
Can you reuse a saved identity without passing raw cookies around?

If your scrape is really an authenticated action on a common site, Wire treats the login identity as part of the managed action instead of making every caller ship cookies or replay a login script.

You may not need that. For internal tools or one-off admin exports, a Playwright script with secrets loaded from your vault may be enough. For a multi-tenant app, credential storage and auditability become part of the scraping architecture, not an implementation detail.

What to compare before choosing a scraping API

For any scraping provider, ignore the biggest credit number until you answer these questions:

Does JavaScript rendering change the credit cost?
Do premium, residential, or stealth proxies change the cost?
Are failed requests charged?
What counts as failed: timeout only, non-2xx, CAPTCHA page, empty extraction?
Is there response caching, and does a cache hit cost credits?
Can you submit batches or async jobs, or do you need to build that queue?
Does structured extraction charge extra?
How do you handle logins without exposing credentials everywhere?
What are the concurrency limits, and do retries count against them?
What is the cost per valid record on your actual target sites?

Scraping APIs are useful because running your own browser fleet, proxy rotation, and anti-bot handling is annoying operational work. The tradeoff is that pricing models encode those same operational costs in ways that are easy to underestimate.

Before committing to a plan, run a representative 100 URL test set: static pages, JavaScript-heavy pages, blocked pages, repeated URLs, and authenticated pages if you have them. Log credits spent, valid records returned, and failure reasons. Then choose based on cost per valid record, not credits on the pricing page.

Measuring ChatGPT answers means measuring the product, not the model API

Anakin — Tue, 20 Oct 2026 10:00:02 +0000

Your monitoring script calls the OpenAI API with the same prompt users type into ChatGPT, but the result does not match what people see on chatgpt.com. The API returns a model response. ChatGPT the product may run a web search, rewrite the query, pick sources, add citations, and vary the answer by country. If you are trying to measure whether a brand appears in ChatGPT answers, that difference matters.

The API and the product are not the same target

The OpenAI API is the right tool when you want to build with a model. It is not a reliable way to answer questions like:

Did ChatGPT mention our product for this category prompt?
Which domains did ChatGPT cite?
Did the answer change in the US versus the UK?
Did ChatGPT run a web search, or answer from model context?

Those are product-layer questions.

For example, this prompt looks simple:

best project management tool for small engineering teams

A direct model call might return a generic list based on training data and whatever tools you explicitly configure. ChatGPT in the browser may decide to search the web, query something like:

best project management tools small engineering teams 2026 Linear Jira Asana ClickUp comparison

Then it may cite review sites, vendor docs, or recent comparison posts. That retrieval behavior is the thing you are measuring. If you skip it, your numbers can look precise while measuring the wrong system.

What usually breaks when you scrape it yourself

The direct approach is browser automation. You open ChatGPT with Playwright, submit a prompt, wait for the streamed answer to finish, parse the DOM, and store the result.

That can work for small internal checks. It gets annoying when you put it on a schedule.

Common failure modes are not subtle:

TimeoutError: locator.waitFor: Timeout 30000ms exceeded

or:

Error: Execution context was destroyed, most likely because of a navigation

Sometimes the script does not crash. It just captures half an answer because the stream was still running. That is worse, because bad data enters your metrics quietly.

You also have to handle login state, anti-bot checks, rate limits, changing selectors, regional proxies, and answer variants. A selector like this may work today:

const answer = await page.locator('[data-message-author-role="assistant"]').last().innerText();

Then ChatGPT ships a frontend change, and your job starts returning empty strings.

For this specific measurement problem, Wire exposes ChatGPT scraping as an async action that returns the answer text, citations, and whether web search ran.

Store answers as evidence, not just scores

A useful tracking pipeline should keep the raw answer and the parsed fields. Do not store only mentioned: true and throw everything else away. You will want to debug why a score changed.

A minimal record might look like this:

{
  "prompt": "best project management tool for small engineering teams",
  "country": "us",
  "run_at": "2026-08-14T10:30:00Z",
  "answer_text": "For small engineering teams, I would shortlist Linear...",
  "web_search_triggered": true,
  "web_search_query": "best project management tools small engineering teams 2026 Linear Jira Asana",
  "citations": [
    {
      "url": "https://example.com/project-management-tools",
      "domain": "example.com",
      "title": "Best Project Management Tools for Engineering Teams",
      "position": 1,
      "cited": true
    }
  ]
}

That gives you three useful measurements:

Mention share: how often the answer names your brand.
Source share: how often ChatGPT cites your domain.
Competitor/source analysis: which brands and publishers show up when you do not.

Here is a small Python pass over stored responses:

import json
from pathlib import Path

BRAND = "linear"
DOMAIN = "linear.app"

runs = [json.loads(p.read_text()) for p in Path("runs").glob("*.json")]

def mentions_brand(run):
    return BRAND.lower() in run["answer_text"].lower()

def cites_domain(run):
    return any(
        DOMAIN in citation.get("url", "")
        for citation in run.get("citations", [])
        if citation.get("cited")
    )

total = len(runs)
brand_mentions = sum(mentions_brand(run) for run in runs)
domain_citations = sum(cites_domain(run) for run in runs)

print({
    "total_runs": total,
    "mention_share": brand_mentions / total if total else 0,
    "citation_share": domain_citations / total if total else 0,
})

This is intentionally boring. The hard part is not the aggregation. The hard part is collecting comparable answers reliably over time.

Treat location as part of the input

ChatGPT answers can change by market. If your customers are in Canada, Germany, and Australia, one US-based run does not tell you enough.

Make country a first-class field in your prompt schedule:

{
  "prompt": "best payroll software for remote teams",
  "countries": ["us", "ca", "gb", "au"],
  "frequency": "weekly"
}

Then report by country instead of averaging everything together too early. Otherwise a strong US presence can hide that you never appear in UK answers.

If you already handle scheduling and storage, Wire can sit in the collection layer where each prompt and country pair becomes an async job with structured output.

Be careful with the metric

A ChatGPT visibility score is not a source of truth. It is a sampled measurement of a changing product.

You should expect variance from:

prompt wording
country
time of day
whether search runs
recently published pages
changes in ChatGPT retrieval behavior

The fix is not to pretend the data is deterministic. Run enough prompts, keep the raw evidence, and compare changes over consistent prompt sets.

A practical starting point: choose 25 real customer questions, run them weekly in your main markets, store answer_text, citations, web_search_triggered, and web_search_query, then review both the aggregate numbers and the raw answers before you draw conclusions.

Browser automation vs network calls for sites without public APIs

Anakin — Mon, 19 Oct 2026 10:00:02 +0000

You need data from a portal that never shipped a public API. The first version is usually a browser script: open the page, log in, click the button, scrape the table. It works until the site ships a harmless UI change and your job starts failing with TimeoutError: locator.click: Timeout 30000ms exceeded because the button text changed.

The real choice is the layer

There are two common ways to automate a site that only exposes a web UI.

Browser automation operates at the presentation layer. Tools like Playwright, Puppeteer, Selenium, and managed platforms such as CloudCruise drive a real browser. They click buttons, type into inputs, wait for navigation, handle downloads, and read data from the DOM.

Network automation operates one layer lower. Instead of clicking the UI, you inspect the requests the frontend already makes and call those endpoints directly. The page might not have a public API, but the JavaScript app usually talks to some backend API.

The difference matters because the failure modes are different.

A browser script depends on the page structure. If the app changes a label, moves a button, delays hydration, or swaps a table for virtualized rows, your script can break even if the underlying business action still works.

A network call depends on request shape, auth, and backend behavior. If the endpoint path changes, a CSRF token expires, the API starts requiring a new header, or the response schema changes, your integration breaks there instead.

For example, Wire is a maintained network-layer catalog for calling site actions directly when the useful behavior sits behind the page rather than in a public API.

What browser automation failure looks like

This is a typical Playwright flow:

import { chromium } from 'playwright';

const browser = await chromium.launch();
const page = await browser.newPage();

await page.goto('https://example-portal.test/login');
await page.getByLabel('Email').fill(process.env.PORTAL_USER);
await page.getByLabel('Password').fill(process.env.PORTAL_PASS);
await page.getByRole('button', { name: 'Sign in' }).click();

await page.getByRole('link', { name: 'Reports' }).click();
await page.getByRole('button', { name: 'Download CSV' }).click();

await browser.close();

This can fail even when the portal still works for a person:

TimeoutError: locator.click: Timeout 30000ms exceeded.
Call log:
  - waiting for getByRole('button', { name: 'Download CSV' })

Maybe the product team renamed the button to Export. Maybe the button is now an icon with no accessible name. Maybe a cookie banner covers it. The backend endpoint that generates the CSV may not have changed at all.

The browser approach is still the right tool when you need the actual user flow. Login with SSO, TOTP prompts, healthcare payer portals, EHR systems, visual confirmations, file pickers, and workflows that only exist as UI interactions all fit here. CloudCruise, for example, focuses on managed browser workflows for healthcare portals, including credential handling and 2FA support. That is a real use case, not something you should replace with a fragile copied request.

What network automation failure looks like

If you inspect DevTools while clicking Download CSV, you might find a request like this:

curl 'https://example-portal.test/api/reports/claims?month=2026-01' \
  -H 'authorization: Bearer eyJhbGciOi...' \
  -H 'x-csrf-token: 7d1b0...' \
  -H 'accept: application/json'

A safer implementation checks status codes and schema instead of assuming the request worked:

async function fetchClaimsReport({ month, token, csrf }) {
  const res = await fetch(
    `https://example-portal.test/api/reports/claims?month=${encodeURIComponent(month)}`,
    {
      headers: {
        authorization: `Bearer ${token}`,
        'x-csrf-token': csrf,
        accept: 'application/json'
      }
    }
  );

  if (res.status === 401 || res.status === 403) {
    throw new Error(`auth failed: ${res.status}`);
  }

  if (res.status === 429) {
    throw new Error('rate limited');
  }

  if (!res.ok) {
    throw new Error(`report request failed: ${res.status}`);
  }

  const data = await res.json();

  if (!Array.isArray(data.claims)) {
    throw new Error('unexpected response shape: claims is missing');
  }

  return data.claims;
}

This version avoids DOM breakage, browser startup time, screenshots, and selector maintenance. It also gives you clearer operational errors: 401, 403, 429, or unexpected response shape instead of a generic click timeout.

But it has edges. You must understand authentication. You need to respect rate limits. You may need to refresh CSRF tokens. Some endpoints are intentionally tied to short-lived browser sessions. Some sites prohibit automated access in their terms. Direct calls are not automatically safer or more acceptable just because they are faster.

Long-running actions need a job pattern

Browser jobs and network jobs can both outlive a single HTTP request. A report might take 45 seconds to generate. A search might fan out across multiple pages. Do not keep one request open forever if the caller can lose the connection.

Use a task creation endpoint and poll by job ID:

async function waitForJob(jobId, getStatus, { timeoutMs = 120000 } = {}) {
  const started = Date.now();
  let delay = 1000;

  while (Date.now() - started < timeoutMs) {
    const job = await getStatus(jobId);

    if (job.status === 'succeeded') return job.result;
    if (job.status === 'failed') throw new Error(job.error || 'job failed');

    await new Promise(resolve => setTimeout(resolve, delay));
    delay = Math.min(delay * 1.5, 5000);
  }

  throw new Error(`job timed out: ${jobId}`);
}

Wire models long-running site actions this way: create a task, keep the returned job ID, then poll until the action returns structured JSON or a failure.

How to choose

Use browser automation when:

The workflow requires real user interaction, MFA, SSO, or visual confirmation
The site does not expose a clean backend request for the action
You need screenshots, video replay, or DOM-level debugging
Compliance requirements point you toward a managed browser platform built for that domain

Use network automation when:

The frontend calls a stable JSON endpoint for the data or action you need
You care about latency and concurrency more than visual fidelity
You want structured errors based on HTTP status and response schema
UI changes should not break the integration

For your next portal automation, record one successful manual run with DevTools open. If the important action maps to a clear backend request, prototype the network call and validate auth, rate limits, and response shape. If it does not, write the browser workflow and make selector failure an expected operational case rather than a surprise.

A scraping API should do more than start a scraper you built somewhere else

Anakin — Fri, 16 Oct 2026 10:00:02 +0000

A common trap with scraping tools is assuming that “has an API” means “can be driven entirely from code.” Sometimes it only means the API can start a project you already built in a GUI. That difference matters once scraping becomes part of a backend job, data pipeline, agent workflow, or monitoring system.

The difference between a trigger API and a scraping API

A trigger API wraps an existing scraper. You build the scraper somewhere else, usually by clicking through pages in a desktop or web UI, then your code starts that saved project.

The API flow usually looks like this:

# Start a saved scraping project
curl -X POST "https://api.example.com/v2/projects/PROJECT_TOKEN/run" \
  -d "api_key=$API_KEY"

# Poll for results
curl "https://api.example.com/v2/runs/RUN_TOKEN/data?api_key=$API_KEY"

That can be useful. If a non-developer built a scraper once and wants it scheduled or triggered from another system, this model works fine.

But it has a hard boundary: your code does not define the extraction. It only presses run.

A scraping API should let your code describe the job at runtime:

curl -X POST "https://api.example.com/v1/scrape" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/products/widget-123",
    "renderJavascript": true,
    "extract": {
      "title": "product name",
      "price": "current price",
      "availability": "stock status"
    }
  }'

That distinction changes how you maintain the system. With the first model, a new URL pattern may require someone to reopen the visual builder and edit selectors. With the second, the extraction contract lives next to the code that consumes the result.

Where the trigger model breaks down

The trigger model usually fails in boring production ways.

A selector changes and the job still “succeeds,” but returns an empty CSV. A site adds a login step and the run starts timing out. A product page template changes and half the rows get shifted into the wrong columns. Your pipeline sees HTTP 200, then discovers that price is missing downstream.

The failure mode is not always an exception. Often it is bad data.

That makes observability harder. If your code only knows “run project 123,” it cannot easily explain which selector failed, which extraction field was missing, or whether the target page changed. You end up debugging in the visual tool instead of in logs, tests, and version control.

For authenticated or repeated site-specific workflows, Wire treats those actions as API-callable units rather than desktop projects, which is the model you want when job inputs, credentials, and outputs need to stay inside a pipeline.

A better contract for scraper jobs

For code-driven scraping, I want the job request to contain enough information to reproduce the run:

{
  "url": "https://example.com/products/widget-123",
  "javascript": true,
  "country": "US",
  "timeoutMs": 30000,
  "schema": {
    "title": "string",
    "price": "number",
    "currency": "string",
    "inStock": "boolean"
  }
}

And I want the response to separate transport success from extraction success:

{
  "status": "completed",
  "httpStatus": 200,
  "extractionStatus": "partial",
  "data": {
    "title": "Widget 123",
    "price": null,
    "currency": null,
    "inStock": true
  },
  "errors": [
    {
      "field": "price",
      "message": "No matching price text found"
    }
  ]
}

That gives your pipeline something useful to do. You can reject partial results, retry with a different region, alert on missing fields, or store the raw HTML for inspection.

You can also write tests against the contract:

import assert from "node:assert";

function validateProduct(row) {
  assert.equal(typeof row.title, "string");
  assert.ok(row.title.length > 0);
  assert.equal(typeof row.inStock, "boolean");

  if (row.inStock) {
    assert.equal(typeof row.price, "number");
    assert.ok(row.price > 0);
  }
}

This is much harder when the real extraction logic lives in a GUI project that your test suite cannot inspect.

Security is part of the API shape

Scraping often starts with public pages, then drifts into authenticated pages: admin dashboards, partner portals, internal tools, vendor accounts.

That is where visual scrapers can get uncomfortable. If the tool asks someone to type credentials into a recorded login step, you need to know exactly how those secrets are stored, who can view them, whether they are encrypted, and whether the API can ever return them.

At minimum, look for these properties:

Secrets are stored outside the scraper definition
The API can reference a credential by ID, not by raw password
Secrets are never returned by read endpoints
Runs produce audit logs
Free or shared projects are not public by default

A safer request shape looks like this:

{
  "url": "https://portal.example.com/invoices",
  "auth": {
    "credentialId": "cred_vendor_portal_readonly"
  },
  "extract": {
    "invoiceNumber": "string",
    "total": "number",
    "dueDate": "date"
  }
}

The scraper gets access at run time. The pipeline never handles the password directly.

Wire is relevant here because its prebuilt actions can use vault-backed credentials without returning the secret through the API, which avoids mixing login material with extraction code.

When a visual scraper is still the right tool

Visual scrapers are not bad. They are often the right choice for a bounded task, especially when the person doing the work does not write code.

If someone needs to scrape one directory once a week, export a spreadsheet, and adjust the workflow manually when the site changes, a point-and-click builder may be faster than building an internal tool.

The tradeoff appears when the scraper becomes infrastructure. If the output feeds billing, search, analytics, alerts, model evaluation, or customer-facing features, you probably want the scrape definition, job inputs, validation, retries, and errors in code.

What to check before choosing

Before you depend on a scraping API, answer these questions with actual requests and responses:

Can I define a new scrape entirely through the API?
Can I pass a schema or extraction instructions at runtime?
What does a partial extraction failure look like?
Are JavaScript rendering, proxy region, timeout, and retries request parameters?
Can I run the same job in CI with fixture URLs?
Where are credentials stored, and can any endpoint read them back?
Does the pricing model punish failed runs or repeated cached fetches?
Are there page-per-run or worker caps that affect batch jobs?

The practical next step: take one real target page, write the API request you wish you could send, then compare tools against that contract instead of comparing feature lists.

Your competitor's price is not a float

Anakin — Thu, 15 Oct 2026 10:00:01 +0000

You add competitor pricing to a dashboard and start with the obvious model: SKU, competitor, price. It works for retailer pages. Then someone asks why the same product costs $6.49 on one site, $7.29 on a delivery app, and $14.74 at checkout. The bug is not in your scraper. The bug is the assumption that a product has one competitor price.

A price is an observation, not a property

For a nationally distributed product, customers may see different prices on your site, Amazon, Walmart, Instacart in one zip code, Instacart in another zip code, and DoorDash or Uber Eats from a nearby store.

Those are not copies of the same decision. Different systems set them:

Your site price is usually controlled by you.
Amazon may expose a Buy Box price affected by third-party sellers and marketplace rules.
A retailer site reflects that retailer's margin and promo calendar.
Instacart can vary by store, banner, zip code, fulfillment path, and shopper context.
Delivery apps often add platform markup, delivery fees, service fees, small-basket fees, and sometimes demand-based charges.

If your schema stores products.price, you will overwrite useful data or average it into something no customer actually saw.

A better primitive is a price observation:

create table price_observations (
  id bigserial primary key,
  sku text not null,
  competitor_sku text,
  surface text not null,              -- amazon, walmart, instacart, doordash
  retailer text,
  postcode text,
  observed_at timestamptz not null,
  subscriber_context text not null,   -- none, prime, instacart_plus, dashpass
  item_price_cents integer not null,
  delivery_fee_cents integer,
  service_fee_cents integer,
  surcharge_cents integer,
  currency char(3) not null default 'USD',
  raw_url text,
  raw_payload jsonb
);

This looks more annoying than price numeric(10,2), because it is. It also matches the thing you are measuring.

For teams collecting these delivery and quick-commerce prices as structured data, Wire treats the surface, postcode, fee components, and shopper context as part of the price record rather than metadata to recover later.

The checkout total can reverse the signal

On delivery surfaces, the listed item price is not the number the customer pays. That matters when you compare competitors.

Say your item is $7.29 on a grocery delivery app. A competitor's item is $5.99. If you track only item price, you report a 17.8% gap.

But the checkout may look like this:

from dataclasses import dataclass

@dataclass
class DeliveryPrice:
    item: int
    delivery_fee: int
    service_fee: int
    surcharge: int = 0

    @property
    def delivered_total(self) -> int:
        return self.item + self.delivery_fee + self.service_fee + self.surcharge

mine = DeliveryPrice(item=729, delivery_fee=399, service_fee=146, surcharge=200)
theirs = DeliveryPrice(item=599, delivery_fee=399, service_fee=146, surcharge=200)

item_gap = (mine.item - theirs.item) / theirs.item
total_gap = (mine.delivered_total - theirs.delivered_total) / theirs.delivered_total

print(f"item gap: {item_gap:.1%}")
print(f"delivered gap: {total_gap:.1%}")

Output:

item gap: 21.7%
delivered gap: 9.1%

The fee stack did not eliminate the difference, but it changed its size. In some baskets it compresses the competitive gap. In others it amplifies it, especially when fees depend on basket size or subscription status.

The common failure mode is a dashboard that says "we are 20% more expensive" while a customer sees a much smaller checkout difference. Or the reverse: item prices look close, but one platform's local fees make the delivered total much worse.

Location is part of the key

A national average is often useful for retailer sites. It is much less useful for delivery apps.

A delivery price depends on the local store, local platform markup, delivery supply, fee rules, and sometimes the exact address. Two zip codes in the same metro area can produce different item prices and different fees for the same SKU.

That means this query is usually wrong:

select sku, avg(item_price_cents) as avg_price
from price_observations
where surface = 'instacart'
group by sku;

It creates a number that no shopper paid. Use it only if you explicitly want a statistical summary.

For pricing decisions, keep geography in the grouping key:

select
  sku,
  postcode,
  percentile_cont(0.5) within group (order by item_price_cents) as median_item_price,
  percentile_cont(0.5) within group (
    order by item_price_cents
      + coalesce(delivery_fee_cents, 0)
      + coalesce(service_fee_cents, 0)
      + coalesce(surcharge_cents, 0)
  ) as median_delivered_total
from price_observations
where surface in ('instacart', 'doordash', 'ubereats')
group by sku, postcode;

This is not just a reporting detail. If a pricing team can only change prices regionally, collect data regionally. If it can act at store or zip-code level, collect at that level. The measurement grain should match the decision grain.

Subscription context is another split

Instacart+, DashPass, Uber One, Walmart+, and Prime can create different effective prices for the same product at the same time.

Sometimes the item price changes. Sometimes the delivery fee disappears. Sometimes the platform applies member-only promotions at checkout. If your scraper runs from a logged-in account, you may collect a subscriber price and label it as the public price. If it runs logged out, you may miss the price your most loyal customers see.

This is the kind of bug that does not throw an exception. It just poisons comparisons quietly.

Store the context explicitly:

{
  "sku": "DOGFOOD-12LB",
  "surface": "doordash",
  "postcode": "77002",
  "subscriber_context": "dashpass",
  "item_price_cents": 1899,
  "delivery_fee_cents": 0,
  "service_fee_cents": 285,
  "surcharge_cents": 0
}

If a vendor or internal pipeline cannot tell you whether an observation came from a subscriber or non-subscriber session, do not mix those rows without labeling the result.

Wire separates subscriber and non-subscriber delivery observations so downstream pricing analysis can compare like with like.

What I would build first

Before building a large monitoring system, run a small audit.

Pick one SKU. Collect prices from your own site, Amazon, one major retailer, Instacart in two distant zip codes, and one delivery app. For delivery surfaces, record item price, delivery fee, service fee, surcharge, postcode, timestamp, and subscription context.

Then calculate two spreads:

item price spread
delivered total spread

If those numbers are close, your current item-price tracker may be good enough for that category. If they differ materially, your data model needs to capture fees, location, and shopper context before the dashboard can answer the pricing question correctly.

Turning Ride-Hailing Scrapes Into Useful Pricing Signals

Anakin — Wed, 14 Oct 2026 10:00:01 +0000

Ride-hailing pricing changes quickly enough that a scrape from ten minutes ago can already be misleading. A competitor may surge in one neighborhood, show no cars in another, and offer a coupon only to a new account. If you treat those values as clean facts, your pricing or forecasting model will learn the wrong thing.

The useful pattern is to treat competitor data as timestamped observations with context, uncertainty, and failure states. That sounds obvious, but many scraping pipelines lose this information early.

Scraped fares are observations, not ground truth

A ride quote is not just price = 18.42.

It depends on at least:

pickup and dropoff coordinates
request time
service type, such as economy, comfort, XL, electric, accessibility
account state, such as new user, frequent rider, corporate account
app version, locale, currency, and payment method
whether the quote included surge, fees, taxes, discounts, or credits

Two scrapes can disagree and both can be correct if they used different accounts or slightly different coordinates.

A useful raw schema keeps those dimensions intact:

CREATE TABLE competitor_quote_observations (
  id BIGSERIAL PRIMARY KEY,
  competitor TEXT NOT NULL,
  observed_at TIMESTAMPTZ NOT NULL,
  pickup_lat DOUBLE PRECISION NOT NULL,
  pickup_lng DOUBLE PRECISION NOT NULL,
  dropoff_lat DOUBLE PRECISION NOT NULL,
  dropoff_lng DOUBLE PRECISION NOT NULL,
  service_type TEXT NOT NULL,
  persona TEXT NOT NULL,
  currency TEXT,
  total_fare_cents INTEGER,
  base_fare_cents INTEGER,
  distance_fare_cents INTEGER,
  time_fare_cents INTEGER,
  surge_multiplier NUMERIC,
  promo_cents INTEGER,
  eta_seconds INTEGER,
  raw_response JSONB NOT NULL,
  status TEXT NOT NULL,
  error_code TEXT
);

The status column matters. A failed scrape is still data. If one platform returns quotes in most areas but times out or hides availability in a specific zone, that tells you something different from “fare is null.”

For extraction teams that do not want to build every app and web collection path themselves, Wire fits this problem when the output needs to preserve quote context like location, persona, timestamp, ETA, and failure state rather than just a flattened price.

Normalize before feeding models

A pricing model usually needs a smaller table than the raw scrape table. You might aggregate by geohash, hour, competitor, and service type.

The important part is not to average stale and fresh observations together as if they have equal value. A quote observed 90 seconds ago should usually count more than one observed 25 minutes ago.

Here is a simple Python example that applies a freshness weight before producing features:

from datetime import datetime, timezone
import math

HALF_LIFE_SECONDS = 300  # after 5 minutes, weight is 0.5

def freshness_weight(observed_at: datetime, now: datetime) -> float:
    age = max(0, (now - observed_at).total_seconds())
    return 0.5 ** (age / HALF_LIFE_SECONDS)


def weighted_average(rows, field):
    numerator = 0.0
    denominator = 0.0

    now = datetime.now(timezone.utc)

    for row in rows:
        if row["status"] != "ok":
            continue
        value = row.get(field)
        if value is None:
            continue

        w = freshness_weight(row["observed_at"], now)
        numerator += value * w
        denominator += w

    return numerator / denominator if denominator else None


quotes = [
    {
        "competitor": "example_ride",
        "observed_at": datetime(2026, 8, 18, 10, 0, 0, tzinfo=timezone.utc),
        "status": "ok",
        "total_fare_cents": 2140,
        "eta_seconds": 360,
        "surge_multiplier": 1.4,
    },
    {
        "competitor": "example_ride",
        "observed_at": datetime(2026, 8, 18, 9, 42, 0, tzinfo=timezone.utc),
        "status": "ok",
        "total_fare_cents": 1750,
        "eta_seconds": 780,
        "surge_multiplier": 1.0,
    },
]

print(weighted_average(quotes, "total_fare_cents"))
print(weighted_average(quotes, "eta_seconds"))

This is not a production pricing model. It is the kind of guardrail you want before modeling starts. Without it, old low-fare quotes can dampen a current surge signal and make your system react late.

The useful features are usually boring

For ride-hailing, the most useful competitor features tend to be straightforward:

median competitor fare by area, route bucket, and hour
competitor ETA by area and service type
observed surge multiplier
quote success rate
promo availability by persona
fare component split, if visible
variance across accounts for the same route

The last one catches persona-based pricing. If a new account sees a lower fare than an existing account for the same trip, your model should not collapse both into one number without recording the persona.

A simple demand model might use your own demand plus competitor signals:

predicted_demand(area, hour) =
  baseline
  + beta_1 * own_completed_rides(area, hour - 1)
  + beta_2 * competitor_median_eta(area, hour)
  + beta_3 * competitor_surge_multiplier(area, hour)
  + beta_4 * competitor_quote_failure_rate(area, hour)

The sign of each coefficient depends on your market. A high competitor ETA can mean unmet demand that you can capture. It can also mean bad weather or an event that affects everyone, including your own drivers.

How scraping fails in this domain

The common failures are not subtle.

You get HTTP 429 responses or app-level throttling when you sample too aggressively. You get stale cached quotes that look valid but repeat the same fare and ETA for too long. You get partial responses where fare exists but ETA is missing. You get account contamination when a test account receives a promo and then stops representing a normal user.

You should log these as first-class states:

{
  "status": "failed",
  "error_code": "rate_limited",
  "competitor": "example_ride",
  "observed_at": "2026-08-18T10:04:12Z",
  "pickup_geohash": "dr5ru",
  "service_type": "economy"
}

Do not hide these rows from analysts. A spike in rate_limited may mean your scraper broke, but a spike in no_drivers_available in one area can be a useful competitive signal.

This is also where vendor output quality matters. Wire is relevant here if you need ride-hailing extraction that distinguishes “no quote available,” “blocked,” “timed out,” and “valid zero promo” instead of returning a blank field for all of them.

Use competitor data carefully

Competitor scraping should not directly set prices without constraints. Feed it into a pricing system as one group of features, then cap changes and monitor outcomes.

A safer control loop looks like this:

Collect quotes and ETAs for fixed route samples.
Validate freshness, persona, and failure rates.
Aggregate features by area and time window.
Use those features in pricing, supply allocation, or promotion models.
Compare outcomes against holdout areas before broad rollout.

The practical next step is to define one route grid, one service type, and one persona, then build the raw observation table before touching any pricing logic. If the data model cannot explain where a fare came from, the model built on top of it will not be trustworthy.

Treat product assortment as a data problem, not a catalog spreadsheet

Anakin — Tue, 13 Oct 2026 10:00:01 +0000

A catalog usually gets messy slowly. Someone adds five near-identical variants because a supplier sent them. Another team keeps a slow seller because it had one good month last year. Search results fill up with products nobody buys, while customers still complain that you do not carry the thing they came for.

That is a product assortment problem, but for developers it often shows up as a data problem: unclear SKU relationships, missing category mappings, stale inventory, and dashboards that count products without explaining whether the catalog is useful.

Start with a catalog model you can query

Before you can reason about assortment, you need a few basic relationships to be explicit. At minimum, separate the sellable SKU from the product family and category taxonomy.

create table categories (
  category_id bigint primary key,
  parent_category_id bigint,
  name text not null
);

create table products (
  product_id bigint primary key,
  parent_sku text not null,
  category_id bigint not null references categories(category_id),
  brand text,
  title text not null,
  status text not null check (status in ('active', 'inactive'))
);

create table skus (
  sku_id bigint primary key,
  product_id bigint not null references products(product_id),
  color text,
  size text,
  price numeric(10,2) not null,
  inventory_count int not null,
  created_at timestamp not null
);

create table order_items (
  order_id bigint,
  sku_id bigint not null references skus(sku_id),
  quantity int not null,
  unit_price numeric(10,2) not null,
  created_at timestamp not null
);

If your catalog cannot answer “which SKUs are variants of the same product?” then depth metrics will lie. If category_id is nullable or overloaded with values like misc, breadth metrics will tell you that you sell in more categories than you really do.

The failure mode is familiar: the dashboard says you have 12,000 active SKUs, but customers see 40 copies of the same black phone case and still cannot find a charger for their laptop.

Measure breadth, depth, and length separately

Product assortment has a few useful dimensions:

Breadth: how many categories you cover
Depth: how many variants you carry inside a product family or category
Length: total active SKUs
Quality mix: distribution across price bands
Consistency: how related the categories are to each other

You do not need a complicated model to start. These queries give you a baseline.

-- Catalog length
select count(*) as active_skus
from skus s
join products p on p.product_id = s.product_id
where p.status = 'active';

-- Breadth by active category
select count(distinct p.category_id) as active_categories
from products p
join skus s on s.product_id = p.product_id
where p.status = 'active'
  and s.inventory_count > 0;

-- Depth by category
select
  c.name as category,
  count(distinct p.parent_sku) as product_families,
  count(s.sku_id) as sku_count,
  round(count(s.sku_id)::numeric / nullif(count(distinct p.parent_sku), 0), 2) as avg_skus_per_family
from categories c
join products p on p.category_id = c.category_id
join skus s on s.product_id = p.product_id
where p.status = 'active'
group by c.name
order by avg_skus_per_family desc;

The last query is where problems often appear. A high avg_skus_per_family can be good for apparel, where size and color matter. It can be noise for commodity accessories, where 18 tiny variations split demand and make search results worse.

Find over-assortment with sales and inventory together

Counting SKUs is not enough. You need to know whether variants sell, sit, or cannibalize each other.

This query flags product families where many variants exist, but most have low sales in the last 90 days.

with sku_sales as (
  select
    s.sku_id,
    p.parent_sku,
    p.category_id,
    s.inventory_count,
    coalesce(sum(oi.quantity), 0) as units_90d
  from skus s
  join products p on p.product_id = s.product_id
  left join order_items oi
    on oi.sku_id = s.sku_id
   and oi.created_at >= now() - interval '90 days'
  where p.status = 'active'
  group by s.sku_id, p.parent_sku, p.category_id, s.inventory_count
), family_stats as (
  select
    parent_sku,
    count(*) as variant_count,
    count(*) filter (where units_90d = 0 and inventory_count > 0) as stocked_dead_variants,
    sum(inventory_count) as total_stock
  from sku_sales
  group by parent_sku
)
select *
from family_stats
where variant_count >= 8
  and stocked_dead_variants::numeric / variant_count >= 0.5
order by stocked_dead_variants desc;

Do not automatically delete these products. The query creates a review list. A variant might have zero sales because it launched yesterday, because it is seasonal, or because the product page is broken. The data points to a question, not the answer.

Detect under-assortment from demand signals

Under-assortment is harder because the missing product is not in your catalog. You need demand signals from searches, product listing views, support tickets, or competitor catalogs.

Suppose you track search events like this:

create table search_events (
  search_id bigint primary key,
  user_id bigint,
  query text not null,
  result_count int not null,
  created_at timestamp not null
);

A simple under-assortment signal is repeated searches with poor results.

select
  lower(query) as normalized_query,
  count(*) as searches,
  avg(result_count) as avg_results
from search_events
where created_at >= now() - interval '30 days'
group by lower(query)
having count(*) >= 50
   and avg(result_count) < 3
order by searches desc;

This catches cases like customers searching for “usb c gan charger” while your taxonomy only has a generic “chargers” category. The symptom is not just low conversion. It is demand with no useful landing place.

External catalog and price data can help here, but only if you store it as structured observations with timestamps and source URLs. Wire fits this part of the workflow when you need competitor assortment and pricing data as rows you can join against your own SKU and category tables.

Normalize competitor data before comparing it

Competitor matching fails in boring ways. Titles differ. Sizes use different units. One site says “midnight,” another says “black.” Bundles look cheaper until you compare unit counts.

A practical table for external observations might look like this:

create table competitor_offers (
  observed_at timestamp not null,
  source text not null,
  source_url text not null,
  external_title text not null,
  normalized_brand text,
  normalized_category text,
  normalized_size text,
  price numeric(10,2),
  in_stock boolean,
  match_confidence numeric(4,3),
  matched_sku_id bigint references skus(sku_id)
);

Keep external_title even after normalization. When a match looks wrong, the raw title is usually the fastest way to debug it.

Also store match_confidence. A low-confidence match should not drive repricing or assortment decisions. It should go into a manual review queue.

Turn metrics into an assortment review queue

The useful output is not a dashboard with five charts. It is a list of actions someone can inspect.

create table assortment_review as
select
  'possible_over_assortment' as reason,
  parent_sku as entity_id,
  json_build_object(
    'variant_count', variant_count,
    'stocked_dead_variants', stocked_dead_variants,
    'total_stock', total_stock
  ) as evidence,
  now() as created_at
from family_stats
where variant_count >= 8
  and stocked_dead_variants::numeric / variant_count >= 0.5;

You can add other reasons over time: missing category demand, price gaps, stale inventory, poor variant coverage, or competitor-only products with repeated search demand.

The important part is to keep the evidence with the recommendation. “Remove SKU 123” is not useful. “Review this product family because 9 of 12 stocked variants sold zero units in 90 days” is useful.

A good next step is to run the breadth, depth, and dead-variant queries against one category you know well. If the output surprises the category owner, fix the data model before you build more assortment logic on top of it.

Retail pricing data is messier than a price field

Anakin — Mon, 12 Oct 2026 10:00:03 +0000

You can have a clean pricing model and still make bad pricing decisions because the input data is wrong. In retail, the problem usually starts with a simple assumption: one product page has one price. Then you discover variants, coupons, member pricing, out-of-stock items, regional pricing, marketplace sellers, and JavaScript-rendered pages that sometimes return nothing at all.

Retail teams talk about competitive pricing, dynamic pricing, discount pricing, and value-based pricing. Developers usually inherit the operational part: collect enough reliable data so those strategies do not run on guesses.

Treat price as an observation, not a product attribute

A common first version of a price monitor stores this:

CREATE TABLE competitor_products (
  sku TEXT PRIMARY KEY,
  competitor_url TEXT NOT NULL,
  price_cents INTEGER,
  currency TEXT,
  updated_at TIMESTAMP
);

That works until the same URL shows different prices depending on size, location, login state, promotion, or seller. Updating one row destroys useful history and hides uncertainty.

A better pattern is to store every scrape result as an observation:

CREATE TABLE price_observations (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  internal_sku TEXT NOT NULL,
  competitor TEXT NOT NULL,
  competitor_url TEXT NOT NULL,
  observed_at TIMESTAMP NOT NULL,

  variant_key TEXT,
  seller_name TEXT,
  price_cents INTEGER,
  list_price_cents INTEGER,
  currency TEXT,
  in_stock BOOLEAN,
  promo_text TEXT,

  http_status INTEGER,
  extraction_status TEXT NOT NULL,
  raw_hash TEXT,
  error_message TEXT
);

This schema lets you answer different questions later:

What was the lowest available price today?
Did the competitor discount the item or change the base price?
Was the product actually purchasable?
Did our scraper fail, or did the site stop showing the product?

That distinction matters. A missing price is not the same thing as a zero price, and an out-of-stock product should not always trigger an aggressive price drop.

Capture the effective price, not just the visible number

Retail pages often expose several numbers:

List price: $129.99
Sale price: $99.99
Coupon: Save 10% at checkout
Loyalty price: $89.99 with membership
Bundle price: 2 for $170

If your parser grabs the first currency-looking string, it will eventually make a bad decision. You might parse the crossed-out list price instead of the sale price. Or you might treat a coupon as unavailable because it appears outside the price container.

A simple extractor should make the decision explicit:

from bs4 import BeautifulSoup
from decimal import Decimal
import re

PRICE_RE = re.compile(r"(\d+[\d,]*\.\d{2})")

def cents(text: str) -> int | None:
    if not text:
        return None
    match = PRICE_RE.search(text.replace(",", ""))
    if not match:
        return None
    return int(Decimal(match.group(1)) * 100)

def extract_price(html: str) -> dict:
    soup = BeautifulSoup(html, "html.parser")

    sale = soup.select_one('[data-testid="sale-price"]')
    list_price = soup.select_one('[data-testid="list-price"]')
    stock = soup.select_one('[data-testid="stock-status"]')
    promo = soup.select_one('[data-testid="promo-message"]')

    price_cents = cents(sale.get_text(" ")) if sale else None
    list_price_cents = cents(list_price.get_text(" ")) if list_price else None

    if price_cents is None:
        raise ValueError("sale price selector matched nothing or contained no parseable price")

    return {
        "price_cents": price_cents,
        "list_price_cents": list_price_cents,
        "in_stock": bool(stock and "in stock" in stock.get_text(" ").lower()),
        "promo_text": promo.get_text(" ", strip=True) if promo else None,
    }

This is intentionally boring code. The important part is that extraction failure is explicit. If the selector breaks, you get a ValueError instead of silently inserting NULL and letting downstream pricing code interpret it however it wants.

Expect matching to be harder than scraping

Competitive pricing depends on comparing the same product. That sounds obvious, but matching breaks in small ways:

A competitor lists a 12-pack while you sell a single unit.
Color names differ: navy, midnight blue, blue black.
Product titles include accessories or bundles.
A marketplace page changes the default seller.
Variant URLs canonicalize back to the parent product.

Do not match only on title similarity. Store the evidence you used for the match:

{
  "internal_sku": "SHOE-123-BLK-10",
  "competitor": "example-store",
  "competitor_url": "https://example.com/products/running-shoe",
  "match_confidence": 0.92,
  "matched_fields": ["brand", "model", "color", "size"],
  "unit_quantity": 1
}

If a price suddenly drops by 70%, the first debugging question is not “what did the pricing algorithm do?” It is “did we compare the right item?”

Separate collection failures from business signals

Scrapers fail for normal reasons. You get 403 Forbidden, 429 Too Many Requests, stale cached HTML, bot checks, changed markup, and pages that render prices only after client-side API calls. These are collection failures, not market events.

Your pipeline should classify them separately:

try:
    response = fetch(url)
    data = extract_price(response.text)
    status = "ok"
    error = None
except TimeoutError as exc:
    data = {}
    status = "fetch_timeout"
    error = str(exc)
except ValueError as exc:
    data = {}
    status = "extraction_failed"
    error = str(exc)

Then add a rule: pricing decisions can use only observations where extraction_status = 'ok' and observed_at is recent enough for the category. Grocery prices might need hourly checks. Furniture might not.

If you are buying rather than building this extraction layer, Wire fits this part of the problem: competitor price monitoring where variant matching, promotion capture, and extraction failure handling all affect the final dataset.

Make pricing rules conservative by default

Even if the business wants dynamic pricing, the first implementation should avoid automatic overreaction. Put guardrails around changes:

SELECT internal_sku,
       MIN(price_cents) AS competitor_min_price
FROM price_observations
WHERE extraction_status = 'ok'
  AND in_stock = 1
  AND observed_at > datetime('now', '-6 hours')
GROUP BY internal_sku;

Then apply limits outside SQL:

Do not change price if fewer than two competitors have fresh observations.
Do not match an out-of-stock competitor.
Do not drop below margin floor.
Do not react to one anomalous observation without a second confirmation.
Log the input observations used for every price change.

That last point saves time. When someone asks why an item changed from $49.99 to $44.99, you need more than “the job ran.”

A practical next step

Pick ten SKUs and track them across two competitors for a week. Store observations instead of overwriting rows. Include stock, promo text, variant, seller, extraction status, and raw response hashes. Before writing any pricing algorithm, inspect the weird cases manually. Those weird cases are the system requirements.