Scraping API pricing: read the defaults, not just the headline rate

A scraping API can look cheap in the pricing table and still cost several times more in production. The usual reason is not the base price. It is the default request mode: JavaScript rendering, residential proxies, geo targeting, retries, or parsing options that change how many credits one URL actually consumes.

If you only compare “requests per dollar,” you can miss the part that matters: what the API does when you call it with the simplest possible integration.

The default request is the real unit price

Most teams start with the shortest working request. Something like this:

curl "https://api.example-scraper.com/scrape?api_key=$API_KEY&url=https://example.com"

That request might mean “fetch plain HTML.” It might also mean “start a browser, render JavaScript, route through a premium proxy pool, and return parsed output.” Those are very different cost profiles.

Scrapingdog is a useful concrete example. Its pricing is low on paper, but its JavaScript rendering option is on by default according to its docs. A default request costs 5 credits. If you set dynamic=false, the same plain HTML request drops to 1 credit.

# Default: dynamic rendering enabled, billed as 5 credits
curl "https://api.scrapingdog.com/scrape?api_key=$API_KEY&url=https://example.com"

# Plain HTML: billed as 1 credit
curl "https://api.scrapingdog.com/scrape?api_key=$API_KEY&url=https://example.com&dynamic=false"

That does not make the product bad. It means the integration has a sharp edge. If your workload is mostly static pages and you remember to disable rendering, the low price can be real. If you forget, your credit burn is 5x before you have scraped anything interesting.

This is the first thing I check with any scraping API:

const urls = 100_000;

const headlineCreditsPerUrl = 1;
const defaultCreditsPerUrl = 5;

console.log({
  headlineCredits: urls * headlineCreditsPerUrl,
  defaultCredits: urls * defaultCreditsPerUrl,
  multiplier: defaultCreditsPerUrl / headlineCreditsPerUrl
});

// { headlineCredits: 100000, defaultCredits: 500000, multiplier: 5 }

A pricing page tells you the best case. Your first curl command tells you the likely case.

Rendering is not optional for many modern pages

Turning JavaScript off saves money, but it can also silently break extraction.

You usually notice it as missing data, not as a clean error. The HTTP request succeeds. The API returns HTML. Your parser runs. Then the fields are empty because the page shipped a shell like this:

<div id="root"></div>
<script src="/assets/app.js"></script>

For pages like that, dynamic=false gives you a successful response with the wrong content. That is worse than a failure in some pipelines because it can pollute downstream data.

A quick test is to compare the rendered and non-rendered response for the fields you need:

function hasProductPrice(html) {
  return /class="price"|data-testid="price"|itemprop="price"/.test(html);
}

const staticHtml = await fetchStatic(url);
const renderedHtml = await fetchRendered(url);

console.log({
  staticHasPrice: hasProductPrice(staticHtml),
  renderedHasPrice: hasProductPrice(renderedHtml)
});

If rendered output is required for most targets, treat rendering as part of the base request when you estimate cost. Some providers charge the same for rendered and non-rendered requests. Anakin’s URL scraper, for example, lists JavaScript rendering at the same 1-credit cost as a basic scrape. Others charge a multiplier.

Neither model is universally better. The cheaper static option works well when you control the target list and know the pages do not need a browser. Flat pricing is easier to budget when your target mix changes or product pages vary by region, device, or A/B test.

Proxy tiers can change the math again

The next multiplier is usually proxy type.

Datacenter proxies are cheaper, but some sites block them quickly. Residential proxies cost more, but they often get through sites that reject obvious automation. In Scrapingdog’s public docs, premium residential proxies cost 10 credits, or 25 credits when combined with rendering.

That matters because hard pages tend to need both. A site that blocks basic requests may also require JavaScript rendering. Your “1 credit per URL” estimate can become 25 credits per URL for the exact pages that your business actually cares about.

You can model this before committing:

const workload = {
  staticPages: 60_000,
  renderedPages: 30_000,
  premiumRenderedPages: 10_000
};

const credits =
  workload.staticPages * 1 +
  workload.renderedPages * 5 +
  workload.premiumRenderedPages * 25;

console.log(credits); // 460000
console.log(credits / 100_000); // 4.6 average credits per URL

The average request cost is what hits your bill, not the cheapest row in the table.

Failures, caching, and batches are billing features too

Two APIs can charge the same per successful scrape but behave differently around repeated work.

Check these details:

• Do failed requests cost credits?
• What counts as a failure: timeout, 403, CAPTCHA page, empty extraction?
• Does the API cache recent fetches?
• Can you submit batches, or do you pay request overhead one URL at a time?
• Does async mode return a job ID you can retry safely?

A typical async scraping flow looks like this:

# Submit work
curl -X POST "https://api.example.com/scrape" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com","render":true}'

# Response
# { "jobId": "job_abc123" }

# Poll result
curl "https://api.example.com/scrape/jobs/job_abc123" \
  -H "Authorization: Bearer $API_KEY"

# { "status": "completed", "html": "..." }

This pattern is easier to operate than long blocking requests, but you still need idempotency. If your worker crashes after submitting the job and before storing the jobId, you may submit the same URL again. A 24-hour cache or your own URL/job table can prevent duplicate spend.

Authenticated scraping needs a credential plan

Cookie headers work until they do not. Sessions expire. Sites rotate CSRF tokens. Some flows require 2FA. If developers paste cookies into environment variables, those cookies eventually end up in logs, crash dumps, or shared config.

For authenticated extraction, decide where credentials live before you build the scraper. A minimal internal table might look like this:

create table scrape_identity (
  id uuid primary key,
  site text not null,
  username text not null,
  encrypted_secret bytea not null,
  created_at timestamptz not null default now(),
  rotated_at timestamptz
);

You also need rules for who can read, rotate, and use those identities. The scraper should receive short-lived session material, not raw passwords, when possible.

If you need managed identities for authenticated web actions rather than passing cookies through request headers, Wire treats credentials as named identities backed by a vault instead of returning secrets from an API.

That is not necessary for every scraper. Public pages do not need it. Small internal jobs may be fine with a locked-down secret manager and careful logging. But once multiple services or teams use the same authenticated targets, credential handling becomes part of the scraping architecture, not a detail.

A practical comparison checklist

Before choosing a scraping API, run this against your own URLs:

1. Pick 100 representative pages, including the annoying ones.
2. Fetch each page with rendering off and rendering on.
3. Validate the actual fields you need, not just HTTP 200.
4. Record credits used per mode.
5. Repeat a subset to see whether caching changes cost.
6. Test blocked pages and confirm whether they are billed.
7. Test one authenticated flow if your product needs it.

Put the result in a small table with url, mode, status, fields_found, credits_used, and error. That table will tell you more than any pricing page.

The next useful step: take your current scraper target list, sample 50 URLs, and calculate the average credits per valid extraction rather than the advertised credits per request.

Comments

[Lucas Him]

The root-only HTML shell is the costly false success here. For BrowserAct Cloud exports, I'd gate a run on required-field coverage and keep failed samples separate from empty-but-valid results; that keeps cost-per-valid-row honest.

[Paul Irolla]

The same trap sits in model APIs: the default you never set can be the line that bills.