Web Scraping API Pricing Compared: Pay-Per-Result vs Subscription vs Per-GB

If you are about to commit to a web scraping API, the pricing model matters more than the per-request rate. Pay-per-result charges for each successfully delivered record, subscription charges a flat monthly fee for a request quota, and per-GB charges for the bandwidth the provider uses on your behalf. Each model fits a different traffic shape, and each hides a different failure mode — the subscription you underuse, the per-GB bucket you overshoot, or the pay-per-result invoice that surprises you when retries stack up. This guide walks through the three models with the same evaluation axes so you can pick the one that matches your workload before you sign up.

TL;DR

• Pay-per-result is best when record volume is unpredictable and your priority is paying only for clean, parsed outputs.
• Subscription is best when traffic is steady, predictable, and you want a known monthly line item.
• Per-GB is best when payloads are large, formats vary, or you consume a lot of full-page renders and proxy bandwidth.
• The right choice is rarely the cheapest per-record headline price; it is the model that matches the variance, retries, and failure modes of your actual workload.

Why pricing model beats headline rate

Most comparison pages stop at the per-request or per-record rate. That misses three real cost drivers:

1. Retry cost. When a target site returns a partial response or a CAPTCHA challenge, a pay-per-result bill may not charge for the failure, while a per-GB bill still charges for the bandwidth you burned. A subscription typically charges you whether the call succeeded or not.
2. Bandwidth shape. A SERP result is small. A full-page render, a residential proxy session, or a video metadata response can be hundreds of kilobytes. Per-GB vendors accrue cost on every byte, including images and CSS you did not ask for.
3. Tier jump. Subscription pricing often includes a soft cap that triggers an automatic overage tier or a hard throttle. Pay-per-result scales linearly with usage — easier to forecast but harder to cap.

A practical comparison therefore includes the variance of your workload, not just the rate card. Treat any advertised price as a starting point and verify the current value on the provider's official pricing page before you budget.

The three pricing models explained

Pay-per-result (record-based)

You pay per successfully delivered record. The provider absorbs bandwidth, proxy rotation, parsing, and retries for the same record. If the record fails validation, you typically do not pay for it.

Strengths:

• Clean alignment between invoice and business outcome. Each record maps to a lead, a product, or a row.
• No cost when you idle. If you stop scraping, your bill flattens.
• Proxy, browser rendering, and CAPTCHA handling are usually bundled.

Watch for:

• Per-record rates are higher than per-request rates on raw APIs. The bundled infrastructure has to be paid for somewhere.
• Some vendors define a "result" loosely (per page, per item) — clarify the unit before you benchmark.
• Retries can multiply records if the parser returns partial output. Ask whether the provider deduplicates.

Subscription (request-based quota)

You pay a flat monthly fee for a request quota. Additional requests either roll over, get throttled, or trigger overage charges at a stated rate.

Strengths:

• Predictable monthly line item, which finance teams prefer.
• Often the lowest per-request rate at high, steady utilization.
• Easier capacity planning when traffic is forecastable.

Watch for:

• Idle quota is wasted spend. If usage is seasonal or project-driven, you still pay in slow months.
• Overage tiers can be steep. Read the contract end and look for the auto-upgrade trigger.
• Quotas sometimes reset on a calendar month, not a rolling 30-day window. That affects month-end spikes.

Per-GB (bandwidth-based)

You pay for the bytes the provider fetches on your behalf, including pages, assets, headers, and proxy overhead.

Strengths:

• Natural fit for large payloads — full pages, image-heavy sites, residential sessions.
• Marginal cost is near zero per request, so exploratory scraping becomes cheap.
• Often the simplest pricing when the scraping layer is in your hands and the provider is just transport.

Watch for:

• Asset bloat (CSS, fonts, tracking pixels) inflates the bill. A 50 KB HTML page can become 1.5 MB once rendered assets are added.
• Retries, redirects, and 404s still consume bandwidth even when nothing useful is returned.
• The model rewards engineering that strips assets — a cost lever you may not want to pull on your side.

Side-by-side comparison

Dimension	Pay-per-result	Subscription	Per-GB
Best for	Outcome-based workloads (leads, products, listings)	Predictable, steady ingestion pipelines	Large payloads and exploratory scraping
Setup model	Send record spec; receive JSON	Send requests against quota; receive raw or parsed	Send URLs; receive raw HTML or rendered output
Data coverage	Targeted records (parsed)	Raw or parsed depending on plan	Raw HTTP / rendered output only
Output format	Structured JSON or CSV	JSON / HTML	HTML / binary
Maintenance burden	Low — vendor parses	Medium — you parse	High — you parse and clean
Integration path	Direct to data store	Queue + worker	Pipeline + filter stage
Quota or freshness	Unlimited by default	Monthly quota + overage	Bandwidth ceiling per cycle
Cost variance	Linear with success records	Fixed monthly + variable overage	Linear with payload size
Pricing verification source	Vendor's record-based pricing page	Vendor's plan-and-quota page	Vendor's bandwidth-pricing page

A reusable cost-calculator workflow

The fastest way to evaluate these models in your own context is to convert your workload into the same unit each vendor uses. Use environment variables for the inputs and rates so the script stays portable across providers.

"""
Estimate monthly cost under three pricing models for the same workload.

Inputs are read from environment variables so the script does not hard-code
any provider name, endpoint, or current rate. Use the verified, current
values from the vendor's official pricing page; replace the placeholders.
"""

import os


def env_float(name: str, default: float = 0.0) -> float:
    raw = os.environ.get(name)
    if raw is None or raw == "":
        return default
    return float(raw)


def main() -> None:
    # --- Workload shape (replace with your measured values) ---
    requests_per_month = env_float("REQUESTS_PER_MONTH", 100_000)
    records_per_request = env_float("RECORDS_PER_REQUEST", 10.0)
    success_rate = env_float("SUCCESS_RATE", 0.85)  # 0.0 to 1.0
    avg_payload_kb = env_float("AVG_PAYLOAD_KB", 250.0)
    retry_multiplier = env_float("RETRY_MULTIPLIER", 1.2)

    # --- Vendor rates (set per provider from their official pricing page) ---
    # Pay-per-result: USD per 1,000 successful records.
    rate_per_result_per_1k = env_float("RATE_PER_RESULT_PER_1K", 0.0)
    # Subscription: USD per month + USD per 1,000 requests over the quota.
    subscription_fee = env_float("SUBSCRIPTION_FEE", 0.0)
    subscription_quota = env_float("SUBSCRIPTION_QUOTA", 200_000)
    overage_per_1k = env_float("OVERAGE_PER_1K", 0.0)
    # Per-GB: USD per GB of bandwidth.
    rate_per_gb = env_float("RATE_PER_GB", 0.0)

    # --- Derived totals ---
    total_records = (
        requests_per_month * records_per_request * success_rate * retry_multiplier
    )

    # Pay-per-result cost.
    ppr_cost = (total_records / 1_000.0) * rate_per_result_per_1k

    # Subscription cost.
    billed_requests = requests_per_month * retry_multiplier
    overage_requests = max(0.0, billed_requests - subscription_quota)
    sub_cost = subscription_fee + (overage_requests / 1_000.0) * overage_per_1k

    # Per-GB cost.
    total_gb = (requests_per_month * retry_multiplier * avg_payload_kb) / 1024.0 / 1024.0
    gb_cost = total_gb * rate_per_gb

    summary = {
        "workload": {
            "requests_per_month": requests_per_month,
            "records_per_request": records_per_request,
            "success_rate": success_rate,
            "avg_payload_kb": avg_payload_kb,
            "retry_multiplier": retry_multiplier,
        },
        "pay_per_result_usd": round(ppr_cost, 2),
        "subscription_usd": round(sub_cost, 2),
        "per_gb_usd": round(gb_cost, 2),
    }

    print(__import__("json").dumps(summary, indent=2))


if __name__ == "__main__":
    main()

Run it with measured inputs and the rates pulled from the official pricing pages you are comparing. The script returns a JSON object so you can drop the output straight into a spreadsheet or a dashboard panel.

Example output

The numbers below are illustrative — substitute the rates from the vendor you are evaluating. They are not CoreClaw rates and they are not a quote for any third-party provider.

{
  "workload": {
    "requests_per_month": 100000,
    "records_per_request": 10,
    "success_rate": 0.85,
    "avg_payload_kb": 250,
    "retry_multiplier": 1.2
  },
  "pay_per_result_usd": 1020.0,
  "subscription_usd": 540.0,
  "per_gb_usd": 286.0
}

In this shape, a steady ingestion workload running near the subscription quota looks cheapest on paper but is also the most fragile — one spike pushes you into overage, and idle months still bill. Pay-per-result scales linearly with records, which is what you want when record volume is the unit your team plans against.

Picking a model by use case

Use case	Recommended model	Why
B2B lead generation, sales prospecting	Pay-per-result	Billable unit is the record; budget per lead is known
Daily SERP rank tracking on a fixed keyword set	Subscription	Steady, forecastable traffic; flat fee wins
Long-tail product research with bursty crawls	Pay-per-result	Idle days should not bill; success is the unit
Image-heavy e-commerce scraping	Per-GB	Payloads are large and bandwidth is the limiting resource
Compliance and policy archives (one-off captures)	Per-GB	One-time captures with large PDF or HTML footprint
Continuous monitoring with seasonal spikes	Subscription with overage cap	Predictable base, defined blow-up cost
AI agent pipelines that fetch context on demand	Pay-per-result	Records are the unit the agent consumes

If you are running multi-tenant data products, split the bill between the team and the unit the team owns. Sales can budget per lead. SEO can budget per keyword. Engineering can budget per GB. That split is the real reason pricing model matters more than headline rate.

Limits, freshness, and compliance

A pricing comparison is only useful if the numbers stay valid. Three things change fast:

• Rate cards. Vendors change plans quarterly. Treat any rate in this article as a starting point and re-verify on the provider's official pricing page before you commit.
• Quotas. Subscription tiers are updated, soft caps shift, and overage thresholds change. Read the current plan terms, not last quarter's blog post.
• Coverage. Pay-per-result pricing depends on what the vendor defines as a record. Pages with thousands of nested items can produce wildly different record counts on different vendors.

Compliance is independent of the model but worth pinning down:

• Public web data collection still requires that you respect the target site's terms, robots directives where relevant, applicable privacy law, and any contractual terms that apply to your use case.
• Pricing that bundles residential proxies does not absolve the buyer of the responsibility to use those proxies lawfully.
• For a structured walkthrough of the legal, ethical, and operational considerations around public web data — including freshness, attribution, and audit trails — the CoreClaw Web Data compliance guide lays out the patterns in Chinese, useful as a second-language reference for global teams.

FAQ

Which model is cheapest for a startup?

For most early-stage workloads where traffic is bursty and record volume is small, pay-per-result is the cheapest because you do not pay for idle days. Subscription wins only when utilization stays above ~70% of the quota every month.

Should I pick subscription or pay-per-result for rank tracking?

Subscription. Rank tracking is steady, daily, and predictably sized to your keyword set. A flat monthly fee makes capacity planning easier than tracking per-record invoices.

How do I estimate retry overhead?

Run a one-week pilot with the provider at your real traffic shape. Measure successful records, failed records, and bytes consumed. Feed those numbers into a cost calculator like the one above. Vendors' published retry policies are guidelines, not guarantees.

What happens when my site layout changes?

Pricing does not change, but record quality does. A pay-per-result vendor absorbs the parsing cost; a per-GB vendor still bills for the bandwidth while you fix your parser. Treat layout-change risk as a separate line item in your cost model.

Can I switch models mid-contract?

Usually yes, but watch for grandfathered rates. Some vendors lock pricing at signup; others keep the new plan on the new rate. Read the renewal terms before you commit to a year.

How do I compare a bundled quote to an unbundled quote?

Force them onto the same unit. Compute the implied per-record rate for a subscription plan by dividing the monthly fee by the records you actually deliver. Compute the implied per-GB rate for a pay-per-result plan by dividing the bill by the bytes the provider fetched. Once the units match, the comparison is honest.

Summary

Web scraping API pricing is a model problem before it is a rate problem. Pay-per-result aligns cost with outcomes and rewards efficient, targeted scraping. Subscription rewards steady, forecastable traffic and gives finance a clean monthly number. Per-GB rewards large payloads and engineering that keeps bandwidth lean. Pick the model that matches the variance and failure modes of your workload, then verify the current rates on each vendor's official page before you sign.

For teams whose unit of value is the record, CoreClaw's pay-per-result scraper pricing is built around that model. If you want to see the live catalog of ready-made workers, the CoreClaw product store lists 100+ scrapers that already map to these cost shapes, and you can try the workflow yourself from the CoreClaw Workers console.

Related reading

• Residential Proxy Scraper: How to Scrape the Web Without Managing Proxies — the mechanism that powers most pay-per-result vendors today.
• Build vs Buy Web Scraping: When to Use a Scraper API Instead of Self-Hosting — the architectural decision that precedes the pricing decision.
• No-Code Web Scraping Tools: How to Choose a Ready-Made Scraper Marketplace — how marketplace pricing and per-worker pricing differ from per-record pricing.