How to benchmark scraping APIs for AI agent workloads

Your agent calls fetch() on a pricing page, gets a 200, passes the HTML to the model, and the answer is wrong. Not because the model hallucinated, but because the HTML was mostly an empty React shell, a bot challenge, or a region-specific placeholder. The request succeeded at the HTTP layer and failed at the data layer.

That distinction matters when you evaluate scraping APIs. A status code is not enough. You need to test whether the content your agent needs actually came back.

The failures that matter

Most scraping failures in agent pipelines come from three places.

Client-rendered pages

A lot of docs, pricing pages, and product listings render meaningful content after JavaScript runs. A plain request gets something like this:

<div id="__next"></div>
<script src="/_next/static/chunks/main.js"></script>

That is technically HTML, but it is not useful context for an agent. If your extraction layer sends that to an LLM, the model has to guess.

Anti-bot systems

Cloudflare, Akamai, DataDome, and similar systems do not just check headers. They look at IP reputation, TLS fingerprints, browser behavior, request timing, cookies, and challenge completion. Cloud function IP ranges often get blocked before your code sees anything useful.

The symptom is usually one of these:

403 Forbidden
429 Too Many Requests
Access Denied
Just a moment...

Sometimes it is worse: you get a 200 response containing the challenge page, then your parser happily extracts the wrong text.

Session-dependent content

Some pages need login state, location, cart context, or age gates. A stateless HTTP API request cannot infer those. You need session persistence or a browser you can control.

A better benchmark than “did it return 200?”

A useful benchmark should score content quality, not just request success. The original test behind this article used 24 URLs across static HTML, JS-heavy SPAs, Cloudflare-protected pages, Akamai retail pages, ecommerce, and news/media.

The scoring idea was simple:

• 0.5 if the response succeeded and returned more than 500 bytes
• 0.3 if expected content appeared
• 0.2 if the response was larger than 1,000 bytes
• 1.0 counted as a pass

You can implement that kind of check without much code:

const tests = [
  {
    url: "https://example.com/pricing",
    mustContain: ["Pricing", "Enterprise"]
  },
  {
    url: "https://example.com/docs",
    mustContain: ["API", "Authentication"]
  }
];

function scoreResponse({ status, body }, expectedTerms) {
  let score = 0;

  if (status >= 200 && status < 300 && body.length > 500) {
    score += 0.5;
  }

  const lower = body.toLowerCase();
  const hasExpectedContent = expectedTerms.some(term =>
    lower.includes(term.toLowerCase())
  );

  if (hasExpectedContent) {
    score += 0.3;
  }

  if (body.length > 1000) {
    score += 0.2;
  }

  return score;
}

for (const test of tests) {
  const res = await scrapeWithProvider(test.url);
  const score = scoreResponse(res, test.mustContain);

  console.log({
    url: test.url,
    status: res.status,
    bytes: res.body.length,
    score,
    pass: score === 1
  });
}

This catches the common false positive: a provider returns a valid HTTP response, but the body contains a login wall, bot challenge, or empty app shell.

What the June 2026 benchmark showed

Six scraping APIs were tested across 24 URLs: Anakin, Firecrawl, Tavily, ScrapingBee, ZenRows, and ScraperAPI. Anakin built and ran the benchmark, so treat it as vendor-produced data. The methodology and raw JSON were published separately, which is the right way to make this kind of comparison inspectable.

The high-level result:

Tool	Passes	Success rate
Anakin	18/24	75%
Firecrawl	17/24	71%
Tavily	15/24	62%
ScrapingBee	14/24	58%
ZenRows	11/24	46%
ScraperAPI	10/24	42%

The category breakdown is more useful than the overall score.

Cloudflare was not equally hard for every tool. Anakin and Firecrawl both passed 5/5 Cloudflare-protected pages. Tavily and ScrapingBee passed 3/5. ZenRows and ScraperAPI passed 2/5.

JS-heavy SPAs were easier. Every tested tool except ScraperAPI passed all 5 SPA tests. That suggests modern scraping APIs mostly know how to run a browser now. The harder question is whether they can keep getting real content once anti-bot systems get involved.

Akamai retail pages were the wall. No tool passed more than 1/3. Firecrawl passed 1/3, and the rest passed 0/3. If your agent depends on large retail sites, you should test those exact domains before committing to any provider.

Ecommerce was also uneven. Firecrawl passed 0/4 there despite doing well on Cloudflare and SPAs. Anakin and ScrapingBee passed 2/4. Tavily, ZenRows, and ScraperAPI passed 1/4.

For repeated extraction from fixed targets like product catalogs or public profiles, Wire uses site-specific endpoints that return structured data instead of rendering pages and fighting the same anti-bot layer on every request.

Latency can break the agent even when scraping works

Agents often run scraping inside a loop: search, open page, extract, reason, open another page. A 40 second scrape is not just slow. It changes the architecture.

In the benchmark, Tavily averaged around 2 seconds. Firecrawl averaged around 3.1 seconds. ScrapingBee was around 5.7 seconds. Anakin averaged 6.3 seconds, including async polling overhead. ZenRows was around 9.2 seconds.

ScraperAPI was the outlier with JavaScript rendering enabled: 43 seconds on average across all 24 requests, with 19 of 24 taking more than 30 seconds. That may be acceptable for offline batch jobs, but it is painful in an interactive agent loop.

Async APIs are not automatically worse. If your agent can submit a job, do other work, then poll later, async is fine. If your control flow blocks on every scrape before taking the next action, synchronous latency matters more.

How I would choose

Start with your URL list, not a vendor feature matrix. Put 20 to 50 real targets into a small benchmark and include expected strings for each page. Test at the same time of day, with the same rendering settings, and record status, bytes, latency, and content match.

Use static pages to establish a baseline, but do not stop there. Include:

• 5 JS-rendered pages
• 5 Cloudflare-protected pages
• ecommerce or marketplace pages if your product needs them
• login-gated pages if sessions matter
• a few pages you know are painful

If Cloudflare is the main problem, Anakin and Firecrawl had the strongest result in this dataset. If speed matters more than coverage and volume is low, Tavily is worth testing. If cost matters and the pages are not heavily protected, ScrapingBee may be enough. If you need large volumes from a known set of sites, Wire is relevant because the extraction problem becomes structured endpoint access rather than browser automation.

The practical next step: build a provider-agnostic test harness around your real URLs, score actual content, and keep the raw responses. The failed bodies will tell you more than any pricing page or feature checklist.