<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Sid Mishra</title>
    <description>The latest articles on DEV Community by Sid Mishra (@sid_mishra).</description>
    <link>https://dev.to/sid_mishra</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4154228%2F618374b2-57ca-4e4e-9bd4-c3ae889409a9.jpg</url>
      <title>DEV Community: Sid Mishra</title>
      <link>https://dev.to/sid_mishra</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/sid_mishra"/>
    <language>en</language>
    <item>
      <title>Scrape a Cloudflare-protected site for $0.40 per 1,000 pages: the session rung</title>
      <dc:creator>Sid Mishra</dc:creator>
      <pubDate>Mon, 09 Nov 2026 19:30:00 +0000</pubDate>
      <link>https://dev.to/sid_mishra/scrape-a-cloudflare-protected-site-for-040-per-1000-pages-the-session-rung-887</link>
      <guid>https://dev.to/sid_mishra/scrape-a-cloudflare-protected-site-for-040-per-1000-pages-the-session-rung-887</guid>
      <description>&lt;p&gt;I kept watching scraping bills grow faster than page counts, and the culprit was almost always the same: a Cloudflare "Just a moment…" page, answered by sending every single page through a real browser, often from a residential IP, at the premium rate. So when I built MeshArc, I wanted a Cloudflare-walled site to cost a browser render once, not ten thousand times. Here's how that works, what it costs, and where it doesn't help.&lt;/p&gt;

&lt;p&gt;The short answer: one real browser waits out the challenge once, earns the site's session, and every page after it is read with plain HTTP carrying those cookies. That is the second rung of the ladder, "fetch with session", and it costs 2 credits a page. On the Growth plan that is $0.40 per 1,000 pages.&lt;/p&gt;

&lt;p&gt;This post explains how the rung works, what it costs at each plan, where it stops working, and how it compares with the alternatives, with every competitor figure checked on 26 October 2026.&lt;/p&gt;

&lt;h2&gt;
  
  
  The wall: what a Cloudflare managed challenge actually asks for
&lt;/h2&gt;

&lt;p&gt;A managed challenge is not a captcha you have to solve. It is an interstitial that runs JavaScript in the visitor's browser, decides the visitor looks like a browser, and sets a clearance cookie. Plain HTTP clients fail it because they never run the script. A headless browser usually passes it, because it does.&lt;/p&gt;

&lt;p&gt;That gives you two expensive options and one cheap one:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Render every page.&lt;/strong&gt; Correct, slow, and priced at the render rate on every page.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Render every page from a residential IP.&lt;/strong&gt; Also correct, slower, and priced at the residential rate on every page.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Render once, then reuse the clearance.&lt;/strong&gt; The browser does the one thing only a browser can do, and HTTP does the rest.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;MeshArc does the third automatically. You do not configure a session, store cookies or write retry logic.&lt;/p&gt;

&lt;h2&gt;
  
  
  How the session rung works
&lt;/h2&gt;

&lt;p&gt;Every page starts at the cheapest method and climbs one rung per refusal, never two. On a Cloudflare-protected site the first page goes like this:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Fetch (1 credit)&lt;/strong&gt; — plain HTTP. Cloudflare answers with a challenge. The page is refused, so this attempt is free.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Render (4 credits)&lt;/strong&gt; — a real Chrome loads the page and waits the challenge out. No solver is involved. On MeshArc's recorded run this took about 6 seconds, and 10 out of 10 Cloudflare managed challenges were passed, direct and via a residential exit.&lt;/li&gt;
&lt;li&gt;The browser now holds the site's session. From the next page onward the request is &lt;strong&gt;fetch with session (2 credits)&lt;/strong&gt;: the same HTTP request, carrying the cookies the browser earned.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The rung that worked is remembered for the site across runs, so next week's crawl starts at the session rung rather than climbing from the bottom again.&lt;/p&gt;

&lt;p&gt;The arithmetic for a 100-page Cloudflare site, from the &lt;a href="https://mesharc.dev/docs/billing" rel="noopener noreferrer"&gt;billing docs&lt;/a&gt;: one render plus 99 session reads, &lt;strong&gt;4 + 198 = 202 credits&lt;/strong&gt;. A render-every-page approach costs 400 credits for the same 100 pages on MeshArc's own ladder, so the session rung roughly halves it. Against residential rendering (16 credits a page, 1,600 for the site) it is about an eighth.&lt;/p&gt;

&lt;h2&gt;
  
  
  What 1,000 pages cost, plan by plan
&lt;/h2&gt;

&lt;p&gt;For a 1,000-page Cloudflare site read with one render and 999 session reads, the bill is 4 + 1,998 = &lt;strong&gt;2,002 credits&lt;/strong&gt;. Converted at each plan's included-credit rate from &lt;a href="https://mesharc.dev/pricing" rel="noopener noreferrer"&gt;pricing&lt;/a&gt;:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Plan&lt;/th&gt;
&lt;th&gt;Price&lt;/th&gt;
&lt;th&gt;Credits&lt;/th&gt;
&lt;th&gt;Cost of 2,002 credits&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Free&lt;/td&gt;
&lt;td&gt;$0&lt;/td&gt;
&lt;td&gt;1,000 once&lt;/td&gt;
&lt;td&gt;Covers about 499 pages of such a site&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Starter&lt;/td&gt;
&lt;td&gt;$29/mo&lt;/td&gt;
&lt;td&gt;100,000&lt;/td&gt;
&lt;td&gt;$0.58&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Growth&lt;/td&gt;
&lt;td&gt;$99/mo&lt;/td&gt;
&lt;td&gt;500,000&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;$0.40&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Scale&lt;/td&gt;
&lt;td&gt;$349/mo&lt;/td&gt;
&lt;td&gt;2,500,000&lt;/td&gt;
&lt;td&gt;$0.28&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Two details matter here. First, fetch with session is on every plan and render is on Free and up, so a site that only needs a managed challenge waited out is readable on the free plan. Residential rendering (16 credits) needs Growth or higher. Second, refused pages are free "whatever the ladder spent finding out", which is why the failed plain fetch on page one does not appear in the sum.&lt;/p&gt;

&lt;h2&gt;
  
  
  How this compares (checked on 26 October 2026)
&lt;/h2&gt;

&lt;p&gt;Per 1,000 pages of a walled site, at roughly the $99 price point, using each vendor's live pricing page and docs:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Vendor and plan&lt;/th&gt;
&lt;th&gt;How a walled page is read&lt;/th&gt;
&lt;th&gt;Per 1,000 pages&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;MeshArc Growth, $99 / 500k credits&lt;/td&gt;
&lt;td&gt;One render, then session at 2 credits&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;$0.40&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Firecrawl Standard, $83/mo annual or $99 monthly / 100k credits&lt;/td&gt;
&lt;td&gt;1 credit a page, enhanced proxy included&lt;/td&gt;
&lt;td&gt;$0.83 annual, $0.99 monthly&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;ZenRows Launch, $99 / 500k credits&lt;/td&gt;
&lt;td&gt;JS rendering 5 credits; JS + premium proxy 25&lt;/td&gt;
&lt;td&gt;$0.99 rendered; $4.95 rendered + premium&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;ScrapingBee Startup, $99 / 1M credits&lt;/td&gt;
&lt;td&gt;Premium proxy + JS 25 credits; stealth proxy 75&lt;/td&gt;
&lt;td&gt;$2.48 premium; $7.43 stealth&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Where the others are better, said plainly:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Firecrawl's price is flat and simple.&lt;/strong&gt; One credit a page, and its docs say enhanced proxy and auto-escalated retries carry no surcharge. If a site needs heavy treatment on every page, a flat $0.83 can beat a ladder that has to climb. The trade-off is that Firecrawl bills a page that comes back 403 or 404 at 1 credit; MeshArc bills neither. We worked through that in &lt;a href="https://mesharc.dev/blog/firecrawl-pricing-explained" rel="noopener noreferrer"&gt;Firecrawl pricing explained&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;ScrapingBee and ZenRows are cheaper for plain pages.&lt;/strong&gt; ScrapingBee's Startup plan works out at about $0.10 per 1,000 one-credit requests, against MeshArc's $0.20. If the site is not walled, the session rung is irrelevant and their base rate wins.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;ScrapingBee's Auto-Mode&lt;/strong&gt; also escalates cheapest-first and, per its docs, costs 0 credits when every configuration fails. MeshArc is not alone in escalating. What it adds is replaying the earned session at a lower price instead of paying the premium rate on every page.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The full side-by-side, with the dated snapshot, is on &lt;a href="https://mesharc.dev/compare" rel="noopener noreferrer"&gt;compare&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Running it: a crawl that stays on the cheap rung
&lt;/h2&gt;

&lt;p&gt;Install the client:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;pip &lt;span class="nb"&gt;install &lt;/span&gt;mesharc
&lt;span class="c"&gt;# or: npm install mesharc&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then crawl, cap what any one page may cost, and count which engine read each page:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;collections&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Counter&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;mesharc&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;MeshArc&lt;/span&gt;

&lt;span class="n"&gt;arc&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;MeshArc&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;  &lt;span class="c1"&gt;# reads MESHARC_API_KEY
&lt;/span&gt;
&lt;span class="n"&gt;job&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;arc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;crawl&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://www.example.com&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;            &lt;span class="c1"&gt;# a Cloudflare-protected site
&lt;/span&gt;    &lt;span class="n"&gt;limit&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;1000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;                            &lt;span class="c1"&gt;# page budget
&lt;/span&gt;    &lt;span class="n"&gt;crawlMode&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;sitemap_first&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;             &lt;span class="c1"&gt;# declared pages first, then links
&lt;/span&gt;    &lt;span class="n"&gt;maxTier&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;browser&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;                     &lt;span class="c1"&gt;# never climb to residential
&lt;/span&gt;    &lt;span class="n"&gt;scrapeOptions&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;formats&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;markdown&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]},&lt;/span&gt;
    &lt;span class="n"&gt;config&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;max_credits_per_page&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;         &lt;span class="c1"&gt;# anything dearer is TIER_LIMIT, at 0
&lt;/span&gt;        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;crawl_delay_ms&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;            &lt;span class="c1"&gt;# the default; be a polite visitor
&lt;/span&gt;    &lt;span class="p"&gt;},&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;engines&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Counter&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;page&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;job&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;pages&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
    &lt;span class="n"&gt;engines&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;page&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;method&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]]&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;page&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;errorCode&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;OK&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;page&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;url&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;page&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;verdict&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;page&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;errorCode&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;page&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;credits&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;

&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;engines&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;                       &lt;span class="c1"&gt;# expect one "browser", the rest "minted"
&lt;/span&gt;&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;job&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;envelope&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;creditsUsed&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;method&lt;/code&gt; is the engine that read the page: &lt;code&gt;browser&lt;/code&gt; for the render, &lt;code&gt;minted&lt;/code&gt; for the session rung (the docs also call it &lt;code&gt;http-jar&lt;/code&gt;). &lt;code&gt;credits&lt;/code&gt; is what that page cost, and &lt;code&gt;creditsUsed&lt;/code&gt; on the job envelope is the running total, so you can check the 4 + 2n arithmetic against your own run rather than taking ours on trust.&lt;/p&gt;

&lt;p&gt;Two settings do the guarding. &lt;code&gt;maxTier="browser"&lt;/code&gt; stops a refused page climbing to the residential, solver or unlocked rungs. &lt;code&gt;max_credits_per_page&lt;/code&gt; is the finer cap: engines dearer than it are never tried, and a page only they could read is recorded as &lt;code&gt;TIER_LIMIT&lt;/code&gt; and costs nothing. You find out that a page needed more, without paying for it.&lt;/p&gt;

&lt;p&gt;If the site turns out to be worth watching, keep the crawl as a project and every later run is diffed against this one:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;project&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;job&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;keep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Walled site&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;schedule&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;weekly&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Where the session rung does not help
&lt;/h2&gt;

&lt;p&gt;Be clear about the limits before you plan a budget around $0.40:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Sites that challenge every request.&lt;/strong&gt; Cloudflare, not MeshArc, decides how long a clearance lasts and whether it is accepted from HTTP. If the session is refused, that page climbs again to a render at 4 credits, so the worst case is the render price, not a failure.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Sites that refuse datacenter IPs outright.&lt;/strong&gt; Those need the residential rung at 16 credits a page (1 MB included, 4 credits per extra MB), which is on Growth and Scale only. The session rung cannot fix an IP reputation problem.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Interactive captchas.&lt;/strong&gt; Turnstile or reCAPTCHA widgets on forms are a separate, opt-in feature (&lt;code&gt;solve_captcha&lt;/code&gt;, 22 credits when a token is bought). hCaptcha is detected and refused: MeshArc does not sell a solver it cannot run reliably.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Logged-in content.&lt;/strong&gt; A session earned from a challenge is not a login. Pages behind a sign-in need a saved login (&lt;code&gt;login_id&lt;/code&gt;); without one they are judged a &lt;code&gt;login_wall&lt;/code&gt;, not read.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Hosted interactive browsers.&lt;/strong&gt; If you need to click through a flow in a live browser you control, a browser-session product is the better tool.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;And a note on manners: a site behind Cloudflare has usually chosen to be careful about automated traffic. Respect robots.txt (MeshArc does, and robots-blocked URLs are free because they are never fetched), keep the crawl delay sensible, and read only what you are entitled to read.&lt;/p&gt;

&lt;h2&gt;
  
  
  The short version
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;A Cloudflare managed challenge needs a browser once, not on every page.&lt;/li&gt;
&lt;li&gt;MeshArc renders once at 4 credits, then reads the rest at 2 credits with the earned session: 202 credits for 100 pages, 2,002 for 1,000.&lt;/li&gt;
&lt;li&gt;That is $0.40 per 1,000 pages on Growth, $0.28 on Scale and $0.58 on Starter, and refused pages and 404s cost nothing.&lt;/li&gt;
&lt;li&gt;Flat-rate APIs are simpler and plain-page APIs are cheaper on unwalled sites. The session rung wins where a site walls you once and then lets you in.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Try it on the site that keeps walling you: paste the URL into the &lt;a href="https://mesharc.dev/playground" rel="noopener noreferrer"&gt;playground&lt;/a&gt; and see which rung read it and what it cost, or &lt;a href="https://mesharc.dev/signup" rel="noopener noreferrer"&gt;start free&lt;/a&gt; with 1,000 credits and no card, which is enough for about 499 pages of a Cloudflare site.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://mesharc.dev/blog/cloudflare-scraping-session-rung" rel="noopener noreferrer"&gt;mesharc.dev&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>webscraping</category>
      <category>python</category>
      <category>api</category>
      <category>tutorial</category>
    </item>
    <item>
      <title>How to keep a RAG knowledge base fresh without re-embedding everything</title>
      <dc:creator>Sid Mishra</dc:creator>
      <pubDate>Wed, 28 Oct 2026 05:50:59 +0000</pubDate>
      <link>https://dev.to/sid_mishra/how-to-keep-a-rag-knowledge-base-fresh-without-re-embedding-everything-18fa</link>
      <guid>https://dev.to/sid_mishra/how-to-keep-a-rag-knowledge-base-fresh-without-re-embedding-everything-18fa</guid>
      <description>&lt;p&gt;Most RAG pipelines I look at are refreshed by brute force: re-crawl the docs site every night, re-chunk everything, re-embed everything, overwrite the index. It works, eventually, but it's wasteful and it hides three bugs a nightly rebuild never fixes.&lt;/p&gt;

&lt;p&gt;Below is what actually goes stale, what a full rebuild really costs (less than you'd think in embeddings, more elsewhere), a do-it-yourself content-hash pattern you can drop into any scraper, and how we handle it in MeshArc, the crawler I'm building. The first half needs no sign-up for anything.&lt;/p&gt;

&lt;h2&gt;
  
  
  What goes wrong when a knowledge base ages
&lt;/h2&gt;

&lt;p&gt;A stale chunk does not announce itself. It still sits close to the query in vector space, so retrieval still ranks it highly and the model still quotes it with confidence. Three failure modes are worth separating:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Stale chunks.&lt;/strong&gt; The page changed; the index holds the old wording. The assistant recommends a deprecated flag or an old price.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Zombie chunks.&lt;/strong&gt; The page was deleted, but its vectors were not. Nightly &lt;em&gt;upserts&lt;/em&gt; never remove anything, so deleted pages live on indefinitely unless something deletes them on purpose.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Flicker re-embeds.&lt;/strong&gt; A cookie banner, a "last updated 3 minutes ago" line or a rotating testimonial makes every page look modified on every run, so a "changed pages only" pipeline quietly turns back into a full rebuild.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;There is a fourth, nastier one: &lt;strong&gt;the blocked crawl that deletes your index.&lt;/strong&gt; If the site rate-limits you halfway through a run and your pipeline treats "not seen this run" as "removed", you wipe half the knowledge base. Any deletion logic needs a guard against partial runs.&lt;/p&gt;

&lt;h2&gt;
  
  
  What a full nightly rebuild costs
&lt;/h2&gt;

&lt;p&gt;Here is the arithmetic for a mid-sized documentation site. The assumptions are ours and stated so you can swap in your own:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;2,000 pages, averaging 1,200 words each.&lt;/li&gt;
&lt;li&gt;Chunks of about 350 words with a 40-word overlap, which gives roughly 4 chunks per page and 8,000 chunks in all.&lt;/li&gt;
&lt;li&gt;About 1.3 tokens per English word.&lt;/li&gt;
&lt;li&gt;OpenAI &lt;code&gt;text-embedding-3-small&lt;/code&gt; at &lt;strong&gt;$0.02 per million input tokens&lt;/strong&gt; (OpenAI's model page, checked on 23 October 2026).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That is roughly 2.8 million embedded words, or 3.7 million tokens, per full rebuild: &lt;strong&gt;about $0.07&lt;/strong&gt;. Thirty nights is about &lt;strong&gt;$2.20 a month&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;So, to be fair to the brute-force approach: with a small embedding model, the embedding bill is not what hurts. A larger model, a bigger corpus or a hosted embedding service with per-request pricing changes that, but the real costs of rebuilding everything are usually elsewhere:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Fetching.&lt;/strong&gt; 2,000 pages × 30 nights is 60,000 page reads a month, every one of which can be blocked, rate-limited or billed.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Index writes.&lt;/strong&gt; 8,000 chunk upserts a night, most of them writing identical vectors over identical vectors. Managed vector databases tend to price writes, and self-hosted ones still pay for them in compaction and I/O.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Correctness.&lt;/strong&gt; A rebuild that overwrites by ID still leaves zombies behind, and a rebuild that drops and recreates the index has a window in which the assistant answers from nothing.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The fix for all of these is the same: find out what changed, and touch only that.&lt;/p&gt;

&lt;h2&gt;
  
  
  The do-it-yourself version: hash, compare, delete deliberately
&lt;/h2&gt;

&lt;p&gt;You do not need any particular vendor for this. The pattern is:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Normalise each page's text (strip boilerplate, collapse whitespace) and hash it.&lt;/li&gt;
&lt;li&gt;Compare against the hash you stored last time.&lt;/li&gt;
&lt;li&gt;Re-chunk and re-embed only pages whose hash moved; give every chunk an ID that starts with the page URL.&lt;/li&gt;
&lt;li&gt;Before adding a changed page's new chunks, delete &lt;em&gt;all&lt;/em&gt; of its old ones. The chunk count can shrink, so overwriting by ID is not enough.&lt;/li&gt;
&lt;li&gt;Delete pages that have disappeared, but only if the run reached most of the site.
&lt;/li&gt;
&lt;/ol&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;hashlib&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pathlib&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Path&lt;/span&gt;

&lt;span class="n"&gt;STATE&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Path&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;hashes.json&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;normalise&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;text&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sub&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Last updated[^\n]*&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;   &lt;span class="c1"&gt;# add your own volatile lines
&lt;/span&gt;    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sub&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;\s+&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt; &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;plan_refresh&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pages&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;expected_urls&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;pages: {url: markdown} from this run. Returns what to embed and delete.&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;
    &lt;span class="n"&gt;old&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;loads&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;STATE&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;read_text&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;STATE&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;exists&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="p"&gt;{}&lt;/span&gt;
    &lt;span class="n"&gt;new&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;u&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;hashlib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sha256&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;normalise&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;encode&lt;/span&gt;&lt;span class="p"&gt;()).&lt;/span&gt;&lt;span class="nf"&gt;hexdigest&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;u&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;pages&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;items&lt;/span&gt;&lt;span class="p"&gt;()}&lt;/span&gt;

    &lt;span class="n"&gt;changed&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;u&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;u&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;h&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;new&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;items&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;old&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;u&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="n"&gt;removed&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;u&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;u&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;old&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;u&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;new&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

    &lt;span class="c1"&gt;# Guard: a run that saw under 90% of the site must not delete anything.
&lt;/span&gt;    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pages&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mf"&gt;0.9&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;expected_urls&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;removed&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;
        &lt;span class="n"&gt;new&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="n"&gt;old&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="n"&gt;new&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;          &lt;span class="c1"&gt;# keep unseen pages' hashes for next time
&lt;/span&gt;
    &lt;span class="n"&gt;STATE&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;write_text&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dumps&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;new&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;changed&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;removed&lt;/span&gt;

&lt;span class="c1"&gt;# changed -&amp;gt; delete chunks with IDs starting f"{url}#", then embed and add new chunks
# removed -&amp;gt; delete chunks with IDs starting f"{url}#"
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two honest caveats. First, this still fetches every page every night to compute the hashes; it saves embedding and index writes, not crawling. Second, the &lt;code&gt;normalise&lt;/code&gt; function is where the work lives. Every site has its own flicker, and you will be adding regexes for months.&lt;/p&gt;

&lt;p&gt;If you are already on LangChain, its indexing API (a record manager plus &lt;code&gt;index()&lt;/code&gt; with incremental cleanup) does the hash-and-skip part for documents you hand it. It does not fetch pages or know whether a crawl was partial, so the first and last steps above remain yours.&lt;/p&gt;

&lt;h2&gt;
  
  
  Skipping the fetch as well: sitemap &lt;code&gt;lastmod&lt;/code&gt;
&lt;/h2&gt;

&lt;p&gt;Many sites publish a &lt;code&gt;lastmod&lt;/code&gt; date per URL in their sitemap. If it is trustworthy, you can skip fetching pages that have not changed at all. "If" is doing real work there: plenty of sitemaps stamp every URL with today's date, and some never update it. A sensible incremental crawler checks how many entries carry &lt;code&gt;lastmod&lt;/code&gt;, falls back to a full crawl when coverage is poor, and runs a full pass periodically anyway to catch pages whose dates lie.&lt;/p&gt;

&lt;h2&gt;
  
  
  How MeshArc does it
&lt;/h2&gt;

&lt;p&gt;This is the part where we describe our own product, so weigh it accordingly. Everything below is from &lt;a href="https://mesharc.dev/docs/configuration" rel="noopener noreferrer"&gt;our docs&lt;/a&gt; and &lt;a href="https://mesharc.dev/pricing" rel="noopener noreferrer"&gt;pricing page&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;A &lt;strong&gt;project&lt;/strong&gt; is a site plus its settings, run on a schedule (&lt;code&gt;hourly&lt;/code&gt;, &lt;code&gt;daily&lt;/code&gt;, &lt;code&gt;weekly&lt;/code&gt; or &lt;code&gt;manual&lt;/code&gt;). Every run is diffed against the previous one, page by page: added, modified, removed, same, withheld. Boilerplate such as cookie banners, newsletter prompts and share bars is stripped by default, and a modified page with fewer than &lt;code&gt;notify_min_words&lt;/code&gt; changed words (default 5) does not count as changed.&lt;/p&gt;

&lt;p&gt;With &lt;code&gt;sitemap_incremental&lt;/code&gt; on, a scheduled run fetches only pages the sitemap shows as new or with a newer &lt;code&gt;lastmod&lt;/code&gt;, and carries the rest forward from the last full run. A full run is forced every &lt;code&gt;sitemap_full_every&lt;/code&gt; runs (default 10), whenever fewer than 90% of entries carry &lt;code&gt;lastmod&lt;/code&gt;, or when a carried page is over 30 days old. &lt;strong&gt;Carried-forward pages cost 0 credits.&lt;/strong&gt; So do refused pages, 404s and 410s.&lt;/p&gt;

&lt;p&gt;Removals are only reported when a run reached at least 90% of the site. Below that, missing pages are &lt;em&gt;withheld&lt;/em&gt;, not removed, so a blocked crawl cannot empty your index.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;pip &lt;span class="nb"&gt;install &lt;/span&gt;mesharc langchain-mesharc
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;mesharc&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;MeshArc&lt;/span&gt;

&lt;span class="n"&gt;arc&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;MeshArc&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;  &lt;span class="c1"&gt;# reads MESHARC_API_KEY
&lt;/span&gt;
&lt;span class="n"&gt;project&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;arc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;projects&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://docs.example.com&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Docs KB&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;schedule&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;daily&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;config&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;include_paths&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/docs/*&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;sitemap_incremental&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;run&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;arc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;runs&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;start&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;project&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;wait&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;   &lt;span class="c1"&gt;# first run is the baseline
&lt;/span&gt;&lt;span class="n"&gt;record&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;arc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;changes&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;project&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;record&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;change&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;counts&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then, after each scheduled run (from the &lt;code&gt;run.finished&lt;/code&gt; webhook or a cron of your own), load only what changed and apply it to any LangChain vector store:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;langchain_mesharc&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;MeshArcLoader&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;refresh&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;store&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;project_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;docs&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;MeshArcLoader&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;mode&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;changes&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;project_id&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;project_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                         &lt;span class="n"&gt;include_removed&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;load&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="n"&gt;gone&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;url&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;docs&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;change&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;removed&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="n"&gt;fresh&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;docs&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;change&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;removed&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;gone&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;store&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;delete&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ids&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;gone&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;fresh&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;store&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;add_documents&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fresh&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ids&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;url&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;fresh&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That keys one document per URL. If you split pages into chunks before embedding, use chunk IDs that start with the URL and delete a page's old chunks first, exactly as in the DIY version. The &lt;a href="https://mesharc.dev/docs/langchain" rel="noopener noreferrer"&gt;LangChain integration docs&lt;/a&gt; have the details.&lt;/p&gt;

&lt;p&gt;If you would rather not run that code at all, MeshArc can write straight into &lt;strong&gt;Pinecone, Qdrant, Weaviate or pgvector&lt;/strong&gt; as a &lt;a href="https://mesharc.dev/docs/connectors-destinations" rel="noopener noreferrer"&gt;destination&lt;/a&gt;. It splits Markdown into paragraph-aligned chunks of about 350 words with a 40-word overlap, embeds them with your own OpenAI-compatible endpoint and key (we add no markup on embeddings), and by default writes only pages classed as added, modified or removed. Destinations are included on every plan.&lt;/p&gt;

&lt;h2&gt;
  
  
  The same docs site, priced
&lt;/h2&gt;

&lt;p&gt;Back to the 2,000-page site, read daily on our Growth plan ($99 for 500,000 credits, so about $0.20 per 1,000 plain fetches). Assume the pages fetch plainly at 1 credit each, and that around 2% change on a typical day — your site's real churn will differ, which is the point of measuring it.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Approach&lt;/th&gt;
&lt;th&gt;Page reads a month&lt;/th&gt;
&lt;th&gt;Credits&lt;/th&gt;
&lt;th&gt;Cost on Growth&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Full re-crawl every night&lt;/td&gt;
&lt;td&gt;60,000&lt;/td&gt;
&lt;td&gt;60,000&lt;/td&gt;
&lt;td&gt;~$11.88&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;sitemap_incremental&lt;/code&gt;, ~40 changed pages a night, plus 3 forced full runs&lt;/td&gt;
&lt;td&gt;~1,080 + 6,000&lt;/td&gt;
&lt;td&gt;~7,080 (plus sitemap reads at 1 credit per file)&lt;/td&gt;
&lt;td&gt;~$1.40&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Embeddings follow the same shape: only the ~40 changed pages a day are re-embedded, instead of all 2,000. Pages that need a real browser cost 4 credits rather than 1, so multiply accordingly; our &lt;a href="https://mesharc.dev/blog/firecrawl-pricing-explained" rel="noopener noreferrer"&gt;Firecrawl pricing breakdown&lt;/a&gt; works through the per-rung maths in more detail.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where this does not fit
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;One-off corpora.&lt;/strong&gt; If you index a site once for a demo, none of this matters. Any scraper will do.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Sites without a usable sitemap.&lt;/strong&gt; Incremental fetching needs &lt;code&gt;lastmod&lt;/code&gt;. Without it, MeshArc still diffs and still writes only changed pages to your store, but it has to fetch every page to know.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Logged-in content.&lt;/strong&gt; Pages behind your own SSO are better pulled from the source system (Confluence, Notion, a CMS export) than scraped.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Self-hosting.&lt;/strong&gt; MeshArc is not open source. If you need the crawler on your own hardware, the DIY approach above with an open-source crawler is the honest answer; our &lt;a href="https://mesharc.dev/compare" rel="noopener noreferrer"&gt;comparison page&lt;/a&gt; lists the trade-offs by vendor.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Try it on your own docs
&lt;/h2&gt;

&lt;p&gt;The quickest test is to point a project at a site you already index, let it run twice, and look at the change counts: that number, not the page count, is what your pipeline should be embedding each night. The &lt;a href="https://mesharc.dev/use-cases/rag-pipeline" rel="noopener noreferrer"&gt;RAG pipeline use case&lt;/a&gt; has the overview; if your assistant reads whole sites at once, the &lt;a href="https://mesharc.dev/tools/llms-txt-generator" rel="noopener noreferrer"&gt;llms.txt generator&lt;/a&gt; is a free alternative to embeddings altogether.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;a href="https://mesharc.dev/playground" rel="noopener noreferrer"&gt;Try a page in the playground&lt;/a&gt;&lt;/strong&gt; · &lt;strong&gt;&lt;a href="https://mesharc.dev/signup" rel="noopener noreferrer"&gt;Start free — 1,000 credits, no card&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;I drafted this post with AI assistance and reviewed and edited it myself; the code and numbers come from our own docs and pricing.&lt;/em&gt;&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://mesharc.dev/blog/keep-rag-knowledge-base-fresh" rel="noopener noreferrer"&gt;mesharc.dev&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>rag</category>
      <category>ai</category>
      <category>python</category>
      <category>webscraping</category>
    </item>
    <item>
      <title>Firecrawl Pricing Explained: What 1,000 Pages Really Cost</title>
      <dc:creator>Sid Mishra</dc:creator>
      <pubDate>Tue, 27 Oct 2026 07:48:58 +0000</pubDate>
      <link>https://dev.to/sid_mishra/firecrawl-pricing-explained-what-1000-pages-really-cost-2am8</link>
      <guid>https://dev.to/sid_mishra/firecrawl-pricing-explained-what-1000-pages-really-cost-2am8</guid>
      <description>&lt;p&gt;I keep getting asked what Firecrawl actually costs per 1,000 pages, and the honest answer is "it depends" — on your plan, your billing cycle, the output formats you request and how many of your URLs turn out to be 404s. So I went through Firecrawl's pricing page and billing docs line by line and worked it all out in dollars, with a small Python estimator at the end you can point at your own workload.&lt;/p&gt;

&lt;p&gt;Every Firecrawl figure below comes from &lt;a href="https://www.firecrawl.dev/pricing" rel="noopener noreferrer"&gt;firecrawl.dev/pricing&lt;/a&gt; and the &lt;a href="https://docs.firecrawl.dev/billing" rel="noopener noreferrer"&gt;Firecrawl billing docs&lt;/a&gt;, &lt;strong&gt;checked on 21 October 2026&lt;/strong&gt;. Prices change; if you are reading this much later, check them again.&lt;/p&gt;

&lt;p&gt;Disclosure: I build MeshArc, which competes with Firecrawl. I have tried to be fair, and I say plainly where Firecrawl is the cheaper choice.&lt;/p&gt;

&lt;h2&gt;
  
  
  The plans, converted to dollars per 1,000 credits
&lt;/h2&gt;

&lt;p&gt;Firecrawl sells monthly credit allowances. Converting each plan to a cost per 1,000 credits is the first step, because a credit is a page only in the simplest case.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Plan&lt;/th&gt;
&lt;th&gt;Monthly billing&lt;/th&gt;
&lt;th&gt;Annual billing (per month)&lt;/th&gt;
&lt;th&gt;Credits/month&lt;/th&gt;
&lt;th&gt;$ per 1,000 credits (monthly / annual)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Free&lt;/td&gt;
&lt;td&gt;$0&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;1,000&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Hobby&lt;/td&gt;
&lt;td&gt;$19&lt;/td&gt;
&lt;td&gt;$16&lt;/td&gt;
&lt;td&gt;5,000&lt;/td&gt;
&lt;td&gt;$3.80 / $3.20&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Standard&lt;/td&gt;
&lt;td&gt;$99&lt;/td&gt;
&lt;td&gt;$83&lt;/td&gt;
&lt;td&gt;100,000&lt;/td&gt;
&lt;td&gt;$0.99 / $0.83&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Growth&lt;/td&gt;
&lt;td&gt;$399&lt;/td&gt;
&lt;td&gt;$333&lt;/td&gt;
&lt;td&gt;500,000&lt;/td&gt;
&lt;td&gt;$0.80 / $0.67&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Scale&lt;/td&gt;
&lt;td&gt;$749&lt;/td&gt;
&lt;td&gt;$599&lt;/td&gt;
&lt;td&gt;1,000,000&lt;/td&gt;
&lt;td&gt;$0.75 / $0.60&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Two things stand out. First, Hobby is nearly four times dearer per credit than Standard, so anyone doing real volume ends up on Standard or above. Second, the annual discount is worth about 16% and is how most comparisons (including ours on &lt;a href="https://mesharc.dev/compare" rel="noopener noreferrer"&gt;/compare&lt;/a&gt;) arrive at "$0.83 per 1,000 pages".&lt;/p&gt;

&lt;p&gt;Firecrawl's concurrency also scales with plan: 2 on Free, 5 on Hobby, 25 on Standard, 50 on Growth, 100 on Scale. If you need throughput rather than volume, that can push you up a tier before credits do.&lt;/p&gt;

&lt;h2&gt;
  
  
  What a credit buys
&lt;/h2&gt;

&lt;p&gt;From the billing docs, the per-page costs that matter for most scraping jobs:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Operation&lt;/th&gt;
&lt;th&gt;Credits&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Scrape or crawl, Markdown/HTML&lt;/td&gt;
&lt;td&gt;1 per page&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Enhanced proxy (including &lt;code&gt;auto&lt;/code&gt; escalation)&lt;/td&gt;
&lt;td&gt;+0 — billed at the base 1 credit&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;JSON format (LLM extraction)&lt;/td&gt;
&lt;td&gt;+4 per page&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;PDF parsing&lt;/td&gt;
&lt;td&gt;+1 per PDF page&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Zero Data Retention&lt;/td&gt;
&lt;td&gt;+1 per page&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Map&lt;/td&gt;
&lt;td&gt;1 per call&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Search&lt;/td&gt;
&lt;td&gt;2 per 10 results, plus scrape costs for each scraped result&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Monitor&lt;/td&gt;
&lt;td&gt;1 per page per check&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The enhanced-proxy line deserves credit. Firecrawl does not charge extra when a page needs a better proxy, and escalated retries are not billed separately. For a site that needs residential-grade routing on every page, that flat 1 credit is very hard to beat — more on that below.&lt;/p&gt;

&lt;p&gt;The JSON line is the one that changes bills. A page with &lt;code&gt;formats: ["json"]&lt;/code&gt; costs 5 credits, not 1. Modifiers stack, so JSON plus Zero Data Retention is 6.&lt;/p&gt;

&lt;h2&gt;
  
  
  The 404 question
&lt;/h2&gt;

&lt;p&gt;This is the line most people miss. From Firecrawl's billing docs: if a document is returned — including a 403 or a 404 — the page costs 1 credit plus any option costs. Only when no document comes back at all (the site never responded) is it free.&lt;/p&gt;

&lt;p&gt;The docs suggest checking &lt;code&gt;metadata.statusCode&lt;/code&gt; to spot blocked URLs. That is good advice, but by then you have paid. On a crawl driven by an old sitemap, or a list of product URLs scraped six months ago, 404s are not rare. And if you asked for JSON, each 404 costs 5 credits.&lt;/p&gt;

&lt;h2&gt;
  
  
  Worked examples in dollars
&lt;/h2&gt;

&lt;p&gt;All Firecrawl prices below use Standard at $0.99 per 1,000 credits (monthly) with the annual $0.83 in brackets.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. 1,000 plain pages, Markdown.&lt;/strong&gt;&lt;br&gt;
1,000 credits → &lt;strong&gt;$0.99&lt;/strong&gt; ($0.83).&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. 1,000 product pages, extracted to JSON.&lt;/strong&gt;&lt;br&gt;
1,000 × 5 = 5,000 credits → &lt;strong&gt;$4.95&lt;/strong&gt; ($4.15). The model cost is included, which is convenient.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. The same 1,000 JSON pages, where 80 URLs are now 404s.&lt;/strong&gt;&lt;br&gt;
Still 5,000 credits → &lt;strong&gt;$4.95&lt;/strong&gt; ($4.15). The 80 dead URLs cost 400 credits between them and returned nothing useful.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. A 50-page PDF manual.&lt;/strong&gt;&lt;br&gt;
50 credits for parsing on top of the scrape → about &lt;strong&gt;$0.05&lt;/strong&gt;. Cheap; PDFs are not where Firecrawl bills hurt.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. Re-checking 2,000 pages weekly with Monitor.&lt;/strong&gt;&lt;br&gt;
2,000 × 4 checks = 8,000 credits a month → &lt;strong&gt;$7.92&lt;/strong&gt; ($6.64), whether or not anything changed.&lt;/p&gt;
&lt;h2&gt;
  
  
  Expiry, rollover and top-ups
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Plan credits reset monthly and &lt;strong&gt;do not roll over&lt;/strong&gt; on Free, Hobby, Standard or Growth. Annual Scale rolls unused credits over one month; annual Enterprise, two.&lt;/li&gt;
&lt;li&gt;Pay-as-you-go top-ups come in $5 increments: 1,000 credits on Hobby, 2,000 on Standard, 2,500 on Growth, 5,000 on Scale. On Standard that is &lt;strong&gt;$2.50 per 1,000 credits&lt;/strong&gt; — 2.5× the plan rate. Top-ups stay on the account until used, but expire if you cancel the subscription.&lt;/li&gt;
&lt;li&gt;Pay-as-you-go needs a paid plan; on Free, requests return HTTP 402 when credits run out.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The practical effect: size your plan to your typical month, not your peak, and treat top-ups as expensive insurance.&lt;/p&gt;
&lt;h2&gt;
  
  
  The same jobs on MeshArc
&lt;/h2&gt;

&lt;p&gt;MeshArc bills by the rung a page actually needed, and does not bill pages that did not deliver. From &lt;a href="https://mesharc.dev/pricing" rel="noopener noreferrer"&gt;mesharc.dev/pricing&lt;/a&gt;, on Growth ($99 for 500,000 credits, so $0.198 per 1,000 credits):&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Rung&lt;/th&gt;
&lt;th&gt;Credits/page&lt;/th&gt;
&lt;th&gt;$ per 1,000 pages&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Fetch&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;$0.20&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Fetch with session (walled sites, after one render)&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;$0.40&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Render (real Chrome)&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;$0.79&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Render, residential (incl. 1 MB)&lt;/td&gt;
&lt;td&gt;16&lt;/td&gt;
&lt;td&gt;$3.17&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Refused pages, 404s and 410s, pages served from cache, URLs blocked by robots.txt and pages carried forward unchanged on an incremental run cost nothing. Structured extraction is 1 credit per 1,000 tokens, with the model bill paid to your own provider.&lt;/p&gt;

&lt;p&gt;Running the examples again:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;1,000 plain pages:&lt;/strong&gt; $0.20 if they fetch plainly, $0.79 if every one needs a browser.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;1,000 JSON pages, 80 of them 404s:&lt;/strong&gt; the 80 cost 0. The other 920, at about 3,000 tokens each (the example on our pricing page), are 3 credits each for extraction plus the read — but you also pay your model provider. If that model bill matters to you, Firecrawl's all-in 5 credits may well be simpler.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A 100-page Cloudflare-walled site:&lt;/strong&gt; one render earns a session, then 99 pages read with it: 202 credits, about &lt;strong&gt;$0.04&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;2,000 pages, 4 runs a month, 20% needing a browser, 2% residential:&lt;/strong&gt; our pricing page puts that at about 15,200 credits, or roughly &lt;strong&gt;$3.01&lt;/strong&gt;. Firecrawl Monitor on the same pages is $7.92 monthly — though Monitor gives you diffs, and so does MeshArc, so compare like with like.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2&gt;
  
  
  Where Firecrawl is cheaper
&lt;/h2&gt;

&lt;p&gt;Be honest about the residential case. A page that needs a residential IP costs MeshArc 16 credits ($3.17 per 1,000 on Growth). Firecrawl charges 1 credit with enhanced proxy, so $0.83–$0.99 per 1,000. If your whole job is hard sites that need residential routing on every page and cannot be read with a session, Firecrawl is three to four times cheaper per page. Our &lt;a href="https://mesharc.dev/firecrawl-alternative" rel="noopener noreferrer"&gt;Firecrawl alternative page&lt;/a&gt; also says Firecrawl is the better pick if you want to self-host an open-source crawler; MeshArc is not open source.&lt;/p&gt;

&lt;p&gt;Firecrawl's JSON format also includes the model. If you do not want a model key of your own, that is a real convenience.&lt;/p&gt;

&lt;p&gt;Where the maths goes the other way is plain pages (about 4× cheaper on MeshArc), walled sites that a session can read (about 2×), anything with a meaningful 404 rate, and repeat crawls where most pages have not changed.&lt;/p&gt;
&lt;h2&gt;
  
  
  Estimate your own bill
&lt;/h2&gt;

&lt;p&gt;Here is a small, dependency-free calculator for Firecrawl's side, using the rates above. Change the inputs to your workload.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# Firecrawl monthly cost estimate (rates checked 21 Oct 2026)
&lt;/span&gt;&lt;span class="n"&gt;PLAN_USD_PER_1K&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;standard_monthly&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;0.99&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;standard_annual&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;0.83&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                   &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;growth_monthly&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;0.80&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;growth_annual&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;0.67&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;firecrawl_cost&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pages&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;json_share&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;0.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;pdf_pages&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;zdr&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;False&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                   &lt;span class="n"&gt;plan&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;standard_monthly&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;per_page&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;zdr&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;credits&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;pages&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;per_page&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;pages&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;json_share&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;pdf_pages&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;credits&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;round&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;credits&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;1000&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;PLAN_USD_PER_1K&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;plan&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# 10,000 pages, 30% as JSON, 200 PDF pages
&lt;/span&gt;&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;firecrawl_cost&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;10_000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;json_share&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;0.3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;pdf_pages&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;200&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;span class="c1"&gt;# (22200.0, 21.98)
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Note that 404s are not discounted in that formula — because Firecrawl does not discount them.&lt;/p&gt;

&lt;p&gt;To see what the same URLs cost on MeshArc, run them through the SDK and add up the per-page &lt;code&gt;credits&lt;/code&gt; field. Every row tells you the HTTP status, the verdict and what it cost.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;pip &lt;span class="nb"&gt;install &lt;/span&gt;mesharc        &lt;span class="c"&gt;# or: npm install mesharc&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;mesharc&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;MeshArc&lt;/span&gt;

&lt;span class="n"&gt;arc&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;MeshArc&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;  &lt;span class="c1"&gt;# reads MESHARC_API_KEY
&lt;/span&gt;
&lt;span class="n"&gt;urls&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;line&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;urls.txt&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;()]&lt;/span&gt;
&lt;span class="n"&gt;batch&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;arc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;scrape&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;urls&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;config&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;concurrency&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;

&lt;span class="n"&gt;total&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;batch&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;pages&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;
    &lt;span class="n"&gt;total&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;credits&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;url&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;httpStatus&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;verdict&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;credits&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;

&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;urls&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; URLs, &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;total&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; credits, ~$&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;total&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;1000&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mf"&gt;0.198&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; on Growth&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A 404 or a refused page shows up in that list with &lt;code&gt;credits&lt;/code&gt; of 0.&lt;/p&gt;

&lt;h2&gt;
  
  
  The short version
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Firecrawl Standard is &lt;strong&gt;$0.99 per 1,000 pages&lt;/strong&gt; billed monthly, &lt;strong&gt;$0.83&lt;/strong&gt; billed annually.&lt;/li&gt;
&lt;li&gt;JSON extraction makes it &lt;strong&gt;$4.95 / $4.15&lt;/strong&gt; per 1,000, model included.&lt;/li&gt;
&lt;li&gt;403s and 404s are billed. Plan credits do not roll over below Scale. Top-ups cost 2.5× the plan rate on Standard.&lt;/li&gt;
&lt;li&gt;Enhanced proxy is free on top, which makes Firecrawl the cheaper choice for hard, residential-only sites.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you want to compare on your own URLs rather than mine, paste a few into the &lt;a href="https://mesharc.dev/playground" rel="noopener noreferrer"&gt;MeshArc playground&lt;/a&gt; and look at the credits column, or see the full vendor table on &lt;a href="https://mesharc.dev/compare" rel="noopener noreferrer"&gt;/compare&lt;/a&gt;. Re-crawling the same sites every week? The &lt;a href="https://mesharc.dev/use-cases/website-change-monitoring" rel="noopener noreferrer"&gt;website change monitoring use case&lt;/a&gt; shows how unchanged pages drop to zero.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;a href="https://mesharc.dev/playground" rel="noopener noreferrer"&gt;Try it in the playground&lt;/a&gt;&lt;/strong&gt; · &lt;strong&gt;&lt;a href="https://mesharc.dev/signup" rel="noopener noreferrer"&gt;Start free — 1,000 credits, no card&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://mesharc.dev/blog/firecrawl-pricing-explained" rel="noopener noreferrer"&gt;mesharc.dev&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>webscraping</category>
      <category>api</category>
      <category>python</category>
      <category>webdev</category>
    </item>
  </channel>
</rss>
