<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: DatLab</title>
    <description>The latest articles on DEV Community by DatLab (@datlab).</description>
    <link>https://dev.to/datlab</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4242446%2F66ae438e-e579-431b-beba-af751f2a9400.png</url>
      <title>DEV Community: DatLab</title>
      <link>https://dev.to/datlab</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/datlab"/>
    <language>en</language>
    <item>
      <title>Taking full-page screenshots of a URL list for $0.60 per 1,000</title>
      <dc:creator>DatLab</dc:creator>
      <pubDate>Sat, 14 Nov 2026 18:03:41 +0000</pubDate>
      <link>https://dev.to/datlab/taking-full-page-screenshots-of-a-url-list-for-060-per-1000-257l</link>
      <guid>https://dev.to/datlab/taking-full-page-screenshots-of-a-url-list-for-060-per-1000-257l</guid>
      <description>&lt;p&gt;I built the Apify actor I use in the examples. I will say that now, so you can read the rest with the right amount of skepticism.&lt;/p&gt;

&lt;p&gt;The job is simple. You have a list of public URLs. You want a full-page PNG, a mobile JPEG, or a PDF of each one, plus a public image URL you can store. You do not want to start a new browser for every page, and you want to know the bill before you press Run.&lt;/p&gt;

&lt;p&gt;If that is your job, you can do it for $0.60 per 1,000 screenshots, plus a $0.003 start fee per run. Platform usage is included.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I actually use
&lt;/h2&gt;

&lt;p&gt;I use &lt;a href="https://apify.com/datlab/bulk-website-screenshot" rel="noopener noreferrer"&gt;Bulk Website Screenshot&lt;/a&gt; on Apify Store. It opens one shared browser for the whole list. Each URL becomes a dataset row with a &lt;code&gt;screenshotUrl&lt;/code&gt;, the final loaded URL, width, height, title, HTTP status, and an &lt;code&gt;error&lt;/code&gt; if the capture failed. The files themselves live in the run's key-value store.&lt;/p&gt;

&lt;p&gt;What it will do:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;PNG, JPEG, or PDF. JPEG has a &lt;code&gt;jpegQuality&lt;/code&gt; setting, default 80.&lt;/li&gt;
&lt;li&gt;Full page (the default) or just the viewport.&lt;/li&gt;
&lt;li&gt;Desktop at 1280x800 by default, or emulate iPhone 15, Pixel 7, or iPad Pro 11.&lt;/li&gt;
&lt;li&gt;Dark mode, and retina-quality images if you set &lt;code&gt;deviceScaleFactor&lt;/code&gt; to 2.&lt;/li&gt;
&lt;li&gt;Capture one element with a CSS selector, for example &lt;code&gt;#pricing&lt;/code&gt; or &lt;code&gt;main&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Hide cookie banners by default, and any extra selectors you pass. Nothing is clicked or accepted.&lt;/li&gt;
&lt;li&gt;Scroll first so lazy images load, then wait for load, network idle, a selector, or a fixed delay.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;What it will not do:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Log in, inject cookies, or bypass captchas or paywalls.&lt;/li&gt;
&lt;li&gt;Ignore robots.txt or a crawl-delay.&lt;/li&gt;
&lt;li&gt;Use residential proxies. An optional datacenter proxy is the only proxy option.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If the page is behind a sign-in, this is the wrong tool.&lt;/p&gt;

&lt;h2&gt;
  
  
  An honest comparison
&lt;/h2&gt;

&lt;p&gt;I am not going to crown a winner. These tools overlap, they do not replace each other.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://apify.com/cheapapi/website-screenshot-api" rel="noopener noreferrer"&gt;cheapapi/website-screenshot-api&lt;/a&gt; is the one people often pick when they want a cheap Store actor with more device profiles and WebP. I re-checked its pricing page on 14 November 2026 (Asia/Jerusalem). It is $0.89 per 1,000 successful screenshots on the Free plan, $0.83 on Starter, $0.74 on Scale, and $0.61 on Business. The start fee is $0.00001, and you also pay Apify platform usage on top. It can click cookie consent, pass cookies for authenticated pages, and emulate more devices. If you need those, use it.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://apify.com/apify/screenshot-url" rel="noopener noreferrer"&gt;apify/screenshot-url&lt;/a&gt; has no actor event fee. You pay platform usage only. That can be cheap or not, depending on memory and how long the run takes. I am not going to invent a compute bill I did not measure today.&lt;/p&gt;

&lt;p&gt;The DatLab actor is $0.60 per 1,000 screenshots, plus $0.003 per run, with platform usage included. Failed pages, robots.txt blocks, and HTTP errors are not charged. The start fee is higher than cheapapi's, so a single URL costs $0.0036. In a batch of 1,000, the all-in price is $0.603 on any plan.&lt;/p&gt;

&lt;h2&gt;
  
  
  Walkthrough: a list of public URLs
&lt;/h2&gt;

&lt;p&gt;You will need an Apify account and an API token from Console, under Settings, then API and Integrations. The &lt;a href="https://apify.com/pricing" rel="noopener noreferrer"&gt;Free plan&lt;/a&gt; is $0, includes $5 of prepaid usage each month, and does not require a card. Unused credit expires at the end of the cycle. You can also run the actor from its Store page with no code. The examples below are the API path.&lt;/p&gt;

&lt;p&gt;The URLs are public pages from the actor's own input prefill. Check robots.txt before you point this at a host you do not run.&lt;/p&gt;

&lt;h3&gt;
  
  
  JavaScript, official client
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nc"&gt;ApifyClient&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="s2"&gt;"apify-client"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;ApifyClient&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;process&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;env&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;APIFY_TOKEN&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;run&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;actor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"datlab/bulk-website-screenshot"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;urls&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"https://apify.com"&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"https://crawlee.dev"&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;format&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"png"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;fullPage&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;items&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dataset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;defaultDatasetId&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;listItems&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;item&lt;/span&gt; &lt;span class="k"&gt;of&lt;/span&gt; &lt;span class="n"&gt;items&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nb"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;item&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;item&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;screenshotUrl&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;item&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;error&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Python, official client
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;apify_client&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;ApifyClient&lt;/span&gt;

&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;ApifyClient&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;environ&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"APIFY_TOKEN"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;run&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;actor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"datlab/bulk-website-screenshot"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;run_input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="s2"&gt;"urls"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
            &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"url"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"https://apify.com"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"url"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"https://crawlee.dev"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"format"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"png"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"fullPage"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;items&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dataset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"defaultDatasetId"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;list_items&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;items&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;item&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;items&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;item&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"url"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;item&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"screenshotUrl"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;item&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"error"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A successful item looks like this (trimmed, from a real run shape):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"url"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"https://crawlee.dev/"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"loadedUrl"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"https://crawlee.dev/"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"screenshotUrl"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"https://api.apify.com/v2/key-value-stores/…/records/crawlee-dev-0d35627b.png"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"screenshotKey"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"crawlee-dev-0d35627b.png"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"format"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"png"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"fullPage"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"truncatedAtHeight"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"width"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1280&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"height"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;2874&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"sizeBytes"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;335092&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"device"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"desktop"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"title"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Crawlee · Build reliable crawlers. Fast."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"httpStatusCode"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;200&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"error"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Failed or blocked URLs come back with &lt;code&gt;screenshotUrl: null&lt;/code&gt; and an &lt;code&gt;error&lt;/code&gt;. You are not charged for those.&lt;/p&gt;

&lt;h2&gt;
  
  
  Mobile, lazy images, or just one element
&lt;/h2&gt;

&lt;p&gt;Same actor, different input. &lt;code&gt;device&lt;/code&gt; switches the viewport, pixel ratio, user agent, and touch. &lt;code&gt;scrollToBottom&lt;/code&gt; walks the page first so lazy-loaded images have a chance to render. &lt;code&gt;elementSelector&lt;/code&gt; crops to the first match.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;run&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;actor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"datlab/bulk-website-screenshot"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;run_input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="s2"&gt;"urls"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"url"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"https://crawlee.dev"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"format"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"jpeg"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"jpegQuality"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;80&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"fullPage"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"device"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"iPhone 15"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"scrollToBottom"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"hideCookieBanners"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"selectorsToHide"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;".chat-widget, #newsletter-popup"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A few input notes, because the field names are easy to guess wrong:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;urls&lt;/code&gt; is required. Each item can be &lt;code&gt;{"url": "..."}&lt;/code&gt; or a plain string.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;waitUntil&lt;/code&gt; is &lt;code&gt;load&lt;/code&gt; (default), &lt;code&gt;domcontentloaded&lt;/code&gt;, or &lt;code&gt;networkidle&lt;/code&gt;. Puppeteer-style &lt;code&gt;networkidle0&lt;/code&gt; and &lt;code&gt;networkidle2&lt;/code&gt; are accepted and treated as &lt;code&gt;networkidle&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;maxFullPageHeight&lt;/code&gt; defaults to 15,000 px. Taller pages are cut, and &lt;code&gt;truncatedAtHeight&lt;/code&gt; tells you where.&lt;/li&gt;
&lt;li&gt;PDF output is sized to the screenshot viewport, so the page does not reflow to a paper width and clip wide content. &lt;code&gt;pdfFormat&lt;/code&gt; is kept for compatibility and does not change that.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;maxConcurrency&lt;/code&gt; defaults to 2, which fits the default 1 GB memory. Raise it only if you also raise memory.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;minDelayPerHostMs&lt;/code&gt; defaults to 500 ms. A larger robots.txt crawl-delay wins.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you want an agent to call this as a tool, Apify hosts an MCP server at &lt;code&gt;https://mcp.apify.com&lt;/code&gt;. You can pin the actor with &lt;code&gt;https://mcp.apify.com/?tools=fetch-actor-details,datlab/bulk-website-screenshot&lt;/code&gt;. See the &lt;a href="https://docs.apify.com/mcp" rel="noopener noreferrer"&gt;Apify MCP docs&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  What a run actually costs
&lt;/h2&gt;

&lt;p&gt;The prices are pay per event. I am doing the arithmetic in the open, not quoting a made-up invoice.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;1,000 URLs in one run: 1,000 x $0.0006 + $0.003 = $0.603.&lt;/li&gt;
&lt;li&gt;50 URLs in one run: 50 x $0.0006 + $0.003 = $0.033.&lt;/li&gt;
&lt;li&gt;A single URL in its own run: $0.0006 + $0.003 = $0.0036.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Failed, robots-blocked, and HTTP-error rows are not billed. The $0.003 start fee is why I batch URLs instead of starting a run per page. If you only need one shot, that start fee is most of the bill, and a usage-only actor can come out cheaper. If you have a list, the start fee is a rounding error.&lt;/p&gt;

&lt;h2&gt;
  
  
  When a shot comes out wrong
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;The row has an error and no image.&lt;/strong&gt; Some sites block automated browsers (HTTP 403) or disallow the path in robots.txt. The actor reports that and does not try to get around it. You are not charged.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The screenshot is cut off.&lt;/strong&gt; The page is taller than &lt;code&gt;maxFullPageHeight&lt;/code&gt;. Raise the limit, or capture a single element instead.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Images are missing at the bottom.&lt;/strong&gt; Turn on &lt;code&gt;scrollToBottom&lt;/code&gt;. You can also set &lt;code&gt;waitUntilNetworkIdleAfterScroll&lt;/code&gt; if a fixed delay is not enough, or wait for a selector with &lt;code&gt;waitForSelector&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The image looks soft.&lt;/strong&gt; Set &lt;code&gt;deviceScaleFactor&lt;/code&gt; to 2, or pick a mobile device. Those already use a 2.6x to 3x pixel ratio.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;You wanted a logged-in view.&lt;/strong&gt; It cannot do that, by design.&lt;/p&gt;

&lt;p&gt;You are responsible for having the right to capture and use the screenshots (copyright, trademarks, site terms, applicable law). The actor only captures public pages you list.&lt;/p&gt;

&lt;p&gt;If a public page comes out messy, leave a comment with the URL. I will take a look.&lt;/p&gt;

&lt;p&gt;The actor is here: &lt;a href="https://apify.com/datlab/bulk-website-screenshot" rel="noopener noreferrer"&gt;datlab/bulk-website-screenshot&lt;/a&gt;. If you need the page as Markdown instead of a picture, I also have &lt;a href="https://apify.com/datlab/web-page-to-markdown" rel="noopener noreferrer"&gt;Web Page to Markdown&lt;/a&gt; and &lt;a href="https://apify.com/datlab/site-to-markdown-crawler" rel="noopener noreferrer"&gt;Website to Markdown Crawler&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>python</category>
      <category>javascript</category>
      <category>webscraping</category>
      <category>automation</category>
    </item>
    <item>
      <title>Turning a docs site into clean Markdown for RAG for about 10 cents per 1,000 pages</title>
      <dc:creator>DatLab</dc:creator>
      <pubDate>Fri, 13 Nov 2026 07:57:38 +0000</pubDate>
      <link>https://dev.to/datlab/turning-a-docs-site-into-clean-markdown-for-rag-for-about-10-cents-per-1000-pages-1nfp</link>
      <guid>https://dev.to/datlab/turning-a-docs-site-into-clean-markdown-for-rag-for-about-10-cents-per-1000-pages-1nfp</guid>
      <description>&lt;p&gt;I built the two Apify actors I use in the examples. I will say that now, so you can read the rest with the right amount of skepticism.&lt;/p&gt;

&lt;p&gt;The job is simple to describe and easy to overpay for. You have a documentation site, or a list of URLs. You want clean Markdown, one record per page, with a title and a source URL, so you can chunk it and embed it. You do not want nav bars, cookie banners, or a bill you cannot explain to yourself before you press Run.&lt;/p&gt;

&lt;p&gt;If that is your job, you can do it for about $0.10 per 1,000 static pages.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why raw HTML makes RAG worse
&lt;/h2&gt;

&lt;p&gt;Most "load this website" tutorials fetch the HTML and throw it at a splitter. The splitter then dutifully indexes the footer, the cookie dialog, and three copies of the sidebar. Your bot will someday answer a pricing question with "Accept all cookies," and it will sound very confident.&lt;/p&gt;

&lt;p&gt;Markdown helps because headings, lists, tables, and code blocks are still there, and the chrome is mostly gone. You still have to chunk carefully (I will get to that), but you are no longer embedding CSS class names.&lt;/p&gt;

&lt;p&gt;Two other traps:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Some tools also search the web. That is useful when you do not have URLs. It is wasted money when you already do.&lt;/li&gt;
&lt;li&gt;Some tools start a headless browser on every page. That is required for a React shell that ships an empty &lt;code&gt;#root&lt;/code&gt;. It is not required for a lot of docs sites, which are still just HTML.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  What I actually use
&lt;/h2&gt;

&lt;p&gt;I use two actors on &lt;a href="https://apify.com/datlab" rel="noopener noreferrer"&gt;Apify Store&lt;/a&gt;:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A list of known URLs: &lt;a href="https://apify.com/datlab/web-page-to-markdown" rel="noopener noreferrer"&gt;Web Page to Markdown&lt;/a&gt;, $0.50 per 1,000 HTTP pages, $1.00 per 1,000 browser pages.&lt;/li&gt;
&lt;li&gt;A docs site, blog, or help center: &lt;a href="https://apify.com/datlab/site-to-markdown-crawler" rel="noopener noreferrer"&gt;Website to Markdown Crawler&lt;/a&gt;, $0.10 per 1,000 HTTP pages, $0.35 per 1,000 browser pages, $0.50 per 1,000 saved files. That last one is only if you turn on file saving.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Both try plain HTTP first. A headless Chromium starts only when a page looks like a client-rendered shell, and only that page is billed at the browser price. Platform usage is included in the event price. Failed pages, duplicates, and robots.txt blocks are not billed. There is a $0.00005 start fee per run.&lt;/p&gt;

&lt;p&gt;They stay on public pages. robots.txt and crawl-delay are honoured. There is no login, no cookie injection, and no captcha bypass. If the docs are behind a sign-in, this is the wrong tool.&lt;/p&gt;

&lt;p&gt;The rest of this post uses the crawler, because "here is the docs root, give me the site" is the common case.&lt;/p&gt;

&lt;h2&gt;
  
  
  An honest comparison
&lt;/h2&gt;

&lt;p&gt;I am not going to crown a winner. These tools overlap, they do not replace each other.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://www.firecrawl.dev/pricing" rel="noopener noreferrer"&gt;Firecrawl&lt;/a&gt; is the polished hosted API a lot of people reach for. A standard scrape or crawl is 1 credit per page (checked 13 November 2026). The free plan is 1,000 credits a month, no card, which is a genuine way to try it. After that, Hobby is $19 a month for 5,000 credits, about $3.80 per 1,000 pages if you use the allotment. Extra Hobby credits are $5 per 1,000. Standard is $99 for 100,000 credits, about $0.99 per 1,000, or $5 per 2,000 extra credits. JSON extraction and some add-ons cost extra credits. Firecrawl also searches, maps, and interacts with pages. I do not do those things.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://apify.com/apify/rag-web-browser/pricing" rel="noopener noreferrer"&gt;Apify RAG Web Browser&lt;/a&gt; starts at $1.33 per 1,000 fetched pages. It can search Google and then fetch the results. Use it when you need search. My actors do not search. They only fetch the URLs or the site you give them.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://apify.com/apify/website-content-crawler/pricing" rel="noopener noreferrer"&gt;Apify Website Content Crawler&lt;/a&gt; is the official full-featured crawler. It is billed on platform usage, not a fixed per-page event. That can be cheap or not, depending on memory, proxies, and how long the run takes. I wanted a number I could write down before the run, so I went pay-per-page instead. I am not claiming I beat it on every site.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://r.jina.ai/" rel="noopener noreferrer"&gt;Jina Reader&lt;/a&gt; is still the least ceremony for a single URL: prefix the URL and you get Markdown. I use it for one-off fetches. It is not what I use to crawl a 2,000-page docs site with a cap and a sitemap.&lt;/p&gt;

&lt;p&gt;If you want to self-host, &lt;a href="https://github.com/unclecode/crawl4ai" rel="noopener noreferrer"&gt;Crawl4AI&lt;/a&gt; is the open-source tool I would try first. You pay in your own time and machines, not per page.&lt;/p&gt;

&lt;h2&gt;
  
  
  Walkthrough: crawl a docs site
&lt;/h2&gt;

&lt;p&gt;You will need an Apify account and an API token from Console, under Settings, then API and Integrations. The &lt;a href="https://apify.com/pricing" rel="noopener noreferrer"&gt;Free plan&lt;/a&gt; is $0, includes $5 of prepaid usage each month, and does not require a card. Unused credit expires at the end of the cycle.&lt;/p&gt;

&lt;p&gt;The example below starts at a public docs root, stays on that host and path, reads sitemaps and &lt;code&gt;llms.txt&lt;/code&gt; if they exist, and stops at 200 pages so a first run cannot run away.&lt;/p&gt;

&lt;h3&gt;
  
  
  JavaScript, official client
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nc"&gt;ApifyClient&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="s2"&gt;"apify-client"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;ApifyClient&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;process&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;env&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;APIFY_TOKEN&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;run&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;actor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"datlab/site-to-markdown-crawler"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;startUrls&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"https://docs.apify.com/academy"&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;crawlerType&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"auto"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;useSitemaps&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;useLlmsTxt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;maxCrawlPages&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;200&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;outputFormats&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"markdown"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;items&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dataset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;defaultDatasetId&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;listItems&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;item&lt;/span&gt; &lt;span class="k"&gt;of&lt;/span&gt; &lt;span class="n"&gt;items&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="n"&gt;item&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;markdown&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;continue&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="nb"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;item&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;item&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;metadata&lt;/span&gt;&lt;span class="o"&gt;?&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;title&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;item&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;tokenEstimate&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Python, official client
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;apify_client&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;ApifyClient&lt;/span&gt;

&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;ApifyClient&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;environ&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"APIFY_TOKEN"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;run&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;actor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"datlab/site-to-markdown-crawler"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;run_input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="s2"&gt;"startUrls"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"url"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"https://docs.apify.com/academy"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"crawlerType"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"auto"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"useSitemaps"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"useLlmsTxt"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"maxCrawlPages"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;200&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"outputFormats"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"markdown"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;items&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dataset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"defaultDatasetId"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;list_items&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;items&lt;/span&gt;
&lt;span class="n"&gt;pages&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;item&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;item&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;items&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;item&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"markdown"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"saved {len(pages)} pages"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A successful item looks like this (trimmed):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"url"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"https://crawlee.dev/js/docs/quick-start"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"crawl"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nl"&gt;"httpStatusCode"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;200&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"renderedWith"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"raw-http"&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"metadata"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nl"&gt;"title"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Quick Start | Crawlee for JavaScript"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"canonicalUrl"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"https://crawlee.dev/js/docs/quick-start"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"languageCode"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"en"&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"markdown"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"# Quick Start\n\nWith this short tutorial you can start scraping…"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"wordCount"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;756&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"tokenEstimate"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;2373&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;renderedWith&lt;/code&gt; is the field I look at after a first run. If almost everything is &lt;code&gt;raw-http&lt;/code&gt;, you got the $0.10 per 1,000 price. If almost everything is the browser, the site is probably a client-rendered app, and you should expect the $0.35 per 1,000 price. You can force &lt;code&gt;playwright:chromium&lt;/code&gt; when you already know that.&lt;/p&gt;

&lt;p&gt;Respect the site. Use a real &lt;code&gt;maxCrawlPages&lt;/code&gt;. Do not point this at a host you are not allowed to crawl. Check their robots.txt first. Hacker News, for example, publishes a 30 second crawl-delay, which will make discovery slow on purpose.&lt;/p&gt;

&lt;h2&gt;
  
  
  Hook it to a vector store
&lt;/h2&gt;

&lt;p&gt;Import paths in LangChain move around. If a name below has changed, the idea has not: each page becomes a &lt;code&gt;Document&lt;/code&gt;, you split on Markdown headings, you prepend the title, you embed, you store.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;langchain_core&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;documents&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;Document&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;langchain_text_splitters&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;MarkdownHeaderTextSplitter&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;RecursiveCharacterTextSplitter&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;langchain_openai&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;OpenAIEmbeddings&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;langchain_community&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;vectorstores&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;Chroma&lt;/span&gt;

&lt;span class="n"&gt;docs&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;item&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;pages&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;meta&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;item&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"metadata"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="n"&gt;docs&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="nc"&gt;Document&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="n"&gt;page_content&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;item&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"markdown"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="n"&gt;metadata&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
                &lt;span class="s2"&gt;"source"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;item&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"url"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="s2"&gt;"title"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;meta&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"title"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="s2"&gt;"canonical"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;meta&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"canonicalUrl"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="s2"&gt;"rendered_with"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;item&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"crawl"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"renderedWith"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="s2"&gt;"token_estimate"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;item&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"tokenEstimate"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;header_splitter&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;MarkdownHeaderTextSplitter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;headers_to_split_on&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;
        &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"#"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"h1"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"##"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"h2"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"###"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"h3"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;chunker&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;RecursiveCharacterTextSplitter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;chunk_size&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;800&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;chunk_overlap&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;120&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;chunks&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;doc&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;docs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;sections&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;header_splitter&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;split_text&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;doc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;page_content&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;section&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;sections&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;title&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;doc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"title"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="s2"&gt;""&lt;/span&gt;
        &lt;span class="n"&gt;h1&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;section&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"h1"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="s2"&gt;""&lt;/span&gt;
        &lt;span class="n"&gt;h2&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;section&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"h2"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="s2"&gt;""&lt;/span&gt;
        &lt;span class="n"&gt;prefix&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;" &amp;gt; "&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;part&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;part&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;title&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;h1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;h2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;part&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;section&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;page_content&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"{prefix}\n\n{section.page_content}"&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;prefix&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="n"&gt;section&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;page_content&lt;/span&gt;
        &lt;span class="n"&gt;section&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;metadata&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="n"&gt;doc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="n"&gt;section&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="n"&gt;chunks&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;extend&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;chunker&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;split_documents&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;section&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;vectorstore&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Chroma&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;from_documents&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;chunks&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;OpenAIEmbeddings&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three notes, because this is the part tutorials usually skip:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;If you split only on character count, a chunk in the middle of "Authentication" loses the heading, and retrieval gets worse. Split on headings first.&lt;/li&gt;
&lt;li&gt;Put the page title and the parent headings into the chunk text, not only into metadata. Many embeddings never see your metadata.&lt;/li&gt;
&lt;li&gt;I am not going to tell you Chroma is the right store. Use whatever you already run. The mapping is the same for Pinecone, Qdrant, or Weaviate.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you would rather stay in Python with no LangChain, the loop is the same: keep &lt;code&gt;url&lt;/code&gt;, &lt;code&gt;title&lt;/code&gt;, and &lt;code&gt;markdown&lt;/code&gt;, split on headings, embed the chunks.&lt;/p&gt;

&lt;p&gt;LlamaIndex can consume the same records. The official &lt;a href="https://docs.apify.com/integrations/llama-index" rel="noopener noreferrer"&gt;Apify LlamaIndex integration&lt;/a&gt; and the &lt;a href="https://docs.apify.com/integrations/langchain" rel="noopener noreferrer"&gt;LangChain integration&lt;/a&gt; both work with a named actor, including these two.&lt;/p&gt;

&lt;p&gt;If you want an agent to call the crawler as a tool, Apify hosts an MCP server at &lt;code&gt;https://mcp.apify.com&lt;/code&gt;. You can pin the actor with a tools query like &lt;code&gt;https://mcp.apify.com/?tools=fetch-actor-details,datlab/site-to-markdown-crawler&lt;/code&gt;. See the &lt;a href="https://docs.apify.com/mcp" rel="noopener noreferrer"&gt;Apify MCP docs&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  What a run actually costs
&lt;/h2&gt;

&lt;p&gt;These are event prices, not a made-up crawl log. I did not invent CU figures for Website Content Crawler, and I will not.&lt;/p&gt;

&lt;p&gt;Example A. 1,000 static docs pages, all HTTP:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;1,000 * $0.0001 = $0.10&lt;/code&gt;, plus the $0.00005 start fee.&lt;/p&gt;

&lt;p&gt;Example B. 200 pages, 160 HTTP and 40 browser:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;160 * $0.0001 + 40 * $0.00035 = $0.016 + $0.014 = $0.03&lt;/code&gt;, plus the start fee.&lt;/p&gt;

&lt;p&gt;Example C. A 5,000-page docs site, mostly static, in &lt;code&gt;auto&lt;/code&gt; mode: about $0.50 if almost all pages stay on HTTP. That is the example I put on the Store page. If the site is a SPA, expect closer to &lt;code&gt;5,000 * $0.00035 = $1.75&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;The URL-list actor is more expensive per page because it is a smaller, fetch-only job: $0.50 / $1.00 per 1,000. Use it when an agent already has the links, or when you recrawl a known set after a sitemap diff.&lt;/p&gt;

&lt;p&gt;The $5 Free plan covers a serious test. 1,000 HTTP pages is a dime. Even a few thousand browser pages stay well under $5. Cap a run with &lt;code&gt;maxCrawlPages&lt;/code&gt; or Apify's maximum cost per run, so a bad glob cannot surprise you.&lt;/p&gt;

&lt;p&gt;For comparison, 1,000 Firecrawl credits on the Hobby allotment are about $3.80, or $5 if you buy extra credits, and a standard scrape is 1 credit (pricing page, 13 November 2026). That gap is real for static docs. It shrinks if you need Firecrawl's search, extract, or interact features, which I do not provide.&lt;/p&gt;

&lt;h2&gt;
  
  
  Recrawling later
&lt;/h2&gt;

&lt;p&gt;A full recrawl of a docs site every night is usually a waste. Footers and build hashes change even when the article does not, and you will re-embed pages for no reason.&lt;/p&gt;

&lt;p&gt;If the site has a sitemap, watch that first: new URLs, removed URLs, and lastmod changes. lastmod can lie, so I also hash the extracted content when I care. I shipped a &lt;a href="https://apify.com/datlab/sitemap-change-monitor" rel="noopener noreferrer"&gt;Sitemap Change Monitor&lt;/a&gt; for that job. Then I send only the changed URLs back through the URL-list actor.&lt;/p&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;p&gt;Is this a Google-search browser? No. You give it URLs or a site root.&lt;/p&gt;

&lt;p&gt;Will it log in or skip captchas? No.&lt;/p&gt;

&lt;p&gt;What about PDFs and other files? The crawler can save same-site documents when you turn that on, at the file event price.&lt;/p&gt;

&lt;p&gt;Can I schedule this? Yes, with Apify schedules and webhooks.&lt;/p&gt;

&lt;p&gt;Can I use it from n8n, Make, or Zapier? Yes, through Apify's integrations. r/n8n does not want product pitches, so if you want an n8n example, I would rather publish a workflow with the JSON than drop a Store link in that subreddit.&lt;/p&gt;

&lt;h2&gt;
  
  
  If a page comes out wrong
&lt;/h2&gt;

&lt;p&gt;Open an issue on the actor page, or comment here with the URL (public pages only). I am more interested in bad Markdown, missed tables, and wrong HTTP-versus-browser decisions than in compliments.&lt;/p&gt;

&lt;p&gt;Actors:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://apify.com/datlab/site-to-markdown-crawler" rel="noopener noreferrer"&gt;Website to Markdown Crawler&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://apify.com/datlab/web-page-to-markdown" rel="noopener noreferrer"&gt;Web Page to Markdown&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>rag</category>
      <category>llm</category>
      <category>python</category>
      <category>webscraping</category>
    </item>
  </channel>
</rss>
