<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: SteadyScrape</title>
    <description>The latest articles on DEV Community by SteadyScrape (@steadyscrape).</description>
    <link>https://dev.to/steadyscrape</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4209307%2Fcb5abd14-1d82-43f2-b4e1-0c168889098d.png</url>
      <title>DEV Community: SteadyScrape</title>
      <link>https://dev.to/steadyscrape</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/steadyscrape"/>
    <language>en</language>
    <item>
      <title>Is my business on Google Maps? How to check in 10 seconds</title>
      <dc:creator>SteadyScrape</dc:creator>
      <pubDate>Sun, 15 Nov 2026 23:23:56 +0000</pubDate>
      <link>https://dev.to/steadyscrape/is-my-business-on-google-maps-how-to-check-in-10-seconds-4gnd</link>
      <guid>https://dev.to/steadyscrape/is-my-business-on-google-maps-how-to-check-in-10-seconds-4gnd</guid>
      <description>&lt;h2&gt;
  
  
  Is my business on Google Maps? How to check in 10 seconds
&lt;/h2&gt;

&lt;p&gt;Short answer: type your business name and city into a listing checker and look at two things — whether the listing exists, and whether it's &lt;em&gt;claimed&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;Free check (no signup): &lt;strong&gt;&lt;a href="https://steadyscrape.dev/tools/google-maps/" rel="noopener noreferrer"&gt;https://steadyscrape.dev/tools/google-maps/&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Here's why those two things matter, and what to do about each result.&lt;/p&gt;

&lt;h3&gt;
  
  
  "Found and claimed" — you're in control
&lt;/h3&gt;

&lt;p&gt;This is the good outcome. It means a Google Business Profile exists for the business and someone has verified ownership of it, so you (or whoever claimed it) can edit the hours, photos, phone number and website, and respond to reviews.&lt;/p&gt;

&lt;p&gt;Example from a real check: Café du Monde in New Orleans comes back &lt;strong&gt;Found · Claimed&lt;/strong&gt;, 4.6★ across 5,294 reviews, with its website attached. That's a listing doing its job.&lt;/p&gt;

&lt;h3&gt;
  
  
  "Found but unclaimed" — the listing exists, nobody owns it
&lt;/h3&gt;

&lt;p&gt;Google builds listings from public data even when the owner never signed up. An unclaimed listing still shows up in Maps and search — with whatever details Google scraped, which may be an old address, wrong hours, or no website at all. Anyone searching for you sees it; you just can't fix it or reply to its reviews.&lt;/p&gt;

&lt;p&gt;Fixing it is free: search for the business on Google Maps, open the listing, and use "Claim this business" / "Own this business?" and follow the verification steps. Until it's claimed, it's a storefront you don't have the keys to.&lt;/p&gt;

&lt;h3&gt;
  
  
  "Not found" — three usual causes
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;The business is new&lt;/strong&gt; and Google hasn't picked it up yet. Create the Business Profile yourself rather than waiting.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The name or city doesn't match&lt;/strong&gt; how Google knows it (a trading name vs. the registered name is the classic one). Try the exact name on the storefront signage, and the exact city the address is in.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;It's genuinely not listed.&lt;/strong&gt; Same fix as #1 — creating the profile puts you on the map, literally.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;One honest caveat: an automated check returns Google's best match for the name + city you gave it. If a business has several locations, the check may surface a different branch than the one you meant — read the address in the result before concluding anything.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why agencies check this in bulk
&lt;/h3&gt;

&lt;p&gt;Everything above is one business at a time. The interesting version is a whole market: &lt;em&gt;every&lt;/em&gt; plumber in Denver, say, with their rating, review count, website (or lack of one) and claimed status in a single table. The unclaimed and no-website rows in that table are, not coincidentally, the businesses most likely to need marketing help.&lt;/p&gt;

&lt;p&gt;That's the step-up from the free check: the &lt;a href="https://apify.com/steady-scrape/google-maps-business-segments" rel="noopener noreferrer"&gt;Google Maps Leads&lt;/a&gt; Actor runs the same lookup across a metro + niche and returns the full list (from $0.002 per business). The free checker is one business, answered in seconds; the Actor is the whole map, in about two minutes.&lt;/p&gt;

&lt;h3&gt;
  
  
  The 10-second version
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Go to the &lt;a href="https://steadyscrape.dev/tools/google-maps/" rel="noopener noreferrer"&gt;free Maps check&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Enter business name + city&lt;/li&gt;
&lt;li&gt;Read three things: Found?, Claimed?, and the rating/review count&lt;/li&gt;
&lt;li&gt;If it's unclaimed or missing, that's a 15-minute fix with an outsized payoff — it's the listing your customers actually find&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;em&gt;SteadyScrape builds small, pay-per-use data tools for local marketing and SEO work. The free checks live at &lt;a href="https://steadyscrape.dev/tools" rel="noopener noreferrer"&gt;steadyscrape.dev/tools&lt;/a&gt; — no signup, and the paid tools are linked from each result if you ever need the same job at scale.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>seo</category>
      <category>marketing</category>
      <category>webdev</category>
      <category>tooling</category>
    </item>
    <item>
      <title>How to Pack Public File URLs Into a Checksummed ZIP for Catalog Work</title>
      <dc:creator>SteadyScrape</dc:creator>
      <pubDate>Tue, 10 Nov 2026 23:18:37 +0000</pubDate>
      <link>https://dev.to/steadyscrape/how-to-pack-public-file-urls-into-a-checksummed-zip-for-catalog-work-55m8</link>
      <guid>https://dev.to/steadyscrape/how-to-pack-public-file-urls-into-a-checksummed-zip-for-catalog-work-55m8</guid>
      <description>&lt;p&gt;Catalog imports, vendor handoffs, and offline reviews often start from a spreadsheet of file addresses you already have: product images you are licensed to reuse, public PDF spec sheets, a CSV or two. The messy part is not finding the URLs. It is fetching them one by one, noticing that one link returned an HTML page, and handing someone a folder of unnamed partial files.&lt;/p&gt;

&lt;p&gt;This walkthrough uses SteadyScrape’s &lt;strong&gt;Bulk Image and File Downloader&lt;/strong&gt;. You supply a short list of &lt;strong&gt;public direct file URLs&lt;/strong&gt;. The Actor fetches the files that actually download, packs them into one ZIP, and writes a status row for every URL with the byte count, detected type, SHA-256, and a sanitized source URL. The same manifest is stored as JSON and inside the ZIP.&lt;/p&gt;

&lt;p&gt;The Actor does not crawl, search, follow page links, render HTML, or log in. It does not read cookies, tokens, or custom headers. It does not bypass DRM or paywalls, unpack archives, or execute anything it downloads. Public access is not a copyright license. You need the right to fetch and reuse each file. This Actor does not scan for malware and does not clear copyright.&lt;/p&gt;

&lt;h2&gt;
  
  
  The buyer problem
&lt;/h2&gt;

&lt;p&gt;If you already have the file addresses, you want one bounded pack that answers:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Which URLs were actually saved?&lt;/li&gt;
&lt;li&gt;What type was detected from the bytes, not just the filename?&lt;/li&gt;
&lt;li&gt;What is the SHA-256 of each saved file?&lt;/li&gt;
&lt;li&gt;Which URLs were HTML pages, private addresses, or credential links, and therefore left out?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A site crawler is the wrong tool. So is pasting signed URLs that embed tokens into a dataset you will store. This Actor is for direct public files you are allowed to fetch.&lt;/p&gt;

&lt;p&gt;SteadyScrape publishes a separate public Actor for turning a page into JSON: &lt;a href="https://apify.com/steady-scrape/ai-page-to-json" rel="noopener noreferrer"&gt;AI Page To JSON&lt;/a&gt;. That is a different job. This downloader only retrieves the file URLs you supply.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the downloader returns
&lt;/h2&gt;

&lt;p&gt;The dataset has one row per processed URL. The default key-value store has three fixed keys: &lt;code&gt;OUTPUT&lt;/code&gt; (counts only), &lt;code&gt;MANIFEST&lt;/code&gt; (the full JSON manifest), and &lt;code&gt;PACK&lt;/code&gt; (the ZIP, when it fits). File bytes are not copied into the key-value store a second time. Keys are not built from the source URLs.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Field&lt;/th&gt;
&lt;th&gt;What it means&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;index&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Zero-based position of the URL in the input list&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;status&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Explicit outcome, such as &lt;code&gt;saved&lt;/code&gt;, &lt;code&gt;saved_type_mismatch&lt;/code&gt;, or &lt;code&gt;rejected_type_mismatch&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;saved&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Whether the file is in the ZIP&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;partial&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Whether bytes were counted even though the file was not saved&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;sourceUrl&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Sanitized URL, or &lt;code&gt;null&lt;/code&gt; when the URL contained credential material&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;sourceUrlRedacted&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Whether the source URL was omitted because it contained credential material&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;finalUrl&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Sanitized URL after redirects&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;httpStatus&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Last HTTP status, when one was received&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;declaredContentType&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;Content-Type&lt;/code&gt; header, not treated as proof of the file type&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;detectedMediaType&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Type read from the decoded bytes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;typeMatch&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Whether the declared type and detected type match, differ, or were generic&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;byteLength&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Decoded bytes read, including partial failures&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;sha256&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;SHA-256 of those decoded bytes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;zipEntryName&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Sanitized name inside the ZIP&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;errorCode&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Same as &lt;code&gt;status&lt;/code&gt; when the file was not saved; &lt;code&gt;null&lt;/code&gt; when it was saved&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;redirectCount&lt;/code&gt;, &lt;code&gt;attemptCount&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Redirects followed, and connection attempts&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;saved&lt;/code&gt; and &lt;code&gt;saved_type_mismatch&lt;/code&gt; are the only statuses that put a file in the ZIP. A row is not labeled as a successful image when the body is an HTML page.&lt;/p&gt;

&lt;h2&gt;
  
  
  How to run it in Apify Console
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Open &lt;a href="https://apify.com/steady-scrape/bulk-file-downloader" rel="noopener noreferrer"&gt;Bulk Image and File Downloader&lt;/a&gt; on the Apify Store.&lt;/li&gt;
&lt;li&gt;Paste up to 20 direct &lt;code&gt;http&lt;/code&gt; or &lt;code&gt;https&lt;/code&gt; file URLs. Use port 80 for http and port 443 for https.&lt;/li&gt;
&lt;li&gt;Leave the caps at their defaults unless you want a &lt;em&gt;smaller&lt;/em&gt; run. You can lower &lt;code&gt;maxUrls&lt;/code&gt;, &lt;code&gt;maxFileBytes&lt;/code&gt;, &lt;code&gt;maxTotalBytes&lt;/code&gt;, &lt;code&gt;maxConcurrency&lt;/code&gt;, &lt;code&gt;maxRuntimeSeconds&lt;/code&gt;, &lt;code&gt;maxRedirects&lt;/code&gt;, &lt;code&gt;connectTimeoutSeconds&lt;/code&gt;, and &lt;code&gt;readIdleTimeoutSeconds&lt;/code&gt;. You cannot raise them.&lt;/li&gt;
&lt;li&gt;Run the Actor.&lt;/li&gt;
&lt;li&gt;Read the dataset for per-URL status. Download the &lt;code&gt;PACK&lt;/code&gt; record for the ZIP and &lt;code&gt;MANIFEST&lt;/code&gt; for the JSON copy.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;There is no field for headers, cookies, proxy, a custom DNS resolver, localhost, or a fixture path. A URL with userinfo, a nonempty credential query value (&lt;code&gt;token&lt;/code&gt;, &lt;code&gt;signature&lt;/code&gt;, &lt;code&gt;api_key&lt;/code&gt;, &lt;code&gt;password&lt;/code&gt;, and similar names), or a fragment that carries token material is rejected and is not written to the dataset, manifest, or log.&lt;/p&gt;

&lt;p&gt;Example input using two well-known public files (a W3C PNG and the RFC 2606 text file):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"urls"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="s2"&gt;"https://www.w3.org/Icons/w3c_home.png"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"https://www.rfc-editor.org/rfc/rfc2606.txt"&lt;/span&gt;
  &lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The Store page's example input downloads three small public files (this PNG, a W3C test PDF, and the RFC text file), so a first run shows a real ZIP and manifest.&lt;/p&gt;

&lt;h2&gt;
  
  
  How to call it from Python
&lt;/h2&gt;

&lt;p&gt;This uses the official Apify Python client and the Actor's real input fields:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;apify_client&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;ApifyClient&lt;/span&gt;

&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;ApifyClient&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;environ&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"APIFY_TOKEN"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;run&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;actor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"steady-scrape/bulk-file-downloader"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;run_input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="s2"&gt;"urls"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
            &lt;span class="s2"&gt;"https://www.w3.org/Icons/w3c_home.png"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="s2"&gt;"https://www.rfc-editor.org/rfc/rfc2606.txt"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dataset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"defaultDatasetId"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;iterate_items&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"saved"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"sourceUrl"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"detectedMediaType"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"byteLength"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"sha256"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"zipEntryName"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"errorCode"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then download &lt;code&gt;PACK&lt;/code&gt; and &lt;code&gt;MANIFEST&lt;/code&gt; from the run’s default key-value store.&lt;/p&gt;

&lt;h2&gt;
  
  
  Output examples
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Measured sample
&lt;/h3&gt;

&lt;p&gt;A test run fetched the W3C PNG and RFC 2606 text file above, plus three URLs that should not be saved: an HTML page, a non-public address, and a credential URL. Two files were saved. The HTML page was rejected. The credential URL was omitted from &lt;code&gt;sourceUrl&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Measured saved PNG row (the request followed one public redirect to the current W3C asset URL):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"index"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"saved"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"saved"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"partial"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"sourceUrl"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"https://www.w3.org/Icons/w3c_home.png"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"sourceUrlRedacted"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"finalUrl"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"https://www.w3.org/assets/logos/w3c-2025-transitional/w3c-72x48.png"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"finalUrlRedacted"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"httpStatus"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;200&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"declaredContentType"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"image/png"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"detectedMediaType"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"png"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"typeMatch"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"match"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"byteLength"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1848&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"sha256"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"590608bf256237f22d84bbab2a0a1afc0d4f20db6350bbac2c6841f1f3e13b01"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"zipEntryName"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"w3c-72x48.png"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"errorCode"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"redirectCount"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Measured saved text row:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"index"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"saved"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"saved"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"partial"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"sourceUrl"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"https://www.rfc-editor.org/rfc/rfc2606.txt"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"httpStatus"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;200&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"declaredContentType"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"text/plain"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"detectedMediaType"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"text"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"typeMatch"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"match"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"byteLength"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;8008&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"sha256"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"b6869c8984701701bc2e6973b6ffc750d497f845cc1a65a106e9301590a13ab0"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"zipEntryName"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"rfc2606.txt"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"errorCode"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"redirectCount"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Measured non-success outcomes from the same run:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;https://www.iana.org/help/example-domains&lt;/code&gt; returned HTTP 200 with &lt;code&gt;declaredContentType&lt;/code&gt; &lt;code&gt;text/html&lt;/code&gt; and &lt;code&gt;detectedMediaType&lt;/code&gt; &lt;code&gt;html&lt;/code&gt;. &lt;code&gt;status&lt;/code&gt; was &lt;code&gt;rejected_disallowed_type&lt;/code&gt;. The file was not added to the ZIP (&lt;code&gt;zipEntryName&lt;/code&gt; was &lt;code&gt;null&lt;/code&gt;, &lt;code&gt;partial&lt;/code&gt; was &lt;code&gt;true&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;A non-public address was &lt;code&gt;rejected_nonpublic_address&lt;/code&gt;, with &lt;code&gt;sourceUrl&lt;/code&gt; &lt;code&gt;null&lt;/code&gt; and no bytes saved.&lt;/li&gt;
&lt;li&gt;A credential URL was &lt;code&gt;rejected_credentials&lt;/code&gt;, with &lt;code&gt;sourceUrl&lt;/code&gt; &lt;code&gt;null&lt;/code&gt; and &lt;code&gt;sourceUrlRedacted&lt;/code&gt; &lt;code&gt;true&lt;/code&gt;. The secret URL was not stored.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Trimmed &lt;code&gt;OUTPUT&lt;/code&gt; fields from that run: &lt;code&gt;packKey&lt;/code&gt; &lt;code&gt;PACK&lt;/code&gt;, &lt;code&gt;manifestKey&lt;/code&gt; &lt;code&gt;MANIFEST&lt;/code&gt;, &lt;code&gt;packBytes&lt;/code&gt; 6262, &lt;code&gt;savedCount&lt;/code&gt; 2, &lt;code&gt;failedCount&lt;/code&gt; 3, &lt;code&gt;omittedUrlCount&lt;/code&gt; 0, &lt;code&gt;finishedReason&lt;/code&gt; &lt;code&gt;completed&lt;/code&gt;. The ZIP contains the two saved files plus &lt;code&gt;manifest.json&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Those checksums describe the bytes stored in that run. If a publisher replaces a file, a later run can produce a different &lt;code&gt;sha256&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it costs
&lt;/h2&gt;

&lt;p&gt;The Actor uses pay-per-event pricing:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;$0.004&lt;/strong&gt; per saved file (a file that actually lands in the ZIP).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;$0.003&lt;/strong&gt; when a run includes at least one saved file.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Worked examples:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;What you run&lt;/th&gt;
&lt;th&gt;Event math&lt;/th&gt;
&lt;th&gt;Price&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;2 saved files (the measured W3C PNG + RFC text case)&lt;/td&gt;
&lt;td&gt;(2 × $0.004) + $0.003&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;$0.011&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;20 saved files in one run&lt;/td&gt;
&lt;td&gt;(20 × $0.004) + $0.003&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;$0.083&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Rejected HTML pages, private addresses, and credential URLs are not saved files under this model.&lt;/p&gt;

&lt;p&gt;With pay-per-event pricing these events are the whole Actor price; there is no separate start fee and no compute charge on top. A run that saves nothing costs nothing. The &lt;a href="https://apify.com/steady-scrape/bulk-file-downloader" rel="noopener noreferrer"&gt;Store page&lt;/a&gt; shows the same event list.&lt;/p&gt;

&lt;h2&gt;
  
  
  Limits and allowed types
&lt;/h2&gt;

&lt;p&gt;Defaults are the hard maximums. Input can lower them and cannot raise them.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Cap&lt;/th&gt;
&lt;th&gt;Maximum&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;URLs processed&lt;/td&gt;
&lt;td&gt;20&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Decoded bytes per file&lt;/td&gt;
&lt;td&gt;10 MB (10485760)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Decoded plus partial bytes per run&lt;/td&gt;
&lt;td&gt;25 MB (26214400)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;ZIP size&lt;/td&gt;
&lt;td&gt;32 MB (33554432)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Concurrency&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Runtime&lt;/td&gt;
&lt;td&gt;120 seconds&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Redirects&lt;/td&gt;
&lt;td&gt;5&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Connect and header timeout&lt;/td&gt;
&lt;td&gt;10 seconds&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Read idle timeout&lt;/td&gt;
&lt;td&gt;10 seconds&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Saved types are JPEG, PNG, GIF, WebP, BMP, TIFF, PDF, plain text, CSV, JSON, ZIP, gzip, 7z, RAR, DOCX, XLSX, PPTX, and legacy OLE Office files when the name or content type says so. HTML, SVG, JavaScript, scripts, WASM, and executables are rejected. An unknown binary is rejected. If the declared type and the sniffed type disagree and either side is an active rejected type, the file is not saved.&lt;/p&gt;

&lt;p&gt;Only public &lt;code&gt;http&lt;/code&gt; and &lt;code&gt;https&lt;/code&gt; are allowed. Private, loopback, and other non-public addresses are rejected before they are used. Archives that &lt;em&gt;are&lt;/em&gt; allowed are stored whole; they are not extracted.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;manifest.json&lt;/code&gt; is reserved inside the ZIP. Entry names are sanitized, and collisions get a numeric suffix.&lt;/p&gt;

&lt;h2&gt;
  
  
  Scheduling and integrations
&lt;/h2&gt;

&lt;p&gt;A run is a normal Apify Actor run. You can start it from Console, the Apify API, or an Apify schedule (for example, to keep dated, hashed copies of the same public files every week). Apify's Make, Zapier and n8n integrations and the Apify MCP server call Actors through that same API. The Actor itself does not create a webhook receiver or expose its own MCP endpoint, and standby mode is not enabled.&lt;/p&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Does it crawl a site or discover files?&lt;/strong&gt;&lt;br&gt;
No. You provide each direct file URL.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Can it use a signed or expiring link?&lt;/strong&gt;&lt;br&gt;
Query names such as &lt;code&gt;token&lt;/code&gt;, &lt;code&gt;signature&lt;/code&gt;, &lt;code&gt;X-Amz-Signature&lt;/code&gt;, and &lt;code&gt;X-Goog-Signature&lt;/code&gt; are rejected so those secrets are not stored. Use a URL you are allowed to fetch that does not put a credential in the query or fragment.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What if the URL is an HTML page?&lt;/strong&gt;&lt;br&gt;
The body is not treated as a saved image or document. In the measured sample, an HTML page was &lt;code&gt;rejected_disallowed_type&lt;/code&gt; and left out of the ZIP. That is type rejection, not a general challenge-page detector.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Will it open a ZIP it downloaded?&lt;/strong&gt;&lt;br&gt;
No. Archive bytes are stored. They are not extracted, and nothing downloaded is executed.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Are the files checked for malware or copyright?&lt;/strong&gt;&lt;br&gt;
No. You must be entitled to fetch and reuse the files. The Actor does not grant that right and does not claim the files are safe to open.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Does a failed URL still produce a status row?&lt;/strong&gt;&lt;br&gt;
Yes. The row records the failure and any bytes already read. Other URLs keep the status they earned. Only saved files are charged ($0.004 each, plus $0.003 once per run that saves at least one file). A rejected URL is not a saved file.&lt;/p&gt;

&lt;h2&gt;
  
  
  Try it
&lt;/h2&gt;

&lt;p&gt;Open &lt;a href="https://apify.com/steady-scrape/bulk-file-downloader" rel="noopener noreferrer"&gt;Bulk Image and File Downloader on the Apify Store&lt;/a&gt;, keep the example input, and run it. You get a ZIP with three small public files, a &lt;code&gt;MANIFEST&lt;/code&gt; with their SHA-256 checksums, and one status row per URL.&lt;/p&gt;

&lt;p&gt;If you need text out of the PDFs you downloaded, &lt;a href="https://apify.com/steady-scrape/pdf-text-markdown" rel="noopener noreferrer"&gt;PDF to Text Markdown Extractor&lt;/a&gt; is the companion Actor.&lt;/p&gt;

</description>
      <category>python</category>
      <category>ecommerce</category>
      <category>automation</category>
      <category>tutorial</category>
    </item>
    <item>
      <title>How to Capture Hashed Desktop and Mobile Screenshots for Release QA</title>
      <dc:creator>SteadyScrape</dc:creator>
      <pubDate>Tue, 10 Nov 2026 23:02:10 +0000</pubDate>
      <link>https://dev.to/steadyscrape/how-to-capture-hashed-desktop-and-mobile-screenshots-for-release-qa-p6g</link>
      <guid>https://dev.to/steadyscrape/how-to-capture-hashed-desktop-and-mobile-screenshots-for-release-qa-p6g</guid>
      <description>&lt;p&gt;After a docs change, a pricing-page update or a client preview goes live, someone usually asks: &lt;em&gt;can you show me what the public page looked like on desktop and on a phone?&lt;/em&gt; A screenshot dropped into Slack answers the visual question. It doesn't answer the follow-ups: is this the file we archived, when was it taken, which viewport produced it, and did the page actually load or was it a 404?&lt;/p&gt;

&lt;p&gt;This walkthrough uses the &lt;strong&gt;Website Screenshot Evidence Packs&lt;/strong&gt; Actor from SteadyScrape (steady-scrape) on Apify. You give it up to five public &lt;code&gt;http&lt;/code&gt; or &lt;code&gt;https&lt;/code&gt; URLs. For each one it saves a PNG in a fixed viewport, the SHA-256 of that file, the page's HTTP status and final URL, a UTC time from the run's clock, a JSON manifest and a ZIP. The result is an ordinary webpage capture with integrity hashes. It is &lt;strong&gt;not notarized&lt;/strong&gt;, not legal proof, and not a guarantee of historical truth.&lt;/p&gt;

&lt;h2&gt;
  
  
  The problem
&lt;/h2&gt;

&lt;p&gt;QA engineers, agency account teams and developers need a repeatable way to:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Capture the same public URL at a fixed desktop size and a fixed mobile size.&lt;/li&gt;
&lt;li&gt;Hand a teammate both the PNG and a checksum they can recompute.&lt;/li&gt;
&lt;li&gt;Know whether the page really answered &lt;code&gt;200&lt;/code&gt;, or whether the "screenshot" is of an error page or a redirect.&lt;/li&gt;
&lt;li&gt;Keep a manifest that says what happened to every attempt.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;It's a poor fit for logged-in pages, personal profiles or long-term legal archives. The Actor opens each URL directly. It doesn't crawl links, submit forms, log in, solve captchas or run JavaScript you supply. It also doesn't try to decide whether a &lt;code&gt;200&lt;/code&gt; page is a login wall or a challenge page; it records the status the server sent.&lt;/p&gt;

&lt;h2&gt;
  
  
  What a capture record contains
&lt;/h2&gt;

&lt;p&gt;Each dataset row is one URL and one viewport.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Field&lt;/th&gt;
&lt;th&gt;What it means&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;status&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;captured&lt;/code&gt;, &lt;code&gt;partial&lt;/code&gt;, &lt;code&gt;refused&lt;/code&gt;, &lt;code&gt;blocked&lt;/code&gt;, &lt;code&gt;unsupported&lt;/code&gt;, &lt;code&gt;failed&lt;/code&gt;, &lt;code&gt;empty&lt;/code&gt; or &lt;code&gt;limit_exceeded&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;url&lt;/code&gt;, &lt;code&gt;finalUrl&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Sanitized URL you asked for, and where the page ended up after redirects&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;viewport&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;desktop&lt;/code&gt; or &lt;code&gt;mobile&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;httpStatus&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;HTTP status of the main page, recorded by the browser session&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;sha256&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;SHA-256 of the stored PNG&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;capturedAt&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;UTC time from the run's clock, not a timestamp authority&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;byteLength&lt;/code&gt;, &lt;code&gt;width&lt;/code&gt;, &lt;code&gt;height&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Size of the stored PNG&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;refusedSubresourceCount&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Requests the browser refused because they pointed at private or local addresses&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;usable&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;true&lt;/code&gt; only for a 2xx page with a stored PNG and nothing refused&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;kvKey&lt;/code&gt;, &lt;code&gt;archiveEntry&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Where the PNG lives in the run's key-value store and in the ZIP&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;reasonCode&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Why a row is partial, refused or failed&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The run summary is in &lt;code&gt;OUTPUT&lt;/code&gt;. The ZIP holds &lt;code&gt;manifest.json&lt;/code&gt; plus PNGs named like &lt;code&gt;captures/01-desktop-&amp;lt;hash&amp;gt;.png&lt;/code&gt;. It contains no HTML or scripts.&lt;/p&gt;

&lt;h2&gt;
  
  
  Running it in Apify Console
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Open &lt;a href="https://apify.com/steady-scrape/website-screenshot-evidence-packs" rel="noopener noreferrer"&gt;Website Screenshot Evidence Packs&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;Paste one to five public URLs. Don't add usernames, passwords or tokens; credential URLs are refused.&lt;/li&gt;
&lt;li&gt;Set &lt;code&gt;viewports&lt;/code&gt; to &lt;code&gt;desktop&lt;/code&gt;, &lt;code&gt;mobile&lt;/code&gt; or both. Desktop is &lt;strong&gt;1280×800&lt;/strong&gt;, mobile is &lt;strong&gt;390×844&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;Leave &lt;code&gt;fullPage&lt;/code&gt; off for a viewport-only shot. With it on, the capture is the full page, cut off at 4,000 pixels tall.&lt;/li&gt;
&lt;li&gt;Run it, then open the dataset (one row per URL and viewport) and the key-value store (&lt;code&gt;OUTPUT&lt;/code&gt;, the manifest, the PNGs and the ZIP).
&lt;/li&gt;
&lt;/ol&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"urls"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"https://example.com/"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"viewports"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"desktop"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"mobile"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"fullPage"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Calling it from Python
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;apify_client&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;ApifyClient&lt;/span&gt;

&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;ApifyClient&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;environ&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"APIFY_TOKEN"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;run&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;actor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"steady-scrape/website-screenshot-evidence-packs"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;run_input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="s2"&gt;"urls"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"https://example.com/"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"https://www.iana.org/help/example-domains"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"viewports"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"desktop"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"mobile"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"fullPage"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;False&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dataset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"defaultDatasetId"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;iterate_items&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"viewport"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"httpStatus"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"sha256"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"url"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;To check a file later, download the PNG (or the ZIP from the &lt;code&gt;archiveKey&lt;/code&gt; in &lt;code&gt;OUTPUT&lt;/code&gt;) and compare:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;sha256sum captures/01-desktop-*.png
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  A real row
&lt;/h2&gt;

&lt;p&gt;From a public run on &lt;code&gt;https://example.com/&lt;/code&gt; (desktop):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"url"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"https://example.com/"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"finalUrl"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"https://example.com/"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"viewport"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"desktop"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"captured"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"reasonCode"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"httpStatus"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;200&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"capturedAt"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"2026-11-10T22:49:06.620Z"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"sha256"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"9dab6e635ca5a3f4c2220115121db3387da42e6890aae19da30d0395e2039066"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"byteLength"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;74812&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"width"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1280&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"height"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;800&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"fullPage"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"archiveEntry"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"captures/01-desktop-9dab6e635ca5.png"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"refusedSubresourceCount"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"usable"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If a URL points at a private address (say &lt;code&gt;http://169.254.169.254/&lt;/code&gt;), the row is &lt;code&gt;refused&lt;/code&gt; with &lt;code&gt;reasonCode&lt;/code&gt; &lt;code&gt;non_public_address&lt;/code&gt; and nothing is fetched. A page that answers &lt;code&gt;404&lt;/code&gt; or &lt;code&gt;500&lt;/code&gt; still gets a PNG, but the row is &lt;code&gt;partial&lt;/code&gt; with &lt;code&gt;not_success_http_status&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Re-running later against a live page can produce different bytes and a different &lt;code&gt;sha256&lt;/code&gt;. That's the point: the hash tells you exactly which file you're looking at.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it costs
&lt;/h2&gt;

&lt;p&gt;Pay per event:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;$0.003&lt;/strong&gt; per usable screenshot (one viewport of a page that answered 2xx, with nothing refused).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;$0.003&lt;/strong&gt; once per run that saved at least one usable screenshot.&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;What you run&lt;/th&gt;
&lt;th&gt;Math&lt;/th&gt;
&lt;th&gt;Total&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1 URL, desktop only&lt;/td&gt;
&lt;td&gt;1 × $0.003 + $0.003&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;$0.006&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;1 URL, desktop + mobile&lt;/td&gt;
&lt;td&gt;2 × $0.003 + $0.003&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;$0.009&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;5 URLs × both viewports&lt;/td&gt;
&lt;td&gt;10 × $0.003 + $0.003&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;$0.033&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Refused URLs, failed loads, timeouts, non-2xx pages and partial captures are free, and a run with no usable screenshot pays nothing. There's no start fee or compute charge on top, and you can set a maximum charge per run.&lt;/p&gt;

&lt;h2&gt;
  
  
  Limits
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Up to 5 URLs and 10 screenshots per run, one at a time.&lt;/li&gt;
&lt;li&gt;Desktop 1280×800, mobile 390×844, device scale 1.&lt;/li&gt;
&lt;li&gt;About 15 seconds to load each page; 180 seconds per run.&lt;/li&gt;
&lt;li&gt;Public &lt;code&gt;http&lt;/code&gt;/&lt;code&gt;https&lt;/code&gt; only. Userinfo, credential query keys, &lt;code&gt;localhost&lt;/code&gt;, &lt;code&gt;.local&lt;/code&gt;, &lt;code&gt;.internal&lt;/code&gt; and private or reserved IPs are refused.&lt;/li&gt;
&lt;li&gt;Retention follows your Apify plan. The manifest's timestamp doesn't delete anything, so download the ZIP if you need it longer.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Scheduling and integrations
&lt;/h2&gt;

&lt;p&gt;Start runs from Console, the Apify API or an Apify schedule (for example, every Monday on your top pages). Make, Zapier and n8n Apify modules, or Apify's MCP server for AI agents, can start the same run.&lt;/p&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Is a screenshot legal proof?&lt;/strong&gt;&lt;br&gt;
No. It's a webpage capture with a hash of the bytes this run stored, timed by the run's clock.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Can I screenshot a logged-in page?&lt;/strong&gt;&lt;br&gt;
No. Cookies, headers, passwords and proxy credentials aren't accepted.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Am I charged for a 404 or a failed page?&lt;/strong&gt;&lt;br&gt;
No. Only 2xx pages with a stored PNG and nothing refused are charged.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What sizes are desktop and mobile?&lt;/strong&gt;&lt;br&gt;
1280×800 and 390×844, fixed.&lt;/p&gt;




&lt;p&gt;Try it: &lt;a href="https://apify.com/steady-scrape/website-screenshot-evidence-packs" rel="noopener noreferrer"&gt;Website Screenshot Evidence Packs on Apify&lt;/a&gt;. More tools from SteadyScrape (steady-scrape) are on &lt;a href="https://apify.com/steady-scrape" rel="noopener noreferrer"&gt;our Apify profile&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>qa</category>
      <category>webdev</category>
      <category>testing</category>
      <category>python</category>
    </item>
    <item>
      <title>Get Google Images Results as JSON in Python, With Real Image Sizes, for $0.012 a Search</title>
      <dc:creator>SteadyScrape</dc:creator>
      <pubDate>Tue, 10 Nov 2026 05:31:08 +0000</pubDate>
      <link>https://dev.to/steadyscrape/get-google-images-results-as-json-in-python-with-real-image-sizes-for-0012-a-search-1lga</link>
      <guid>https://dev.to/steadyscrape/get-google-images-results-as-json-in-python-with-real-image-sizes-for-0012-a-search-1lga</guid>
      <description>&lt;p&gt;Product teams often need one question answered: &lt;em&gt;which images show up when shoppers search this product phrase, and from which sites?&lt;/em&gt; Building that yourself means wiring up a search source, handling country and language codes, removing duplicate URLs, and deciding what to do when a search comes back empty.&lt;/p&gt;

&lt;p&gt;Usually you don't need the image files. For assortment research, marketplace checks or a creative brief, the useful output is metadata: the image URL, the page it came from, the site, the title, the rank, and the pixel size.&lt;/p&gt;

&lt;p&gt;This walkthrough uses SteadyScrape's &lt;strong&gt;&lt;a href="https://apify.com/steady-scrape/google-images-api" rel="noopener noreferrer"&gt;Google Images API&lt;/a&gt;&lt;/strong&gt; Actor on Apify. It returns that metadata as JSON for public, non-personal keywords. It does &lt;strong&gt;not&lt;/strong&gt; download full files, rehost or license images. A result row grants no reuse, training or copyright rights.&lt;/p&gt;

&lt;h2&gt;
  
  
  What a run returns
&lt;/h2&gt;

&lt;p&gt;Each keyword writes one &lt;strong&gt;status row&lt;/strong&gt; and one &lt;strong&gt;image row&lt;/strong&gt; per result, up to 100.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Field&lt;/th&gt;
&lt;th&gt;What it means&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;rank&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Position in the image results&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;imageUrl&lt;/code&gt;, &lt;code&gt;thumbnailUrl&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Source image URL and Google's thumbnail&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;sourcePageUrl&lt;/code&gt;, &lt;code&gt;sourceDomain&lt;/code&gt;, &lt;code&gt;sourceName&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;The page that shows the image, its domain, and the site name Google shows&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;title&lt;/code&gt;, &lt;code&gt;altText&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Result title and alt text&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;width&lt;/code&gt;, &lt;code&gt;height&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Pixel size, read from the image file header. Null when it couldn't be read&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;queryStatus&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;success&lt;/code&gt;, &lt;code&gt;partial&lt;/code&gt;, &lt;code&gt;empty&lt;/code&gt;, &lt;code&gt;timeout&lt;/code&gt;, &lt;code&gt;failed&lt;/code&gt;, &lt;code&gt;blocked_operator&lt;/code&gt;, &lt;code&gt;not_run&lt;/code&gt;, and more&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;relatedSearches&lt;/code&gt;, &lt;code&gt;googleImagesUrl&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;On status rows: related searches and a link to the same Google Images page&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h3&gt;
  
  
  Where width and height come from
&lt;/h3&gt;

&lt;p&gt;The search results don't include pixel sizes. So the Actor asks each image host for only the &lt;strong&gt;first bytes&lt;/strong&gt; of the file (an HTTP range request) and stops reading as soon as the size is in the header. It reads PNG, JPEG, GIF, WebP, BMP and AVIF, contacts public hosts only, and spends about 12 seconds per keyword on this step.&lt;/p&gt;

&lt;p&gt;Some hosts refuse range requests or answer slowly, so expect a share of nulls. In my test run for &lt;code&gt;blue ceramic bowl&lt;/code&gt;, 51 of 100 images got a size, reading about 165 KB in total. You can turn the step off with &lt;code&gt;"fillDimensions": false&lt;/code&gt; for the fastest runs. It never changes the status or the price.&lt;/p&gt;

&lt;h2&gt;
  
  
  Run it from Python
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;apify_client&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;ApifyClient&lt;/span&gt;

&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;ApifyClient&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;environ&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"APIFY_TOKEN"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;run&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;actor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"steady-scrape/google-images-api"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;run_input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="s2"&gt;"keywords"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"red ceramic mug"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"blue ceramic bowl"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"country"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"us"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"language"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"en"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"depth"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dataset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"defaultDatasetId"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;iterate_items&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"recordType"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;"image"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"rank"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"sourceName"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"width"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"height"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"imageUrl"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"STATUS"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"keyword"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"queryStatus"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"imageCount"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Need a different country per keyword? Use the advanced form:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"queries"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"keyword"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"red ceramic mug"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"country"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"us"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"language"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"en"&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"keyword"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"rote Keramiktasse"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"country"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"de"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"language"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"de"&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Want to try it with no search? &lt;code&gt;"dryRun": true&lt;/code&gt; returns labeled sample rows and nothing is charged.&lt;/p&gt;

&lt;h2&gt;
  
  
  A real run
&lt;/h2&gt;

&lt;p&gt;A single keyword (&lt;code&gt;blue ceramic bowl&lt;/code&gt;, US, English, depth 100):&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;100 image rows plus a status row with 20 related searches&lt;/li&gt;
&lt;li&gt;search took about 28 seconds, then about 13 seconds of size fill-in&lt;/li&gt;
&lt;li&gt;51 of the 100 images got a width and height&lt;/li&gt;
&lt;li&gt;the same run included &lt;code&gt;site:example.com mug&lt;/code&gt;, which was rejected as &lt;code&gt;blocked_operator&lt;/code&gt; and not charged&lt;/li&gt;
&lt;li&gt;charged: one $0.010 query event plus the $0.002 run event = &lt;strong&gt;$0.012&lt;/strong&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What it costs
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;$0.010&lt;/strong&gt; per keyword that returns at least one image (including a &lt;code&gt;partial&lt;/code&gt; result with at least one image)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;$0.002&lt;/strong&gt; once per run, only if at least one keyword returned images&lt;/li&gt;
&lt;li&gt;Empty, failed and timed-out searches are &lt;strong&gt;free&lt;/strong&gt;, and so is test mode&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;What you run&lt;/th&gt;
&lt;th&gt;Cost&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1 keyword with images&lt;/td&gt;
&lt;td&gt;$0.012&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;10 keywords with images in one run&lt;/td&gt;
&lt;td&gt;$0.102&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;10 keywords, 2 of them find nothing&lt;/td&gt;
&lt;td&gt;$0.082&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;a run where nothing is found&lt;/td&gt;
&lt;td&gt;$0.000&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Apify also adds its standard &lt;code&gt;apify-actor-start&lt;/code&gt; event ($0.00005 per run at the default 256 MB). You can set a maximum cost per run, and the Actor stops sending new searches before it would go over.&lt;/p&gt;

&lt;h2&gt;
  
  
  Limits worth knowing
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Up to 10 keywords per run, up to 100 images each, one search at a time.&lt;/li&gt;
&lt;li&gt;Each search can take up to about 95 seconds. A search that may already have been answered is never sent twice, so you aren't charged twice.&lt;/li&gt;
&lt;li&gt;Advanced operators (&lt;code&gt;site:&lt;/code&gt;, a leading minus and similar) are rejected for free. Don't submit person or face lookups. Emails, phone numbers and profile links are rejected.&lt;/li&gt;
&lt;li&gt;Live search needs a paid Apify plan.&lt;/li&gt;
&lt;li&gt;Chinese must be &lt;code&gt;zh-CN&lt;/code&gt; or &lt;code&gt;zh-TW&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Does this download the images?&lt;/strong&gt; No. It returns links. With size fill-in on, it reads only the first bytes of each file.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Can I use the images in my store or for training?&lt;/strong&gt; Not based on this output. Rows are search metadata, so check each source page for rights.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What does &lt;code&gt;partial&lt;/code&gt; mean?&lt;/strong&gt; The search returned images, but a few results were incomplete and were left out. It's charged like a success because you got images.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Can I schedule it?&lt;/strong&gt; Yes. It's a normal Apify Actor, so you can use Apify schedules, the API, webhooks, Make, Zapier, n8n, or the Apify MCP server for AI agents.&lt;/p&gt;




&lt;p&gt;Try it here: &lt;strong&gt;&lt;a href="https://apify.com/steady-scrape/google-images-api" rel="noopener noreferrer"&gt;Google Images API on Apify&lt;/a&gt;&lt;/strong&gt;. Results come from Google Images via the DataForSEO API. The Actor is not affiliated with Google.&lt;/p&gt;

</description>
      <category>python</category>
      <category>ecommerce</category>
      <category>api</category>
      <category>seo</category>
    </item>
    <item>
      <title>Remote for Whom? Building a Remote-Eligibility Classifier for Job Postings in Python</title>
      <dc:creator>SteadyScrape</dc:creator>
      <pubDate>Tue, 10 Nov 2026 00:59:37 +0000</pubDate>
      <link>https://dev.to/steadyscrape/remote-for-whom-building-a-remote-eligibility-classifier-for-job-postings-in-python-10lp</link>
      <guid>https://dev.to/steadyscrape/remote-for-whom-building-a-remote-eligibility-classifier-for-job-postings-in-python-10lp</guid>
      <description>&lt;p&gt;If you've ever searched for remote jobs from outside the US, you know the problem. The listing says &lt;strong&gt;Remote&lt;/strong&gt;. You read the whole description, and in the last paragraph: &lt;em&gt;\"Candidates must reside in one of the following states.\"&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;If you're a job seeker, our earlier post on &lt;a href="https://dev.to/steadyscrape/how-to-find-remote-jobs-you-can-actually-apply-to-from-your-country-python-58m6"&gt;finding remote jobs you can actually apply to from your country&lt;/a&gt; is the short version: one script, one filter, done.&lt;/p&gt;

&lt;p&gt;This post is for the people building the filter: a remote job board, a newsletter or a job-matching tool. For them, "is it remote?" is the easy question. The useful question is &lt;strong&gt;"remote for whom?"&lt;/strong&gt; Below are the signals to look at, a small Python classifier you can extend, and the false positives that will catch you out. If you need the raw postings first, &lt;a href="https://dev.to/steadyscrape/how-to-pull-job-postings-straight-from-company-career-sites-greenhouse-lever-ashby-workday-as-2co0"&gt;this post shows how to pull them from Greenhouse, Lever, Ashby and Workday&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Three questions, not one
&lt;/h2&gt;

&lt;p&gt;For each posting you want to know:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Remote type:&lt;/strong&gt; fully remote, remote-optional (remote possible but offices listed), or hybrid.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Scope:&lt;/strong&gt; worldwide, a region (EMEA, LATAM, "Americas"), specific countries, or unspecified.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Time zone:&lt;/strong&gt; some roles say "UTC-3 to UTC+1" or "must work PST hours", which matters as much as the country.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Where the signals are
&lt;/h2&gt;

&lt;p&gt;In order of reliability:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The ATS flag.&lt;/strong&gt; Greenhouse, Lever, Ashby and others let employers mark a workplace type (remote, hybrid, on-site). When it's set, it's the best signal, but many postings leave it empty.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The location field.&lt;/strong&gt; &lt;code&gt;Remote - US&lt;/code&gt;, &lt;code&gt;Anywhere in EMEA&lt;/code&gt;, &lt;code&gt;Brazil (Remote)&lt;/code&gt;, &lt;code&gt;Hybrid - Berlin&lt;/code&gt;, or a list mixing offices and "Remote".&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The title.&lt;/strong&gt; &lt;code&gt;Senior Engineer (Remote)&lt;/code&gt;, &lt;code&gt;Work From Home&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The description.&lt;/strong&gt; "3 days a week in our office", "must be located in", "working hours overlap with CET". It's the richest source and the noisiest.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  A small classifier
&lt;/h2&gt;

&lt;p&gt;Here's a heuristic version that handles the location field, which is where most of the eligibility information lives:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;

&lt;span class="no"&gt;REGIONS&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"emea"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"EMEA"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"europe"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Europe"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"latam"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"LATAM"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"apac"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"APAC"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
           &lt;span class="s2"&gt;"americas"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Americas"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"north america"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"North America"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="no"&gt;COUNTRIES&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"us"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"US"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"usa"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"US"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"united states"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"US"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"canada"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"CA"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
             &lt;span class="s2"&gt;"uk"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"GB"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"united kingdom"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"GB"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"germany"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"DE"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"brazil"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"BR"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="no"&gt;FALSE_POSITIVES&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nb"&gt;compile&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="s2"&gt;"remote (sensing|pilot|patient|monitoring|support technician)"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;I&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;classify&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;location&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;title&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;""&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;text&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"{title} | {location}"&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;lower&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="no"&gt;FALSE_POSITIVES&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;search&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"remoteType"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="s2"&gt;"hybrid"&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;rtype&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"HYBRID"&lt;/span&gt;
    &lt;span class="k"&gt;elif&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;search&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="s2"&gt;"\b(remote|anywhere|work from home|wfh)\b"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;rtype&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"FULLY_REMOTE"&lt;/span&gt;
    &lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"remoteType"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="n"&gt;regions&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;sorted&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="no"&gt;REGIONS&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;items&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;search&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rf&lt;/span&gt;&lt;span class="s2"&gt;"\b{k}\b"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;countries&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;sorted&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="no"&gt;COUNTRIES&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;items&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;search&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rf&lt;/span&gt;&lt;span class="s2"&gt;"\b{k}\b"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;search&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="s2"&gt;"\b(anywhere|worldwide|global)\b"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;countries&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="n"&gt;regions&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;scope&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"WORLDWIDE"&lt;/span&gt;
    &lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;scope&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"COUNTRY"&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;countries&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="s2"&gt;"REGION"&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;regions&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="s2"&gt;"UNSPECIFIED"&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"remoteType"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;rtype&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"scope"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;scope&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"countries"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;countries&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"regions"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;regions&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;classify&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Remote - US"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="c1"&gt;# {'remoteType': 'FULLY_REMOTE', 'scope': 'COUNTRY', 'countries': ['US'], 'regions': []}&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;classify&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Anywhere in EMEA"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="c1"&gt;# {'remoteType': 'FULLY_REMOTE', 'scope': 'REGION', 'countries': [], 'regions': ['EMEA']}&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;classify&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Houston, TX"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;title&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"Remote Sensing Analyst"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="c1"&gt;# {'remoteType': None}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It's a starting point. Things it doesn't handle yet, and you'll hit all of them:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;"Remote" + office cities in one list.&lt;/strong&gt; &lt;code&gt;["Remote", "New York, NY", "London"]&lt;/code&gt; usually means remote is possible but not the default (remote-optional).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One role, twelve postings.&lt;/strong&gt; "Remote - Ohio", "Remote - Texas", … for the same job. Merge them and keep all locations.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Corporate "Home Office".&lt;/strong&gt; In some postings "Home Office" is the headquarters address, not working from home.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;"Temporarily remote" or "remote until relocation".&lt;/strong&gt; Not really remote.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Bare "Remote".&lt;/strong&gt; No scope at all. Treat it as unspecified rather than worldwide; many US employers mean "in the US".&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Time zones.&lt;/strong&gt; "PST hours", "UTC-3 to UTC+1", "±2 hours of CET" all need parsing into a UTC offset range if you want to filter by where someone lives.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Doing this at scale
&lt;/h2&gt;

&lt;p&gt;&lt;em&gt;Disclosure: we (SteadyScrape) built the tool below and get paid when it's used.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://apify.com/steady-scrape/remote-jobs-api" rel="noopener noreferrer"&gt;Remote Jobs API &amp;amp; Scraper: Filter by Country&lt;/a&gt; (&lt;code&gt;steady-scrape/remote-jobs-api&lt;/code&gt;) is an Apify Actor that runs this kind of classification, with the guards above, over an index of &lt;strong&gt;1.3 million+ open jobs from 37,000+ company career sites&lt;/strong&gt;, refreshed daily from the employers' public job board APIs. Each remote job gets:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;a &lt;code&gt;remoteType&lt;/code&gt; (&lt;code&gt;FULLY_REMOTE&lt;/code&gt;, &lt;code&gt;REMOTE_OPTIONAL&lt;/code&gt;, &lt;code&gt;HYBRID&lt;/code&gt;),&lt;/li&gt;
&lt;li&gt;a &lt;code&gt;remoteConfidence&lt;/code&gt; (high when the ATS flag and the text agree) and the &lt;code&gt;remoteEvidence&lt;/code&gt; behind it,&lt;/li&gt;
&lt;li&gt;a &lt;code&gt;remoteScope&lt;/code&gt; (&lt;code&gt;WORLDWIDE&lt;/code&gt;, &lt;code&gt;REGION&lt;/code&gt;, &lt;code&gt;COUNTRY&lt;/code&gt;, &lt;code&gt;UNSPECIFIED&lt;/code&gt;) with &lt;code&gt;eligibleCountries&lt;/code&gt; and &lt;code&gt;eligibleRegions&lt;/code&gt;,&lt;/li&gt;
&lt;li&gt;any stated time zones as a UTC offset range (&lt;code&gt;utcOffsetMin&lt;/code&gt;, &lt;code&gt;utcOffsetMax&lt;/code&gt;).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Duplicate postings of the same role are merged, with every location kept in &lt;code&gt;mergedPostings&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;For a board or newsletter, the filter is the audience you serve. Here's a feed of senior data roles for readers in Europe who want at most two hours' difference from Lisbon:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;apify_client&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;ApifyClient&lt;/span&gt;

&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;ApifyClient&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;environ&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"APIFY_TOKEN"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;run&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;actor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"steady-scrape/remote-jobs-api"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run_input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="s2"&gt;"keywords"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"data"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"analytics"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"regions"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"Europe"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"workerTimezone"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Europe/Lisbon"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"maxTimezoneDiffHours"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"seniority"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"SENIOR"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"LEAD"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"postedWithinDays"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"maxResults"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;40&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;job&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dataset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"defaultDatasetId"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;iterate_items&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;job&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"remoteConfidence"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;job&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"remoteScope"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;job&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"remoteEvidence"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
          &lt;span class="n"&gt;job&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"title"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"|"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;job&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"company"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  What came back
&lt;/h2&gt;

&lt;p&gt;We ran exactly that input on November 9, 2026, with the run's maximum charge capped at $0.02, so it stopped after 15 jobs. The run's &lt;code&gt;OUTPUT&lt;/code&gt; record shows what the classifier and the dedupe did:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;103 postings matched, and 79 were unique roles.&lt;/strong&gt; The other 24 were the same role posted in several locations, which is the "one role, twelve postings" problem from the list above.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;All 15 returned jobs were &lt;code&gt;FULLY_REMOTE&lt;/code&gt;.&lt;/strong&gt; 9 were high confidence and 6 medium.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;11 had a &lt;code&gt;COUNTRY&lt;/code&gt; scope and 4 a &lt;code&gt;REGION&lt;/code&gt; scope.&lt;/strong&gt; For example, one senior product analyst role was open in 10 European countries (PT, IE, IT, RO, FR, NL, DE, CH, GB, ES), and three roles were limited to Spain (ES).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The evidence is readable.&lt;/strong&gt; Each job lists the signals it was classified on. Five of the 15 had two, such as &lt;code&gt;["ats:REMOTE", "location:Remote - Europe"]&lt;/code&gt;, where the ATS flag and the location text agree. The rest had the ATS flag alone, with the eligible countries taken from the location list.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The search itself took about 4 seconds.&lt;/p&gt;

&lt;p&gt;Set &lt;code&gt;onlyNewSinceLastRun: true&lt;/code&gt; and schedule it daily, and you have the feed for a remote job newsletter or a board. Add &lt;code&gt;includeDescriptions: true&lt;/code&gt; to fetch full descriptions for Greenhouse, Lever, Ashby and Workday jobs. That also refines the classification ("3 days a week in our office", "must be located in…") and skips jobs that have closed. If your board shows pay or tech stacks, &lt;code&gt;enrichJobs: true&lt;/code&gt; adds an annualized salary range, skills, seniority and a short summary, with recruiter names and contact details stripped.&lt;/p&gt;

&lt;p&gt;Honest limits:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The index only covers the supported ATS platforms.&lt;/li&gt;
&lt;li&gt;Eligibility is parsed from what the employer wrote, so check &lt;code&gt;remoteConfidence&lt;/code&gt; and &lt;code&gt;remoteEvidence&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Bare "Remote" postings come back as &lt;code&gt;UNSPECIFIED&lt;/code&gt;. These are excluded from country, region and time-zone filters unless you set &lt;code&gt;includeUnspecifiedEligibility: true&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What it costs
&lt;/h2&gt;

&lt;p&gt;On Apify's free plan it's &lt;strong&gt;$0.0015 per remote job returned&lt;/strong&gt; ($1.50 per 1,000), plus Apify's $0.00005 start fee per run. On paid Apify plans it's $0.0013 to $0.001 per job. Jobs that turn out to be closed during the description fetch are skipped and not charged.&lt;/p&gt;

&lt;p&gt;Enrichment is an optional add-on: &lt;strong&gt;+$0.004 per enriched job&lt;/strong&gt; on the free plan ($0.0036 to $0.003 on paid plans), so an enriched job is $0.0055 in total.&lt;/p&gt;

&lt;p&gt;Examples at free-plan prices: a newsletter that runs daily and gets about 100 new jobs a day pays 30 × 100 × $0.0015 = &lt;strong&gt;$4.50 a month&lt;/strong&gt;. The 40-job run above, uncapped, costs $0.06.&lt;/p&gt;

&lt;p&gt;If you've found a "remote" phrasing that breaks classifiers, post it in the comments. That's how the guard list grows.&lt;/p&gt;

</description>
      <category>python</category>
      <category>jobs</category>
      <category>remote</category>
      <category>datascience</category>
    </item>
    <item>
      <title>Who's Hiring for This Job Title, and What Do They Pay? Searching 37,000 Company Career Sites With Python</title>
      <dc:creator>SteadyScrape</dc:creator>
      <pubDate>Tue, 10 Nov 2026 00:56:53 +0000</pubDate>
      <link>https://dev.to/steadyscrape/whos-hiring-for-this-job-title-and-what-do-they-pay-searching-37000-company-career-sites-with-4fh2</link>
      <guid>https://dev.to/steadyscrape/whos-hiring-for-this-job-title-and-what-do-they-pay-searching-37000-company-career-sites-with-4fh2</guid>
      <description>&lt;p&gt;The earlier posts in this series start from a list of companies. &lt;a href="https://dev.to/steadyscrape/how-to-pull-job-postings-straight-from-company-career-sites-greenhouse-lever-ashby-workday-as-2co0"&gt;Pulling job postings straight from Greenhouse, Lever, Ashby and Workday&lt;/a&gt; shows the raw ATS endpoints, and &lt;a href="https://dev.to/steadyscrape/how-to-get-a-slack-alert-when-a-company-posts-a-new-job-greenhouse-lever-ashby-with-n8n-or-129a"&gt;getting a Slack alert when a company posts a new job&lt;/a&gt; turns that into a daily alert for the companies you're watching.&lt;/p&gt;

&lt;p&gt;This post covers the opposite question: &lt;strong&gt;you know the job title, and you want to find out which companies are hiring for it and what they pay.&lt;/strong&gt; You can't loop over a company list for that, because you don't know the companies yet. You have to search by title across every career site you can reach.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why search the employer's career site at all
&lt;/h2&gt;

&lt;p&gt;Job boards repost listings late, show the same role five times and keep jobs up after the employer has closed them. The employer's own applicant tracking system (ATS) is the source of truth: the job shows up there the day it opens and disappears the day it closes. Since many US states now require pay ranges in postings, the ATS is also where the salary data is.&lt;/p&gt;

&lt;p&gt;The catch is that "every career site" means tens of thousands of separate boards on a dozen ATS platforms, each with its own field names.&lt;/p&gt;

&lt;h2&gt;
  
  
  Part 1: a title matcher that isn't too literal
&lt;/h2&gt;

&lt;p&gt;Searching by title only works if the matcher handles real titles. "engineer" should match &lt;em&gt;Engineers&lt;/em&gt; and &lt;em&gt;Engineering Manager&lt;/em&gt;; "data engineer" should match &lt;em&gt;Senior Data Engineer, Platform&lt;/em&gt;; and you need a way to say "exactly this phrase". This version matches a term when the title contains &lt;em&gt;all&lt;/em&gt; of its words, on word starts, so plurals and "-ing" forms count. Quoted terms are exact phrases.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;compile_term&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;term&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;term&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;term&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;term&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;startswith&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'"'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;term&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;endswith&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'"'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nb"&gt;compile&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rf&lt;/span&gt;&lt;span class="s2"&gt;"(?&amp;lt;!\w){re.escape(term[1:-1])}(?!\w)"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;I&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nb"&gt;compile&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rf&lt;/span&gt;&lt;span class="s2"&gt;"(?&amp;lt;!\w){re.escape(w)}"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;I&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;w&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;term&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;matches&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;title&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;terms&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;exclude&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;any&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;search&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rf&lt;/span&gt;&lt;span class="s2"&gt;"(?&amp;lt;!\w){re.escape(x)}"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;title&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;I&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;exclude&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kc"&gt;False&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nb"&gt;any&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;all&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;search&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;title&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;compile_term&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;terms&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;matches&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Senior Data Engineer, Platform"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"data engineer"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;       &lt;span class="c1"&gt;# True&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;matches&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Engineering Manager"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"engineer"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;                       &lt;span class="c1"&gt;# True&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;matches&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Data Engineering Intern"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"data engineer"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"intern"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;  &lt;span class="c1"&gt;# False&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;matches&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Analytics Engineer II"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;'"analytics engineer"'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;         &lt;span class="c1"&gt;# True&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Part 2: count roles, not postings
&lt;/h2&gt;

&lt;p&gt;If you're counting who is hiring, duplicates skew everything. Employers often post one role in several cities, so a company with one opening can look like it has eight. A key of company plus normalized title is usually enough to collapse them:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;role_key&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;job&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;title&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sub&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="s2"&gt;"\W+"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;" "&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;job&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"title"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;lower&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;job&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"company"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;lower&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;title&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;unique_roles&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;jobs&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;seen&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;out&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kt"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;j&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;jobs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nf"&gt;role_key&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;j&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;seen&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;seen&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;role_key&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;j&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;j&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;out&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Part 3: normalize the pay
&lt;/h2&gt;

&lt;p&gt;Pay ranges come in every shape: structured fields on some ATS platforms, free text like "$70K - $100K" or "$55/hour" in the description on others. For a benchmark you want one shape: a minimum and maximum per year, in one currency, with a note of where the number came from. Hourly rates need annualizing (2,080 hours is the usual convention), and anything you parse from text deserves less trust than a structured field.&lt;/p&gt;

&lt;h2&gt;
  
  
  Doing the search part without building a crawler
&lt;/h2&gt;

&lt;p&gt;Parts 1 to 3 are a few dozen lines. The part that isn't is the index: a list of tens of thousands of board identifiers, a parser per platform, a daily refresh with retries and rate limits, and a store you can search in seconds. That's a crawler and a database, not a script.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Disclosure: we (SteadyScrape) built the tool below and get paid when it's used.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://apify.com/steady-scrape/career-jobs-search" rel="noopener noreferrer"&gt;Job Search API: Career Site Job Listings&lt;/a&gt; (&lt;code&gt;steady-scrape/career-jobs-search&lt;/code&gt;) is an Apify Actor that searches an index of &lt;strong&gt;1.3 million+ open jobs from 37,000+ company career sites&lt;/strong&gt; (Greenhouse, Lever, Ashby, Workday, BambooHR, Workable, Recruitee, Personio, Rippling, Teamtailor, Breezy HR, Gem and Oracle Cloud HCM), rebuilt daily from the employers' public job board APIs. It uses the same title rules as the matcher above and removes duplicates. With &lt;code&gt;enrichJobs&lt;/code&gt; on, it reads each returned job live from the employer's board and adds an &lt;code&gt;enrichment&lt;/code&gt; object: an annualized salary range with its source, a list of skills and tools, seniority, years of experience, work arrangement and a short summary. Recruiter names, emails and phone numbers are stripped.&lt;/p&gt;

&lt;p&gt;Here's a small salary and skills benchmark for one title:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;collections&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;Counter&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;apify_client&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;ApifyClient&lt;/span&gt;

&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;ApifyClient&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;environ&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"APIFY_TOKEN"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;run&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;actor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"steady-scrape/career-jobs-search"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run_input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="s2"&gt;"keywords"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;'"analytics engineer"'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"countries"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"US"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"postedWithinDays"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;14&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"salaryOnly"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;     &lt;span class="c1"&gt;# only postings that publish pay&lt;/span&gt;
    &lt;span class="s2"&gt;"enrichJobs"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;     &lt;span class="c1"&gt;# salary, skills, seniority&lt;/span&gt;
    &lt;span class="s2"&gt;"maxResults"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;50&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;jobs&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kt"&gt;list&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dataset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"defaultDatasetId"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;iterate_items&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;skills&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Counter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;j&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;jobs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;e&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;j&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"enrichment"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="n"&gt;pay&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"salary"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="n"&gt;skills&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;update&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"skills"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s1"&gt;'{j["title"]} | {j["company"]} | {j["locationText"]} | '&lt;/span&gt;
          &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s1"&gt;'${pay.get("annualMin"):,}-${pay.get("annualMax"):,} ({pay.get("source")})'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;skills&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;most_common&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  What came back
&lt;/h2&gt;

&lt;p&gt;We ran it on November 9, 2026 with the run's maximum charge capped at $0.02, so it stopped after 3 jobs. Seven US postings with a published pay range matched in the previous 14 days. One had closed by the time the Actor fetched it, so it was skipped and not charged. The first three:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Title&lt;/th&gt;
&lt;th&gt;Company board&lt;/th&gt;
&lt;th&gt;Location&lt;/th&gt;
&lt;th&gt;Annual pay&lt;/th&gt;
&lt;th&gt;Source&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Senior Analytics Engineer&lt;/td&gt;
&lt;td&gt;instructure&lt;/td&gt;
&lt;td&gt;US-REMOTE&lt;/td&gt;
&lt;td&gt;$145,000 to $155,000&lt;/td&gt;
&lt;td&gt;ATS pay field&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Analytics Engineer II&lt;/td&gt;
&lt;td&gt;branchinsurance&lt;/td&gt;
&lt;td&gt;United States - Remote&lt;/td&gt;
&lt;td&gt;$70,000 to $100,000&lt;/td&gt;
&lt;td&gt;ATS pay field&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Staff Analytics Engineer, GTM&lt;/td&gt;
&lt;td&gt;scribe&lt;/td&gt;
&lt;td&gt;San Francisco&lt;/td&gt;
&lt;td&gt;$200,000 to $250,000&lt;/td&gt;
&lt;td&gt;ATS pay field&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The skills extracted from those three descriptions overlapped a lot: &lt;strong&gt;dbt and SQL were in all three, and Snowflake and Looker were in two&lt;/strong&gt;, alongside BigQuery, Airflow, Mixpanel and Python. Three jobs aren't a benchmark, but the same script with a higher cap gives you one for any title and country. The run took about 10 seconds, and its &lt;code&gt;OUTPUT&lt;/code&gt; record reported an index of 1,373,175 jobs from 37,515 career sites, refreshed that morning.&lt;/p&gt;

&lt;p&gt;Each job also has the same normalized fields regardless of ATS: &lt;code&gt;title&lt;/code&gt;, &lt;code&gt;company&lt;/code&gt;, &lt;code&gt;department&lt;/code&gt;, &lt;code&gt;locationText&lt;/code&gt;, parsed &lt;code&gt;locations&lt;/code&gt; with country codes, &lt;code&gt;isRemote&lt;/code&gt;, &lt;code&gt;workplaceType&lt;/code&gt;, &lt;code&gt;employmentType&lt;/code&gt;, &lt;code&gt;salary&lt;/code&gt; (when the employer publishes it), &lt;code&gt;postedAt&lt;/code&gt;, &lt;code&gt;jobUrl&lt;/code&gt; and &lt;code&gt;applyUrl&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;A few honest limits:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Titles, not full text.&lt;/strong&gt; Keywords match job titles (and optionally departments). Descriptions can be added to results, but they aren't searched.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Coverage is the supported ATS platforms.&lt;/strong&gt; Companies with a custom career site or another ATS aren't in the index. If you already have a fixed list of companies, &lt;a href="https://apify.com/steady-scrape/ats-job-feed" rel="noopener noreferrer"&gt;ATS Job Feed&lt;/a&gt; reads named boards live instead (that's the tool behind the Slack alert post).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Pay is what the employer wrote.&lt;/strong&gt; Check &lt;code&gt;salary.source&lt;/code&gt;: a structured ATS field is more reliable than a range parsed from text. Postings without pay are excluded when &lt;code&gt;salaryOnly&lt;/code&gt; is on, so the sample leans toward states and companies that publish ranges.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Some platforms don't publish everything.&lt;/strong&gt; BambooHR and Rippling don't publish a posting date, and Workday only gives relative ages.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What it costs
&lt;/h2&gt;

&lt;p&gt;On Apify's free plan the price is &lt;strong&gt;$0.002 per job returned&lt;/strong&gt; ($2 per 1,000), plus Apify's $0.00005 start fee per run. On paid Apify plans it's $0.0017 to $0.0013 per job.&lt;/p&gt;

&lt;p&gt;Enrichment is an optional add-on: &lt;strong&gt;+$0.004 per enriched job&lt;/strong&gt; on the free plan ($0.0036 to $0.003 on paid plans), charged only when &lt;code&gt;enrichJobs&lt;/code&gt; is on and the job comes back with its &lt;code&gt;enrichment&lt;/code&gt; object. An enriched job therefore costs $0.006 in total on the free plan.&lt;/p&gt;

&lt;p&gt;Examples at free-plan prices: the 50-job benchmark above, uncapped, would cost at most 50 × $0.006 = &lt;strong&gt;$0.30&lt;/strong&gt;. A one-off pull of 10,000 jobs without enrichment is $20. You can cap any run with &lt;em&gt;Max results&lt;/em&gt; and the run's maximum charge.&lt;/p&gt;

&lt;h2&gt;
  
  
  Other uses for the same search
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Sales signals:&lt;/strong&gt; a company hiring its first "data engineer" or a "Salesforce administrator" is a likely buyer for related services.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Niche job boards and newsletters:&lt;/strong&gt; fill a board with fresh roles that link straight to the employer's apply page.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Labor-market research:&lt;/strong&gt; count unique roles by title, country, remote share or ATS.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you try a title the matching rules don't handle well, tell us in the comments.&lt;/p&gt;

</description>
      <category>python</category>
      <category>api</category>
      <category>jobs</category>
      <category>career</category>
    </item>
    <item>
      <title>Tracking Prediction Market Odds With Python: Polymarket Prices as a Clean Time Series</title>
      <dc:creator>SteadyScrape</dc:creator>
      <pubDate>Tue, 10 Nov 2026 00:53:42 +0000</pubDate>
      <link>https://dev.to/steadyscrape/tracking-prediction-market-odds-with-python-polymarket-prices-as-a-clean-time-series-a7p</link>
      <guid>https://dev.to/steadyscrape/tracking-prediction-market-odds-with-python-polymarket-prices-as-a-clean-time-series-a7p</guid>
      <description>&lt;p&gt;Prediction markets have become a common data source for journalists, analysts and dashboards. "Markets give a 93% chance the Fed holds rates" is a more concrete line than "economists expect". The data is public, but going from "I want the odds" to "I have a clean table I can chart every hour" takes more work than you'd expect.&lt;/p&gt;

&lt;p&gt;This post walks through it in Python: reading Polymarket's public market data, the quirks that trip people up, and building a time series. At the end there's a hosted option if you'd rather not maintain it yourself.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 1: read markets from the Gamma API
&lt;/h2&gt;

&lt;p&gt;Polymarket publishes market metadata through a public, no-login API (Gamma). A first pass looks like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;fetch_markets&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;limit&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;offset&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"https://gamma-api.polymarket.com/markets"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;params&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="s2"&gt;"active"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"true"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"closed"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"false"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"limit"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;limit&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"offset"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;offset&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;raise_for_status&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;fetch_markets&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;outcomes&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;loads&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"outcomes"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="s2"&gt;"[]"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;        &lt;span class="c1"&gt;# e.g. '["Yes", "No"]'&lt;/span&gt;
    &lt;span class="n"&gt;prices&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="kt"&gt;float&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;loads&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"outcomePrices"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="s2"&gt;"[]"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="s2"&gt;"question"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"question"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"yes"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;prices&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;prices&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"volume"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;float&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"volume"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"liquidity"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;float&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"liquidity"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"closes"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"endDate"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"outcomes"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;zip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;outcomes&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;prices&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;sorted&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;lambda&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"volume"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s1"&gt;'{r["yes"]:.3f}  {r["question"][:70]}'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Step 2: handle the quirks
&lt;/h2&gt;

&lt;p&gt;That works for a demo. For real use, here's what you'll run into:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Prices and outcomes are JSON strings inside JSON.&lt;/strong&gt; &lt;code&gt;outcomePrices&lt;/code&gt; is &lt;code&gt;'["0.93", "0.07"]'&lt;/code&gt;, not a list of numbers. Parse it twice.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Events vs markets.&lt;/strong&gt; "Fed decision in October" is one &lt;em&gt;event&lt;/em&gt; made of several yes/no &lt;em&gt;markets&lt;/em&gt; (no change, cut 25 bps, hike 25 bps, …). If you want "the probability of each outcome", group markets by event.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Paging.&lt;/strong&gt; There are thousands of open markets. You need offset paging and a stopping rule.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Thin markets lie.&lt;/strong&gt; A market with $300 of liquidity and a 20-cent spread doesn't tell you much. Keep &lt;code&gt;liquidity&lt;/code&gt; and the bid/ask spread, and filter on them.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Keyword search is loose.&lt;/strong&gt; Searching "rate" matches every market that mentions a rate of anything. You'll want your own strict word matching afterwards.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;History is a separate API.&lt;/strong&gt; The current price is on the market, but a price series needs a per-market call to the CLOB prices-history endpoint, keyed by token ID.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Step 3: build a time series
&lt;/h2&gt;

&lt;p&gt;The simplest robust approach is a snapshot: run on a schedule, append the markets you care about to a table, and chart it.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;csv&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;datetime&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;dt&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;snapshot&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"odds.csv"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;now&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;dt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;datetime&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;now&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;dt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;timezone&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;utc&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;isoformat&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;timespec&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"minutes"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"a"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;newline&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;""&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;w&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;csv&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;writer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;writerow&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;now&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"question"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"yes"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"volume"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"liquidity"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Run it hourly with cron and you have your own dataset, one that also keeps markets after they resolve.&lt;/p&gt;

&lt;h2&gt;
  
  
  The hosted version
&lt;/h2&gt;

&lt;p&gt;&lt;em&gt;Disclosure: we (SteadyScrape) built the tool below and get paid when it's used.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;If you don't want to own the paging, parsing and history calls, &lt;a href="https://apify.com/steady-scrape/prediction-markets" rel="noopener noreferrer"&gt;Polymarket Scraper &amp;amp; Odds API&lt;/a&gt; (&lt;code&gt;steady-scrape/prediction-markets&lt;/code&gt;) is an Apify Actor that reads the same official public APIs (Gamma, Data API and CLOB) and returns &lt;strong&gt;one flat row per market&lt;/strong&gt;: question, every outcome with price and token ID, &lt;code&gt;impliedProbability&lt;/code&gt;, best bid and ask, spread, 24-hour and 1-week price change, volume, liquidity, open interest, close date, status and the result once resolved. It has strict keyword search (every word has to match; quoted phrases are exact), normalized categories, filters for closing soon, minimum volume and minimum liquidity, and optional daily or hourly price history.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;apify_client&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;ApifyClient&lt;/span&gt;

&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;ApifyClient&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;environ&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"APIFY_TOKEN"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;run&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;actor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"steady-scrape/prediction-markets"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run_input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="s2"&gt;"searchTerms"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"fed rates"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"maxItems"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"includePriceHistory"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"priceHistoryDays"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dataset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"defaultDatasetId"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;iterate_items&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s1"&gt;'{m["impliedProbability"]:.1%}  spread {m["spread"]}  {m["question"]}'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Here's what a "fed rates" search with &lt;code&gt;maxItems: 10&lt;/code&gt; returned in one of our test runs on November 9, 2026 (the odds will have moved by the time you read this):&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Market&lt;/th&gt;
&lt;th&gt;Implied probability&lt;/th&gt;
&lt;th&gt;Volume (USD)&lt;/th&gt;
&lt;th&gt;Spread&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Will the Fed increase interest rates by 25 bps after the December 2026 meeting?&lt;/td&gt;
&lt;td&gt;79.5%&lt;/td&gt;
&lt;td&gt;5.2M&lt;/td&gt;
&lt;td&gt;0.01&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Will there be no change in Fed interest rates after the December 2026 meeting?&lt;/td&gt;
&lt;td&gt;18.5%&lt;/td&gt;
&lt;td&gt;4.2M&lt;/td&gt;
&lt;td&gt;0.01&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Will no Fed rate cuts happen in 2026?&lt;/td&gt;
&lt;td&gt;97.95%&lt;/td&gt;
&lt;td&gt;9.5M&lt;/td&gt;
&lt;td&gt;0.001&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Will 1 Fed rate cut happen in 2026?&lt;/td&gt;
&lt;td&gt;0.55%&lt;/td&gt;
&lt;td&gt;3.7M&lt;/td&gt;
&lt;td&gt;0.001&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Will 2 Fed rate cuts happen in 2026?&lt;/td&gt;
&lt;td&gt;0.65%&lt;/td&gt;
&lt;td&gt;3.6M&lt;/td&gt;
&lt;td&gt;0.001&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The run took about 4 seconds. The first two rows belong to the "Fed Decision in December?" event and the rest to "How many Fed rate cuts in 2026?", which is why &lt;code&gt;eventTitle&lt;/code&gt; matters when you group results.&lt;/p&gt;

&lt;p&gt;Each row also includes &lt;code&gt;eventTitle&lt;/code&gt; and &lt;code&gt;outcomeLabel&lt;/code&gt;, so multi-outcome events group cleanly, plus &lt;code&gt;url&lt;/code&gt; back to the market. Other useful inputs: &lt;code&gt;categories&lt;/code&gt; (politics, economy, crypto, sports, …), &lt;code&gt;closingWithinDays&lt;/code&gt;, &lt;code&gt;minVolume&lt;/code&gt;, &lt;code&gt;minLiquidity&lt;/code&gt;, and &lt;code&gt;marketIds&lt;/code&gt;, which accepts pasted market or event URLs. An event URL returns every market in the event.&lt;/p&gt;

&lt;p&gt;Some limits to know about. It's Polymarket only right now (Kalshi is listed in the input but currently unavailable, and runs say so in their summary). Data is read live when the run starts, so for a time series you schedule it. It's meant for research, journalism, dashboards and personal analysis. SteadyScrape isn't affiliated with Polymarket; if you plan to redistribute the data commercially, read Polymarket's own terms first.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it costs
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;$1 per 1,000 markets returned&lt;/strong&gt; ($0.001 per market) on Apify's free plan, and $0.0009 to $0.0007 per market on paid Apify plans (Bronze and up), plus Apify's $0.00005 start fee per run. Price history is included in the per-market price, and references that aren't found return no row and aren't charged.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Use&lt;/th&gt;
&lt;th&gt;Markets per run&lt;/th&gt;
&lt;th&gt;Monthly cost (free-plan price)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;A daily snapshot of the 50 markets you follow&lt;/td&gt;
&lt;td&gt;50&lt;/td&gt;
&lt;td&gt;~$1.50&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;The same 50 markets every hour&lt;/td&gt;
&lt;td&gt;50&lt;/td&gt;
&lt;td&gt;~$36&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;A one-off pull of 5,000 open markets&lt;/td&gt;
&lt;td&gt;5,000&lt;/td&gt;
&lt;td&gt;$5 (once)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Hourly tracking of a short watchlist is where most dashboards end up. Cap runs with &lt;code&gt;maxItems&lt;/code&gt; if you want a hard ceiling.&lt;/p&gt;

&lt;p&gt;Whichever route you take, DIY or hosted, keep the liquidity and spread columns next to the probability. A 95% in a deep market and a 95% in a $500 market mean very different things.&lt;/p&gt;

</description>
      <category>python</category>
      <category>api</category>
      <category>datascience</category>
      <category>tutorial</category>
    </item>
    <item>
      <title>Make YouTube Videos Searchable by What Was Said: Timestamped Transcripts to Jump Links (Python)</title>
      <dc:creator>SteadyScrape</dc:creator>
      <pubDate>Sun, 08 Nov 2026 03:02:45 +0000</pubDate>
      <link>https://dev.to/steadyscrape/make-youtube-videos-searchable-by-what-was-said-timestamped-transcripts-to-jump-links-python-5ei</link>
      <guid>https://dev.to/steadyscrape/make-youtube-videos-searchable-by-what-was-said-timestamped-transcripts-to-jump-links-python-5ei</guid>
      <description>&lt;p&gt;You remember a speaker explaining something in one of forty conference talks, but not which talk or where. YouTube's own search won't find a phrase inside a video. Transcripts will, and if you keep the timestamps, you can link to the exact second.&lt;/p&gt;

&lt;p&gt;This post builds a small "search inside videos" tool:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;getting captions yourself, and why that breaks,&lt;/li&gt;
&lt;li&gt;pulling timestamped captions in bulk with a hosted Actor, with real output,&lt;/li&gt;
&lt;li&gt;building a phrase search that returns &lt;code&gt;&amp;amp;t=&lt;/code&gt; jump links, plus cost and caveats.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  The DIY version
&lt;/h2&gt;

&lt;p&gt;Open-source transcript libraries call YouTube's internal caption endpoints. They work on a laptop, then start failing from cloud servers when requests get blocked, and they break when YouTube changes its page. The official Data API only lets you download captions for videos you own.&lt;/p&gt;

&lt;h2&gt;
  
  
  The hosted version
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://apify.com/steady-scrape/youtube-transcripts" rel="noopener noreferrer"&gt;YouTube Transcript API: Captions with Timestamps&lt;/a&gt; returns the captions that already exist on public videos, whether creator-uploaded or YouTube auto-generated. &lt;em&gt;Disclosure: I build this Actor at SteadyScrape.&lt;/em&gt; &lt;strong&gt;It doesn't do AI or speech-to-text transcription.&lt;/strong&gt; If a video has no captions, you get a &lt;code&gt;no_captions&lt;/code&gt; row with no transcript charge.&lt;/p&gt;

&lt;p&gt;Input matching the current schema:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"videos"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="s2"&gt;"https://www.youtube.com/watch?v=dQw4w9WgXcQ"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"https://youtu.be/9bZkp7q19f0"&lt;/span&gt;
  &lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"includeSegments"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;videos&lt;/code&gt; accepts watch, youtu.be, shorts, embed and live links, or plain 11-character IDs, up to 200 per run. Duplicates are removed.&lt;/p&gt;

&lt;p&gt;Real output from a verified run (trimmed):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"videoId"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"dQw4w9WgXcQ"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"url"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"https://www.youtube.com/watch?v=dQw4w9WgXcQ"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"ok"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"language"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"requestedLanguage"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"en"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"chars"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;2089&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"segmentCount"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;61&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"durationSec"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;211.32&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"segments"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"start"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;1.36&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"end"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;3.04&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"text"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"[♪♪♪]"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"start"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;18.64&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"end"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;21.88&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"text"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"♪ We're no strangers to love ♪"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"chargeEvent"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"transcript"&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;language&lt;/code&gt; is deliberately &lt;code&gt;null&lt;/code&gt;: the data source doesn't reliably report which caption track it returned, so the Actor doesn't guess.&lt;/p&gt;

&lt;h2&gt;
  
  
  Phrase search with jump links
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;apify_client&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;ApifyClient&lt;/span&gt;

&lt;span class="no"&gt;VIDEOS&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"talks.txt"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;read&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;      &lt;span class="c1"&gt;# one URL or ID per line&lt;/span&gt;

&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;ApifyClient&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;environ&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"APIFY_TOKEN"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;run&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;actor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"steady-scrape/youtube-transcripts"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;run_input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"videos"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="no"&gt;VIDEOS&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"includeSegments"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;segments&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;skipped&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dataset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"defaultDatasetId"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;iterate_items&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="s2"&gt;"ok"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;skipped&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"input"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;continue&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"segments"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;segments&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"videoId"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"start"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"text"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;search&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;phrase&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;phrase&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;phrase&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;lower&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;vid&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;segments&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;phrase&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;lower&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;yield&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"https://www.youtube.com/watch?v={vid}&amp;amp;t={int(start)}s  {text}"&lt;/span&gt;

&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;hit&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;search&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"rate limit"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;hit&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"no transcript:"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;skipped&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Caption lines are short, so a phrase can be split across two segments. For better recall, join each segment with its neighbour before matching, or embed windows of about 30 seconds for semantic search in a RAG index. Store &lt;code&gt;videoId&lt;/code&gt; and &lt;code&gt;start&lt;/code&gt; with every chunk so each answer can cite a timestamp.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it costs
&lt;/h2&gt;

&lt;p&gt;From the Store pricing:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;$0.005 per transcript returned&lt;/strong&gt; (status &lt;code&gt;ok&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;$0.003 per run&lt;/strong&gt;, only if the run returned at least one transcript&lt;/li&gt;
&lt;li&gt;Apify's standard $0.00005 start event&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So 40 talks that all return captions cost &lt;strong&gt;$0.20305&lt;/strong&gt; in one run (40 × $0.005 + $0.003 + $0.00005). Videos without captions, unavailable or invalid videos, timeouts and skipped videos incur no transcript charge, and a run with no transcripts doesn't pay the $0.003 run fee; Apify's $0.00005 start event still applies to every run.&lt;/p&gt;

&lt;h2&gt;
  
  
  Caveats
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Captions only.&lt;/strong&gt; Auto-generated captions contain recognition errors and usually no punctuation, so quote carefully.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;no_captions&lt;/code&gt; is ambiguous.&lt;/strong&gt; The data source often can't tell a video without captions from a deleted, private or mistyped ID. None of those incur a transcript charge.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;No translation&lt;/strong&gt; in this version.&lt;/li&gt;
&lt;li&gt;Public videos only: no logins, cookies, or private or members-only videos.&lt;/li&gt;
&lt;li&gt;Videos still pending after 5 minutes come back as &lt;code&gt;timeout&lt;/code&gt; (no transcript charge) so you can retry them. The free Apify plan allows up to 10 videos per run.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Integrations
&lt;/h2&gt;

&lt;p&gt;Run it via the Apify API, on a schedule (for example, new uploads from a channel list), or from Make, Zapier or n8n through Apify's integrations. AI agents can call it through the Apify MCP server.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;SteadyScrape (steady-scrape) builds small, pay-per-result data tools on Apify.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>youtube</category>
      <category>python</category>
      <category>rag</category>
      <category>api</category>
    </item>
    <item>
      <title>Daily Google News Alerts for Your Brand That Only Show New Articles (Python + Slack)</title>
      <dc:creator>SteadyScrape</dc:creator>
      <pubDate>Sun, 08 Nov 2026 03:01:35 +0000</pubDate>
      <link>https://dev.to/steadyscrape/daily-google-news-alerts-for-your-brand-that-only-show-new-articles-python-slack-6l5</link>
      <guid>https://dev.to/steadyscrape/daily-google-news-alerts-for-your-brand-that-only-show-new-articles-python-slack-6l5</guid>
      <description>&lt;p&gt;Google Alerts is free, but it's an email digest you can't easily pipe into anything. Hand-rolled news scrapers have the opposite problem: they return the same top ten headlines every day, and you end up building deduplication yourself.&lt;/p&gt;

&lt;p&gt;This post builds a daily "only new articles" monitor:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;the DIY approach and why deduplication is harder than it looks,&lt;/li&gt;
&lt;li&gt;a hosted Actor that keeps the memory for you, with real output,&lt;/li&gt;
&lt;li&gt;posting new articles to a Slack channel, plus cost and limits.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  The DIY version
&lt;/h2&gt;

&lt;p&gt;The naive approach is to fetch the results for each query and keep a set of seen URLs:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;seen&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;load_seen&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;                 &lt;span class="c1"&gt;# set of URLs from last time&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;article&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;fetch_news&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;article&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"url"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;seen&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="nf"&gt;alert&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;article&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;seen&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;article&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"url"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nf"&gt;save_seen&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;seen&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In practice:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The same story has many URLs&lt;/strong&gt;: tracking parameters, &lt;code&gt;www.&lt;/code&gt; vs. bare domain, AMP versions. Exact URL matching sends duplicates.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Failed fetches poison the state.&lt;/strong&gt; If a fetch times out and you still save, you can lose articles.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The first run floods you&lt;/strong&gt; with everything currently ranking.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Several schedules or clients&lt;/strong&gt; need separate memories.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The hosted version
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://apify.com/steady-scrape/google-news-monitor" rel="noopener noreferrer"&gt;Google News Monitor: Brand &amp;amp; Topic Alerts, Only New Articles&lt;/a&gt; keeps that memory in your own Apify account. &lt;em&gt;Disclosure: I build this Actor at SteadyScrape.&lt;/em&gt; For each query it returns the title, source, publish time and original link, and on scheduled runs &lt;strong&gt;only the articles that are new since the last run with the same Monitor ID&lt;/strong&gt;. It returns metadata only, not article text.&lt;/p&gt;

&lt;p&gt;Input matching the current schema:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"queries"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"Acme Robotics"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"\"warehouse automation\""&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"mode"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"monitor"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"monitorId"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"acme-daily"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"location"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"United States"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"language"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"en"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"maxResultsPerQuery"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"excludeDomains"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"example-press-release-site.com"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;How the memory works:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;First run:&lt;/strong&gt; records a baseline and returns no rows (set &lt;code&gt;firstRunEmitsAll: true&lt;/code&gt; if you want them). Baseline articles are free; the query check is charged.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Later runs:&lt;/strong&gt; an article is new if its normalised link (tracking parameters and &lt;code&gt;www.&lt;/code&gt;/AMP variants removed), or its title on the same site, hasn't been seen.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A failed or timed-out query doesn't update memory&lt;/strong&gt;, so its articles come back next time.&lt;/li&gt;
&lt;li&gt;Use one &lt;code&gt;monitorId&lt;/code&gt; per schedule or client, and don't run the same one twice at the same time.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;There's also &lt;code&gt;financeCompanies&lt;/code&gt; for watchlists (&lt;code&gt;"AAPL: Apple"&lt;/code&gt;): each company becomes one query and rows carry &lt;code&gt;ticker&lt;/code&gt; and &lt;code&gt;company&lt;/code&gt;. It returns headlines only: no stock prices, no market data and no advice.&lt;/p&gt;

&lt;p&gt;For a quick look without memory, use &lt;code&gt;"mode": "oneOff"&lt;/code&gt;. Real rows from a verified one-off run for &lt;code&gt;SpaceX&lt;/code&gt; (trimmed):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;[&lt;/span&gt;
  &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"query"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"SpaceX"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"title"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"SpaceX says Google exec Don Harrison has resigned from its board"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
   &lt;span class="nl"&gt;"source"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"CNBC"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"domain"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"www.cnbc.com"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
   &lt;span class="nl"&gt;"url"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"https://www.cnbc.com/2026/11/06/tesdonald-harrison.html"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
   &lt;span class="nl"&gt;"publishedAt"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"2026-11-06T22:35:34Z"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"isTopStory"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"query"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"SpaceX"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"title"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Google executive Donald Harrison resigns from SpaceX board"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
   &lt;span class="nl"&gt;"source"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Reuters"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"domain"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"www.reuters.com"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
   &lt;span class="nl"&gt;"publishedAt"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"2026-11-06T23:35:34Z"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"isTopStory"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Posting new articles to Slack
&lt;/h2&gt;

&lt;p&gt;Schedule the Actor daily in Apify (Schedules → cron &lt;code&gt;0 8 * * *&lt;/code&gt;), then run a small script after it, or call it directly from a cron job:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;apify_client&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;ApifyClient&lt;/span&gt;

&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;ApifyClient&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;environ&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"APIFY_TOKEN"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;run&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;actor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"steady-scrape/google-news-monitor"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run_input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="s2"&gt;"queries"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"Acme Robotics"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"\"warehouse automation\""&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"mode"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"monitor"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"monitorId"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"acme-daily"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;webhook&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;environ&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"SLACK_WEBHOOK_URL"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;   &lt;span class="c1"&gt;# your Slack incoming webhook&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dataset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"defaultDatasetId"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;iterate_items&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;text&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"*{a['query']}* · {a['source']}\n&amp;lt;{a['url']}|{a['title']}&amp;gt;"&lt;/span&gt;
    &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;post&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;webhook&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"text"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Because the Actor only returns new rows, the script needs no state of its own. If you'd rather not write code, connect the dataset to Slack, email, Make, Zapier or n8n with an Apify integration or webhook.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it costs
&lt;/h2&gt;

&lt;p&gt;From the Store pricing:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;$0.004 per query checked&lt;/strong&gt; (also when nothing new turned up)&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;$0.002 per new article returned&lt;/strong&gt;&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;$0.005 once per run&lt;/strong&gt;, only if at least one query was answered&lt;/li&gt;
&lt;li&gt;Apify's standard $0.00005 start event&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Ten queries in one daily run, with about three new articles each, cost &lt;strong&gt;$0.10505 a day&lt;/strong&gt; (10 × $0.004 + 30 × $0.002 + $0.005 + $0.00005), or about &lt;strong&gt;$3.15 over 30 days&lt;/strong&gt;. Failed or timed-out queries and first-run baseline articles aren't charged, and a run where no query is answered (for example, invalid input) doesn't pay the $0.005 run fee; Apify's $0.00005 start event still applies to every run. Test mode isn't charged.&lt;/p&gt;

&lt;h2&gt;
  
  
  Limits
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Results are Google News search results for your query, location and language, not every article ever published.&lt;/li&gt;
&lt;li&gt;Advanced operators like &lt;code&gt;site:&lt;/code&gt; are rejected; use plain words or "exact phrases".&lt;/li&gt;
&lt;li&gt;Runs usually take 1–2 minutes because results come from a standard data queue.&lt;/li&gt;
&lt;li&gt;No article text, and no paywall or login access.&lt;/li&gt;
&lt;li&gt;The free Apify plan allows up to 5 queries per run, with a daily limit.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;em&gt;SteadyScrape (steady-scrape) builds small, pay-per-result data tools on Apify.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>python</category>
      <category>automation</category>
      <category>news</category>
      <category>slack</category>
    </item>
    <item>
      <title>Bulk-Convert Digital PDFs to Text and Markdown for a RAG Index (and Catch the Scanned Ones)</title>
      <dc:creator>SteadyScrape</dc:creator>
      <pubDate>Sun, 08 Nov 2026 03:00:14 +0000</pubDate>
      <link>https://dev.to/steadyscrape/bulk-convert-digital-pdfs-to-text-and-markdown-for-a-rag-index-and-catch-the-scanned-ones-5ee4</link>
      <guid>https://dev.to/steadyscrape/bulk-convert-digital-pdfs-to-text-and-markdown-for-a-rag-index-and-catch-the-scanned-ones-5ee4</guid>
      <description>&lt;p&gt;Most "chat with your documents" projects die on step one: getting text out of the PDFs. A handful of files is easy. A few hundred reports, manuals and specs from different sources is where you find the edge cases: encrypted files, 400-page monsters, and scans that look like documents but contain only images.&lt;/p&gt;

&lt;p&gt;This post walks through:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;a DIY extraction script and its traps,&lt;/li&gt;
&lt;li&gt;the same batch job with a hosted Actor and real output,&lt;/li&gt;
&lt;li&gt;chunking the result for a search or RAG index, plus cost and limits.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  The DIY version
&lt;/h2&gt;

&lt;p&gt;With &lt;code&gt;pypdf&lt;/code&gt; the basic extraction is a few lines:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;io&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pypdf&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;PdfReader&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;pdf_text&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;data&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;60&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;content&lt;/span&gt;
    &lt;span class="n"&gt;reader&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;PdfReader&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;io&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;BytesIO&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="s2"&gt;"\n\n"&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;page&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;extract_text&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="s2"&gt;""&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;page&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;reader&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pages&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;What bites you in production:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Scanned PDFs return an empty string, silently.&lt;/strong&gt; Your index now holds a "document" with no text, and nobody notices.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Encrypted files&lt;/strong&gt; raise errors or return nothing.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Size and page limits.&lt;/strong&gt; One huge file can stall a whole batch, so you need byte and page caps.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Bookkeeping.&lt;/strong&gt; You want page counts, title/author metadata and a status per file, not a pile of strings.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The hosted version
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://apify.com/steady-scrape/pdf-text-markdown" rel="noopener noreferrer"&gt;PDF to Text Markdown Extractor (Digital, No OCR)&lt;/a&gt; does this batch job with explicit statuses. &lt;em&gt;Disclosure: I build this Actor at SteadyScrape.&lt;/em&gt; It downloads each direct PDF URL and extracts text locally. There's no OCR in this version, so scanned files are labelled &lt;code&gt;image_only&lt;/code&gt; and incur &lt;strong&gt;no extraction charge&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Input matching the current schema:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"pdfUrls"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="s2"&gt;"https://www.orimi.com/pdf-test.pdf"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"https://www.rfc-editor.org/rfc/pdfrfc/rfc8259.txt.pdf"&lt;/span&gt;
  &lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"format"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"both"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"maxFiles"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"includeMetadata"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;format&lt;/code&gt; is &lt;code&gt;text&lt;/code&gt;, &lt;code&gt;markdown&lt;/code&gt; or &lt;code&gt;both&lt;/code&gt;. You can also upload files in the Apify Console instead of passing links.&lt;/p&gt;

&lt;p&gt;From Python (token from an environment variable):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;apify_client&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;ApifyClient&lt;/span&gt;

&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;ApifyClient&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;environ&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"APIFY_TOKEN"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;run&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;actor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"steady-scrape/pdf-text-markdown"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run_input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="s2"&gt;"pdfUrls"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
        &lt;span class="s2"&gt;"https://www.orimi.com/pdf-test.pdf"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"https://www.rfc-editor.org/rfc/pdfrfc/rfc8259.txt.pdf"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"format"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"both"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kt"&gt;list&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dataset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"defaultDatasetId"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;iterate_items&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Real output from a verified run of exactly those three files (text trimmed):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;[&lt;/span&gt;
  &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"source"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"https://www.orimi.com/pdf-test.pdf"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"ok"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
   &lt;span class="nl"&gt;"pageCount"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"chars"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;367&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"title"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"PDF Test Page"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
   &lt;span class="nl"&gt;"text"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"PDF Test File\n\nCongratulations, your computer is equipped with a PDF ..."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
   &lt;span class="nl"&gt;"chargeEvent"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"pdf-extracted"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"source"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
   &lt;span class="nl"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"image_only"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
   &lt;span class="nl"&gt;"note"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"No extractable text — PDF looks image-only/scanned. OCR is not included in v1. Not charged."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
   &lt;span class="nl"&gt;"text"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"chargeEvent"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"source"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"https://www.rfc-editor.org/rfc/pdfrfc/rfc8259.txt.pdf"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"ok"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
   &lt;span class="nl"&gt;"pageCount"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;16&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"chars"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;23656&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
   &lt;span class="nl"&gt;"text"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Internet Engineering Task Force (IETF) T. Bray, Ed.\nRequest for Comments: 8259 ..."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
   &lt;span class="nl"&gt;"chargeEvent"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"pdf-extracted"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That run charged two PDF extractions plus the run fee and the standard start event; the image-only file added no charge, and the row tells you why.&lt;/p&gt;

&lt;h2&gt;
  
  
  Chunking for a search or RAG index
&lt;/h2&gt;

&lt;p&gt;Keep the &lt;code&gt;ok&lt;/code&gt; rows, route &lt;code&gt;image_only&lt;/code&gt; rows to an OCR queue (or a human), and split the Markdown on headings, with a size fallback:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;chunks&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;md&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;max_chars&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;2000&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;parts&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="s2"&gt;"\n(?=#{1,3} )"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;md&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;part&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;parts&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;part&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;max_chars&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;yield&lt;/span&gt; &lt;span class="n"&gt;part&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;max_chars&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="n"&gt;index&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;needs_ocr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;"ok"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;enumerate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;chunks&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"markdown"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"text"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;index&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"source"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"source"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"chunk"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"text"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;elif&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;"image_only"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;needs_ocr&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"source"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;index&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"chunks;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;needs_ocr&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"files need OCR"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Embed &lt;code&gt;index&lt;/code&gt; with whatever model you use and store &lt;code&gt;source&lt;/code&gt; with each chunk so answers can cite the file.&lt;/p&gt;

&lt;p&gt;The Markdown headings are &lt;strong&gt;heuristic&lt;/strong&gt;: the PDF format doesn't reliably mark headings, so treat them as hints for splitting, not as a guaranteed document outline. Tables and multi-column layouts can come out in reading-order text rather than structured tables.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it costs
&lt;/h2&gt;

&lt;p&gt;From the Store pricing:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;$0.004 per successfully extracted digital PDF&lt;/strong&gt;&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;$0.003 once per run&lt;/strong&gt;, only if at least one PDF was extracted&lt;/li&gt;
&lt;li&gt;Apify's standard $0.00005 Actor start event&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;200 successfully extracted digital PDFs in one run cost &lt;strong&gt;$0.80305&lt;/strong&gt; (200 × $0.004 + $0.003 + $0.00005). Scanned/image-only, empty, encrypted, oversized and failed files incur no extraction charge, and a run where no PDF is extracted doesn't pay the $0.003 run fee; Apify's $0.00005 start event still applies to every run.&lt;/p&gt;

&lt;h2&gt;
  
  
  Limits
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;No OCR.&lt;/strong&gt; Scanned documents are flagged, not read.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Direct PDF links only.&lt;/strong&gt; It doesn't crawl a website looking for PDFs.&lt;/li&gt;
&lt;li&gt;Per-run file caps and per-file page and size caps apply. Oversized files return &lt;code&gt;too_large&lt;/code&gt; / &lt;code&gt;too_many_pages&lt;/code&gt; (no extraction charge).&lt;/li&gt;
&lt;li&gt;Only process documents you're allowed to use.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Integrations
&lt;/h2&gt;

&lt;p&gt;Run it via the Apify API, on a schedule, or from Make, Zapier or n8n through Apify's integrations. AI agents can call it through the Apify MCP server, which is handy as a "read this PDF" tool.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;SteadyScrape (steady-scrape) builds small, pay-per-result data tools on Apify.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>python</category>
      <category>pdf</category>
      <category>rag</category>
      <category>markdown</category>
    </item>
    <item>
      <title>Turn Any Product Page Into the Same JSON Shape With Python (Schema-First AI Extraction)</title>
      <dc:creator>SteadyScrape</dc:creator>
      <pubDate>Sun, 08 Nov 2026 02:58:48 +0000</pubDate>
      <link>https://dev.to/steadyscrape/turn-any-product-page-into-the-same-json-shape-with-python-schema-first-ai-extraction-5ed4</link>
      <guid>https://dev.to/steadyscrape/turn-any-product-page-into-the-same-json-shape-with-python-schema-first-ai-extraction-5ed4</guid>
      <description>&lt;p&gt;Every scraping project starts the same way: one site, a few CSS selectors, done. Then a second site arrives with different markup. Then a third redesigns its product page. A few weeks later you are maintaining a pile of per-site parsers whose only job is to produce &lt;strong&gt;the same five fields&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;This post shows a schema-first approach: you describe the &lt;em&gt;shape&lt;/em&gt; of the data you want once, then extract it from many different pages. We will:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;look at the DIY version (fetch + LLM + validation) and where it gets tricky,&lt;/li&gt;
&lt;li&gt;run the same workflow with a hosted Actor and read the real output,&lt;/li&gt;
&lt;li&gt;cover cost and the caveats you should know before trusting AI extraction.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  The DIY version
&lt;/h2&gt;

&lt;p&gt;The core loop is simple:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;jsonschema&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;validate&lt;/span&gt;

&lt;span class="no"&gt;SCHEMA&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="s2"&gt;"type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"object"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"properties"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="s2"&gt;"title"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"string"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"price"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"string"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"availability"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"string"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"required"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"title"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"price"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="n"&gt;html&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;
&lt;span class="n"&gt;text&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;html_to_text&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;html&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;              &lt;span class="c1"&gt;# strip tags, scripts, nav&lt;/span&gt;
&lt;span class="n"&gt;raw&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;call_your_llm&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;prompt&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="no"&gt;SCHEMA&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;data&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;loads&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;raw&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nf"&gt;validate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="no"&gt;SCHEMA&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;                  &lt;span class="c1"&gt;# raises if the shape is wrong&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The hard parts aren't in that snippet:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Bad JSON.&lt;/strong&gt; Models sometimes return prose, trailing commas or the wrong types, so you need a repair-and-retry loop with a cap.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Pages that can't work.&lt;/strong&gt; Login walls, captchas, 429s and JavaScript-only apps return junk text. If you don't detect them, you will "extract" garbage and pay for it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Long pages.&lt;/strong&gt; You have to truncate input, and that silently drops fields near the bottom.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Missing vs. empty.&lt;/strong&gt; A field the page doesn't state should be &lt;code&gt;null&lt;/code&gt; and flagged, not hallucinated.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The hosted version
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://apify.com/steady-scrape/ai-page-to-json" rel="noopener noreferrer"&gt;AI Page to JSON: URL + Schema Extractor&lt;/a&gt; handles that loop for you. &lt;em&gt;Disclosure: I build this Actor at SteadyScrape.&lt;/em&gt; It fetches each public URL over plain HTTP (no browser), sends the page text to an LLM, validates the result against your schema, and returns one row per URL with a clear status.&lt;/p&gt;

&lt;p&gt;Input matching the current schema (&lt;code&gt;urls&lt;/code&gt; is required; give &lt;code&gt;fields&lt;/code&gt; and/or &lt;code&gt;jsonSchema&lt;/code&gt;):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"urls"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"fields"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"title"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"price"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"availability"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Fields can carry a hint, e.g. &lt;code&gt;"price: Product price in USD"&lt;/code&gt;. For stricter pipelines, pass a full &lt;code&gt;jsonSchema&lt;/code&gt; object instead; rows that miss required fields come back as &lt;code&gt;partial&lt;/code&gt; with &lt;code&gt;missingFields&lt;/code&gt; listed.&lt;/p&gt;

&lt;p&gt;Calling it from Python with the official client (token read from an environment variable, never hard-coded):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;apify_client&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;ApifyClient&lt;/span&gt;

&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;ApifyClient&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;environ&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"APIFY_TOKEN"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;run&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;actor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"steady-scrape/ai-page-to-json"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run_input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="s2"&gt;"urls"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
        &lt;span class="s2"&gt;"https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"fields"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"title"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"price"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"availability"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dataset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"defaultDatasetId"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;iterate_items&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"url"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"data"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Real output from a verified run (trimmed):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"url"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"ok"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"data"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nl"&gt;"title"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Tipping the Velvet"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"price"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"£53.74"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"availability"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"In stock (20 available)"&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"missingFields"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"nullFields"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"truncated"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"pageTitle"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Tipping the Velvet | Books to Scrape - Sandbox"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"chargeEvent"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"page-extracted"&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  A real workflow: a weekly price sheet across different sites
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Keep a CSV of product URLs from several shops.&lt;/li&gt;
&lt;li&gt;Run the Actor weekly with the same &lt;code&gt;fields&lt;/code&gt; (Apify's scheduler works fine).&lt;/li&gt;
&lt;li&gt;Keep only rows where &lt;code&gt;status&lt;/code&gt; is &lt;code&gt;ok&lt;/code&gt;, and append &lt;code&gt;url, title, price, availability, run date&lt;/code&gt; to a sheet.&lt;/li&gt;
&lt;li&gt;Diff against last week and alert on price or stock changes.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Because you ask for the same schema every time, step 3 can be the same code for every site, and a shop redesign usually doesn't mean writing a new parser. Still spot-check results after a site changes, since extraction quality can shift with the page.&lt;/p&gt;

&lt;p&gt;The statuses matter here: &lt;code&gt;refused&lt;/code&gt; (login/paywall), &lt;code&gt;blocked&lt;/code&gt; (captcha/429), &lt;code&gt;js_required&lt;/code&gt; (needs a browser) and &lt;code&gt;failed&lt;/code&gt; rows incur &lt;strong&gt;no page-extraction charge&lt;/strong&gt;. Filter them out and review them separately; don't treat them as "price missing".&lt;/p&gt;

&lt;h2&gt;
  
  
  What it costs
&lt;/h2&gt;

&lt;p&gt;Pricing is pay-per-event and shown on the Store page:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;$0.010 per extracted page&lt;/strong&gt; (&lt;code&gt;ok&lt;/code&gt; or &lt;code&gt;partial&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;$0.001 once per run&lt;/strong&gt;, only if at least one page was extracted&lt;/li&gt;
&lt;li&gt;Apify's standard $0.00005 Actor start event&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So a weekly run where all 100 product pages are extracted costs &lt;strong&gt;$1.00105&lt;/strong&gt; (100 × $0.010 + $0.001 + $0.00005). Failed, refused, blocked, JavaScript-only and empty pages incur no page-extraction charge, and a run where every page fails doesn't pay the $0.001 run fee; Apify's $0.00005 start event still applies to every run. You can also set a maximum cost per run in Apify's run options.&lt;/p&gt;

&lt;h2&gt;
  
  
  Caveats (read these)
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;AI extraction is not guaranteed to be correct.&lt;/strong&gt; Validation checks the &lt;em&gt;shape&lt;/em&gt;, not the truth. Spot-check values, especially prices and dates, before you act on them automatically.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;HTTP only in this version.&lt;/strong&gt; Single-page apps that render content with JavaScript return &lt;code&gt;js_required&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Public pages only.&lt;/strong&gt; It doesn't bypass logins, paywalls or captchas, and it has no personal-data harvesting features. Check each site's terms before collecting at scale.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Long pages&lt;/strong&gt; may be cut at &lt;code&gt;maxInputChars&lt;/code&gt;; those rows carry &lt;code&gt;truncated: true&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Integrations
&lt;/h2&gt;

&lt;p&gt;You can run it through the Apify API, schedules and webhooks, or connect it to Make, Zapier or n8n through Apify's own integrations. AI agents can call it through the Apify MCP server.&lt;/p&gt;

&lt;p&gt;If you try it on a page that should have worked and didn't, open an issue on the Actor page. I read them.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;SteadyScrape (steady-scrape) builds small, pay-per-result data tools on Apify.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>python</category>
      <category>webscraping</category>
      <category>ai</category>
      <category>json</category>
    </item>
    <item>
      <title>How to get a Slack alert when a company posts a new job (Greenhouse, Lever, Ashby) with n8n or Python</title>
      <dc:creator>SteadyScrape</dc:creator>
      <pubDate>Sat, 07 Nov 2026 16:03:02 +0000</pubDate>
      <link>https://dev.to/steadyscrape/how-to-get-a-slack-alert-when-a-company-posts-a-new-job-greenhouse-lever-ashby-with-n8n-or-129a</link>
      <guid>https://dev.to/steadyscrape/how-to-get-a-slack-alert-when-a-company-posts-a-new-job-greenhouse-lever-ashby-with-n8n-or-129a</guid>
      <description>&lt;p&gt;Whether you're job hunting, selling to companies that are hiring, or keeping an eye on competitors, "tell me when Stripe posts a new role" is a surprisingly annoying problem. Career pages don't have RSS. Job board alerts miss postings, duplicate them, and arrive days late.&lt;/p&gt;

&lt;p&gt;Most companies host their career page on an applicant tracking system (ATS) like Greenhouse, Lever or Ashby, and those have public job feeds. The tricky part isn't fetching jobs. It's &lt;strong&gt;remembering which ones you've already seen&lt;/strong&gt;, so the alert only contains new ones.&lt;/p&gt;

&lt;p&gt;This post shows two ways to do it: an n8n workflow and a short Python script. Both use the &lt;a href="https://apify.com/steady-scrape/ats-job-feed" rel="noopener noreferrer"&gt;ATS Job Feed Actor&lt;/a&gt; on Apify, which keeps the "already seen" state for you.&lt;/p&gt;

&lt;h2&gt;
  
  
  How the "only new" logic works
&lt;/h2&gt;

&lt;p&gt;Three inputs do the work:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;onlyNewSinceLastRun: true&lt;/code&gt;: return only jobs not returned before.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;stateStoreName&lt;/code&gt;: a named key-value store in &lt;em&gt;your&lt;/em&gt; Apify account that remembers seen job IDs, one key per board. Use one name per watchlist.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;firstRunEmitsAll: false&lt;/code&gt;: the first run records every current job silently and returns nothing. You aren't charged for that baseline.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Removed jobs can be reported too (&lt;code&gt;emitRemovedJobs&lt;/code&gt;), and those records are free.&lt;/p&gt;

&lt;h2&gt;
  
  
  Option 1: n8n
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Install the verified community node &lt;strong&gt;Apify&lt;/strong&gt; (&lt;code&gt;@apify/n8n-nodes-apify&lt;/code&gt;) and add an Apify credential with your API token.&lt;/li&gt;
&lt;li&gt;Build five nodes: &lt;strong&gt;Schedule Trigger&lt;/strong&gt; (daily, 8:00) → &lt;strong&gt;Apify: Run an Actor and get dataset&lt;/strong&gt; (Actor &lt;code&gt;steady-scrape/ats-job-feed&lt;/code&gt;, input JSON below) → &lt;strong&gt;Filter&lt;/strong&gt; (&lt;code&gt;changeType&lt;/code&gt; is not &lt;code&gt;removed&lt;/code&gt;; the workflow stops here when nothing is new) → &lt;strong&gt;Code&lt;/strong&gt; (build one message, below) → &lt;strong&gt;Slack: Send message&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;Run once to save the baseline, then activate the workflow.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Apify node input:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"companies"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="s2"&gt;"https://job-boards.greenhouse.io/stripe"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"https://jobs.lever.co/spotify"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"https://jobs.ashbyhq.com/ramp"&lt;/span&gt;
  &lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"onlyNewSinceLastRun"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"firstRunEmitsAll"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"stateStoreName"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"my-target-companies"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"includeDescription"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"maxJobsPerCompany"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;200&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Code node (JavaScript, "Run once for all items"):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;jobs&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nv"&gt;$input&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;all&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;byCompany&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;j&lt;/span&gt; &lt;span class="k"&gt;of&lt;/span&gt; &lt;span class="n"&gt;jobs&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;byCompany&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;j&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;company&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;??=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;push&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;j&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;Object&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;entries&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;byCompany&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;company&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;list&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt;
  &lt;span class="sb"&gt;`*${company}* (${list.length})\n`&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt;
  &lt;span class="n"&gt;list&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;j&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="sb"&gt;`• &amp;lt;${j.jobUrl}|${j.title}&amp;gt; (${j.locationText || 'n/a'})`&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'\n'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'\n\n'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sb"&gt;`${jobs.length} new jobs\n\n${text}`&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In the Slack node, set the message text to &lt;code&gt;{{ $json.text }}&lt;/code&gt;. To keep a log as well, add a &lt;strong&gt;Google Sheets: Append row&lt;/strong&gt; node after the Filter and match on &lt;code&gt;jobId&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Option 2: Python
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;apify_client&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;ApifyClient&lt;/span&gt;

&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;ApifyClient&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;environ&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"APIFY_TOKEN"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="no"&gt;SLACK_WEBHOOK&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;environ&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"SLACK_WEBHOOK_URL"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;  &lt;span class="c1"&gt;# optional&lt;/span&gt;

&lt;span class="n"&gt;run&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;actor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"steady-scrape/ats-job-feed"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;logger&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;run_input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="s2"&gt;"companies"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
        &lt;span class="s2"&gt;"https://job-boards.greenhouse.io/stripe"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"https://jobs.lever.co/spotify"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"https://jobs.ashbyhq.com/ramp"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"onlyNewSinceLastRun"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"firstRunEmitsAll"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;False&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;          &lt;span class="c1"&gt;# first run = silent, free baseline&lt;/span&gt;
    &lt;span class="s2"&gt;"stateStoreName"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"my-target-companies"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"includeDescription"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;False&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"maxJobsPerCompany"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;200&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;new_jobs&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;j&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;j&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dataset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;default_dataset_id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;iterate_items&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;j&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"changeType"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="s2"&gt;"removed"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="n"&gt;lines&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s1"&gt;'• *{j["company"]}*: &amp;lt;{j["jobUrl"]}|{j["title"]}&amp;gt; ({j.get("locationText") or "n/a"})'&lt;/span&gt;
         &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;j&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;new_jobs&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"{len(new_jobs)} new jobs"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"\n"&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;lines&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="no"&gt;SLACK_WEBHOOK&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;lines&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;post&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="no"&gt;SLACK_WEBHOOK&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"text"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"{len(lines)} new jobs\n"&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="s2"&gt;"\n"&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;lines&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Set &lt;code&gt;SLACK_WEBHOOK_URL&lt;/code&gt; to a Slack &lt;a href="https://api.slack.com/messaging/webhooks" rel="noopener noreferrer"&gt;incoming webhook&lt;/a&gt; if you want the message posted, then schedule the script with cron or GitHub Actions. Keep &lt;code&gt;stateStoreName&lt;/code&gt; the same between runs. That's what makes the second run return only new jobs.&lt;/p&gt;

&lt;h2&gt;
  
  
  What happened when I ran it
&lt;/h2&gt;

&lt;p&gt;All times Costa Rica (UTC−6), Nov 1, 2026:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;6:14 PM, baseline run.&lt;/strong&gt; Stripe (Greenhouse), Spotify (Lever) and Ramp (Ashby). It recorded 1,006 current jobs (Stripe 746, Ramp 172, Spotify 88) in 4.5 seconds and returned 0. Cost: the $0.00005 Actor start.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;6:25 PM, the same script again.&lt;/strong&gt; 0 new jobs, nothing posted to Slack. Cost: $0.00005 again. (It was a Sunday evening. On a weekday these three companies post several roles a day.)&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;To see what a non-empty alert looks like without waiting, I ran the same companies with &lt;code&gt;postedWithinDays: 3&lt;/code&gt; and &lt;code&gt;maxJobsPerCompany: 4&lt;/code&gt; instead of the state options. That returned 8 jobs (4 from Ramp, 4 from Stripe, none from Spotify), for example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;8 new jobs
• Ramp: Manager, Enterprise Technical Consulting (New York, NY (HQ))
• Stripe: Blockchain Security Engineer, Bridge (Luxembourg)
...
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Cost at the free-plan list price: 8 × $0.0015 + $0.00005 = &lt;strong&gt;$0.012&lt;/strong&gt;. In general you pay $0.0015 per &lt;em&gt;new&lt;/em&gt; job and nothing for jobs already delivered, so a watchlist that gets ~10 new roles a day costs roughly $0.45 a month (estimate: 300 jobs × $0.0015, plus ~$0.0015 in Actor starts).&lt;/p&gt;

&lt;h2&gt;
  
  
  Tips
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Paste whatever career URL you have. The Actor detects the ATS, and also reads Workday, SmartRecruiters, Workable and others.&lt;/li&gt;
&lt;li&gt;Filter early with &lt;code&gt;keywords&lt;/code&gt;, &lt;code&gt;excludeKeywords&lt;/code&gt;, &lt;code&gt;locations&lt;/code&gt;, &lt;code&gt;departments&lt;/code&gt; or &lt;code&gt;remoteOnly&lt;/code&gt;. Filtered-out jobs aren't billed.&lt;/li&gt;
&lt;li&gt;Leave &lt;code&gt;includeDescription&lt;/code&gt; off for alerts. It makes runs faster, and the job URL is enough to click through.&lt;/li&gt;
&lt;li&gt;Each job has a stable &lt;code&gt;jobId&lt;/code&gt;, so dedupe on it if you also append jobs to a sheet or CRM.&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;&lt;em&gt;Disclosure: SteadyScrape publishes the Actor used in this post. Numbers are from real runs on Nov 1, 2026.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>n8n</category>
      <category>automation</category>
      <category>python</category>
      <category>career</category>
    </item>
  </channel>
</rss>
