<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: zachary klein</title>
    <description>The latest articles on DEV Community by zachary klein (@zachary_klein_a8078ed2a7f).</description>
    <link>https://dev.to/zachary_klein_a8078ed2a7f</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4253070%2Fd9fd9753-9406-40dd-ac9f-5d484b82b1b0.jpg</url>
      <title>DEV Community: zachary klein</title>
      <link>https://dev.to/zachary_klein_a8078ed2a7f</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/zachary_klein_a8078ed2a7f"/>
    <language>en</language>
    <item>
      <title>Running PageSpeed Insights on a whole site: the API details that matter</title>
      <dc:creator>zachary klein</dc:creator>
      <pubDate>Thu, 19 Nov 2026 06:22:23 +0000</pubDate>
      <link>https://dev.to/zachary_klein_a8078ed2a7f/running-pagespeed-insights-on-a-whole-site-the-api-details-that-matter-4lc8</link>
      <guid>https://dev.to/zachary_klein_a8078ed2a7f/running-pagespeed-insights-on-a-whole-site-the-api-details-that-matter-4lc8</guid>
      <description>&lt;p&gt;"Is there a tool to run PageSpeed Insights on all my pages at once?" comes up constantly in SEO and web dev forums. The answer is yes, Google gives you an API for exactly this. The harder part is everything around it: finding the pages, staying inside quota, handling 40-second responses, and reading the results without being fooled by noise.&lt;/p&gt;

&lt;p&gt;These are my notes from building a bulk auditor on top of the PageSpeed Insights (PSI) and Chrome UX Report (CrUX) APIs. At the end there is the hosted version I published on Apify. Disclosure: I built it.&lt;/p&gt;

&lt;h2&gt;
  
  
  The API call
&lt;/h2&gt;

&lt;p&gt;One audit is one GET:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight http"&gt;&lt;code&gt;&lt;span class="no"&gt;GET&lt;/span&gt; &lt;span class="n"&gt;https&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="c1"&gt;//www.googleapis.com/pagespeedonline/v5/runPagespeed&lt;/span&gt;
    &lt;span class="na"&gt;?url&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;https&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="c1"&gt;//web.dev/&lt;/span&gt;
    &lt;span class="na"&gt;&amp;amp;strategy&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;mobile&lt;/span&gt;
    &lt;span class="na"&gt;&amp;amp;category&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;performance&lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;category&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;seo&lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;category&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;accessibility&lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;category&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;best&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;practices&lt;/span&gt;
    &lt;span class="na"&gt;&amp;amp;key&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="no"&gt;YOUR_KEY&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Things that are not obvious from the docs:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;One strategy per call.&lt;/strong&gt; Mobile and desktop are two audits. Mobile is what Google uses for ranking, so start there.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Categories must be listed.&lt;/strong&gt; Without &lt;code&gt;category&lt;/code&gt; params you only get performance.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The response is huge.&lt;/strong&gt; A full Lighthouse result runs to megabytes. Use the &lt;code&gt;fields&lt;/code&gt; parameter to keep only what you need, for example &lt;code&gt;id,loadingExperience,originLoadingExperience,lighthouseResult(finalUrl,categories,audits,runtimeError)&lt;/code&gt;. It cuts transfer and parse time a lot.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;It is slow.&lt;/strong&gt; Each call runs a real Lighthouse audit on Google's side: 10 to 40 seconds, and up to 2 minutes for heavy pages. Set your HTTP timeout accordingly and run audits in parallel.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Get a key.&lt;/strong&gt; Without one you get throttled quickly. With a free key from Google Cloud you typically get 25,000 queries a day. Check your project's quota page for the actual limits.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Lab data and field data are different things
&lt;/h2&gt;

&lt;p&gt;Each PSI response has two kinds of data, and mixing them up is the most common mistake I see in bulk reports.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Lab data&lt;/strong&gt; is &lt;code&gt;lighthouseResult&lt;/code&gt;: the Lighthouse performance score (0 to 100) and metrics like LCP, CLS, TBT, FCP and Speed Index, measured in one simulated load on Google's servers. Every page gets lab data.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Field data&lt;/strong&gt; is &lt;code&gt;loadingExperience&lt;/code&gt; (this URL) and &lt;code&gt;originLoadingExperience&lt;/code&gt; (the whole site): 75th percentile values for LCP, INP and CLS from real Chrome users over the last 28 days. This is what the Core Web Vitals assessment uses. Many pages have none, because Google only publishes it for pages with enough Chrome traffic. When a URL has too little, you often still get origin-level data.&lt;/p&gt;

&lt;p&gt;Two consequences for a bulk report:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Do not compute "passes Core Web Vitals" from lab scores. The assessment is: LCP at most 2.5 s, INP at most 200 ms and CLS at most 0.1, all at p75 from field data. If any of the three is missing, the honest answer is "no data", not "pass".&lt;/li&gt;
&lt;li&gt;Record which scope the field data came from (&lt;code&gt;url&lt;/code&gt; or &lt;code&gt;origin&lt;/code&gt;). Otherwise a quiet product page "passes" because the homepage is fast.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Lab scores move between runs
&lt;/h2&gt;

&lt;p&gt;Run the same URL twice and the performance score can differ by 5 to 10 points. Network timing and server response vary, even on Google's own sites. For one-off reports, that is fine if you say so. For monitoring, it matters a lot: a naive "alert if the score dropped" rule fires every day.&lt;/p&gt;

&lt;p&gt;What worked for me in monitor mode:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Alert on a performance drop only above a threshold (10 points by default).&lt;/li&gt;
&lt;li&gt;Alert when a page flips from passing to failing the field-data assessment.&lt;/li&gt;
&lt;li&gt;Alert when a field metric gets clearly worse and is no longer "good".&lt;/li&gt;
&lt;li&gt;Alert when a page that worked last time now errors.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Field data changes slowly (it is a 28-day window), so field-based alerts are much less noisy than lab-based ones.&lt;/p&gt;

&lt;h2&gt;
  
  
  Finding the pages
&lt;/h2&gt;

&lt;p&gt;"Audit my site" means you need the URL list first. The reliable order:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;code&gt;robots.txt&lt;/code&gt;, for &lt;code&gt;Sitemap:&lt;/code&gt; lines.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;/sitemap.xml&lt;/code&gt; if robots.txt lists nothing.&lt;/li&gt;
&lt;li&gt;Follow sitemap indexes into child sitemaps, including &lt;code&gt;.xml.gz&lt;/code&gt; files.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Then sample. A 20,000-URL sitemap is mostly the same few templates (product, category, blog post). Auditing 25 to 50 pages spread across the site's sections, always including the homepage, tells you more per dollar than auditing the first 50 URLs in the sitemap, which are often all from one section.&lt;/p&gt;

&lt;h2&gt;
  
  
  Errors that look like results
&lt;/h2&gt;

&lt;p&gt;Some PSI responses have scores but describe the wrong page:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The URL returns 404 or 500. Lighthouse happily audits the error page.&lt;/li&gt;
&lt;li&gt;The URL redirects to a different site (an expired domain, a login wall).&lt;/li&gt;
&lt;li&gt;Lighthouse fails with a &lt;code&gt;runtimeError&lt;/code&gt; like &lt;code&gt;NO_FCP&lt;/code&gt; (no content painted). That one is sometimes transient and worth one retry.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;I turn all of these into error rows with the reason instead of a score. A bulk report where 30 of 200 rows are secretly 404 pages is worse than useless.&lt;/p&gt;

&lt;h2&gt;
  
  
  Field data only, at 100 URLs a minute
&lt;/h2&gt;

&lt;p&gt;If you only need real-user Core Web Vitals, skip Lighthouse entirely and call the CrUX API (&lt;code&gt;chromeuxreport.googleapis.com/v1/records:queryRecord&lt;/code&gt;). It answers in well under a second, so you can check about 100 URLs a minute. Useful for screening a large list before running full audits on the failing pages.&lt;/p&gt;

&lt;h2&gt;
  
  
  Making the output usable
&lt;/h2&gt;

&lt;p&gt;The rows people actually filter on, in my experience:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;performanceScore&lt;/code&gt;, &lt;code&gt;seoScore&lt;/code&gt;, &lt;code&gt;accessibilityScore&lt;/code&gt;, &lt;code&gt;bestPracticesScore&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;cwvVerdict&lt;/code&gt; (&lt;code&gt;pass&lt;/code&gt;, &lt;code&gt;fail&lt;/code&gt;, &lt;code&gt;no-data&lt;/code&gt;) and &lt;code&gt;originCwvVerdict&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;fieldLcpP75Ms&lt;/code&gt;, &lt;code&gt;fieldInpP75Ms&lt;/code&gt;, &lt;code&gt;fieldClsP75&lt;/code&gt; with their ratings&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;topOpportunity&lt;/code&gt;: the single biggest Lighthouse fix by estimated time saved&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;psiReportUrl&lt;/code&gt;: a link to Google's own report for that page, for whoever has to fix it&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Plus one summary row per site: average and median score, Core Web Vitals pass rate, the 5 worst pages and the most common fixes across pages. That is usually the part that ends up in a client deck.&lt;/p&gt;

&lt;h2&gt;
  
  
  The hosted version
&lt;/h2&gt;

&lt;p&gt;The actor is &lt;a href="https://apify.com/zrk1212/pagespeed-bulk-auditor" rel="noopener noreferrer"&gt;Bulk PageSpeed Insights &amp;amp; Core Web Vitals Checker&lt;/a&gt; on Apify. Give it URLs or a domain. It does sitemap discovery, parallel audits, the field data logic above, error rows, a site summary, and optional monitoring with regression alerts.&lt;/p&gt;

&lt;p&gt;Pricing: $5 per 1,000 audits on the shared Google key, $2 per 1,000 with your own free key (which also gives you your own daily quota), $1 per 1,000 for field-only checks. Errors, summaries and monitor comparisons are free. Runs on the shared key are capped at 1,000 audits each.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;apify_client&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;ApifyClient&lt;/span&gt;

&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;ApifyClient&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"YOUR_APIFY_TOKEN"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;run&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;actor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"zrk1212/pagespeed-bulk-auditor"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run_input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="s2"&gt;"startDomains"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"example.com"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"maxPagesPerSite"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;25&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"strategy"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"mobile"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"includeFieldData"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"includeSiteSummary"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="c1"&gt;# "googleApiKey": "YOUR_GOOGLE_KEY",  # optional: lower price, your own quota&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dataset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"defaultDatasetId"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;iterate_items&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"rowType"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;"audit"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"performanceScore"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"cwvVerdict"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"requestedUrl"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"topOpportunity"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;elif&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"rowType"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;"summary"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"site pass rate:"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"cwvPassRatePct"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"%"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If you want to build your own instead, everything above works with a free key and a thread pool. The parts worth copying are the error handling and the field data rules. Those are where most bulk reports go wrong.&lt;/p&gt;

</description>
      <category>webperf</category>
      <category>seo</category>
      <category>python</category>
      <category>corewebvitals</category>
    </item>
    <item>
      <title>Most Google Search ads in the Transparency Center are screenshots. Here is how I get the text out.</title>
      <dc:creator>zachary klein</dc:creator>
      <pubDate>Thu, 19 Nov 2026 06:19:59 +0000</pubDate>
      <link>https://dev.to/zachary_klein_a8078ed2a7f/most-google-search-ads-in-the-transparency-center-are-screenshots-here-is-how-i-get-the-text-out-17d9</link>
      <guid>https://dev.to/zachary_klein_a8078ed2a7f/most-google-search-ads-in-the-transparency-center-are-screenshots-here-is-how-i-get-the-text-out-17d9</guid>
      <description>&lt;p&gt;The Google Ads Transparency Center (adstransparency.google.com) shows every ad an advertiser has run, by region, format and date. For competitor research it is the best free source there is. The problem shows up the moment you try to export it: for most Search ads, Google does not have the ad text as text. It has a picture of the ad.&lt;/p&gt;

&lt;p&gt;This post covers how the Transparency Center serves its data, why you end up with screenshots, and the two-step extraction I use to get headline, description, display URL and sitelinks as fields. I built this into an Apify actor, linked at the end. Disclosure: I am the developer.&lt;/p&gt;

&lt;h2&gt;
  
  
  The site runs on a small RPC API
&lt;/h2&gt;

&lt;p&gt;The Transparency Center is a JavaScript app that talks to one base URL:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight http"&gt;&lt;code&gt;&lt;span class="no"&gt;POST&lt;/span&gt; &lt;span class="n"&gt;https&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="c1"&gt;//adstransparency.google.com/anji/_/rpc/&amp;lt;Service&amp;gt;/&amp;lt;Method&amp;gt;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The methods you need:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;SearchService/SearchSuggestions&lt;/code&gt;: resolve a name to advertiser IDs.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;SearchService/SearchCreatives&lt;/code&gt;: list ads for an advertiser or a domain, with region, format and date filters. About 100 ads per call, with a cursor for the next page.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;LookupService/GetCreativeById&lt;/code&gt;: details for one ad, including all its variations and the regions it was shown in.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The bodies are protobuf-style JSON with numbered keys instead of names (&lt;code&gt;"1"&lt;/code&gt; is the advertiser ID, &lt;code&gt;"2"&lt;/code&gt; the creative ID, and so on), sent as a form field. No login, no cookies, no signed token. The field layout has been stable for years; every open-source client I looked at uses the same numbers.&lt;/p&gt;

&lt;p&gt;One operational note: Google blocks this endpoint from many cloud IPs. From a home connection it just works. From a server you will likely need a residential IP for the search calls. The image and preview hosts do not have that restriction.&lt;/p&gt;

&lt;h2&gt;
  
  
  Each ad has a preview, and the preview is the problem
&lt;/h2&gt;

&lt;p&gt;A creative in &lt;code&gt;SearchCreatives&lt;/code&gt; comes with a preview in one of two shapes:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;A &lt;code&gt;content.js&lt;/code&gt; URL on &lt;code&gt;displayads-formats.googleusercontent.com&lt;/code&gt;. This is a script that renders the ad, and the headline and description strings are inside it.&lt;/li&gt;
&lt;li&gt;An &lt;code&gt;&amp;lt;img&amp;gt;&lt;/code&gt; pointing at &lt;code&gt;tpc.googlesyndication.com/archive/simgad/...&lt;/code&gt;. A PNG of the rendered ad.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;For image ads, the PNG is the ad, so that is fine. For video ads, the &lt;code&gt;content.js&lt;/code&gt; contains the YouTube thumbnail URL, so you can regex out the video ID and build a YouTube link.&lt;/p&gt;

&lt;p&gt;For text (Search) ads, you want shape 1, and you mostly get shape 2. In my samples:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;GEICO: 39 of 40 text ads had only a screenshot.&lt;/li&gt;
&lt;li&gt;Nike: 43 of 50 ads had no structured text.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So any tool that only reads the preview script returns metadata for most Search ads: advertiser, dates, format, a link. That matches complaints I found in reviews of existing scrapers along the lines of "only returns metadata."&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 1: read the preview data when it exists
&lt;/h2&gt;

&lt;p&gt;When a creative has a &lt;code&gt;content.js&lt;/code&gt; preview, fetch it and pull the strings out. It is a big script (around 150 KB for video ads, smaller for text), but the ad text appears as plain string literals. This path is exact and cheap, so it always runs first.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 2: OCR the screenshot
&lt;/h2&gt;

&lt;p&gt;The screenshots are good OCR material: Google renders them at a consistent size, on a white background, in a standard font. I run Tesseract on them (English and Spanish, plus the region's main language), which takes about 0.3 to 0.5 seconds per image.&lt;/p&gt;

&lt;p&gt;Raw OCR gives you lines, not fields. A rendered Search ad has a predictable layout, so I split lines by position and size:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The first line is the advertiser name row. Skip it.&lt;/li&gt;
&lt;li&gt;Line 2 or 3 that looks like a domain (&lt;code&gt;www.geico.com/&lt;/code&gt;) is the display URL.&lt;/li&gt;
&lt;li&gt;Lines labeled "Sponsored", "Ad", "Anuncio" or "Anzeige" are skipped.&lt;/li&gt;
&lt;li&gt;The headline is the first block of lines with the largest text height. A drop in line height to under 70% of the headline's, or a vertical gap, starts the description.&lt;/li&gt;
&lt;li&gt;A larger gap after the description starts the tail, where short lines (60 characters or less) are sitelinks.&lt;/li&gt;
&lt;li&gt;Footer lines like "Top pages" or "Rating for..." end the ad.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Then a quality gate: lines under 60% confidence are dropped, and if the mean confidence of what is left is under 80%, the whole result is discarded. Local map ads and ads with a product photo overlapping the text produce noise, and returning garbage as "ad copy" is worse than returning nothing.&lt;/p&gt;

&lt;p&gt;Here is a GEICO text ad that only exists as a screenshot, after OCR:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"advertiserName"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"GEICO Corporation"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"format"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"text"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"headline"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"GEICO® Car Insurance - Get a Free GEICO® Quote Today"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"description"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"GEICO® was Named the Best Car Insurance Company for 2026-2027. Get a Free Quote Today."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"displayUrl"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"www.geico.com/"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"adTextSource"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"ocr"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"ocrConfidence"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;95&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;I keep &lt;code&gt;adTextSource&lt;/code&gt; (&lt;code&gt;preview_data&lt;/code&gt; or &lt;code&gt;ocr&lt;/code&gt;) and &lt;code&gt;ocrConfidence&lt;/code&gt; on every row, so you can decide how much to trust each one.&lt;/p&gt;

&lt;h2&gt;
  
  
  A cheaper question: does this domain run Google Ads at all?
&lt;/h2&gt;

&lt;p&gt;A lot of people do not need ad copy. They have a list of 5,000 prospect domains and want to know which ones advertise on Google. That is one &lt;code&gt;SearchCreatives&lt;/code&gt; call per domain with the domain filter. Google returns an approximate ad count range (for example 10,000 to 20,000 for shopify.com) and the ads themselves, from which you get the last shown date and formats. One row per domain: &lt;code&gt;runsGoogleAds&lt;/code&gt;, the count range, &lt;code&gt;activeLast30Days&lt;/code&gt;, formats seen. It is a useful lead qualification signal for agencies and SaaS sales teams.&lt;/p&gt;

&lt;h2&gt;
  
  
  Watching competitors over time
&lt;/h2&gt;

&lt;p&gt;Every creative has a first shown and last shown date. If you store the set of creative IDs per advertiser and run daily, you can output only what changed: &lt;code&gt;new&lt;/code&gt; ads, and &lt;code&gt;stopped&lt;/code&gt; ads whose last shown date is more than a few days old. Be careful with partial fetches. If you only pulled 200 of an advertiser's 2,000 ads, a missing ad means nothing, so only judge "stopped" from complete fetches.&lt;/p&gt;

&lt;h2&gt;
  
  
  The hosted version
&lt;/h2&gt;

&lt;p&gt;The actor is &lt;a href="https://apify.com/zrk1212/google-ads-transparency-ad-copy" rel="noopener noreferrer"&gt;Google Ads Transparency Scraper: Ad Text, Domain Check&lt;/a&gt; on Apify. It does the RPC calls (through residential proxy by default, at no extra charge), preview data first, OCR second, YouTube links for video ads, the domain check mode and the new/stopped monitor.&lt;/p&gt;

&lt;p&gt;Pricing per 1,000: $1.20 per ad row, plus $1.00 per ad where text was actually found, and $2.00 per domain check. OCR attempts that find nothing are not charged. 1,000 ads from a Search-heavy advertiser with text found on 75% of them comes to about $1.95.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;apify_client&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;ApifyClient&lt;/span&gt;

&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;ApifyClient&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"YOUR_APIFY_TOKEN"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# 1) ad copy for a competitor's Search ads&lt;/span&gt;
&lt;span class="n"&gt;run&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;actor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"zrk1212/google-ads-transparency-ad-copy"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run_input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="s2"&gt;"mode"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"ads"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"domains"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"geico.com"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"region"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"US"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"adFormats"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"text"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"maxAdsPerTarget"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"includeAdText"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;ad&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dataset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"defaultDatasetId"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;iterate_items&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ad&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"adTextSource"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"|"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ad&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"headline"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# 2) which prospects run Google Ads&lt;/span&gt;
&lt;span class="n"&gt;run&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;actor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"zrk1212/google-ads-transparency-ad-copy"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run_input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="s2"&gt;"mode"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"domainCheck"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"domains"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"shopify.com"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"geico.com"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"example-small-shop.com"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"region"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"US"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dataset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"defaultDatasetId"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;iterate_items&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"domain"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"runsGoogleAds"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"adCountMin"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"adCountMax"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Limits worth knowing: the date filter only reaches about the last 12 months of shown history, ad counts are ranges because that is all Google publishes, and text hidden behind an ad's own product image cannot be recovered. The free Apify plan allows OCR on up to 200 ads per run.&lt;/p&gt;

&lt;p&gt;This is public transparency data that Google publishes for accountability. The actor is not affiliated with Google, does not log in, and does not solve CAPTCHAs.&lt;/p&gt;

</description>
      <category>webscraping</category>
      <category>marketing</category>
      <category>ocr</category>
      <category>ppc</category>
    </item>
    <item>
      <title>How to get TikTok transcripts in bulk: use the captions TikTok already made</title>
      <dc:creator>zachary klein</dc:creator>
      <pubDate>Thu, 19 Nov 2026 06:11:41 +0000</pubDate>
      <link>https://dev.to/zachary_klein_a8078ed2a7f/how-to-get-tiktok-transcripts-in-bulk-use-the-captions-tiktok-already-made-36g2</link>
      <guid>https://dev.to/zachary_klein_a8078ed2a7f/how-to-get-tiktok-transcripts-in-bulk-use-the-captions-tiktok-already-made-36g2</guid>
      <description>&lt;p&gt;If you want the words spoken in 500 TikTok videos, the obvious plan is to download each video and run it through Whisper. It works, and it is wasteful: a large share of TikTok videos already have a transcript sitting in the page, generated by TikTok itself.&lt;/p&gt;

&lt;p&gt;This post walks through where those captions live, the traps in picking the right track, and when you actually need speech-to-text. I ended up building this into an Apify actor, which I link at the end. Disclosure: I am the developer.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where TikTok keeps its captions
&lt;/h2&gt;

&lt;p&gt;Load a public video page, for example &lt;code&gt;https://www.tiktok.com/@nasa/video/7707774305056083213&lt;/code&gt;, and look at the HTML. Near the bottom is a script tag:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight html"&gt;&lt;code&gt;&lt;span class="nt"&gt;&amp;lt;script&lt;/span&gt; &lt;span class="na"&gt;id&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;"__UNIVERSAL_DATA_FOR_REHYDRATION__"&lt;/span&gt; &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;"application/json"&lt;/span&gt;&lt;span class="nt"&gt;&amp;gt;&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="nt"&gt;&amp;lt;/script&amp;gt;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That JSON is what the web app hydrates from. Inside it, under the video detail, is an &lt;code&gt;itemStruct&lt;/code&gt; with everything the page shows: author, description, hashtags, play counts, duration, and under &lt;code&gt;itemStruct.video&lt;/code&gt; two fields that matter here:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;subtitleInfos&lt;/code&gt;: a list of caption tracks. Each has a &lt;code&gt;Url&lt;/code&gt; to a WebVTT file, a &lt;code&gt;LanguageCodeName&lt;/code&gt; like &lt;code&gt;eng-US&lt;/code&gt;, a &lt;code&gt;Format&lt;/code&gt;, and a &lt;code&gt;Source&lt;/code&gt; such as &lt;code&gt;ASR&lt;/code&gt; or &lt;code&gt;MT&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;claInfo&lt;/code&gt;: TikTok's caption metadata. Its &lt;code&gt;captionInfos&lt;/code&gt; entries say whether each track is the original language and whether it was auto-generated, and &lt;code&gt;originalLanguageInfo&lt;/code&gt; says what language TikTok thinks is spoken.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Fetch the WebVTT URL, strip the cue timing, and you have the transcript with timestamps. No audio download, no model, about the cost of one extra HTTP request.&lt;/p&gt;

&lt;p&gt;Two practical notes. The caption URLs are signed and expire, so fetch them right after the page, not hours later. And TikTok is quick to challenge requests that do not look like a browser; a TLS-impersonating client (I use &lt;code&gt;curl_cffi&lt;/code&gt; with Chrome impersonation) gets clean 200s far more often than plain &lt;code&gt;requests&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Trap 1: the Source field lies
&lt;/h2&gt;

&lt;p&gt;The natural rule is "take the track whose &lt;code&gt;Source&lt;/code&gt; is &lt;code&gt;ASR&lt;/code&gt;, ignore &lt;code&gt;MT&lt;/code&gt; (machine translation)." That breaks on real videos.&lt;/p&gt;

&lt;p&gt;The NASA video above has exactly one track, &lt;code&gt;eng-US&lt;/code&gt;, labeled &lt;code&gt;Source: "MT"&lt;/code&gt;. By that rule it has no original captions. But &lt;code&gt;claInfo.captionInfos&lt;/code&gt; for the same track says &lt;code&gt;isOriginalCaption: true&lt;/code&gt; and &lt;code&gt;isAutoGen: false&lt;/code&gt;. It is a caption NASA uploaded, in English, in the spoken language. Treating it as a translation would throw away the best transcript on the page.&lt;/p&gt;

&lt;p&gt;So I classify tracks from &lt;code&gt;claInfo&lt;/code&gt; first and fall back to &lt;code&gt;Source&lt;/code&gt; only when &lt;code&gt;claInfo&lt;/code&gt; has no matching entry:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;original and auto-generated: &lt;code&gt;ASR&lt;/code&gt; (TikTok's speech recognition)&lt;/li&gt;
&lt;li&gt;original and not auto-generated: &lt;code&gt;CREATOR&lt;/code&gt; (uploaded by the creator)&lt;/li&gt;
&lt;li&gt;not original: &lt;code&gt;MT&lt;/code&gt; (a translation)&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Trap 2: picking the language
&lt;/h2&gt;

&lt;p&gt;A Spanish-language video from @duolingo carries two tracks: &lt;code&gt;spa-ES&lt;/code&gt; (original, auto-generated) and &lt;code&gt;eng-US&lt;/code&gt; (a translation). If you just take the first English track, you get a machine translation and lose the actual words.&lt;/p&gt;

&lt;p&gt;The order I use:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;If the user asked for a language, the matching original track, then a matching translation.&lt;/li&gt;
&lt;li&gt;The track whose language equals &lt;code&gt;claInfo.originalLanguageInfo.language&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Any track marked original.&lt;/li&gt;
&lt;li&gt;Any &lt;code&gt;ASR&lt;/code&gt; track.&lt;/li&gt;
&lt;li&gt;Whatever is first.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;That gives you the spoken language by default and a translation only when you ask for one.&lt;/p&gt;

&lt;h2&gt;
  
  
  When there are no captions
&lt;/h2&gt;

&lt;p&gt;Plenty of videos have no tracks. The page tells you, roughly, why: there is a &lt;code&gt;noCaptionReason&lt;/code&gt; code, and the music info says whether the audio is an original sound or a licensed track. In my test set, a talking-head video from @garyvee had no captions at all, and neither did a @gordonramsay cooking clip.&lt;/p&gt;

&lt;p&gt;For those you do need speech-to-text. The pipeline is short:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Download the audio (same session as the page fetch, so the CDN accepts it).&lt;/li&gt;
&lt;li&gt;Convert to 16 kHz mono with ffmpeg. That is what Whisper wants and it shrinks the upload.&lt;/li&gt;
&lt;li&gt;Transcribe with a Whisper endpoint that returns segments with timestamps.&lt;/li&gt;
&lt;li&gt;Filter the output. Whisper hallucinates on music and silence ("Thank you for watching" over a dance clip is a classic). If a video is music only, mark it &lt;code&gt;NO_SPEECH&lt;/code&gt; instead of returning junk.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Only doing step 1 to 4 for videos without captions changes the economics a lot. A captions-first run costs about one page fetch per video. A Whisper run costs one page fetch plus audio, conversion and per-minute model time.&lt;/p&gt;

&lt;h2&gt;
  
  
  Getting many videos, not one
&lt;/h2&gt;

&lt;p&gt;Three inputs cover most use cases:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Video URLs&lt;/strong&gt;, including &lt;code&gt;vm.tiktok.com&lt;/code&gt; short links (follow the redirect) and bare numeric IDs.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Profiles&lt;/strong&gt;: the latest N videos from a username. I use yt-dlp's flat extraction to list a profile's video IDs, then fetch each page myself.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Another scraper's output&lt;/strong&gt;: if you already ran a hashtag or search scraper, read the video URLs from its dataset and transcribe those.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;At volume, expect a few percent of requests to get challenged. Retry those on a different IP. In my runs, datacenter IPs actually did better than residential ones for TikTok pages, so I start on datacenter and only fall back to residential per video.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the output looks like
&lt;/h2&gt;

&lt;p&gt;For each video you want the transcript plus enough context to analyze it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"url"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"https://www.tiktok.com/@nasa/video/7707774305056083213"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"authorUsername"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"nasa"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"durationSec"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;146&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"views"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;100300&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"language"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"en"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"transcriptSource"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"native"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"captionKind"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"CREATOR"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"transcript"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"From turning up the power at Kennedy Space Center to turning our sights toward the Red Planet..."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"wordCount"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;377&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"segments"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"start"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;0.42&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"end"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;2.66&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"text"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"From turning up the power at Kennedy Space Center"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;transcriptSource&lt;/code&gt; (&lt;code&gt;native&lt;/code&gt;, &lt;code&gt;ai&lt;/code&gt; or &lt;code&gt;none&lt;/code&gt;) and &lt;code&gt;captionKind&lt;/code&gt; let you filter by quality later. For research on hooks, the first two or three segments are usually what you want.&lt;/p&gt;

&lt;h2&gt;
  
  
  The hosted version
&lt;/h2&gt;

&lt;p&gt;The actor is &lt;a href="https://apify.com/zrk1212/tiktok-transcript-scraper" rel="noopener noreferrer"&gt;TikTok Transcript Scraper&lt;/a&gt; on Apify. It does everything above: captions first with the &lt;code&gt;claInfo&lt;/code&gt; logic, Whisper only when there is no caption, music detection, SRT and WebVTT output, and video stats on every row.&lt;/p&gt;

&lt;p&gt;Pricing: $1.50 per 1,000 transcripts from TikTok captions, $0.004 per started minute for AI transcripts, $0.50 per 1,000 rows with stats but no transcript, and errors (deleted, private, invalid) are free. A typical mix comes out around $2 per 1,000 videos. One honest limit: AI transcription only runs on paid Apify plans, because it costs real money per minute. On the free plan you get TikTok captions only.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;apify_client&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;ApifyClient&lt;/span&gt;

&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;ApifyClient&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"YOUR_APIFY_TOKEN"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;run&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;actor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"zrk1212/tiktok-transcript-scraper"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run_input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="s2"&gt;"profiles"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"nasa"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"maxVideosPerProfile"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"aiFallback"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"skipVideosWithoutTranscript"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dataset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"defaultDatasetId"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;iterate_items&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"error"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;continue&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"transcriptSource"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"wordCount"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"transcript"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;80&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It does not search TikTok by hashtag or keyword itself yet. Run a hashtag scraper first and pass its dataset ID in &lt;code&gt;datasetId&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;A last note on use: transcripts are the creators' words. Research, analysis and accessibility are the common uses. Check copyright before republishing them.&lt;/p&gt;

</description>
      <category>tiktok</category>
      <category>python</category>
      <category>webscraping</category>
      <category>ai</category>
    </item>
    <item>
      <title>Workday career sites stop at 2,000 jobs. Here is how to get all of them.</title>
      <dc:creator>zachary klein</dc:creator>
      <pubDate>Thu, 19 Nov 2026 06:10:25 +0000</pubDate>
      <link>https://dev.to/zachary_klein_a8078ed2a7f/workday-career-sites-stop-at-2000-jobs-here-is-how-to-get-all-of-them-ia</link>
      <guid>https://dev.to/zachary_klein_a8078ed2a7f/workday-career-sites-stop-at-2000-jobs-here-is-how-to-get-all-of-them-ia</guid>
      <description>&lt;p&gt;I build data tools on Apify, and one of the first requests I kept seeing was simple: "I want every open job from Company X's Workday site." It sounds like a 30-line script. It is, right up until the company has more than 2,000 openings.&lt;/p&gt;

&lt;p&gt;This post explains the 2,000 cap, why paging cannot get past it, and the approach I ended up shipping: split the search by Workday's own filters until every slice is small enough, then merge and dedupe. At the end there is a hosted version you can call from Python if you do not want to maintain it yourself. Disclosure up front: I built that actor.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Workday jobs API in two minutes
&lt;/h2&gt;

&lt;p&gt;Every career site on &lt;code&gt;*.myworkdayjobs.com&lt;/code&gt; is a single-page app backed by a JSON endpoint. Open any Workday careers page, watch the network tab, and you will see a POST like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight http"&gt;&lt;code&gt;&lt;span class="no"&gt;POST&lt;/span&gt; &lt;span class="n"&gt;https&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="c1"&gt;//nvidia.wd5.myworkdayjobs.com/wday/cxs/nvidia/NVIDIAExternalCareerSite/jobs&lt;/span&gt;
&lt;span class="nc"&gt;Content&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nc"&gt;Type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;application&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;json&lt;/span&gt;

&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"appliedFacets"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"limit"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"offset"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"searchText"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;""&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The URL has three parts you need: the host (&lt;code&gt;nvidia.wd5.myworkdayjobs.com&lt;/code&gt;), the tenant (&lt;code&gt;nvidia&lt;/code&gt;) and the site name (&lt;code&gt;NVIDIAExternalCareerSite&lt;/code&gt;). You can read all three from any job link on the company's careers page.&lt;/p&gt;

&lt;p&gt;The response gives you &lt;code&gt;total&lt;/code&gt;, a page of &lt;code&gt;jobPostings&lt;/code&gt; (title, location, posted text, and an &lt;code&gt;externalPath&lt;/code&gt; you use to fetch the full job), and a &lt;code&gt;facets&lt;/code&gt; array. Keep the facets in mind, because they are the whole trick.&lt;/p&gt;

&lt;p&gt;No login, no cookies, no browser. A loop over &lt;code&gt;offset&lt;/code&gt; in steps of 20 gets you every job. For most companies, that is the end of the story.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where it breaks: total is always 2,000
&lt;/h2&gt;

&lt;p&gt;Try the loop on NVIDIA or Target. &lt;code&gt;total&lt;/code&gt; comes back as exactly &lt;code&gt;2000&lt;/code&gt;. Page past offset 2,000 and you get nothing.&lt;/p&gt;

&lt;p&gt;That number is not real. In my tests NVIDIA had about 2,630 open jobs and Target had over 14,000. The search simply refuses to report or return more than 2,000 results for any one query. A scraper that trusts &lt;code&gt;total&lt;/code&gt; and pages to the end will finish without errors and quietly hand you 14% of Target's jobs.&lt;/p&gt;

&lt;p&gt;People asking about this on Reddit usually get two suggestions: re-scrape every day for a month and hope the older jobs rotate to the top, or search by keyword and hope the keywords cover everything. Both leave gaps you cannot measure.&lt;/p&gt;

&lt;h2&gt;
  
  
  The facets tell you the real size
&lt;/h2&gt;

&lt;p&gt;Here is the useful part. The &lt;code&gt;facets&lt;/code&gt; array in the response lists every filter the site exposes, with a count for each value. For a big site it looks roughly like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"facetParameter"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"jobFamilyGroup"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"values"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"descriptor"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Engineering"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"0c40..."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"count"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1180&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"descriptor"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Sales"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"8f12..."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"count"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;410&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="err"&gt;...&lt;/span&gt;
  &lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Those counts are not capped. If you add up the values of a facet where each job has exactly one value (job family, country, worker type, time type), you get an estimate of the true total. That is how you know NVIDIA has 2,630 jobs while &lt;code&gt;total&lt;/code&gt; says 2,000.&lt;/p&gt;

&lt;p&gt;It also gives you a way in. If you send the same request with &lt;code&gt;"appliedFacets": {"jobFamilyGroup": ["0c40..."]}&lt;/code&gt;, you get a smaller search. If that slice is under 2,000, you can page it completely.&lt;/p&gt;

&lt;h2&gt;
  
  
  Splitting until every slice fits
&lt;/h2&gt;

&lt;p&gt;The algorithm I settled on is a recursive split:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Run the unfiltered search once. Read &lt;code&gt;total&lt;/code&gt; and the facets.&lt;/li&gt;
&lt;li&gt;If &lt;code&gt;total&lt;/code&gt; is under 2,000, page through it and stop.&lt;/li&gt;
&lt;li&gt;Otherwise pick a facet to split on and run one sub-search per value.&lt;/li&gt;
&lt;li&gt;For each sub-search, repeat from step 2, with the parent's filters still applied.&lt;/li&gt;
&lt;li&gt;Merge everything and dedupe by job path.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;A few details decide whether this returns 99.9% or 85%:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Pick the facet whose largest bucket is smallest.&lt;/strong&gt; Splitting Target by country does little if 13,000 jobs are in the US. Splitting by job family might leave every bucket under 2,000 in one step. I score each facet group by its biggest value and take the lowest, with a tiebreak preference for job family, then country, then state, then worker type.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Some facets are multi-valued.&lt;/strong&gt; A job can be listed in several locations, so location counts overlap and add up to more than the real total. They are fine for splitting (dedupe handles the overlap) but useless for estimating size, and you cannot subtract them to find "jobs with no value."&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Some jobs have no value in the facet you split on.&lt;/strong&gt; If job family counts sum to 13,900 and the site has 14,147 jobs, about 250 jobs have no job family. They appear in no bucket. You have to page the unfiltered parent search as a catch-all pass and let dedupe throw away the jobs you already have. For huge sites I skip that pass when the gap is under 1% and would cost hundreds of requests, and I record that the result is incomplete.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Cap the depth.&lt;/strong&gt; I stop at six levels. If a slice is still over 2,000 with no facet left to split by, only its first 2,000 jobs are reachable, and the run says so instead of pretending.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Offset paging is not stable.&lt;/strong&gt; Jobs close while you crawl, which shifts every later offset by one, so a job can slip between two pages. When you diff two runs to find closed jobs, confirm each "missing" job by requesting its detail page before marking it closed. Otherwise a job that dropped out of one page gets reported closed and then "new" again tomorrow.&lt;/p&gt;

&lt;h2&gt;
  
  
  The numbers
&lt;/h2&gt;

&lt;p&gt;With this approach, on the day I tested:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Site&lt;/th&gt;
&lt;th&gt;Workday says&lt;/th&gt;
&lt;th&gt;Real total (facet estimate)&lt;/th&gt;
&lt;th&gt;Returned&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;NVIDIA&lt;/td&gt;
&lt;td&gt;2,000&lt;/td&gt;
&lt;td&gt;about 2,632&lt;/td&gt;
&lt;td&gt;2,632&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Target&lt;/td&gt;
&lt;td&gt;2,000&lt;/td&gt;
&lt;td&gt;14,147&lt;/td&gt;
&lt;td&gt;14,145&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The two missing Target jobs are churn: jobs opening and closing during a crawl that takes a few minutes.&lt;/p&gt;

&lt;h2&gt;
  
  
  Things you get for free once you have every job
&lt;/h2&gt;

&lt;p&gt;The list endpoint only returns a summary. Each job's detail endpoint (the &lt;code&gt;externalPath&lt;/code&gt; under the same &lt;code&gt;/wday/cxs/&lt;/code&gt; base) has the full description, requisition ID, time type, remote type, all locations, country and the hiring legal entity.&lt;/p&gt;

&lt;p&gt;Workday has no salary field. US postings usually include a pay range in the description text, in formats like "172,500 USD - 325,000 USD for Level 4; and 210,000 USD - 403,000 USD for Level 5". A parser that handles that pattern, plus hourly rates and the common "$X to $Y" forms, gives you usable &lt;code&gt;salaryMin&lt;/code&gt; and &lt;code&gt;salaryMax&lt;/code&gt; for most US roles.&lt;/p&gt;

&lt;p&gt;If you run the crawl daily and keep the previous set of job IDs, you can label each job &lt;code&gt;new&lt;/code&gt;, &lt;code&gt;changed&lt;/code&gt;, &lt;code&gt;unchanged&lt;/code&gt; or &lt;code&gt;closed&lt;/code&gt;. For a large site that turns a 14,000-row export into a few hundred rows a day that actually matter, and you only need detail requests for the new and changed jobs.&lt;/p&gt;

&lt;h2&gt;
  
  
  The hosted version
&lt;/h2&gt;

&lt;p&gt;I packaged all of this as an Apify actor: &lt;a href="https://apify.com/zrk1212/workday-jobs-scraper" rel="noopener noreferrer"&gt;Workday Jobs Scraper&lt;/a&gt;. You give it Workday URLs or company names. It runs the split, fetches details, parses salary, and writes a completeness report (jobs found vs Workday's estimate) to the run's &lt;code&gt;OUTPUT&lt;/code&gt; record, so you can see whether you got everything. It also has the monitor mode described above.&lt;/p&gt;

&lt;p&gt;Pricing is $1.50 per 1,000 jobs saved, and unchanged jobs in monitor mode are free. All of NVIDIA costs about $4. The Apify free plan's monthly credit covers a test run.&lt;/p&gt;

&lt;p&gt;From Python with &lt;code&gt;apify-client&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;apify_client&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;ApifyClient&lt;/span&gt;

&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;ApifyClient&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"YOUR_APIFY_TOKEN"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;run&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;actor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"zrk1212/workday-jobs-scraper"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run_input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="s2"&gt;"siteUrls"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"https://target.wd5.myworkdayjobs.com/targetcareers"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"completeResults"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"includeDetails"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;jobs&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kt"&gt;list&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dataset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"defaultDatasetId"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;iterate_items&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;jobs&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"jobs"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# per-site completeness: jobs found vs Workday's own estimate&lt;/span&gt;
&lt;span class="n"&gt;summary&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;key_value_store&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"defaultKeyValueStoreId"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get_record&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"OUTPUT"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"value"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;site&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;summary&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"sites"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;site&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"estimatedTotal"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;site&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"completenessPct"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If you would rather build it yourself, everything above works with plain &lt;code&gt;requests&lt;/code&gt;. Respect the site: keep concurrency modest, you are hitting a company's careers site. And if you find a Workday site where the split does not reach 99%, I would like to see it. Open an issue on the actor page with the URL.&lt;/p&gt;

</description>
      <category>webscraping</category>
      <category>python</category>
      <category>api</category>
      <category>career</category>
    </item>
  </channel>
</rss>
