<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Ricardo Batista</title>
    <description>The latest articles on DEV Community by Ricardo Batista (@rbatista19).</description>
    <link>https://dev.to/rbatista19</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F1418786%2F23772d20-e7df-4f2f-a42e-3b638a582674.png</url>
      <title>DEV Community: Ricardo Batista</title>
      <link>https://dev.to/rbatista19</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/rbatista19"/>
    <language>en</language>
    <item>
      <title>Best ChatGPT scraper in 2026: what breaks, and what 8 tools cost</title>
      <dc:creator>Ricardo Batista</dc:creator>
      <pubDate>Mon, 17 Aug 2026 05:57:00 +0000</pubDate>
      <link>https://dev.to/rbatista19/best-chatgpt-scraper-in-2026-what-breaks-and-what-8-tools-cost-4im6</link>
      <guid>https://dev.to/rbatista19/best-chatgpt-scraper-in-2026-what-breaks-and-what-8-tools-cost-4im6</guid>
      <description>&lt;p&gt;The best ChatGPT scraper for brand monitoring is a managed API that returns parsed citations, and cloro is the only one of the eight below that ships that parsing rather than handing back HTML. For general-purpose scraping at volume, Bright Data's Scraping Browser is the stronger pick. For zero budget with engineering capacity, it is Playwright and a proxy bill. Which of those three you are depends on the table further down, not on the verdict.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Key takeaways&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The official OpenAI API cannot answer brand-monitoring questions, because citations, shopping cards and the search-versus-memory decision only exist in the web UI.&lt;/li&gt;
&lt;li&gt;At 1,000 queries a day, DIY Playwright runs $980-2,140 a month once proxies, CAPTCHA and 8-15 engineer hours are counted, against $100-300 for a managed API.&lt;/li&gt;
&lt;li&gt;Selector maintenance is the hidden line item. OpenAI's dynamic CSS class names change roughly weekly, and the failure is silent rather than loud.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;There are two ChatGPTs. The API developers build on, and the web interface &lt;a href="https://almcorp.com/blog/chatgpt-900-million-weekly-active-users/" rel="noopener noreferrer"&gt;900 million weekly users&lt;/a&gt; actually see. Only the second one browses the web, cites sources, renders &lt;a href="https://cloro.dev/blog/chatgpt-shopping/" rel="noopener noreferrer"&gt;shopping cards&lt;/a&gt;, and decides per-query whether to search or answer from memory.&lt;/p&gt;

&lt;p&gt;If you want to know whether ChatGPT recommends your product or names your competitor instead, the API structurally cannot tell you. You have to read the rendered UI, which means scraping one of the better-defended properties on the public web.&lt;/p&gt;

&lt;h2&gt;
  
  
  What makes chatgpt.com hard to scrape?
&lt;/h2&gt;

&lt;h3&gt;
  
  
  1. Cloudflare fingerprints your TLS handshake
&lt;/h3&gt;

&lt;p&gt;Per &lt;a href="https://scrapfly.io/blog/posts/how-to-bypass-cloudflare-anti-scraping" rel="noopener noreferrer"&gt;Scrapfly's analysis&lt;/a&gt;, the 2026 stack starts with JA4 TLS fingerprinting. Default &lt;code&gt;requests&lt;/code&gt; or &lt;code&gt;urllib&lt;/code&gt; fail on the handshake, before any of your logic runs. Then behavioral profiling of timing and movement, JavaScript challenges a static client cannot execute, and Turnstile CAPTCHAs on anything that smells automated.&lt;/p&gt;

&lt;p&gt;Datacenter IPs are the fastest way to get nowhere. &lt;a href="https://www.proxies.sx/blog/scrape-chatgpt-search-results-mobile-proxies" rel="noopener noreferrer"&gt;proxies.sx testing&lt;/a&gt; has real mobile IPs surviving 50-100+ queries while datacenter ranges get blocked in the first few requests.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. The response streams
&lt;/h3&gt;

&lt;p&gt;ChatGPT does not return HTML. It streams tokens over Server-Sent Events, so you keep the connection alive, parse &lt;code&gt;event: message&lt;/code&gt; frames as they land, assemble the answer, and detect the end signal. Static HTTP clients see nothing. Headless browsers handle it transparently and charge you in compute and latency.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Class names change between deploys
&lt;/h3&gt;

&lt;p&gt;OpenAI's React build emits dynamic classes like &lt;code&gt;._a4b3f&lt;/code&gt;. Anything selecting on them breaks roughly weekly. Semantic selectors (ARIA labels, roles, text content) survive longer, and you still want fallback chains. This is the line item every DIY estimate underprices.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. Login, 2FA, and session persistence
&lt;/h3&gt;

&lt;p&gt;Most useful workflows need an account, most accounts have 2FA, and the login flow sits behind the same Cloudflare checks. Three realistic paths: export cookies after a manual login and refresh on a schedule, pay someone to hold the session, or run headful Playwright with stealth plugins and accept the breakage.&lt;/p&gt;

&lt;h3&gt;
  
  
  5. Proxy economics
&lt;/h3&gt;

&lt;p&gt;&lt;a href="https://databay.com/blog/how-much-do-residential-proxies-cost" rel="noopener noreferrer"&gt;Databay's 2026 breakdown&lt;/a&gt; puts residential proxies at $3-15/GB, with the mobile IPs ChatGPT tolerates at the top of that range. At 1,000 queries a day you burn roughly 3-8 GB a month, plus CAPTCHA credits at $1-3 per 1,000 challenges. That line item alone often exceeds a managed subscription. There is a fuller treatment of &lt;a href="https://cloro.dev/blog/proxies-for-serp-scraping/" rel="noopener noreferrer"&gt;proxy selection for SERP work&lt;/a&gt; elsewhere.&lt;/p&gt;

&lt;h2&gt;
  
  
  The eight options
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Tool&lt;/th&gt;
&lt;th&gt;What it solves&lt;/th&gt;
&lt;th&gt;What you still write&lt;/th&gt;
&lt;th&gt;Starting price&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://cloro.dev/chatgpt/" rel="noopener noreferrer"&gt;cloro&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Access, SSE assembly, &lt;a href="https://cloro.dev/chatgpt/sources/" rel="noopener noreferrer"&gt;citation parsing&lt;/a&gt;, fan-out, auth&lt;/td&gt;
&lt;td&gt;Nothing&lt;/td&gt;
&lt;td&gt;$100/mo, 500 free credits&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Apify&lt;/td&gt;
&lt;td&gt;Access and compute, via marketplace actors&lt;/td&gt;
&lt;td&gt;Cookie refresh; parsing varies by actor&lt;/td&gt;
&lt;td&gt;$49/mo + compute&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Bright Data&lt;/td&gt;
&lt;td&gt;Access at any scale, 72M+ IPs, CAPTCHA&lt;/td&gt;
&lt;td&gt;SSE assembly and all parsing&lt;/td&gt;
&lt;td&gt;~$1.50/1k at volume&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Browserbase&lt;/td&gt;
&lt;td&gt;Persistent sessions, stealth, agent ergonomics&lt;/td&gt;
&lt;td&gt;SSE assembly, parsing, selectors&lt;/td&gt;
&lt;td&gt;$50/mo, free tier&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Browserless&lt;/td&gt;
&lt;td&gt;Headless Chrome, self-hostable&lt;/td&gt;
&lt;td&gt;SSE assembly, parsing, selectors, some evasion&lt;/td&gt;
&lt;td&gt;$50/mo, free Docker image&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;ScrapingBee&lt;/td&gt;
&lt;td&gt;Cloudflare bypass with a clean API&lt;/td&gt;
&lt;td&gt;SSE assembly, parsing, selectors&lt;/td&gt;
&lt;td&gt;$49/mo, 100k credits&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;ZenRows&lt;/td&gt;
&lt;td&gt;Cloudflare bypass, cheaper tier&lt;/td&gt;
&lt;td&gt;SSE assembly, parsing, selectors&lt;/td&gt;
&lt;td&gt;$69/mo, 250k credits&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Playwright&lt;/td&gt;
&lt;td&gt;Nothing. It is the toolkit&lt;/td&gt;
&lt;td&gt;Everything&lt;/td&gt;
&lt;td&gt;Free, plus your time&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;This roundup is published by cloro, which is one of the eight tools in it. The prices and capabilities attributed to the other seven come from their own documentation, linked where they are not obvious.&lt;/p&gt;

&lt;p&gt;The split that matters is the third column. Every tool except the managed APIs leaves you owning SSE assembly and selector maintenance, which is the part that fails weekly rather than the part that fails loudly.&lt;/p&gt;

&lt;p&gt;A few notes the table cannot hold. Apify's actors are community-maintained more often than not, so reliability tracks whoever still cares about that actor. Bright Data's Scraping Browser is genuinely hard for Cloudflare to block and overkill below serious volume. Browserbase is built for agents and its session persistence solves the auth problem cleanly, which is a real advantage here. ScrapingBee's team &lt;a href="https://www.scrapingbee.com/blog/best-chatgpt-scraper-tools/" rel="noopener noreferrer"&gt;published their own ChatGPT scraper roundup&lt;/a&gt;, which is a decent signal they understand the target.&lt;/p&gt;

&lt;p&gt;Out of scope on purpose: the official OpenAI API (different surface), wrappers like LiteLLM and OpenRouter (they wrap the API, not the UI), Cloudflare-bypass libraries like Camoufox and FlareSolverr (they solve the WAF and leave you the rest), and answer APIs like Tavily and Exa (synthesized answers, not the rendered UI).&lt;/p&gt;

&lt;h2&gt;
  
  
  What it costs at 1,000 queries a day
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Tool&lt;/th&gt;
&lt;th&gt;Subscription&lt;/th&gt;
&lt;th&gt;Proxies/credits&lt;/th&gt;
&lt;th&gt;CAPTCHA&lt;/th&gt;
&lt;th&gt;Eng hrs/mo&lt;/th&gt;
&lt;th&gt;Total $/mo&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;cloro&lt;/td&gt;
&lt;td&gt;$30-300&lt;/td&gt;
&lt;td&gt;included&lt;/td&gt;
&lt;td&gt;included&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;$30-300&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Apify (official actor)&lt;/td&gt;
&lt;td&gt;$49&lt;/td&gt;
&lt;td&gt;$30-80&lt;/td&gt;
&lt;td&gt;included&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;$280-410&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;ScrapingBee&lt;/td&gt;
&lt;td&gt;$49-249&lt;/td&gt;
&lt;td&gt;included&lt;/td&gt;
&lt;td&gt;included&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;$449-849&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;ZenRows&lt;/td&gt;
&lt;td&gt;$69-249&lt;/td&gt;
&lt;td&gt;included&lt;/td&gt;
&lt;td&gt;included&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;$469-849&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Browserless&lt;/td&gt;
&lt;td&gt;$50-100&lt;/td&gt;
&lt;td&gt;$100-300&lt;/td&gt;
&lt;td&gt;$30-90&lt;/td&gt;
&lt;td&gt;6&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;$580-890&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Bright Data&lt;/td&gt;
&lt;td&gt;pay-as-you-go&lt;/td&gt;
&lt;td&gt;$200-500&lt;/td&gt;
&lt;td&gt;included&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;$600-900&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Browserbase&lt;/td&gt;
&lt;td&gt;$50-200&lt;/td&gt;
&lt;td&gt;$50-150&lt;/td&gt;
&lt;td&gt;$30-90&lt;/td&gt;
&lt;td&gt;6&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;$730-1,040&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Playwright (DIY)&lt;/td&gt;
&lt;td&gt;$0&lt;/td&gt;
&lt;td&gt;$150-450&lt;/td&gt;
&lt;td&gt;$30-90&lt;/td&gt;
&lt;td&gt;8-15&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;$980-2,140&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Assumptions, because a cost table without them is decoration: engineer time at $100/hour fully loaded, CAPTCHA solver at $2 per 1,000 challenges firing on 5% of requests, mobile residential proxies at $10/GB, requests averaging 4-8 KB.&lt;/p&gt;

&lt;p&gt;Two things fall out. The DIY gap widens with volume, because proxy bandwidth scales linearly while per-call rates flatten or improve with tiers. And the browser-infrastructure tier sits in an awkward middle: cheaper than Bright Data, more expensive than a managed API once you price the parsing work.&lt;/p&gt;

&lt;p&gt;Change the engineer-hours assumption and the ranking moves. If your team already runs Playwright infrastructure and selector patches cost you an hour a month rather than ten, DIY wins outright. That is a real case, not a strawman.&lt;/p&gt;

&lt;h2&gt;
  
  
  Picking
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Already on Apify, ChatGPT is one target of several.&lt;/strong&gt; Use the official actor rather than a community one.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Volume is the binding constraint.&lt;/strong&gt; Bright Data, and budget for writing the parsing layer.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Building an agent that needs a live session.&lt;/strong&gt; Browserbase.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;You want Cloudflare handled and nothing else.&lt;/strong&gt; ScrapingBee, or ZenRows if the budget is tight.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Zero budget, real engineering capacity.&lt;/strong&gt; Playwright with stealth plugins and mobile proxies. Plan for 8-15 hours a month.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;You want parsed citations and fan-out with no maintenance.&lt;/strong&gt; &lt;a href="https://cloro.dev/chatgpt/" rel="noopener noreferrer"&gt;cloro&lt;/a&gt; is built for that case and is a monitoring tool rather than a general scraper. If you need arbitrary browser automation, buy browser infrastructure instead.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;One legal note, since it comes up in every thread: scraping your own session or publicly visible content is generally fine, while bypassing authentication or pulling other people's data is not. &lt;a href="https://cloro.dev/blog/website-scraping-legal/" rel="noopener noreferrer"&gt;Where the line sits&lt;/a&gt; is covered properly elsewhere.&lt;/p&gt;

&lt;p&gt;If you are doing this for brand monitoring rather than data collection, the &lt;a href="https://cloro.dev/blog/llm_visibility_tracking_tools/" rel="noopener noreferrer"&gt;tools built for that job&lt;/a&gt; are a different category and probably the better starting point.&lt;/p&gt;

</description>
      <category>chatgpt</category>
      <category>webscraping</category>
      <category>api</category>
      <category>python</category>
    </item>
    <item>
      <title>How to scrape Google AI Mode: citation pills + place cards</title>
      <dc:creator>Ricardo Batista</dc:creator>
      <pubDate>Fri, 14 Aug 2026 05:49:00 +0000</pubDate>
      <link>https://dev.to/rbatista19/how-to-scrape-google-ai-mode-citation-pills-place-cards-2c7</link>
      <guid>https://dev.to/rbatista19/how-to-scrape-google-ai-mode-citation-pills-place-cards-2c7</guid>
      <description>&lt;p&gt;&lt;strong&gt;Key takeaways&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;AI Mode and AI Overview cite the same URLs only 13.7% of the time, so scraping one tells you nothing reliable about the other.&lt;/li&gt;
&lt;li&gt;Source URLs live in HTML comments keyed by UUID, not in anchor tags, so a scraper reading &lt;code&gt;&amp;lt;a href&amp;gt;&lt;/code&gt; returns an empty citation list rather than an error.&lt;/li&gt;
&lt;li&gt;Strip &lt;code&gt;#:~:text&lt;/code&gt; scroll-to-text fragments before counting, or one article reads as several distinct URLs and inflates your citation numbers.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Google AI Mode does not put its sources in anchor tags. It renders each one as a button, and hides the actual URL in an HTML comment keyed by UUID. A scraper that reads &lt;code&gt;&amp;lt;a href&amp;gt;&lt;/code&gt; finds nothing. That one design choice is most of the work in this post.&lt;/p&gt;

&lt;p&gt;The rest is that nothing is in the initial HTML at all. The shell loads, then the answer streams in over a separate call to &lt;code&gt;/async/folwr&lt;/code&gt;. Parse on page load and you get an empty container.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Build the URL with &lt;code&gt;udm=50&lt;/code&gt; and &lt;code&gt;aep=11&lt;/code&gt; to force the interface.&lt;/li&gt;
&lt;li&gt;Register a network interceptor on &lt;code&gt;https://www.google.com/async/folwr&lt;/code&gt; before navigating.&lt;/li&gt;
&lt;li&gt;Navigate, and treat a non-200 as a CAPTCHA rather than a dead page.&lt;/li&gt;
&lt;li&gt;Poll until the intercepted response lands, with a hard timeout.&lt;/li&gt;
&lt;li&gt;Read the answer text off &lt;code&gt;data-session-thread-id&lt;/code&gt;, not off a snapshot.&lt;/li&gt;
&lt;li&gt;Collect the citation buttons, then rejoin them to their HTML-comment metadata by UUID.&lt;/li&gt;
&lt;li&gt;Rewrite the pills into anchors before converting anything to markdown.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Steps 6 and 7 are where the work is. The rest is plumbing.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why bother with this surface specifically
&lt;/h2&gt;

&lt;p&gt;AI Mode is not AI Overview with a different skin. Per &lt;a href="https://almcorp.com/blog/google-ai-mode-cites-itself-organic-links-seo-2026/" rel="noopener noreferrer"&gt;ALM Corp's analysis of 1.3 million AI Mode citations&lt;/a&gt;, &lt;strong&gt;the two cite the same URLs only 13.7% of the time&lt;/strong&gt; on semantically similar queries, and google.com self-cites in &lt;strong&gt;17.42%&lt;/strong&gt; of AI Mode answers, more than YouTube, Facebook, Reddit, Amazon, Indeed and Zillow combined. Scraping one tells you nothing reliable about the other.&lt;/p&gt;

&lt;p&gt;It is also where the comparison queries go. &lt;a href="https://almcorp.com/blog/google-ai-overviews-organic-ctr-2026/" rel="noopener noreferrer"&gt;Seer Interactive's 2026 trigger data&lt;/a&gt; puts AI surfaces on 95.4% of comparison queries and 85.9% of question-format ones, and AI Mode passed 75 million daily active users by late 2025 per &lt;a href="https://blog.google/company-news/inside-google/message-ceo/alphabet-earnings-q3-2025/" rel="noopener noreferrer"&gt;Alphabet's Q3 2025 earnings call&lt;/a&gt;. The &lt;a href="https://cloro.dev/blog/query-fan-out/" rel="noopener noreferrer"&gt;query fan-out&lt;/a&gt; behind it, where Google splits your question into subtopics and runs many searches at once, is why one answer stitches together so many cited pages.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Surface&lt;/th&gt;
&lt;th&gt;Trigger&lt;/th&gt;
&lt;th&gt;Citation style&lt;/th&gt;
&lt;th&gt;Difficulty&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;AI Mode&lt;/td&gt;
&lt;td&gt;&lt;code&gt;udm=50&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Pills, metadata in HTML comments&lt;/td&gt;
&lt;td&gt;High: async streaming, drifting selectors&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://cloro.dev/blog/how_to_scrape_google_ai_overview/" rel="noopener noreferrer"&gt;AI Overview&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Organic query, varies by class&lt;/td&gt;
&lt;td&gt;Inline source list above the SERP&lt;/td&gt;
&lt;td&gt;Medium: single fetch&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://cloro.dev/blog/scrape-gemini/" rel="noopener noreferrer"&gt;Gemini&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;gemini.google.com&lt;/td&gt;
&lt;td&gt;Grounded sources panel&lt;/td&gt;
&lt;td&gt;High: separate auth domain&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://cloro.dev/blog/scrape-chatgpt/" rel="noopener noreferrer"&gt;ChatGPT Search&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;chatgpt.com with search on&lt;/td&gt;
&lt;td&gt;Numbered citations&lt;/td&gt;
&lt;td&gt;Highest: SSE plus Cloudflare&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  Force the interface
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;udm=50&lt;/code&gt; is the switch. &lt;code&gt;aep=11&lt;/code&gt; goes with it.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;search_url&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;build_url_with_params&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://www.google.com/search&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;udm&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;50&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;        &lt;span class="c1"&gt;# AI Mode
&lt;/span&gt;        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;aep&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;11&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;q&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;prompt&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;hl&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;google_params&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;hl&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;gl&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;google_params&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;gl&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
    &lt;span class="p"&gt;},&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Wait for the stream, then parse
&lt;/h2&gt;

&lt;p&gt;We drive &lt;a href="https://playwright.dev/python/docs/network" rel="noopener noreferrer"&gt;Playwright&lt;/a&gt; and capture the async call, then poll until it lands. Sixty seconds, then fail loudly. A hard timeout beats a silent partial parse, because a half-streamed answer looks like a real answer with fewer citations.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;page_interceptor&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;PlaywrightInterceptor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;do_not_block_resources&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;page_interceptor&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;add_capture_urls&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://www.google.com/async/folwr&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;page_interceptor&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;setup_page_interceptor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;page&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;120&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;                      &lt;span class="c1"&gt;# 500ms intervals, 60s cap
&lt;/span&gt;    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;page_interceptor&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;captured_responses&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="k"&gt;break&lt;/span&gt;
    &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;sleep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;500&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;Exception&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Never received AI Mode response after 60 seconds&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The answer text hangs off &lt;code&gt;data-session-thread-id&lt;/code&gt;. Target that and read its parent, because AI Mode rewrites its own container several times while streaming and a &lt;a href="https://playwright.dev/python/docs/locators" rel="noopener noreferrer"&gt;locator&lt;/a&gt; survives that where a snapshot does not.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;thread_element&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;page&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;locator&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;[data-session-thread-id]&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;text&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;thread_element&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;locator&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;..&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;text_content&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="sh"&gt;""&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  The citation pills
&lt;/h2&gt;

&lt;p&gt;Here is the shape you are working against. A UUID ties the visible button to a comment block carrying the real URLs.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight html"&gt;&lt;code&gt;&lt;span class="c"&gt;&amp;lt;!--Sv6Kpe[["uuid-12345",["label","description"],["https://example.com","source2"]]]--&amp;gt;&lt;/span&gt;
&lt;span class="nt"&gt;&amp;lt;button&lt;/span&gt; &lt;span class="na"&gt;data-icl-uuid=&lt;/span&gt;&lt;span class="s"&gt;"uuid-12345"&lt;/span&gt; &lt;span class="na"&gt;data-amic=&lt;/span&gt;&lt;span class="s"&gt;"true"&lt;/span&gt;&lt;span class="nt"&gt;&amp;gt;&lt;/span&gt;[1]&lt;span class="nt"&gt;&amp;lt;/button&amp;gt;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;So: find the buttons, read their UUIDs, regex the comments back out of the page HTML, and filter Google's own domains out of the URL list. One pill can carry several sources.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;extract_aimode_citation_pills&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;page&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Page&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;Dict&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;List&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;LinkData&lt;/span&gt;&lt;span class="p"&gt;]]:&lt;/span&gt;
    &lt;span class="n"&gt;citation_pills&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Dict&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;List&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;LinkData&lt;/span&gt;&lt;span class="p"&gt;]]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{}&lt;/span&gt;
    &lt;span class="n"&gt;pill_locators&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;page&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;locator&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;button[data-icl-uuid][data-amic=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;true&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;]&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;page_html&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;html&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;unescape&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;page&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;content&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;

    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;pill_locators&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;count&lt;/span&gt;&lt;span class="p"&gt;()):&lt;/span&gt;
        &lt;span class="n"&gt;pill_button&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;pill_locators&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;nth&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;pill_button&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;is_visible&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
            &lt;span class="k"&gt;continue&lt;/span&gt;
        &lt;span class="n"&gt;uuid&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;pill_button&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get_attribute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;data-icl-uuid&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;uuid&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;continue&lt;/span&gt;

        &lt;span class="n"&gt;pattern&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sa"&gt;rf&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;&amp;lt;!--Sv6Kpe\[\[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;escape&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;uuid&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;.*?]]--&amp;gt;&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;
        &lt;span class="n"&gt;current_pill&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;List&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;LinkData&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;

        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;content&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;findall&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pattern&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;page_html&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;DOTALL&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
            &lt;span class="n"&gt;desc_match&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;search&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
                &lt;span class="sa"&gt;rf&lt;/span&gt;&lt;span class="sh"&gt;'"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;escape&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;uuid&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;\s*,\s*\[\s*&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;[^&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;]+&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;\s*,\s*&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;([^&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;]+)&lt;/span&gt;&lt;span class="sh"&gt;"'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;content&lt;/span&gt;
            &lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="n"&gt;url&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;next&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
                &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;u&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;u&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;findall&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;'"&lt;/span&gt;&lt;span class="s"&gt;(https://[^&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;]+)&lt;/span&gt;&lt;span class="sh"&gt;"'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;content&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
                 &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="nf"&gt;any&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;skip&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;u&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;skip&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt;
                            &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;google.com&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;gstatic.com&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;encrypted-tbn&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])),&lt;/span&gt;
                &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="n"&gt;url&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;#:~:text&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
                &lt;span class="n"&gt;url&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="se"&gt;\\&lt;/span&gt;&lt;span class="s"&gt;u003d&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="se"&gt;\\&lt;/span&gt;&lt;span class="s"&gt;u0026&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;&amp;amp;&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
                &lt;span class="n"&gt;current_pill&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;LinkData&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
                    &lt;span class="n"&gt;position&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;current_pill&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                    &lt;span class="n"&gt;label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Source &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;current_pill&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                    &lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                    &lt;span class="n"&gt;description&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;desc_match&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;group&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;desc_match&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="p"&gt;))&lt;/span&gt;

        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;current_pill&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;citation_pills&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;uuid&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;current_pill&lt;/span&gt;

    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;citation_pills&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;#:~:text&lt;/code&gt; strip matters more than it looks. Google appends scroll-to-text fragments, so the same article comes back as several distinct URLs and your citation counts inflate quietly.&lt;/p&gt;

&lt;p&gt;The &lt;a href="https://cloro.dev/ai-mode/sources/" rel="noopener noreferrer"&gt;sources panel&lt;/a&gt; needs its own selector per layout. Web-results pages expose it one way, the dialog another. Playwright's &lt;a href="https://playwright.dev/python/docs/actionability" rel="noopener noreferrer"&gt;actionability checks&lt;/a&gt; handle the waiting, so log and continue on timeout rather than killing the run.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;sources_selector&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;[data-container-id=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;rhs-col&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;] [role=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;dialog&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;] a&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;is_web_results_page&lt;/span&gt;
    &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;a.ZbQNgf&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Converting to markdown without losing the sources
&lt;/h2&gt;

&lt;p&gt;Hand the raw HTML to &lt;a href="https://pypi.org/project/html2text/" rel="noopener noreferrer"&gt;html2text&lt;/a&gt; and every pill becomes a dead button. Rewrite the pills into real anchors first, using the UUID map you already built, then convert.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;soup&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;BeautifulSoup&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;html_content&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;html.parser&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;button&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;soup&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;find_all&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;button&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;attrs&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;data-icl-uuid&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;data-amic&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;true&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;}):&lt;/span&gt;
    &lt;span class="n"&gt;uuid&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;button&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;data-icl-uuid&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;link_data&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;reversed&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;citation_pills&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;uuid&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[])):&lt;/span&gt;
        &lt;span class="n"&gt;anchor&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;soup&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;new_tag&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;a&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;href&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;link_data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;url&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
        &lt;span class="n"&gt;anchor&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;string&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;link_data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;label&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;button&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;insert_after&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;anchor&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;button&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;decompose&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;span class="n"&gt;h&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;html2text&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;HTML2Text&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ignore_links&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="bp"&gt;False&lt;/span&gt;
&lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;body_width&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
&lt;span class="n"&gt;markdown&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;handle&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;str&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;soup&lt;/span&gt;&lt;span class="p"&gt;)).&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  What breaks at scale
&lt;/h2&gt;

&lt;p&gt;A non-200 is usually a &lt;a href="https://cloro.dev/blog/solve-captchas/" rel="noopener noreferrer"&gt;CAPTCHA&lt;/a&gt;, not a dead page, so solve before you give up. One blocked request should not tank a batch.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;response&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;page&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;goto&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;search_url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;20_000&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="nf"&gt;is_http_success&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;solve_captcha&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;page&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;page_interceptor&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;Exception&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;HTTP error: &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; (probably captcha)&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The real running cost is selector drift. AI Mode's DOM changes without notice, and a scraper that worked last month returns empty rather than throwing. Alert on success rate, not on exceptions. A sudden drop is a layout change, and you will find it weeks late if you are only watching for errors. You will also want a &lt;a href="https://cloro.dev/blog/proxies-for-serp-scraping/" rel="noopener noreferrer"&gt;proxy pool&lt;/a&gt; and fingerprint rotation before this runs at any volume.&lt;/p&gt;

&lt;h2&gt;
  
  
  Or skip it
&lt;/h2&gt;

&lt;p&gt;cloro maintains this pipeline as a managed endpoint, which is the disclosure to weigh this section against. One POST, parsed text and sources back, &lt;a href="https://cloro.dev/blog/serp-api-reliability-at-scale/" rel="noopener noreferrer"&gt;P50 under 8 seconds&lt;/a&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;response&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;post&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://api.cloro.dev/v1/monitor/aimode&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Authorization&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Bearer sk_live_your_api_key&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;
    &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;prompt&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;What do you know about Tesla&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;s latest updates?&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
          &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;country&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;US&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
          &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;include&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;markdown&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;}},&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;()[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;result&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;sources&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; citations, fan-out: &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;searchQueries&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Built in-house, this runs $5,000 to $10,000 a month once you count engineering time, browser instances and proxies. Below a few thousand queries a month that maths does not favour us, and you should build it. Above that it does, and &lt;a href="https://cloro.dev/ai-mode/" rel="noopener noreferrer"&gt;cloro's AI Mode endpoint&lt;/a&gt; absorbs the selector drift.&lt;/p&gt;

&lt;p&gt;If you want the dashboard rather than the pipeline, &lt;a href="https://cloro.dev/blog/llm_visibility_tracking_tools/" rel="noopener noreferrer"&gt;the tools that track this surface&lt;/a&gt; are compared separately.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>dataextraction</category>
      <category>aimode</category>
    </item>
    <item>
      <title>SERP API reliability at scale: uptime, latency, concurrency</title>
      <dc:creator>Ricardo Batista</dc:creator>
      <pubDate>Wed, 12 Aug 2026 05:46:10 +0000</pubDate>
      <link>https://dev.to/rbatista19/serp-api-reliability-at-scale-uptime-latency-concurrency-op0</link>
      <guid>https://dev.to/rbatista19/serp-api-reliability-at-scale-uptime-latency-concurrency-op0</guid>
      <description>&lt;p&gt;&lt;strong&gt;Key takeaways&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Two of the four headline reliability numbers in this category are not uptime at all: DataForSEO publishes 99.95% with no credit terms, and Bright Data's 99.99% is a success rate from a benchmark with a disclosed vested interest.&lt;/li&gt;
&lt;li&gt;At 10,000 requests a day the P95 decides your architecture, not the median. Bright Data measures 2.61s P50 against 4.92s P95, and a 2-second median can hide a 15-second tail once queuing starts.&lt;/li&gt;
&lt;li&gt;Google's &lt;code&gt;&amp;amp;num=100&lt;/code&gt; removal plus DataForSEO's depth-based pricing took the Live tier from $2.00 to $17.50 per 1,000 at 100 results with AI Overview, a ninefold jump from a change you did not make.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Below a few hundred requests a day, every SERP API looks fine. The failure modes only show up somewhere past 10,000: queue latency, rate-limit rejections, concurrency ceilings. By then you have built on top of one.&lt;/p&gt;

&lt;p&gt;Three things decide it, and only one of them appears on a pricing page.&lt;/p&gt;

&lt;p&gt;This comparison is published by cloro, which is one of the four providers in it. Every figure attributed to another vendor is sourced below; the cloro rows are self-reported and should be read as such.&lt;/p&gt;

&lt;h2&gt;
  
  
  What does a SERP API uptime SLA actually commit to?
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Provider&lt;/th&gt;
&lt;th&gt;Uptime SLA&lt;/th&gt;
&lt;th&gt;Median response&lt;/th&gt;
&lt;th&gt;Rate limits&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;cloro&lt;/td&gt;
&lt;td&gt;99.9% with service credits&lt;/td&gt;
&lt;td&gt;Sub-second (async queue)&lt;/td&gt;
&lt;td&gt;Parallel lanes, no hard cap&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;SerpApi&lt;/td&gt;
&lt;td&gt;None published&lt;/td&gt;
&lt;td&gt;&lt;a href="https://scrape.do/blog/serpapi-alternatives/" rel="noopener noreferrer"&gt;1.96s&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Hard monthly cap, request queuing&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;DataForSEO&lt;/td&gt;
&lt;td&gt;
&lt;a href="https://nextgrowth.ai/dataforseo-api-guide/" rel="noopener noreferrer"&gt;99.95%&lt;/a&gt;, no credit terms stated&lt;/td&gt;
&lt;td&gt;Live fastest; ~5-min Standard queue&lt;/td&gt;
&lt;td&gt;High default RPS, async batching&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Bright Data&lt;/td&gt;
&lt;td&gt;None published&lt;/td&gt;
&lt;td&gt;&lt;a href="https://hackernoon.com/serp-benchmarks-success-rates-and-latency-at-scale" rel="noopener noreferrer"&gt;2.61s standard, 0.89s premium&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Configurable 5-20+, no documented ceiling&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Two of those four numbers are not what they look like.&lt;/p&gt;

&lt;p&gt;DataForSEO's 99.95% is higher than ours and the documentation states no credit or penalty terms. That makes it a target rather than a remedy. A published SLA without credits tells you what a provider aims for, not what you recover when they miss.&lt;/p&gt;

&lt;p&gt;Bright Data's widely quoted 99.99% is a &lt;strong&gt;success rate&lt;/strong&gt;, not uptime, and it comes from a benchmark whose author &lt;a href="https://hackernoon.com/serp-benchmarks-success-rates-and-latency-at-scale" rel="noopener noreferrer"&gt;disclosed a vested interest&lt;/a&gt;. Success rate measures request completion. Uptime measures whether the endpoint answers at all. They are different failures and only one of them wakes you up.&lt;/p&gt;

&lt;p&gt;So when you compare this column, check three things: is it uptime or success rate, is it contractual or measured, and does anything happen when they miss it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Latency: the median is the wrong number
&lt;/h2&gt;

&lt;p&gt;Almost every published latency figure is measured sequentially, which is the easy case. What decides production behaviour is what happens when you dispatch faster than the provider's queue clears.&lt;/p&gt;

&lt;p&gt;The best public numbers come from that same 30-day benchmark of nine providers: Bright Data's standard P50 at 2.61s against a &lt;strong&gt;P95 of 4.92s&lt;/strong&gt;, in a field ranging from Serper at 2.23s to DataForSEO at 4.54s. Premium routing takes the P50 to 0.89s, and that is the vendor-adjacent figure, so discount it.&lt;/p&gt;

&lt;p&gt;At 10,000 daily requests the P95 is the number that matters. A 2-second median hides a 15-second tail once queuing kicks in. An overnight rank tracker can absorb that tail. Brand monitoring cannot, because a five-minute delay on an incident query defeats the point.&lt;/p&gt;

&lt;p&gt;DataForSEO's Standard queue is the clearest case: one tester measured async callbacks averaging &lt;a href="https://nextgrowth.ai/dataforseo-api-guide/" rel="noopener noreferrer"&gt;12 seconds for 100 keywords&lt;/a&gt;, stretching toward five minutes at production volume. That is fine for scheduled work and useless for anything real-time.&lt;/p&gt;

&lt;h2&gt;
  
  
  Concurrency is an architecture choice, not a setting
&lt;/h2&gt;

&lt;p&gt;A 10k batch survives or fails on this, and each provider picked a different model.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;SerpApi&lt;/strong&gt; queues. Hit the rate limit and requests wait behind one another rather than running in parallel, so latency climbs with volume. Exhaust the monthly cap and requests fail outright until the next billing cycle. If your volume is variable you overprovision for the worst day and pay for headroom you rarely use.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;DataForSEO&lt;/strong&gt; batches asynchronously. Submit, wait, retrieve. It absorbs thousands of tasks without per-request rate errors, and you pay for it in freshness. No requests-per-second ceiling appears in the docs.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Bright Data&lt;/strong&gt; hands you the dial. Its own &lt;a href="https://github.com/brightdata/bright-data-serp-city-accurate-rank-tracking" rel="noopener noreferrer"&gt;rank-tracking demo&lt;/a&gt; defaults to 5 concurrent and recommends 5 to 10, documenting a 1 to 20+ range with no published ceiling. Past the recommended range you are tuning against undocumented limits and watching for rate errors yourself.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;cloro&lt;/strong&gt; runs parallel lanes behind an async job queue. You POST a batch, get a job ID back in milliseconds, and results arrive by webhook as workers finish. Request 10,000 does not wait behind the other 9,999, so adding volume widens the fan-out instead of lengthening the wait. Concurrency scales by tier rather than capping at a fixed lane count.&lt;/p&gt;

&lt;p&gt;The honest summary of that difference: SerpApi and DataForSEO both trade real-time responsiveness for predictable batch cost, in opposite directions. If your workload is nightly and cost-sensitive, DataForSEO's queue is the right trade and cloro is overkill.&lt;/p&gt;

&lt;h2&gt;
  
  
  AI Overview coverage, where the gap is measurable
&lt;/h2&gt;

&lt;p&gt;For GEO work the question is whether you get &lt;a href="https://cloro.dev/ai-overview/" rel="noopener noreferrer"&gt;AI Overview&lt;/a&gt; data as consistent structured fields, not whether the provider returns it at all.&lt;/p&gt;

&lt;p&gt;SerpApi detected AI Overviews in &lt;a href="https://scrape.do/blog/serpapi-alternatives/" rel="noopener noreferrer"&gt;68% of qualifying queries&lt;/a&gt; in a published 25-query benchmark. Roughly a third of pages that should show an Overview returned nothing detectable, and that is the ugly kind of gap: a missing Overview reads identically to an absent one, so your trend lines encode the parser's blind spots as changes in Google.&lt;/p&gt;

&lt;p&gt;DataForSEO makes it opt-in with &lt;a href="https://nextgrowth.ai/dataforseo-api-guide/" rel="noopener noreferrer"&gt;&lt;code&gt;load_async_ai_overview&lt;/code&gt;&lt;/a&gt; at $0.0006 per keyword, which doubles the Live tier to about $4.00 per 1,000 at ten results, and &lt;a href="https://cloro.dev/comparisons/dataforseo-alternatives/" rel="noopener noreferrer"&gt;$17.50 per 1,000 at 100 results with AIO&lt;/a&gt;. Structured, but you submit and poll rather than reading it inline.&lt;/p&gt;

&lt;p&gt;cloro returns Overview text, cited sources and position in the same payload as organic results. The reason that matters is not convenience: Google changed the Overview DOM during its January 2025 JS-rendering update, and anyone parsing raw HTML rebuilt their extraction. Provider-managed fields absorb that server-side. It is the strongest argument for buying rather than building, covered at length in &lt;a href="https://cloro.dev/blog/ai-search-visibility-build-vs-buy/" rel="noopener noreferrer"&gt;this build-versus-buy breakdown&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Cost at volume, which moved twice in 2025
&lt;/h2&gt;

&lt;p&gt;Model this at your depth, not the advertised rate.&lt;/p&gt;

&lt;p&gt;SerpApi is subscription with hard caps and no pay-as-you-go: roughly $2,500 at 100k monthly requests, $25,000 at 1M. DataForSEO is the cheapest at raw volume, about $0.60 per 1,000 on Standard, so 300k requests a month lands near $180. Bright Data prices per call near $0.005 with a minimum commitment &lt;a href="https://thunderbit.com/blog/best-serp-api-for-fast-accurate-data" rel="noopener noreferrer"&gt;cited around $499/month&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Then two changes reset the arithmetic. DataForSEO moved to depth-based pricing on September 19, 2025, and Google deprecated &lt;code&gt;&amp;amp;num=100&lt;/code&gt; on September 11, 2025, forcing pagination at 10 results a page. If you need 100 results plus AI Overview, that Live tier goes from $2.00 to $17.50 per 1,000. A ninefold jump, from a parameter change you did not make. A &lt;a href="https://cloro.dev/blog/cheapest-serp-api/" rel="noopener noreferrer"&gt;fuller pricing breakdown&lt;/a&gt; runs the same numbers at four depths.&lt;/p&gt;

&lt;h2&gt;
  
  
  Pick by workload
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Nightly batches, cost-sensitive:&lt;/strong&gt; DataForSEO. The queue latency is real and irrelevant to you, and $0.60 per 1,000 is hard to argue with. Watch the depth pricing.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;City-level geo-targeting across many markets:&lt;/strong&gt; Bright Data. 195 countries with city-level targeting is the widest coverage published, and you manage concurrency yourself.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Low volume, many engines, want it managed:&lt;/strong&gt; SerpApi. Live inside a tier and the caps never bite.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Real-time monitoring past 10k/day:&lt;/strong&gt; &lt;a href="https://cloro.dev/serp-api/" rel="noopener noreferrer"&gt;cloro&lt;/a&gt; is built for this shape, and its constraint is that it is API-first. If you want a dashboard rather than endpoints, buy a dashboard.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Whatever you pick, run your own 10k-request test against it under concurrent load before you commit. Every number above, cloro's included, was measured by someone with an interest in the result.&lt;/p&gt;

</description>
      <category>api</category>
      <category>llm</category>
      <category>dataextraction</category>
      <category>ai</category>
    </item>
    <item>
      <title>🚀 Launching FidForward! 🚀</title>
      <dc:creator>Ricardo Batista</dc:creator>
      <pubDate>Mon, 08 Jul 2024 16:54:15 +0000</pubDate>
      <link>https://dev.to/rbatista19/launching-fidforward-54gc</link>
      <guid>https://dev.to/rbatista19/launching-fidforward-54gc</guid>
      <description>&lt;p&gt;Today, Bernardo and I are excited to launch FidForward in private beta!&lt;/p&gt;

&lt;p&gt;We're looking for companies with 10-50 employees willing to pilot the next generation of performance management.&lt;/p&gt;

&lt;p&gt;With two pilot programs already running successfully (achieving a 30-50% improvement in eNPS), we’re eager to expand FidForward to more companies to validate these results further.&lt;/p&gt;

&lt;p&gt;If you’re a founder seeking to implement performance management or a larger company aiming to elevate your current system, please reach out to me!&lt;/p&gt;

&lt;p&gt;🔗 Learn more about FidForward at &lt;a href="https://fidforward.com" rel="noopener noreferrer"&gt;FidForward.com&lt;/a&gt;&lt;/p&gt;

</description>
      <category>react</category>
      <category>node</category>
    </item>
    <item>
      <title>We were tired of bureaucracy, so we built an opensource repo for the best guides</title>
      <dc:creator>Ricardo Batista</dc:creator>
      <pubDate>Sun, 23 Jun 2024 21:09:11 +0000</pubDate>
      <link>https://dev.to/rbatista19/we-were-tired-of-bureaucracy-so-we-built-an-opensource-repo-for-the-best-guides-da6</link>
      <guid>https://dev.to/rbatista19/we-were-tired-of-bureaucracy-so-we-built-an-opensource-repo-for-the-best-guides-da6</guid>
      <description>&lt;p&gt;Today we are launching &lt;a href="https://tramitit.com/" rel="noopener noreferrer"&gt;Tramitit&lt;/a&gt;, a shared database on getting all those poor-UX local procedures done in a much simpler way!&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;How it works&lt;/strong&gt;: The community provides detailed walkthroughs on every possible bureaucratic process you might go through.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;For the community by the community&lt;/strong&gt;: The content on this website is curated, verified, and rated by the community.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Welcoming providers that can help our users&lt;/strong&gt;: Although our guides are as simple as possible, users may still prefer to delegate tasks to a curated provider.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;We are in a bit a chicken &amp;amp; egg problem: for the traffic to pick up and have contributors, we need to start by adding some content - some of which we have already gone through and fixed it, other not yet there. We will get there over time, and work with providers that can also bring the quality up.&lt;/p&gt;

&lt;p&gt;You can find our &lt;a href="https://github.com/tramitit/guides" rel="noopener noreferrer"&gt;GitHub repo here&lt;/a&gt;, where you can contribute as well!&lt;br&gt;
Our &lt;a href="https://github.com/tramitit/tramitit.github.io" rel="noopener noreferrer"&gt;website is also open-source&lt;/a&gt;, so feel free to add anything.&lt;/p&gt;

</description>
    </item>
    <item>
      <title>Getting started with Open SaaS</title>
      <dc:creator>Ricardo Batista</dc:creator>
      <pubDate>Sun, 21 Apr 2024 07:34:17 +0000</pubDate>
      <link>https://dev.to/rbatista19/getting-started-with-open-saas-2gnb</link>
      <guid>https://dev.to/rbatista19/getting-started-with-open-saas-2gnb</guid>
      <description>&lt;p&gt;When building &lt;a href="https://article-generation.com/" rel="noopener noreferrer"&gt;Article Generation&lt;/a&gt; and &lt;a href="https://meeting-reminders.com/" rel="noopener noreferrer"&gt;Meeting Reminders&lt;/a&gt;, I decided to get started as fast as possible. So I looked for a free boilerplate and stumbled upon &lt;a href="https://opensaas.sh/" rel="noopener noreferrer"&gt;Open SaaS&lt;/a&gt;, which used YC-backed &lt;a href="https://wasp-lang.dev/" rel="noopener noreferrer"&gt;Wasp&lt;/a&gt;. It is a full-stack React + NodeJS + Prisma that takes 8 hours to get started with.&lt;/p&gt;

&lt;p&gt;Their documentation is brilliant, and great support over Discord. I decided to list here the full walkthrough on how to set up the development environment so that it is a simple copy-paste for you.&lt;/p&gt;

&lt;p&gt;For frontend deployment, I used &lt;a href="https://www.netlify.com/" rel="noopener noreferrer"&gt;Netlify&lt;/a&gt; (for the generous free package) and the recommended &lt;a href="https://fly.io/" rel="noopener noreferrer"&gt;fly.io&lt;/a&gt; for server + database (also cheap package).&lt;/p&gt;

&lt;h2&gt;
  
  
  Run development commands
&lt;/h2&gt;

&lt;p&gt;Start development database:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;cd app &amp;amp;&amp;amp; wasp start db
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;View development database in Prisma:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;wasp db studio
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Start development app:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;cd app &amp;amp;&amp;amp; wasp start
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Start development blog:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;cd blog &amp;amp;&amp;amp; npm install &amp;amp;&amp;amp; npm run dev
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Run frontend deployment commands
&lt;/h2&gt;

&lt;p&gt;Locate the folder:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;cd .wasp/build/web-app
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Install dependencies:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;npm install &amp;amp;&amp;amp; REACT_APP_API_URL={your-server-URL} npm run build
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Deploy preview:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;netlify deploy&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;Deploy to production:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;netlify deploy --prod&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;Every time there is a Wasp app rebuild (with&amp;nbsp;wasp build), it will remove your&amp;nbsp;&lt;code&gt;.wasp/build/&lt;/code&gt;&amp;nbsp;directory. Until the GitHub issue is fixed, you need to update &lt;code&gt;netlify.toml&lt;/code&gt;’s publish line to:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;  publish = ".wasp/build/web-app/build"
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Run backend &amp;amp; db deployment commands
&lt;/h2&gt;

&lt;p&gt;Locate the folder:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;cd .wasp/build/
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Deploy preview:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;flyctl deploy --remote-only --config ../../src/shared/fly-server.toml
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Every time there is a Wasp app rebuild (with&amp;nbsp;wasp build), it will remove your&amp;nbsp;&lt;code&gt;.wasp/build/&lt;/code&gt;&amp;nbsp;directory. That’s why there is a copy of the file in &lt;code&gt;app/src/shared/&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Run blog deployment commands
&lt;/h2&gt;

&lt;p&gt;Locate the folder:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;cd blog
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Build the app:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;npm run build
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Deploy preview:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;netlify deploy
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Deploy to production:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;netlify deploy --prod
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Good luck with building, hopefully it will be faster now!&lt;/p&gt;

</description>
      <category>wasp</category>
      <category>node</category>
      <category>react</category>
    </item>
  </channel>
</rss>
