<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Divyanshu Kumar</title>
    <description>The latest articles on DEV Community by Divyanshu Kumar (@divyanshu_kumar_0704).</description>
    <link>https://dev.to/divyanshu_kumar_0704</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F2606784%2F0ac8e48c-6e76-45c1-8e92-77786b57976f.jpg</url>
      <title>DEV Community: Divyanshu Kumar</title>
      <link>https://dev.to/divyanshu_kumar_0704</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/divyanshu_kumar_0704"/>
    <language>en</language>
    <item>
      <title>My search scraper returned zero results, and HTTP 200 said everything was fine</title>
      <dc:creator>Divyanshu Kumar</dc:creator>
      <pubDate>Sat, 21 Nov 2026 06:36:58 +0000</pubDate>
      <link>https://dev.to/divyanshu_kumar_0704/my-search-scraper-returned-zero-results-and-http-200-said-everything-was-fine-5a0c</link>
      <guid>https://dev.to/divyanshu_kumar_0704/my-search-scraper-returned-zero-results-and-http-200-said-everything-was-fine-5a0c</guid>
      <description>&lt;p&gt;I built MiniSERP to learn how a search results API works underneath. It's a small Rails 8 API that takes a Google results page and turns it into clean JSON: title, link, snippet, position.&lt;/p&gt;

&lt;p&gt;The first version worked on the first page I tested. I wrote the specs and they passed, and I thought the parser was done.&lt;/p&gt;

&lt;p&gt;Then I saved a second results page with a different query and ran it through the same parser. I got back an empty array with HTTP 200.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why that response was dangerous
&lt;/h2&gt;

&lt;p&gt;An empty array with a 200 looks exactly like "Google found nothing for this query." Nothing crashed and nothing showed up in the logs. If I had shipped this, every search would have come back empty, and the API would have reported success every time.&lt;/p&gt;

&lt;p&gt;The cause turned out to be simple. I ran a quick check on the new page:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;grep -c &lt;span class="s1"&gt;'class="g"'&lt;/span&gt; page2.html
&lt;span class="c"&gt;# 0&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;My parser looked for results inside &lt;code&gt;div.g&lt;/code&gt;. On this page, Google had moved results into &lt;code&gt;div.MjjYud&lt;/code&gt;, rearranged what was inside them and rotated its obfuscated class names. My one selector matched nothing, so the parser happily returned nothing.&lt;/p&gt;

&lt;h2&gt;
  
  
  Fix 1: try more than one layout
&lt;/h2&gt;

&lt;p&gt;Instead of trusting one selector, the parser now goes through a list of container strategies in order and uses the first one that finds usable results:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ruby"&gt;&lt;code&gt;&lt;span class="no"&gt;CONTAINER_STRATEGIES&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
  &lt;span class="s2"&gt;"#rso div.g"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="s2"&gt;"#search div.g"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="s2"&gt;"#rso div.MjjYud"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="s2"&gt;"#search div.MjjYud"&lt;/span&gt;
&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nb"&gt;freeze&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That covers the layouts I've seen, but Google will change it again, and a list of class names will always fall behind eventually.&lt;/p&gt;

&lt;h2&gt;
  
  
  Fix 2: a fallback that ignores class names
&lt;/h2&gt;

&lt;p&gt;If every selector misses, the parser stops looking at class names completely and uses the page structure instead. It finds every &lt;code&gt;&amp;lt;h3&amp;gt;&lt;/code&gt;, then walks up to the nearest &lt;code&gt;&amp;lt;div&amp;gt;&lt;/code&gt; that also contains a link, and treats that as one result.&lt;/p&gt;

&lt;p&gt;To make sure this really doesn't depend on class names, one spec wraps the results in a made-up container called &lt;code&gt;div.totally-new-class&lt;/code&gt;. The parser still pulls the results out.&lt;/p&gt;

&lt;p&gt;Two smaller filters keep the output clean:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A block only counts as a result if it has both an &lt;code&gt;&amp;lt;h3&amp;gt;&lt;/code&gt; and an &lt;code&gt;a[href]&lt;/code&gt;. That one rule drops the "People also ask" box.&lt;/li&gt;
&lt;li&gt;Links are normalized. Without JavaScript, Google wraps some links in its own &lt;code&gt;/url?q=...&lt;/code&gt; redirect, so the parser unwraps those, keeps only &lt;code&gt;http(s)&lt;/code&gt; links and drops Google's own navigation links.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Fix 3: empty must never be silent
&lt;/h2&gt;

&lt;p&gt;This was the most important change, and it's the rule I'd keep even if I deleted everything else.&lt;/p&gt;

&lt;p&gt;There are two very different situations that used to produce the same response:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Google really found nothing.&lt;/strong&gt; The page says the search "did not match any documents." The API returns an empty list, HTTP 200 and &lt;code&gt;"no_results": true&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;My parser couldn't read the page.&lt;/strong&gt; No known layout matched and Google didn't say there were no results. The parser raises &lt;code&gt;Scraper::Errors::ParseFailed&lt;/code&gt; and the API returns &lt;strong&gt;502 &lt;code&gt;parse_failed&lt;/code&gt;&lt;/strong&gt;.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The second case is my bug, not the user's empty search, and it should look like one. A 502 shows up in monitoring and someone can go fix the parser. A silent 200 just hides the problem until a customer notices their data is missing.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I took from it
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Test on more than one real page.&lt;/strong&gt; One saved fixture proved my parser worked on that one page. The second page proved it didn't work in general.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Don't let a parser failure look like a valid empty result.&lt;/strong&gt; If "no results" and "I couldn't read this" return the same thing, you can't tell them apart later.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Scraping is maintenance work.&lt;/strong&gt; Writing the first parser was the easy part. Keeping it correct while the page keeps changing is the real job.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That last point is why I have so much respect for search APIs that do this at scale. I hit one markup change on one engine in a side project. They deal with this across many engines, every day, behind a single API call.&lt;/p&gt;

&lt;p&gt;MiniSERP is open source if you want to look at the parser and the specs: &lt;a href="https://github.com/Divyanshu-2907/MiniSERP" rel="noopener noreferrer"&gt;https://github.com/Divyanshu-2907/MiniSERP&lt;/a&gt;&lt;/p&gt;

</description>
      <category>ruby</category>
      <category>rails</category>
      <category>webscraping</category>
      <category>api</category>
    </item>
  </channel>
</rss>
