I built MiniSERP to learn how a search results API works underneath. It's a small Rails 8 API that takes a Google results page and turns it into clean JSON: title, link, snippet, position.
The first version worked on the first page I tested. I wrote the specs and they passed, and I thought the parser was done.
Then I saved a second results page with a different query and ran it through the same parser. I got back an empty array with HTTP 200.
Why that response was dangerous
An empty array with a 200 looks exactly like "Google found nothing for this query." Nothing crashed and nothing showed up in the logs. If I had shipped this, every search would have come back empty, and the API would have reported success every time.
The cause turned out to be simple. I ran a quick check on the new page:
grep -c 'class="g"' page2.html
# 0
My parser looked for results inside div.g. On this page, Google had moved results into div.MjjYud, rearranged what was inside them and rotated its obfuscated class names. My one selector matched nothing, so the parser happily returned nothing.
Fix 1: try more than one layout
Instead of trusting one selector, the parser now goes through a list of container strategies in order and uses the first one that finds usable results:
CONTAINER_STRATEGIES = [
"#rso div.g",
"#search div.g",
"#rso div.MjjYud",
"#search div.MjjYud"
].freeze
That covers the layouts I've seen, but Google will change it again, and a list of class names will always fall behind eventually.
Fix 2: a fallback that ignores class names
If every selector misses, the parser stops looking at class names completely and uses the page structure instead. It finds every <h3>, then walks up to the nearest <div> that also contains a link, and treats that as one result.
To make sure this really doesn't depend on class names, one spec wraps the results in a made-up container called div.totally-new-class. The parser still pulls the results out.
Two smaller filters keep the output clean:
- A block only counts as a result if it has both an
<h3>and ana[href]. That one rule drops the "People also ask" box. - Links are normalized. Without JavaScript, Google wraps some links in its own
/url?q=...redirect, so the parser unwraps those, keeps onlyhttp(s)links and drops Google's own navigation links.
Fix 3: empty must never be silent
This was the most important change, and it's the rule I'd keep even if I deleted everything else.
There are two very different situations that used to produce the same response:
-
Google really found nothing. The page says the search "did not match any documents." The API returns an empty list, HTTP 200 and
"no_results": true. -
My parser couldn't read the page. No known layout matched and Google didn't say there were no results. The parser raises
Scraper::Errors::ParseFailedand the API returns 502parse_failed.
The second case is my bug, not the user's empty search, and it should look like one. A 502 shows up in monitoring and someone can go fix the parser. A silent 200 just hides the problem until a customer notices their data is missing.
What I took from it
- Test on more than one real page. One saved fixture proved my parser worked on that one page. The second page proved it didn't work in general.
- Don't let a parser failure look like a valid empty result. If "no results" and "I couldn't read this" return the same thing, you can't tell them apart later.
- Scraping is maintenance work. Writing the first parser was the easy part. Keeping it correct while the page keeps changing is the real job.
That last point is why I have so much respect for search APIs that do this at scale. I hit one markup change on one engine in a side project. They deal with this across many engines, every day, behind a single API call.
MiniSERP is open source if you want to look at the parser and the specs: https://github.com/Divyanshu-2907/MiniSERP
Top comments (3)
the 502 split is the whole post. 'no results' and 'i could not read the page' should never share a 200.
I was getting a bit annoyed with this. Initially the test cases weren’t catching it, however this is now caught it by saving a second real page. Now we have to actually detect that Google returns its 'no results' message before returning an empty 200. Thanks for reading.
saving a second real page as the fixture is the right call. that google message check is where the real fix lives.