DEV Community

Devil Scrapes
Devil Scrapes

Posted on

grep '"aggregate_rating":' finds zero matches on Zomato. The field is there, just double-escaped.

Quick answer: Zomato embeds its listing data as window.__PRELOADED_STATE__ = JSON.parse("...") — a JSON-encoded JSON string, with every quote inside it backslash-escaped. Grep the raw HTML for "aggregate_rating": and you get zero matches, even though the rating is sitting right there in the page. It's just spelled \"aggregate_rating\": because it's a string inside a string. Miss that and you'll conclude the data isn't server-rendered at all, which is exactly the wrong conclusion.

Zero hits on a field you can see in the browser

Open a Zomato restaurant listing page, view source, and aggregate_rating is visibly there if you scroll to the right script tag. Run grep '"aggregate_rating":' against the same raw response and it comes back empty. That gap — visible to your eyes, invisible to a naive text scan — is the whole bug, and it's worth sitting with for a second because it's a specific, repeatable trap: the field exists, the substring you searched for doesn't, because the escaping changed its literal bytes.

JSON.parse("...") wrapping a hand-escaped string is the tell. Zomato's server serializes the state object to JSON once, then serializes that string again so it can sit safely inside a .parse("…") call in a <script> tag. The result is a JSON string whose payload is itself JSON text, escaped one level deep:

window.__PRELOADED_STATE__ = JSON.parse("{\"pages\":{\"search\":{...\"aggregate_rating\":\"4.1\"...}}}")
Enter fullscreen mode Exit fullscreen mode

The fix is escape-aware isolation, then two parses

You can't just find the closing " with a naive html.find('"', start) either — you'll stop at the first escaped quote inside the payload, truncating everything after it. You need to walk the string respecting backslash-escapes:

def _find_string_end(html: str, start: int) -> int | None:
    index = start + 1
    while index < len(html):
        char = html[index]
        if char == "\\":
            index += 2   # skip the escaped character entirely
            continue
        if char == '"':
            return index
        index += 1
    return None
Enter fullscreen mode Exit fullscreen mode

Once you have the correctly-bounded quoted string, it takes exactly two json.loads calls to get back the object tree Zomato's own frontend consumes — the first unescapes the JS string, the second parses the JSON it contains:

quoted = html[quote_start:quote_end + 1]
inner_text = json.loads(quoted)   # unescape: \" -> "
state = json.loads(inner_text)    # parse the real JSON payload
Enter fullscreen mode Exit fullscreen mode

That's it. No brace-depth counting, no regex trying to guess where the object ends — the string boundary already tells you exactly where the payload stops, because JSON.parse("...") is by definition one JS string literal.

The second problem: two page shapes, different field paths

With the blob decoded, the next surprise is that Zomato doesn't ship one listing-page shape — it ships at least two, and they don't agree on structure:

  • Collection pages (/bangalore/pizza) — records live at pages.collectionDetails.<id>.SECTION_ENTITIES_DATA[], with name, url, rating sitting at the top level of each record.
  • Search pages (/mumbai/andheri-west-restaurants) — records live at pages.search.<path>.sections.SECTION_SEARCH_RESULT[], with the same fields nested one level under info, and the URL relocated to a sibling cardAction.clickUrl instead of info.url.

Even the locality field's key name changes between shapes (text on one, name on the other). Hardcode a path for either shape and the other type of URL silently returns nothing.

The fix we shipped walks the whole parsed state tree and recognizes a restaurant record by signature rather than by fixed path: any dict carrying a rating object with an aggregate_rating key — checked both directly on the record and nested one level under info — gets treated as a restaurant. That's more robust to Zomato redeploying either page template, because it doesn't care which container the record showed up in, only what shape the record itself has.

What a real run produced

A verified cloud run against a Bangalore collection URL and a Mumbai search URL returned 25 rows, 25 unique — 16 from Bangalore, 9 from Mumbai, both page shapes parsing correctly in the same batch. A sample row:

{
  "name": "Lakehouse",
  "aggregate_rating": 4.1,
  "votes": 493,
  "price_for_two": 850,
  "locality": "...",
  "city": "Bangalore"
}
Enter fullscreen mode Exit fullscreen mode

Ratings and vote counts come back as real int/float"2,664" parses to 2664, not a string you have to clean up downstream.

What we handle so you don't have to

We rotate through Chrome / Firefox / Safari TLS fingerprints, retry with exponential backoff on 408/429/5xx, and isolate each listing URL in a batch — one malformed or unreachable record gets skipped and logged, never taking the rest of the run down with it. Every row carries its source_listing_url and a scraped_at timestamp so scheduled runs diff cleanly over time.


🍽️ Zomato Restaurants Scraper turns a list of Zomato city/cuisine URLs into clean rows — name, rating, votes, locality, cuisines, and price for two — with numbers parsed to real types instead of raw comma-formatted strings. $4.20 per 1,000 restaurants, and you only pay for rows that land.

FAQ

Why does grepping the raw HTML for "aggregate_rating": return zero results on Zomato?
Because the field is inside window.__PRELOADED_STATE__ = JSON.parse("...") — a JSON-encoded string where every quote is backslash-escaped. The literal bytes are \"aggregate_rating\":, not the unescaped form.

How do you extract the payload correctly?
Walk the string respecting backslash-escapes to find the real closing quote, then run json.loads twice — once to unescape the JS string, once to parse the JSON payload inside it.

Does Zomato use one consistent JSON shape across all listing pages?
No — collection pages and search pages nest the same fields differently, down to the locality field having a different key name on each. The parser matches records by signature (an aggregate_rating field, direct or under info) rather than a fixed path.

Are ratings and vote counts returned as clean numbers?
Yes — aggregate_rating and votes are parsed to real float/int, including comma-formatted vote strings like "2,664".

Top comments (0)