DEV Community

Abinash Sonowal
Abinash Sonowal

Posted on

How to Build a Google Shopping Scraper (and why it's so hard)

#automation #javascript #tutorial #webscraping

💻 View the full source code on GitHub

🛒 Run the Google Shopping Scraper on Apify

If you’re building an e-commerce price tracker or competitor analysis tool, you’ll quickly realize that scraping individual stores (Amazon, Walmart, Target, etc.) is a nightmare. Every site has a different DOM structure, different anti-bot measures, and different pagination logic.

The easiest solution? Scrape Google Shopping. It aggregates everything for you. But there’s a catch: scraping Google is incredibly difficult.

In this post, I'll explain exactly why it's so hard, the underlying code logic required to bypass Google's defenses, and how you can run my pre-built scraper for fractions of a cent.

Why is scraping Google Shopping so hard?

If you just run a standard fetch or axios request against Google Shopping, you won't get what you expect. Here is what you're up against:

  1. Aggressive Anti-Bot & Captchas: Google heavily monitors traffic. If you make too many requests from a single datacenter IP, you will instantly hit a CAPTCHA wall.
  2. Dynamic DOM & Deferred Execution: The HTML that Google returns is not the HTML you see in the browser. Critical product data isn't in standard HTML tags; it is injected dynamically by JavaScript.
  3. The "OAPV" Async Call & Hidden Data: Google often loads the most important pricing and variant data via asynchronous internal API calls (like OAPV and batchexecute) after the page loads. If you don't use a headless browser (which is slow and expensive), you have to reverse-engineer how Google hides this data in the initial HTML payload before the async calls fire.

The Code Logic: How we beat it (Without a Headless Browser)

To keep costs extremely low ($0.003 per run) and speeds blazing fast (under 10 seconds), I built the Google Shopping Search Scraper entirely in Node.js without using Playwright or Puppeteer.

Here is the exact logic of how it works under the hood:

1. SERP Proxies

Instead of standard datacenter proxies, the scraper uses specialized GOOGLE_SERP proxies provided by Apify. These proxies are designed specifically to rotate IPs optimally and handle Google's fingerprinting, ensuring we rarely hit a block.

const proxyConfiguration = await Actor.createProxyConfiguration({
    groups: ['GOOGLE_SERP'],
    countryCode: 'US', // Localized pricing!
});
const httpClient = new ProxyHttpClient(await proxyConfiguration.newUrl());
Enter fullscreen mode Exit fullscreen mode

2. Crafting the Exact Request

We construct the URL targeting tbm=shop (The Shopping tab). We explicitly pass hl (language) and gl (country) parameters so Google returns the correct localized pricing and currency.

const params = new URLSearchParams({ q: 'iphone 16', tbm: 'shop', hl: 'en', gl: 'us' });
const url = `http://www.google.com/search?${params.toString()}`;
Enter fullscreen mode Exit fullscreen mode

3. Parsing the Deferred HTML Payload

This is where the magic happens. Because we aren't using a browser, the async JS never executes. Instead, we have to extract the data manually from hidden script tags.

For example, Google defers the loading of real product URLs and images by hiding them inside escaped JavaScript blocks (jsl.dh()). Our parser uses a custom regex to find these blocks, decodes the JS strings (\xHH), and loads them into Cheerio:

/** Parse the deferred HTML hidden inside jsl.dh() blocks */
function extractInjectedHtml(html) {
    let injected = '';
    const re = /jsl\.dh\([^,]+,\s*"((?:[^"\\]|\\.)*)"\s*\);/g;
    let m;
    while ((m = re.exec(html)) !== null) {
        // Decode \xHH sequences to reconstruct the HTML
        injected += decodeJsString(m[1]); 
    }
    return cheerio.load(injected);
}
Enter fullscreen mode Exit fullscreen mode

4. Extracting the google.ldi Image Map

High-resolution product images are not stored in <img src="..."> tags. Google stores them in a massive JSON object mapped by IDs (google.ldi). We parse this object out of the raw HTML and map the image URLs back to the DOM nodes.

const ldiMap = extractLdiMap(html); // Parses google.ldi = {...};
// Later, when parsing a product card:
const id = dimg.attr('id');
const imageUrl = ldiMap[id];
Enter fullscreen mode Exit fullscreen mode

5. Constructing the Immersive URL

Finally, to get deep pricing (what Google calls the immersive "oshop" view), we have to extract internal Google IDs (headlineOfferDocid, imageDocid, catalogid) from the DOM's data-* attributes and reconstruct a highly specific prds= query parameter string.

6. Assembling the Final Product Object

Once we have bypassed the blocks, extracted the deferred HTML, mapped the high-res images, and constructed the deep immersive URLs, all that is left is to assemble the clean JSON payload that our API consumers expect:

products.push({
    position: products.length + 1,
    title: text($card.find(SELECTORS.title)),
    url: buildProductUrl(query, country, { headlineOfferDocid, imageDocid, catalogid }),
    price: text($card.find(SELECTORS.price)),
    second_hand_condition: extractCondition($card),
    rating: extractRating($card),
    review_count: extractReviewCount($card),
    source: text($card.find(SELECTORS.source)),
    image: imageUrl || injectedImages[pid] || null,
});
Enter fullscreen mode Exit fullscreen mode

Try it out yourself

By reverse-engineering Google's deferred JS and async payloads, this scraper can extract ~40 products in under 10 seconds, returning a beautiful, clean JSON array of products, prices, ratings, and URLs.

If you are building a price monitor or doing market research, don't waste weeks trying to bypass Google's captchas and decipher their injected JS blocks. You can run my actor directly on Apify's infrastructure, or view the full source code yourself!

Happy Scraping!

Top comments (0)