For the past decade, I’ve been designing data pipelines, and I’ve learned one hard truth: treat a search engine as a simple static HTML page, and your database will be a broken mess within 48 hours. Modern anti-scraping systems are no longer just looking at your IP; they are inspecting your identity.
Why Your Current Approach Is Failing
Most developers try to scrape using standard requests or axios libraries. You’ll be blocked instantly, not necessarily because of your proxy, but because of JA3 fingerprinting.
The server checks your TLS handshake—your elliptic curves, cipher suites, and extensions. If they don't perfectly mimic a real browser, the handshake is rejected. Even if you get past that, Google injects obfuscated JavaScript that renders content asynchronously. If your script doesn't execute that JS, all you’re scraping is an empty shell.
The Foundation of a Resilient Scraper
To build something that actually lasts, stop relying on the visual DOM. Class names in production environments are dynamic and rotate frequently to break fragile CSS selectors.
- Go for the Data Layer: Instead of parsing elements like
div.price-tag, target theapplication/ld+jsonscript blocks. This data is structured, machine-readable, and rarely changes, even when the UI gets a facelift. - Proxy Strategy: Datacenter proxies are a waste of money—they’re flagged by default. You need high-quality residential proxies. Use "sticky sessions" for multi-page crawls to keep your session consistent, but rotate your IPs for broad, parallel searches.
- Geo-targeting: Pricing and shipping are localized. Ensure your proxy gateway supports parameterizing by zip code, or you’ll be scraping incorrect data for your target market.
Architecture for High-Volume Pipelines
Don’t try to do everything in one script. To handle scale:
- Decouple: Use a message broker like Redis or RabbitMQ. Your application should push a "job" to the queue, and a separate worker should handle the scraping.
- Exponential Backoff: If you hit a 429 rate limit, don't just sleep. Implement a wait pattern (5s -> 10s -> 20s). This prevents you from getting your entire proxy pool burned.
- Storage: Forget saving HTML. Normalize your data into a JSON schema (I use Pydantic for validation) and store it in a NoSQL database like MongoDB. This allows your schema to evolve without requiring constant migrations.
Build vs. Buy: The Hidden Cost
Engineering time is your most expensive resource. I’ve seen teams burn $12,000 in dev hours just chasing changing class names before finally switching to a managed service.
If you are building an internal tool for a small project, a custom setup is a great learning experience. But if you’re running a production business, managing proxy rotations, browser fingerprints, and CAPTCHA solvers will eventually eat 70% of your sprint cycle.
For many, leveraging an existing structured search results API is the pragmatic choice. It offloads the "cat and mouse" game of maintenance to specialists, leaving your team free to build actual product features rather than maintaining a scraper that breaks every time a frontend engineer commits a new deploy.
Pro Tip: If you absolutely must build it yourself, always run your scraping logic in an isolated, asynchronous worker thread. Never block your primary web server thread waiting for an HTTP response.
Originally published at How to build a Google Shopping api scraper without blocks
Top comments (1)
I particularly appreciated the section on targeting the
application/ld+jsonscript blocks, as this approach has proven to be more resilient to UI changes in my own scraping projects. The point about leveraging the data layer instead of relying on visual DOM elements is crucial, and I've seen similar success with other search engines. One potential improvement to consider is incorporating a headless browser solution, such as Puppeteer, to more accurately mimic browser behavior and potentially reduce the risk of being flagged by anti-scraping systems. Have you explored this approach in your own work, and if so, what were your findings?