I built an actor that compares tyre prices across multiple retailers in one run. Two of them needed very different parsing strategies than I expected - and neither reason was a blocked request or an anti-bot wall.
Bug 1: the data was in the analytics code, not the display HTML
One retailer's page showed prices in plain HTML, but the layout mixed several product cards with inconsistent structure - brand, model and price weren't reliably tied together in the visible markup. What was complete and consistent was a small script block meant for Google Analytics: a dataLayer.push event carrying a full JSON array of every product on the page - id, brand, model, price, currency, season - as clean structured data.
It's not meant for scraping, it's meant for ad tracking. But it's exactly the shape a scraper wants. Once I stopped fighting the display HTML and read the analytics blob instead, every field showed up reliably.
Bug 2: a "scoped" parser that was silently reading the wrong element
A second retailer's page has a textbook structure: one schema.org Product/Offer JSON-LD block per result, plus a season caption right next to it. The obvious approach: scope each result card as one element, grab the JSON-LD and the caption from inside it. A test with a single result passed cleanly.
Then a real page with 21 results came back with the season field wrong on results 2 through 21.
The HTML parsing library, recovering from a malformed tag somewhere on the page, had nested all 42 script tags under the FIRST result card instead of about 2 each. My "scoped" selector was reading from the wrong card every time except the first.
The fix wasn't to scope by DOM structure at all - it was to pull two flat lists (all JSON-LD blocks, all season captions) in document order and zip them by position. That sounds fragile, until you remember both lists are guaranteed the same length and the same order by how the page is built server-side, malformed tags or not.
Neither bug threw an exception. Neither showed up in a code review. Both only surfaced by checking the actual result count against what the live page really showed - which is why this project runs its offline tests against fixtures captured from real live responses, not hand-written mocks.
Actor page: https://apify.com/0xgollum/tire-market-pulse
Top comments (0)