This actor compares tyre prices across multiple retailers in one run. Give it a tyre size or model, and it returns matching listings from every retailer it covers - price, brand, model, season - normalized into one comparable list, instead of you opening five tabs by hand.
Bug 1: the data was in the analytics code, not the display HTML
One retailer's page showed prices in plain HTML, but the layout mixed several product cards with inconsistent structure - brand, model and price weren't reliably tied together in the visible markup. What was complete and consistent was a small script block meant for Google Analytics: a dataLayer.push event carrying a full JSON array of every product on the page - id, brand, model, price, currency, season - as clean structured data.
It's not meant for scraping, it's meant for ad tracking. But it's exactly the shape a scraper wants. Once I stopped fighting the display HTML and read the analytics blob instead, every field showed up reliably.
Bug 2: a "scoped" parser that was silently reading the wrong element
A second retailer's page has a textbook structure: one schema.org Product/Offer JSON-LD block per result, plus a season caption right next to it. The obvious approach: scope each result card as one element, grab the JSON-LD and the caption from inside it. A test with a single result passed cleanly.
Then a real page with 21 results came back with the season field wrong on results 2 through 21.
The HTML parsing library, recovering from a malformed tag somewhere on the page, had nested all 42 script tags under the FIRST result card instead of about 2 each. My "scoped" selector was reading from the wrong card every time except the first.
The fix wasn't to scope by DOM structure at all - it was to pull two flat lists (all JSON-LD blocks, all season captions) in document order and zip them by position. That sounds fragile, until you remember both lists are guaranteed the same length and the same order by how the page is built server-side, malformed tags or not.
Neither bug threw an exception. Neither showed up in a code review. Both only surfaced by checking the actual result count against what the live page really showed - which is why this project runs its offline tests against fixtures captured from real live responses, not hand-written mocks.
Actor page: https://apify.com/0xgollum/tire-market-pulse
Top comments (2)
The analytics-blob trick is one of the first things I check now before writing a single CSS selector. dataLayer pushes, NEXT_DATA, INITIAL_STATE, anything meant for ads or hydration, usually turns out to be the cleanest structured data on the page, because the site's own code needs it to be correct or their own analytics breaks. Display HTML has no such pressure on it, it just has to look right to a human.
Bug 2 is the sneakier one and I've hit the same shape more than once: a "scoped" selector looks correct on a small sample, then silently drifts once a real page has a malformed tag the parser has to recover from. The zip-by-position fix is the right call specifically because you verified the two lists really are guaranteed the same length by the page's own template, not because it happened to work on this page. Worth a comment right next to that code for whoever debugs it next, since "why not just scope by DOM" is the first question anyone will ask reading it cold.
Good call on documenting the zip-by-position assumption right at the call site — that kind of implicit invariant is exactly what survives a refactor by accident and breaks silently later. I've started adding a one-line assert (len(a) == len(b) or raise) right next to those zips for that reason, doubles as living documentation.