Anyone who has written a scraper knows the feeling. It works perfectly for three weeks, then one morning it returns nothing because the site renamed a CSS class or moved a div.
These are the habits I have picked up so far:
-
Look for a JSON source first. A lot of sites ship their data as embedded JSON (Next.js
__NEXT_DATA__, JSON-LD, or an internal API the page calls). That changes far less often than the HTML around it. - Fail loudly. An empty result should be treated as an error, not a success. A scraper that quietly returns zero rows is worse than one that crashes.
- Run a tiny daily test. One known input with a check that the output still has the fields I expect. It catches breakage before anyone else does.
-
Avoid deep selectors.
div > div:nth-child(3) > spanbreaks the moment anything shifts. Text labels and data attributes hold up better.
I am curious what everyone else does:
- What is your go-to trick for making a scraper last?
- Do you monitor them somehow, or just wait until something looks wrong?
- Has anyone had good results letting an LLM parse the page instead of writing selectors? I keep going back and forth on whether it is worth the cost and the occasional made-up field.
Would love to hear what has worked (and what has burned you).
Top comments (1)
On monitoring: I alert on field-level counts, not just HTTP 200 — if a listing page that normally returns 20 items suddenly returns 0 or 2, that's the failure, even though the request succeeded. On the LLM question: I've had decent results using it once to write the extractor from a sample page, then running the deterministic extractor in production. Letting it parse every page live was too slow, and it occasionally invented a field that wasn't there.