Welcome Oriol. Curious what's holding up best on the FlybyApis stack, the anti-scraping defenses on Amazon and Google Maps have gotten brutal lately. Which source ended up being the most painful to keep stable?
Thanks Valentin! Honestly, none of them are easy, each source comes with its own set of challenges. From HTML structures changing, to managing proxies properly, dealing with rate limits, CAPTCHAs,...
Yeah the rate limit and CAPTCHA arms race is the part that goes non-linear fast, one source toughening up can blow the pipeline budget. The trap I see most teams hit on Amazon/Maps is silent selector changes that return half-empty rows for weeks before anyone notices. Do you alert on row count deltas or just on parse errors?
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
Welcome Oriol. Curious what's holding up best on the FlybyApis stack, the anti-scraping defenses on Amazon and Google Maps have gotten brutal lately. Which source ended up being the most painful to keep stable?
Thanks Valentin! Honestly, none of them are easy, each source comes with its own set of challenges. From HTML structures changing, to managing proxies properly, dealing with rate limits, CAPTCHAs,...
Yeah the rate limit and CAPTCHA arms race is the part that goes non-linear fast, one source toughening up can blow the pipeline budget. The trap I see most teams hit on Amazon/Maps is silent selector changes that return half-empty rows for weeks before anyone notices. Do you alert on row count deltas or just on parse errors?