How to scrape Amazon: Python, the law and official APIs

Originally published on proxymint.

Scrape public Amazon product and price data with Python: what the terms and courts say, the official API routes, polite rate limits and where proxies fit.

TL;DR

• There are three routes to Amazon data. The official APIs if you qualify (Creators API for Associates, SP-API for sellers), your own scraper on public logged-out pages, or a bought dataset.
• Amazon's terms prohibit scraping. The Conditions of Use exclude any collection of product listings or prices and any use of robots or data mining tools, and robots.txt bars 99 named agents, Scrapy included, from the whole site.
• Courts separate access from contract. US rulings since 2021 treat logged-out collection of public pages as mostly outside computer-crime law, but contract, database right and the GDPR still apply.
• The code should stop, not evade. A 503, a robot-check page or a sign-in redirect means log it, back off and end the run after a short streak.
• Most small jobs need no proxy. A few hundred ASINs a day at a slow pace can run from one IP; residential exits fit other countries' marketplaces, and volume past one polite exit belongs on the official APIs or a bought dataset.

Key takeaways

• Check the official APIs before building anything. Associates with 10 qualifying sales in 30 days can use the Creators API, and sellers get pricing data from SP-API.
• Stay on public, logged-out pages. Product pages and search pages only: no cart, no sign-in, no reviews behind the login wall and no reviewer names.
• Set your own slow pace and log every fetch. One request every few seconds per exit with jitter, one at a time, with timestamp, marketplace and status on every row.
• Match marketplace, language and exit country. Prices and currency follow the domain and the request's location, so all three have to agree.
• Measure MB per page before buying traffic. GB needed is pages times MB per page divided by 1,024, and HTML-only fetches are the biggest cost lever.

Read the full guide on proxymint, with the commands, tables and sources.