Key Takeaways
- Running a Zalando and ASOS scraper requires handling an advanced anti-bot system and handling dynamic page content.
- Best practices include ensuring session consistency, reverse-engineering internal and mobile APIs, and selecting high-quality proxies.
- With complex targets like retail websites, hiring a professional scraping company can help get results faster.
Your ASOS scraper may work one day and return a 403 error the next. Big fashion retailers invest heavily in anti-bot protections and update them on a regular basis. If you're building a data-driven product or analytics system, prolonged pipeline downtime can hinder the accuracy of your insights.
In this article, we explain the challenges your retail scraper faces and show how to overcome them when running large-scale collection.
The Main Challenges of Running a Zalando Scraper
Automated data collection from platforms like ASOS and Zalando is often difficult to maintain. Below is a list of the most common issues to help you assess the risks before you start scraping and troubleshoot when something goes wrong.
1. Legal challenges of scraping in Europe
Although collecting publicly available data is generally legal across the world, the particular laws differ depending on use cases, data types, and collection methods. While non-EU European countries often have less strict approaches, the European Union is one of the most regulated markets for web scraping.
Before building a Zalando scraper that covers Germany and other EU states, we recommend consulting a lawyer or doing your own research. Here's a brief overview of regulations to consider.
- Personal data. Names, face photos, and phone numbers are protected by the GDPR. Collecting them requires explicit consent or a documented legitimate interest. When scraping reviews, make sure you do not gather or store any personal or identifiable reviewer details.
- Sources protected by a website's Terms of Service. The EU courts view ToS as a form of a legally binding contract. In Ryanair v. PR Aviation, the price comparison platform was ruled to stop scraping Ryanair's website in violation of its ToS. Note that both ASOS and Zalando prohibit "screen scraping" in their Terms & Conditions.
2. Akamai Bot Manager
If you're clear legally, the next obstacle is websites' anti-bot protections. Both ASOS and Zalando use Akamai Bot Manager, a system that passes each request through an edge server network where detection happens. It assesses traffic using a set of parameters:
- IP and ASN reputation. Akamai owns a dedicated database and records the history of each address's activities across all websites that use Akamai's services.
- TLS and HTTP fingerprints. All the session's signals need to be consistent.
- Device and browser telemetry. The system uses JavaScript Challenges to assess additional signals.
- Behavioral signals. Akamai uses AI to analyze mouse movements and typing patterns and flag artificial sessions.
The system is maintained and regularly updated. Even if you've managed to bypass it once, it doesn't guarantee continuous uptime.
3. Dynamic content
Zalando uses client-side JavaScript rendering for all the product data. That means a page's HTML is empty before a browser loads the dynamic content. If your scraper relies on HTTP requests, it won't return any data.
For ASOS, dynamic rendering is partial. As a result, a BeautifulSoup-based scraper can appear functioning until you try to use the collected data. You can get the product title and image but miss the price, as happened in one Stack Overflow question. For reliable results, use JS rendering or pull data from another source.
Best Practices for Building an ASOS Scraper
Following the right practices from the start can help keep your data pipeline running as your needs grow. Below are a few essential steps to follow, along with additional alternative ways to get fashion data at scale.
1. Select the right proxies
With advanced protection systems like Akamai, cheap datacenter proxies are blocked before a scraper can load a single page. The same can be true for residential or mobile IPs that have already been used to scrape Akamai-protected websites without taking the right precautions.
Choose a reputable proxy provider, read reviews, and try the addresses on a small run before committing.
2. Manage geo-specific signals
Zalando has separate versions for different European countries. When accessing each one, make sure your IP and browser signals match the particular location. Foreign proxies are not blocked automatically, but inconsistent geographic and language markers within a single session can trigger a CAPTCHA or make the system watch you more closely.
3. Reverse engineer internal APIs
Fashion retailers host millions of items. If your analysis needs to cover a large portion of the data, scraping it page-by-page will take a lot of time and resources. In addition, large scale makes it harder to imitate human sessions. If you send too many requests without adding natural pauses and rotating user agents, you'll get blocked. Finding an API is a popular alternative.
Retail websites don't have official public APIs third-party developers could use, but they still use API connections for internal data transfer. Finding the endpoints can help pull data without crawling. Here's a quick guide:
- Open your browser's Developer Tools on a page containing the data you need.
- Inspect network requests and identify the relevant one.
- Copy it as cURL to reproduce the request.
- Test it outside the browser, for example, in a terminal.
- Determine which parameters are required for the request to work consistently by removing them one by one.
- Recreate the request in your scraper.
This is not a universal solution, as the internal API are also often protected by Akamai or other tools. In addition, they can change at any moment, requiring continuous monitoring and maintenance.
4. Try scraping mobile apps
When building an ASOS scraper, app version of the platform can offer another internal API to work with. A reverse-engineered mobile API can deliver clean, structured JSON that is easier to integrate than raw scraped HTML. As protections evolve, one target can become easier than another. Run tests before investing in a full-scale pipeline.
5. Consider professional services
In-house development doesn't make your ASOS scraper free. Even if you have a team necessary to build a reliable pipeline, the costs add up. The scraping market offers ready-to-use solutions as well as custom services that can deliver your scraper fast. The best option for you depends on your use case.
ASOS Scraper Review: Pre-Built Tool vs. Custom Web Scraping Services
Since retail is a common target for data collection, finding a pre-made Zalando scraper is not hard. However, self-serve tools don't fit all projects. Below is a brief overview of their pros and cons compared to custom services.
| Scraping option | Benefits | Downsides |
|---|---|---|
| Custom services | Built around your needs; diverse integration options; transparent costs per-project; dedicated maintenance team; zero tech skills required from you | Delayed start due to development; larger upfront investment |
| Pre-built tool | Same-day data delivery; low costs for small projects; maintenance handled by the provider | Built for generic use cases; limited customization; unpredictable price at scale |
One of the main differences is how much work your team is willing to handle. Even if you find a no-code solution that claims to work straight out of the box, you still have to set it up and process the output. With custom services, you can choose any level of involvement: from data file delivery to end-to-end development where every stage is handled for you.
Before buying an ASOS scraper, review the integration and scalability options. We recommend taking a long-term approach. Pre-built tools work for one-time competitor assessment or small scheduled scraping. If the data needs to be part of your workflow or product, custom development can offer more.
This article is for informational and educational purposes only. Before collecting any data from ASOS, Zalando, or any other platform, consult a qualified legal professional to ensure compliance with the platform's terms of service.
Top comments (0)