Web scraping is a powerful tool for data collection, but let's be honest: anti-bot technology is no joke. It's constantly evolving, with many websites that were once easy targets now employing sophisticated defences. The rise of AI scraping bots means many vendors are outright blocking by default, even without direct intervention from the website owner.
In this post, I want to share my top advice on how to manage these bans, drawing insights from a recent webinar I conducted with two of Zyte's leading anti-ban team members, who collectively boast decades of experience in this challenging field. My goal is to provide actionable tips and illustrate how we approach these challenges at scale.
We'll dive into two key areas:
- Initial steps you need to take when planning your scraping strategy for a site.
- How to implement a better crawling strategy to operate under the radar.
It's this combination of strategic thinking and the right tooling/infrastructure that will enable successful, scalable scraping. Overcoming bans is only useful if you can manage it at scale.
Fingerprinting and Session Usage
To effectively navigate a website's defenses, we must first understand the measures a modern, multi-layer anti-bot solution employs to block access or present a challenge. Put yourself in the shoes of an anti-bot engineer: how would you try to differentiate real human traffic from bot traffic?
This largely breaks down into two core areas: fingerprinting and session usage.
1. Your Digital ID Card
Your "fingerprint" is a unique identifier that the website sees as representing your specific user. A wealth of information can be gathered by running JavaScript on your browser – and similarly, if you're not running a browser at all. There's a lot to consider here, given the sheer number of data points that can be collected from your machine through this method.
More obvious data points include your user agent and operating system, but it goes much deeper:
- Your laptop’s battery life
- Installed fonts
- Your video card
- Even your mouse movements!
That's right, mouse movements play a significant role. When automating a browser, the mouse often jumps between points in a manner impossible for a human, and this data can be used to serve a challenge, effectively blocking your access.
All this data is compiled into a fingerprint to identify you. This is often where newcomers get confused, as we're led to believe that rotating proxies alone is a surefire way around bans. While proxies are crucial, they're not the full picture. This fingerprint can also be used to rate-limit or challenge you, regardless of your IP address.
Speaking of IPs, it's important to think of an IP address as having a quality score associated with it. Rather than just its location or type (though these play a role), it's the quality that will grant you access. At Zyte, we use a mix of proxy IPs through our Zyte API to gain access and optimize costs.
So why is this fingerprint so critical? Consider a common example: scraping data using Selenium or Playwright. Without any modification, both of these browser automation systems have a very unique and distinctive fingerprint, allowing for instant blocking if any requests are made with them. They can be patched, but it's a constant cat-and-mouse game to stay a step ahead.
Another example that often trips people up is the discrepancy between your IP's timezone and your browser's locale. This is especially relevant when rotating different proxies globally within the same browser configuration.
To recap: Your first goal is to maintain a constant and consistent fingerprint, as a real user would. Be mindful of what data is exposed and how you manage it. The objective is to create a session within the website that genuinely appears as if a real human user were operating it.
2. Behaviour and Session Management
The second key point is understanding what anti-bot systems a site has, how it reacts to certain triggers, and building a clear picture of what your crawl strategy should look like.
For instance, common anti-bot systems often issue a session cookie stored locally once the initial checks have been passed. This session cookie is your "ticket" that you need to manage. In some cases, this is all that's required. However, in others, something as seemingly simple as changing the IP address mid-session could trigger a challenge, as it might be considered unusual user behavior.
drop the browser altogether
For some sites, once this session is obtained, you might even be able to drop the browser altogether, switch to a cheaper proxy, and start making requests directly. You'll need to continuously monitor and understand how the site responds, and adapt your actions accordingly.
Also, critically, take into consideration how a real user would browse the site. It's less likely they would go straight to a page deep within the site (though not impossible). It's highly unlikely they would navigate page by page, product by product, in a perfectly linear order.
Your goal is to keep your Browse natural, while maintaining good sessions with consistent fingerprints, and crawling the site in a logical manner. Even visiting the home page first, before navigating to your desired deep pages, can make a difference in some cases.
The Role of AI in Anti-Bot & Scraping
There's one more interesting point I want to mention: the role of AI. Anti-bot vendors have been leveraging machine learning for years to spot patterns in fingerprinting and behavior, but with the advancements in AI, it's now even better at detecting anomalies, much like the ones I mentioned earlier. On the flip side, AI is also proving incredibly useful for deobfuscating the vast amounts of JavaScript on the web that aren't meant to be easily read. These are higher-level considerations, but useful to be aware of.
The Challenge and the Solution
The reality is that staying ahead of modern anti-bot technology is a massive undertaking. Depending on the data required, this level of effort could be out of reach for many businesses. And that's precisely where services like Zyte can help.
The Zyte API handles everything I've just discussed for you. You don't need to worry about IP quality, session management, or maintaining a consistent fingerprint. You simply send a request to our API, and you receive the raw HTML back. We're exceptionally good at this.
Want to see it in action? Sign up for free credit, and let us show you how seamless web scraping can be.
Top comments (0)