Introduction
Imagine this: You've writing a Selenium script to automate a website, only to be blocked by detection services like Cloudflare and PerimeterX.
In this Article, I will share 9 Brilliant Tips to prevent Cloudflare and PerimeterX from detecting your Bots.
Within this article, I will also introduce you to Bose Framework, a framework that I created using my bot development skills to help bot developers in creating unstoppable Bots. Think of it like Swiss Army Knife for Bot Development.
I have concentrated all my Knowledge of Bot Development into a Single Easy to use Framework for the benefit of all Bot Developers.
Now, Let us learn the Tips to avoid bot detection from Cloudflare and PerimeterX like a Real Ninja.
Tips
1. User-Agent and Window Size Rotation
Bot Detection systems analyze the user agent and window size to identify bots.
For instance, making an abnormal number of requests (e.g., 1000 page visits) using a single user agent increases the chances of being flagged as a bot.
Also, using made up user agents that are not in usage also increases the chances of being flagged as a bot.
So, it is best to use commonly used User Agents like Chrome 104 and Chrome 106 on a random basis.
In Bose Framework, the following code snippet demonstrates how to achieve random rotation of user agents and window sizes with each run:
from bose import *
class Task(BaseTask):
browser_config = BrowserConfig(user_agent=UserAgent.RANDOM, window_size=WindowSize.RANDOM)
Additionally, if you are utilizing the Bose Framework instead of plain Selenium, the user agent will be automatically rotated on each run by default.
2. Change your IPs
To evade detection, changing your IP address is another crucial aspect. Websites often track IP addresses to identify and block suspicious activity.
If you have been detected by services like Cloudflare or PerimeterX, you will need to change your IP address to continue automation.
There are both free and paid methods to change your IP address:
Free Methods
Enable and disable airplane mode
In this method, you can utilize the connection between your PC and a mobile hotspot to access the internet via a SIM card and then enable and disable airplane mode on your mobile device to get a new IP address.
This method is quite fast and usually takes 10 seconds to acquire a new IP.
Here's how you can enable and disable airplane mode to change your IP address:
Connect your PC to a mobile hotspot.
On your mobile device, turn airplane mode on and off
Now, Turn the Hotspot On Again.
On and Off WIFI Router
For devices connected to a Wi-Fi network, turning off the router and then turning it back on will change the IP address. Although It is slow methods as Wifi Router will take 2-3 minutes to start.
Paid Method
Paid services like Oxylab and IPRoyal provide a pool of residential IP addresses. However, they can be quite expensive, with prices averaging around $15 per GB for residential IPs. If you opt for this method and are scraping large amounts of data, consider disabling the loading of CSS and images to reduce costs.
In general, the free method of using airplane mode works well for most use cases. It is not only free but also faster than using a proxy
3. Humane Page Flow
To avoid suspicion, replicate organic user behavior by engaging with the website as a human would. In general you should follow these recomendation.
Access pages via search engines
Instead of directly visiting a page, navigate to it through a search engine like Google. This emulates how users typically discover websites by searching on a search engine. The Bose Framework provides a special method called organic_get in the Selenium driver, which achieves this behavior. You can use it as follows:
driver.organic_get("https://example.com")
Navigate using internal links
Browsers can detect the page you came from by checking the document.referrer property. When visiting multiple pages within a site, utilize the get_by_current_page_referrer method from the Bose driver. This JavaScript-based method simulates internal navigation by clicking a link, making it appear more natural:
driver.get_by_current_page_referrer("https://example.com")
4. Use Random Waits and Slow the Bot
Adding random wait times and slowing bot between actions helps mimic human interaction patterns and evade bot detection.
The Bose Framework offers short_random_sleep() and long_random_sleep() methods that pause the bot for a random duration.
Here is how you can use them:
driver.short_random_sleep()
driver.long_random_sleep()
5. Use Captcha Solving Services
Services like Cloudflare may present CAPTCHA challenges. To overcome this obstacle, you can use CAPTCHA solving services like 2captcha, which solves the CAPTCHA challenges.
6. Use Ultrafunkamsterdam’s ChromeDriver
Ultrafunkamsterdam's ChromeDriver, developed by Leon from the Netherlands, is an exceptional tool that enhances the stealth of Selenium Driver . It applies patches making it harder for detection services like Cloudflare and PerimeterX to detect the Bot.
The Bose Framework seamlessly integrates with Ultrafunkamsterdam’s ChromeDriver. To use it, specify use_undetected_driver=True in the browser_config:
class Task(BaseTask):
browser_config = BrowserConfig(use_undetected_driver=True)
7. Do not use a Headless Browser
Avoid using headless browsers, as bot detection systems like PerimeterX can detect them by checking for specific APIs.
For example, the following code in JavaScript determines whether the browser is headless. In the case of headless Chrome, the test will log "Headless Chrome"
navigator.permissions
.query({ name: "notifications" })
.then(function (permissionStatus) {
if (
Notification.permission === "denied" &&
permissionStatus.state === "prompt"
) {
console.log("Headless Chrome")
} else {
console.log("Not Headless Chrome")
}
})
8. Maintain Consistency in User Agent, Window Size, and IP Address
If you are maintaining sessions by using a profile in Selenium, then make sure that the User Agent and Window Size remains same for the Profile.
Sudden and frequent changes in these parameters like one time you are using Chrome 106 and then Chrome 98 and then Chrome 102 can raise red flags and lead to detection.
In the Bose Framework, if you specify a profile in a task, it will automatically retain the same user agent and window size on each run using that profile. Here's an example of how to specify a profile in the Boss Task:
class Task(BaseTask):
# The User Agent and Window Size will remain same on each run with profile 1.
browser_config = BrowserConfig(profile=1)
Also, When using a proxy, it is equally important to maintain consistency in the IP address location. Avoid abrupt changes from one country to another, such as switching from Bharat (India) to Russia and then to Israel.
9. Use Bose Framework
Whenever you are developing any Selenium Script, you should definetly use Bose Framework.
Just like a Thunder Stone evolves Pikachu into Raichu in the Pokémon world, the Bose Framework transforms a normal bot developer into a mega bot developer.
Read the Bose Framework Tutorial at https://www.omkar.cloud/bose/docs/tutorial/ to create powerful bots.
Top comments (0)