DEV Community

Mohammad Waseem
Mohammad Waseem

Posted on

Overcoming IP Bans During Web Scraping in Legacy Python Codebases

#python #webscraping #legacy

Web scraping presents a persistent challenge when it comes to avoiding IP bans, especially in legacy Python projects lacking modern anti-blocking strategies. As a Lead QA Engineer, I’ve faced numerous scenarios where our scraping efforts were thwarted by IP blocking mechanisms implemented by target websites. This article outlines effective techniques and practical code snippets to mitigate IP bans, focusing on legacy codebases that may not be initially designed for such complexities.

Understanding the Root Cause
At the core, IP bans are typically triggered when a server detects unusual activity from an IP address or identifies patterns associated with bots. Classic legacies codes often rely on straightforward request loops without adaptive measures like rate limiting or IP rotation, making them vulnerable.

Implementing IP Rotation with Proxy Pools
The most robust solution involves rotating IP addresses to distribute requests across different identities. Proxy pools can be local proxies or third-party services. Here's an example of integrating a rotating proxy mechanism in a legacy code:

import requests
import itertools

# List of proxies in your pool
proxies = [
    'http://proxy1:port',
    'http://proxy2:port',
    'http://proxy3:port'
]
proxy_pool = itertools.cycle(proxies)

def get_html(url):
    proxy = next(proxy_pool)
    try:
        response = requests.get(url, proxies={'http': proxy, 'https': proxy}, timeout=10)
        response.raise_for_status()
        return response.text
    except requests.RequestException as e:
        print(f"Request failed with proxy {proxy}: {e}")
        return None
# Usage example
url = 'https://example.com/data'
html = get_html(url)
Enter fullscreen mode Exit fullscreen mode

This code cycles through a list of proxies, ensuring each request is sent from a different IP.

Incorporating Randomized Headers & Delays
Bots are also detected via request headers and request frequency. To simulate human-like behavior, randomize your headers and introduce delays:

import random
import time

headers_list = [
    {'User-Agent': 'Mozilla/5.0 ...'},
    {'User-Agent': 'Chrome/89.0 ...'},
    {'User-Agent': 'Safari/537.36 ...'}
]

def make_request(url):
    headers = random.choice(headers_list)
    wait_time = random.uniform(1, 3)  # Random delay between 1-3 seconds
    time.sleep(wait_time)
    try:
        response = requests.get(url, headers=headers, timeout=10)
        response.raise_for_status()
        return response.text
    except requests.RequestException as e:
        print(f"Request error: {e}")
        return None
Enter fullscreen mode Exit fullscreen mode

This technique helps mimic human browsing patterns, reducing the likelihood of triggering bans.

Rotating User Agents & Session Management
Further enhance stealth by dynamically changing user-agents and managing sessions:

session = requests.Session()
session.headers.update({'User-Agent': 'Mozilla/5.0 ...'})

for _ in range(10):
    session.headers.update({'User-Agent': random.choice(headers_list)['User-Agent']})
    html = make_request_with_session(session, url)
    # process html
Enter fullscreen mode Exit fullscreen mode

Handling Legacy Constraints
Legacy codebases often have limitations such as blocking requests on failure or restricted threading capabilities. To maximize efficiency without compromising stealth:

  • Implement exponential backoff retries.
  • Use asynchronous requests if possible (e.g., with aiohttp).
  • Ensure your request routines are modular to integrate new obfuscation features easily.

Conclusion
Dealing with IP bans requires a multi-faceted approach. Combining IP rotation, header randomization, intelligent delays, and session management significantly improves scraping resilience in legacy environments. Always remember, respect target website policies and incorporate appropriate crawl-delay and robots.txt adherence to maintain ethical standards.

By implementing these strategies, QA and development teams can extend the longevity and reliability of their web scraping operations, even within legacy codebases.

🛠️ QA Tip

To test this safely without using real user data, I use TempoMail USA.

Top comments (0)