Set Up Proxies with AIOHTTP

#tutorial #proxies #automation

How to Set Up Proxies with AIOHTTP in Python

When making HTTP requests or performing web scraping tasks, managing proxies effectively is key to maintaining access stability and enhancing security. Using proxies helps avoid IP bans and throttling, especially when your scripts hit the same websites repeatedly.

In this article, we'll explore how to set up proxies with AIOHTTP, a popular asynchronous HTTP client/server library in Python. We'll use DataImpulse as our proxy provider, showcasing practical examples on integrating and rotating proxies for scalable, robust scraping or web service tasks.

Getting Started with AIOHTTP

Before diving into proxy setups, ensure you have the following ready:

Python 3.6 or higher installed
Your DataImpulse proxy plan credentials

To install the required packages, run:

pip install aiohttp asyncio

Then, import the essentials in your Python script:

import aiohttp
import asyncio

Basic HTTP Request with AIOHTTP

Here’s a simple example to fetch your public IP address using the https://ip-api.com/ service:

async def get_response():
    async with aiohttp.ClientSession() as session:
        async with session.get('https://ip-api.com/') as response:
            print('Status Code:', response.status)
            print('Response Body:', await response.text())

loop = asyncio.get_event_loop()
loop.run_until_complete(get_response())

This creates a client session, sends a GET request, and prints both the response status and body.

Adding Proxy Support to AIOHTTP Requests

Websites often implement measures against scraping, including IP blocking when repeated requests come from the same address. Proxies help distribute your traffic and prevent such blocks.

Using DataImpulse proxies, you can easily set up proxy integration with AIOHTTP.

Step 1: Define Your Proxy Credentials

Replace the placeholders with your actual proxy server address and credentials:

PROXY_ENDPOINT = 'gw.dataimpulse.com:823'  # Residential proxy server
USERNAME = 'YourProxyPlanUsername'
PASSWORD = 'YourProxyPlanPassword'

Step 2: Make a Request Using a Proxy

Pass your proxy URL with credentials embedded in the URL string:

async def get_response_with_proxy():
    proxy_url = f'http://{USERNAME}:{PASSWORD}@{PROXY_ENDPOINT}'
    async with aiohttp.ClientSession() as session:
        async with session.get('https://ip-api.com/', proxy=proxy_url) as response:
            print('Status Code:', response.status)
            print('Response Body:', await response.text())

loop = asyncio.get_event_loop()
loop.run_until_complete(get_response_with_proxy())

This method authenticates your requests via the proxy server using your DataImpulse credentials.

Proxy Authentication Using BasicAuth

Alternatively, AIOHTTP supports passing credentials by using the BasicAuth helper instead of embedding them in the URL string:

from aiohttp import BasicAuth

async def get_response_with_basic_auth():
    async with aiohttp.ClientSession() as session:
        async with session.get(
            'https://ip-api.com/',
            proxy=f'http://{PROXY_ENDPOINT}',
            proxy_auth=BasicAuth(USERNAME, PASSWORD)
        ) as response:
            print('Status Code:', response.status)
            print('Response Body:', await response.text())

loop = asyncio.get_event_loop()
loop.run_until_complete(get_response_with_basic_auth())

This is a cleaner and sometimes more secure way to handle proxy authentication.

How to Rotate Proxies with AIOHTTP

Repeatedly using the same proxy can still get your IP blocked by websites. Rotating proxies is essential to maintain uninterrupted scraping or requesting.

Method 1: Random Proxy Selection

Maintain a list of proxy URLs and randomly pick one for each request:

import random

proxy_list = [
    'http://USERNAME:PASSWORD@PROXY_ADDRESS_1:10000',
    'http://USERNAME:PASSWORD@PROXY_ADDRESS_2:10001',
    # Add more proxies as needed
]

async def get_response_random_proxy():
    proxy = random.choice(proxy_list)
    async with aiohttp.ClientSession() as session:
        async with session.get('https://ip-api.com/', proxy=proxy) as response:
            print('Status Code:', response.status)
            print('Response Body:', await response.text())

loop = asyncio.get_event_loop()
loop.run_until_complete(get_response_random_proxy())

This method is simple but may randomly reuse the same proxy multiple times.

Method 2: Round-Robin Proxy Rotation

To ensure more even usage across proxies, cycle through the list sequentially:

proxy_list = [
    'http://USERNAME:PASSWORD@PROXY_ADDRESS_1:10000',
    'http://USERNAME:PASSWORD@PROXY_ADDRESS_2:10001',
    # Add more proxies
]

async def get_response_with_proxy(target_url, proxy):
    async with aiohttp.ClientSession() as session:
        async with session.get(target_url, proxy=proxy) as response:
            print('Status Code:', response.status)
            print('Response Body:', await response.text())

number_of_requests = 10
proxy_count = len(proxy_list)

for i in range(number_of_requests):
    proxy = proxy_list[i % proxy_count]
    loop = asyncio.get_event_loop()
    loop.run_until_complete(get_response_with_proxy('https://ip-api.com/', proxy))

This approach guarantees that proxies are used one by one in a circular manner.

Reusing Proxies Until Failure

Sometimes it makes sense to keep using a proxy until it gets blocked, then switch to the next one.

proxy_list = [
    'http://USERNAME:PASSWORD@PROXY_ADDRESS_1:10000',
    'http://USERNAME:PASSWORD@PROXY_ADDRESS_2:10001',
    # Add more proxies
]

async def get_response_with_proxy(target_url, proxy):
    async with aiohttp.ClientSession() as session:
        async with session.get(target_url, proxy=proxy) as response:
            print('Status Code:', response.status)
            print('Response Body:', await response.text())
            return response.status

proxy_index = 0
number_of_requests = 10
proxy_count = len(proxy_list)

for _ in range(number_of_requests):
    loop = asyncio.get_event_loop()
    status = loop.run_until_complete(get_response_with_proxy('https://ip-api.com/', proxy_list[proxy_index]))
    if status != 200:
        proxy_index = (proxy_index + 1) % proxy_count
    # Continue using the same proxy if status is 200

This logic closely monitors HTTP status codes, switching proxies only when necessary.

Final Thoughts

Setting up proxies with AIOHTTP is straightforward and crucial for resilient web scraping or asynchronous web services. Using DataImpulse residential proxies allows you to manage multiple IPs securely and cost-effectively.

With support for proxy authentication and flexible rotation strategies, you can build scalable and reliable applications that avoid IP bans and maintain continuous access to your target sites.