DEV Community

Cover image for HTTPX: Dump requests library in a junkyard 🚀
Alexey
Alexey

Posted on

1

HTTPX: Dump requests library in a junkyard 🚀

The most common mistake in HTTP requests in Python is still using the requests library. The httpx library is a more advanced, versatile, and modern HTTP client for Python, supporting both synchronous and asynchronous workflows. It's easy to transition from the outdated requests library, which lacks critical features like native async support.

Install httpx:

pip install httpx
Enter fullscreen mode Exit fullscreen mode

Async/Sync

httpx supports both synchronous and asynchronous clients, offering flexibility for various project environments.

  • requests is sync-only, making it unsuitable for high-performance applications.
  • aiohttp is async-only, which lacks simplicity for synchronous needs.

Synchronous Example

import httpx

# Synchronous GET request
response = httpx.get('https://api.example.com/data')
print(response.json())  # Parse JSON data

# Using a persistent Client for better performance
with httpx.Client(base_url='https://api.example.com') as client:
    response = client.get('/data')
    print(response.json())
Enter fullscreen mode Exit fullscreen mode

Asynchronous Example

import httpx
import asyncio

async def fetch_data():
    # Asynchronous GET request
    async with httpx.AsyncClient(base_url='https://api.example.com') as client:
        response = await client.get('/data')
        print(response.json())

# Run the async function
asyncio.run(fetch_data())
Enter fullscreen mode Exit fullscreen mode

Similar to requests

For those familiar with requests, transitioning to httpx is straightforward, as both libraries share similar APIs.

With requests:

import requests

response = requests.get('https://api.example.com/data')
data = response.json()
Enter fullscreen mode Exit fullscreen mode

With httpx:

import httpx

response = httpx.get('https://api.example.com/data')
data = response.json()
Enter fullscreen mode Exit fullscreen mode

Both libraries share similar method names (get, post, etc.), making migration easy.

Client Management

The concept of a Client in httpx is analogous to a Session in requests. However, httpx.Client is more powerful and efficient. You can read the article from the httpx documentation, to learn more about httpx.Client.

Features

  • Performance: Clients reuse underlying connections, making multiple requests faster.
  • Fine-grained control: Manage headers, cookies, timeouts, and other settings.
  • Asynchronous support: Async clients (httpx.AsyncClient) are optimized for modern async workflows.

Perfomance Explanation

When you make a simple request, like the following:

import httpx

response = httpx.get('https://example-api.com', params={'page': 1})
print(response.json())
Enter fullscreen mode Exit fullscreen mode

httpx follows this algorithm:

Diagram

If you make dozens of requests, like this code:

import httpx

urls = [...]
params_list = [{...}, ...]

for url, params in zip(urls, params_list):
    response = httpx.get(url, params=params)
    print(response.json())
Enter fullscreen mode Exit fullscreen mode

httpx will open the client for each request. It slows your application. The better solution is to use a single client instance. You can close it whenever you want.

import httpx

urls = [...]
params_list = [{...}, ...]

with httpx.Client() as client:
    for url, params in zip(urls, params_list):
        response = client.get(url, params=params)
        print(response.json())
Enter fullscreen mode Exit fullscreen mode

Sensei Integration

The sensei Python framework uses httpx as its HTTP client for creating efficient, robust, and flexible API Clients (or API Wrappers). You can inject your own client into each request and close it whenever you want, to achieve the best performance:

from sensei import Manager, Router, Client

manager = Manager()
router = Router('httpx://example-api.com', manager=manager)

@router.get('/users/{id_}')
def get_user(id_: int) -> User:
    pass

with Client(base_url=router.base_url) as client:
    manager.set(client)

    for i in range(10):
        user = get_user(i)
        print(user)

    manager.pop()
Enter fullscreen mode Exit fullscreen mode

Visit sensei repository to read more: https://github.com/CrocoFactory/sensei

Documentation

httpx provides rich, well-organized documentation site with examples and guides, making learning and troubleshooting easier and code docstrings explaining every function and parameter.


With these advantages and code examples, it's clear why httpx is the modern choice for HTTP requests in Python. Replace your requests code with httpx today and enjoy the benefits of a cutting-edge library! 🚀

API Trace View

Struggling with slow API calls?

Dan Mindru walks through how he used Sentry's new Trace View feature to shave off 22.3 seconds from an API call.

Get a practical walkthrough of how to identify bottlenecks, split tasks into multiple parallel tasks, identify slow AI model calls, and more.

Read more →

Top comments (0)

Heroku

Build apps, not infrastructure.

Dealing with servers, hardware, and infrastructure can take up your valuable time. Discover the benefits of Heroku, the PaaS of choice for developers since 2007.

Visit Site

đź‘‹ Kindness is contagious

Immerse yourself in a wealth of knowledge with this piece, supported by the inclusive DEV Community—every developer, no matter where they are in their journey, is invited to contribute to our collective wisdom.

A simple “thank you” goes a long way—express your gratitude below in the comments!

Gathering insights enriches our journey on DEV and fortifies our community ties. Did you find this article valuable? Taking a moment to thank the author can have a significant impact.

Okay