httpx: The Python HTTP Client That Should Have Replaced Requests Years Ago

#python #webdev #tutorial #beginners

Requests Is Showing Its Age

requests is the most popular Python HTTP library. It's also stuck in 2012. No async support. No HTTP/2. No type hints.

httpx is the modern replacement: same simple API, plus async, HTTP/2, and better performance.

Same API, More Features

import httpx

# Synchronous (drop-in replacement for requests)
resp = httpx.get("https://api.openalex.org/works?search=python&per_page=3")
print(resp.json()["results"][0]["title"])

# POST with JSON
resp = httpx.post("https://httpbin.org/post", json={"key": "value"})

# Headers
resp = httpx.get("https://api.example.com/data",
    headers={"Authorization": "Bearer token123"})

# Timeout
resp = httpx.get("https://slow-api.example.com", timeout=30.0)

Async Support (The Killer Feature)

import httpx
import asyncio

async def fetch_papers(queries):
    async with httpx.AsyncClient() as client:
        tasks = [
            client.get("https://api.openalex.org/works",
                       params={"search": q, "per_page": 3})
            for q in queries
        ]
        responses = await asyncio.gather(*tasks)
        for resp in responses:
            papers = resp.json()["results"]
            if papers:
                print(f"Found: {papers[0]['title'][:60]}")

asyncio.run(fetch_papers(["machine learning", "web scraping", "climate change"]))

3 API calls in parallel instead of sequential. For 10 calls, that's 10x faster.

HTTP/2 Support

# HTTP/2 reduces latency for multiple requests to same host
client = httpx.Client(http2=True)
resp = client.get("https://api.example.com/data")
print(resp.http_version)  # "HTTP/2"

Connection Pooling (For Scraping)

import httpx

# Reuse connections — much faster for multiple requests
with httpx.Client(
    headers={"User-Agent": "Mozilla/5.0"},
    follow_redirects=True,
    timeout=10.0
) as client:
    urls = [
        "https://api.openalex.org/works?search=AI",
        "https://api.crossref.org/works?query=AI&rows=3",
        "https://open.er-api.com/v6/latest/USD",
    ]
    for url in urls:
        resp = client.get(url)
        print(f"{resp.status_code} | {url[:50]}")

Real-World: Multi-API Search

import httpx
import asyncio

async def search_all_apis(query):
    async with httpx.AsyncClient(timeout=10.0) as client:
        # Search 3 APIs simultaneously
        openalex, crossref, weather = await asyncio.gather(
            client.get("https://api.openalex.org/works",
                       params={"search": query, "per_page": 3}),
            client.get("https://api.crossref.org/works",
                       params={"query": query, "rows": 3}),
            client.get("https://api.open-meteo.com/v1/forecast",
                       params={"latitude": 40.71, "longitude": -74.01, "current_weather": True}),
        )

        print(f"OpenAlex: {len(openalex.json()['results'])} papers")
        print(f"Crossref: {len(crossref.json()['message']['items'])} papers")
        print(f"Weather: {weather.json()['current_weather']['temperature']}°C")

asyncio.run(search_all_apis("climate change"))

httpx vs requests

Feature	httpx	requests
Sync API	Yes	Yes
Async API	Yes	No
HTTP/2	Yes	No
Type hints	Yes	No
Timeout default	Yes (5s)	No (infinite!)
Follow redirects	Explicit	Always
Streaming	Yes	Yes
File uploads	Yes	Yes

Install

pip install httpx
# For HTTP/2:
pip install httpx[http2]
# For async:
pip install httpx  # Built-in, no extra

Still using requests? Made the switch to httpx? What's your go-to Python HTTP library?

I write about practical Python tools every week. Follow for more.

DEV Community