Mastering the "requests" Library in Python

The requests library is probably the most commonly used library in Python, having been used in countless applications to retrieve data from an API, scrape a website, or send data to a server. It's one of the most downloaded Python packages of all time and for good reason. In this article, you will learn all the important things to know to use it with confidence.

What is requests?

requests is a third-party Python library that makes the sending of HTTP requests super easy. There is a standard library module called urllib in Python, but it's a bit cumbersome to use and has a lot of words. requests will do everything that urllib can do but put all of that into a user-friendly but clean API.

According to its own tagline, the library is HTTP for Humans.

Installation

pip install requests

That's it. Once installed, import it in your script:

import requests

Making Your First Request

GET Request

The most common HTTP method. Use it to retrieve data.

import requests

response = requests.get("https://jsonplaceholder.typicode.com/posts/1")

print(response.status_code)  # 200
print(response.json())       # Parsed JSON response

Response Object

Every request returns a Response object packed with useful attributes:

response = requests.get("https://jsonplaceholder.typicode.com/posts/1")

print(response.status_code)   # HTTP status code (200, 404, 500, etc.)
print(response.text)          # Response body as a string
print(response.json())        # Response body parsed as JSON (dict/list)
print(response.headers)       # Response headers (dict-like)
print(response.url)           # Final URL after redirects
print(response.elapsed)       # Time taken for the request

HTTP Methods

requests supports all standard HTTP methods:

# GET — Retrieve data
requests.get(url)

# POST — Send data to create a resource
requests.post(url, json={"name": "Alice"})

# PUT — Update an entire resource
requests.put(url, json={"name": "Alice Updated"})

# PATCH — Partially update a resource
requests.patch(url, json={"name": "Alice Patched"})

# DELETE — Remove a resource
requests.delete(url)

# HEAD — Like GET, but only fetches headers
requests.head(url)

# OPTIONS — Ask what methods a URL supports
requests.options(url)

Query Parameters

Instead of manually building query strings, pass a params dict:

params = {
    "userId": 1,
    "completed": True
}

response = requests.get(
    "https://jsonplaceholder.typicode.com/todos",
    params=params
)

print(response.url)
# https://jsonplaceholder.typicode.com/todos?userId=1&completed=True

Sending Data

JSON Body (most common for APIs)

payload = {
    "title": "My Post",
    "body": "Hello, world!",
    "userId": 1
}

response = requests.post(
    "https://jsonplaceholder.typicode.com/posts",
    json=payload  # Automatically sets Content-Type: application/json
)

print(response.status_code)  # 201
print(response.json())

Form Data

response = requests.post(
    "https://httpbin.org/post",
    data={"username": "alice", "password": "secret"}
)

Uploading Files

with open("document.pdf", "rb") as f:
    response = requests.post(
        "https://httpbin.org/post",
        files={"file": f}
    )

Custom Headers

Many APIs will require headers, such as authentication tokens, content types, or custom metadata:

headers = {
    "Authorization": "Bearer YOUR_TOKEN_HERE",
    "Accept": "application/json",
    "X-Custom-Header": "my-value"
}

response = requests.get("https://api.example.com/data", headers=headers)

Authentication

Basic Auth

from requests.auth import HTTPBasicAuth

response = requests.get(
    "https://httpbin.org/basic-auth/user/pass",
    auth=HTTPBasicAuth("user", "pass")
)
# Shorthand:
response = requests.get(url, auth=("user", "pass"))

Token / Bearer Auth

headers = {"Authorization": "Bearer my-secret-token"}
response = requests.get("https://api.example.com/protected", headers=headers)

API Key (as query param)

response = requests.get(
    "https://api.example.com/data",
    params={"api_key": "your-api-key"}
)

Handling Timeouts

Always set a timeout:

try:
    response = requests.get("https://httpbin.org/delay/5", timeout=3)
except requests.exceptions.Timeout:
    print("The request timed out!")

The timeout parameter accepts:

• A single number (applies to both connect and read)
• A tuple (connect_timeout, read_timeout)

response = requests.get(url, timeout=(2, 10))  # 2s to connect, 10s to read

Error Handling

Checking Status Codes

response = requests.get("https://httpbin.org/status/404")

if response.status_code == 200:
    print("Success!")
elif response.status_code == 404:
    print("Not found.")
elif response.status_code == 500:
    print("Server error.")

raise_for_status()

A cleaner approach — raises an HTTPError for 4xx and 5xx responses:

try:
    response = requests.get("https://httpbin.org/status/500")
    response.raise_for_status()
except requests.exceptions.HTTPError as e:
    print(f"HTTP error: {e}")

Catching All Request Exceptions

try:
    response = requests.get("https://nonexistent.example.com", timeout=5)
    response.raise_for_status()
    data = response.json()
except requests.exceptions.ConnectionError:
    print("Failed to connect.")
except requests.exceptions.Timeout:
    print("Request timed out.")
except requests.exceptions.HTTPError as e:
    print(f"HTTP error: {e}")
except requests.exceptions.RequestException as e:
    print(f"Something went wrong: {e}")

RequestException is the base class for all requests exceptions, a useful catch-all.

Sessions

For multiple requests to the same host use a Session. It can reuse the underlying tcp connection (higher performance) and allows to configure persistent headers, cookies or auth: (better performance)

with requests.Session() as session:
    session.headers.update({"Authorization": "Bearer my-token"})

    response1 = session.get("https://api.example.com/users")
    response2 = session.get("https://api.example.com/posts")
    response3 = session.post("https://api.example.com/posts", json={"title": "New"})

Working with Cookies

# Send cookies
cookies = {"session_id": "abc123"}
response = requests.get("https://httpbin.org/cookies", cookies=cookies)

# Read cookies from response
print(response.cookies["my-cookie"])

Redirects

By default, requests follows redirects automatically. To disable this:

response = requests.get("https://httpbin.org/redirect/3", allow_redirects=False)
print(response.status_code)  # 302

SSL Verification

By default, SSL certificates are verified to skip verification though not recommended in production:

response = requests.get("https://self-signed.badssl.com/", verify=False)

Streaming Large Responses

For large files, stream the response instead of loading it all into memory:

with requests.get("https://example.com/large-file.zip", stream=True) as response:
    response.raise_for_status()
    with open("large-file.zip", "wb") as f:
        for chunk in response.iter_content(chunk_size=8192):
            f.write(chunk)

Putting It All Together: A Real-World Example

Here's a reusable API client using sessions and proper error handling and it is a production-ready system to fetch specific blog posts from a public API.

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry


def create_session(retries=3, backoff_factor=0.5):
    """Create a session with automatic retries."""
    session = requests.Session()
    retry = Retry(
        total=retries,
        backoff_factor=backoff_factor,
        status_forcelist=[429, 500, 502, 503, 504]
    )
    adapter = HTTPAdapter(max_retries=retry)
    session.mount("https://", adapter)
    session.mount("http://", adapter)
    return session


def fetch_posts(user_id: int) -> list[dict]:
    base_url = "https://jsonplaceholder.typicode.com"

    with create_session() as session:
        session.headers.update({"Accept": "application/json"})

        try:
            response = session.get(
                f"{base_url}/posts",
                params={"userId": user_id},
                timeout=10
            )
            response.raise_for_status()
            return response.json()

        except requests.exceptions.HTTPError as e:
            print(f"HTTP error {response.status_code}: {e}")
        except requests.exceptions.ConnectionError:
            print("Could not connect to the server.")
        except requests.exceptions.Timeout:
            print("Request timed out.")

    return []


posts = fetch_posts(user_id=1)
for post in posts[:3]:
    print(f"- {post['title']}")

Quick Reference Cheat Sheet

Task	Code
GET request	requests.get(url)
POST with JSON	requests.post(url, json=data)
Add headers	requests.get(url, headers={"Auth": "..."})
Query params	requests.get(url, params={"key": "val"})
Set timeout	requests.get(url, timeout=5)
Basic auth	requests.get(url, auth=("user", "pass"))
Check status	response.raise_for_status()
Parse JSON	response.json()
Get text	response.text
Reuse connections	requests.Session()

Conclusion

The requests library deserves to be in the top ten of Python's most favourite packages. Easy to use, well documented and cleanses the bumps from the road of HTTP. From making an API request to scraping data or even creating a CLI tool, requests is almost always the ideal solution.

Here are some important lessons to be learned:

1. Set a timeout on all requests to avoid hanging requests
2. Incorporate a cleanup behavior for errors: use the function raise_for_status()
3. Use a Session for many requests to the same host
4. When sending a JSON payload use json= (not data=)