A Python data collector can run for an hour and then suddenly produce “429 Too Many Requests.” The usual emergency fix is a loop that sleeps for one second and repeats the HTTP call. That can make the situation worse. The code may ignore the server's stated waiting period, multiply load across several workers, or send a write operation twice after the server already processed it.
The better response is specific: inspect the actual failure, respect Retry-After when present, bound the number of attempts, and separate safe read operations from actions that change server state. This tutorial uses Requests and urllib3. Its examples show their intended behavior and include local unit tests instead of making speculative requests to a live API.
Find out what received the 429 before adding retries
A 429 means a server is applying a rate limit. It does not identify which rule was exceeded. The limit might be associated with an account, token, endpoint, organization, or network address. The HTTP standard allows a server to send a Retry-After field, but does not require one. Different services also expose different rate-limit headers, so there is no universal quota header your code can assume.
Begin with one diagnostic request. Record the method, host, path, status, and Retry-After if the response contains it. Do not print bearer tokens or full URLs containing private query parameters. If a batch worker has ten threads, determine whether they all share the same quota before adjusting one thread's settings.
import requests
url = "https://api.example.com/items"
try:
response = requests.get(url, timeout=(3.05, 10))
print("status:", response.status_code)
print("retry-after:", response.headers.get("Retry-After"))
response.raise_for_status()
except requests.ConnectTimeout:
print("Could not establish a connection in time")
except requests.ReadTimeout:
print("Connected server stopped sending data in time")
except requests.HTTPError as exc:
print("Server returned:", exc.response.status_code)
Replace the placeholder URL with an endpoint that you are permitted to use. Requests timeouts should be explicit: otherwise an unresponsive call can wait indefinitely. The pair in this example sets separate connection and read inactivity timeouts. Neither value is a deadline covering the entire download. That distinction matters when retries and slow responses consume far more time than the developer expected.
Interpret Retry-After correctly
The HTTP Retry-After header supports two forms. One is a non-negative number of seconds, such as Retry-After: 120. The other is an HTTP date. A program that simply converts every header to an integer will break on the date form. A program that sleeps for one second regardless of the header may send the next request much earlier than the server asked.
RFC 9110 defines those forms, and RFC 6585 defines the 429 response. Neither establishes a universal waiting time for all APIs. The server may omit Retry-After, or a service may apply a quota that requires a longer pause outside the current request.
You do not normally need to implement both date parsing and retry scheduling yourself. urllib3's Retry component understands the header for certain response codes, including 429 and 503. When no applicable header is present, an exponential backoff setting provides a separate delay policy. These mechanisms are not interchangeable: an explicit server wait should not be silently replaced by an arbitrary short delay.
There is also a deadline problem. If the server asks for an hour and the job has two minutes left, sleeping for an hour is not a successful recovery strategy. Preserve the cursor or checkpoint and defer the remaining work. A healthy queue recognizes when it cannot finish within its allowed time.
Configure a bounded retry policy for GET
Requests uses urllib3 internally, and a Session can be configured with an HTTPAdapter. Make the policy explicit. In particular, list which methods may retry and which status codes are considered temporary. The following example limits automatic retries to GET and HEAD, then handles the exhausted-response case normally.
import requests
from requests.adapters import HTTPAdapter
from urllib3.util import Retry
policy = Retry(
total=3,
connect=3,
read=0,
status=3,
other=0,
allowed_methods=frozenset({"GET", "HEAD"}),
status_forcelist=(429, 502, 503, 504),
respect_retry_after_header=True,
backoff_factor=0.5,
backoff_jitter=0.2,
raise_on_status=False,
)
session = requests.Session()
adapter = HTTPAdapter(max_retries=policy)
session.mount("https://", adapter)
session.mount("http://", adapter)
response = session.get(
"https://api.example.com/items?page=1",
timeout=(3.05, 10),
)
response.raise_for_status()
items = response.json()
print("items:", len(items))
session.close()
The total parameter permits up to three retries after the original request. It does not mean three attempts in total. The selected status codes are deliberately narrow. A validation error such as 400 is not cured by trying the same malformed request repeatedly; a 401 usually needs an authentication fix rather than a backoff delay.
Setting read to zero is conservative. An error while reading can happen after the upstream server received the request. Although a properly implemented GET should be safe to repeat, it is still useful to decide consciously which transport failures your application will replay. Adjust this only after examining the semantics of the upstream endpoint.
Exponential backoff does not necessarily pause before the first retry. The documented urllib3 algorithm commonly begins with an immediate retry when there is no Retry-After instruction, then increases delays for later attempts. Backoff jitter adds small random variation to avoid a synchronized retry wave. Neither feature controls the combined rate of multiple workers in different processes.
Keep POST out of automatic retries until its contract is known
The dangerous case is not a clean 429 received from a GET endpoint. It is a lost response from a write operation. Imagine a POST that creates an invoice: the server stores the invoice, but the client times out while waiting for confirmation. Sending the same POST again could create a duplicate invoice. A timeout is not proof that nothing happened.
Do not widen allowed_methods to include every HTTP method merely because it makes transient errors disappear. The sample intentionally excludes POST. It also does not claim that every GET endpoint is implemented correctly or every PUT is automatically harmless. You have to understand the particular API's documented behavior.
Some providers offer idempotency keys for writes. A client assigns an identifier to one intended operation, retains it across permitted retries, and uses the provider's published contract to prevent duplicate execution. This protection works only when the server explicitly supports it. Adding an invented header to an arbitrary API does not create idempotency.
If the provider offers no dependable idempotency mechanism, persist the local action identifier before submission and query the official object or job status after an ambiguous timeout. If the server confirms completion, record success rather than repeating the request. If the result remains unknown, preserve an unresolved state for reconciliation. This is slower than pressing retry, but prevents silent duplicate side effects.
The same principle applies after retrieving data. Store external object IDs and update existing records instead of inserting a new database row for every repeated page fetch. A retry should not turn one record into two.
Prove the retry decisions without calling a live API
It is wasteful to consume a real service's quota just to find out whether a Python retry configuration behaves as expected. You can inspect the urllib3 policy using synthetic response metadata and Python's built-in unittest. The following tests make no network calls.
import unittest
from urllib3.response import HTTPResponse
from urllib3.util import Retry
class RetryPolicyTests(unittest.TestCase):
def setUp(self):
self.policy = Retry(
total=3,
allowed_methods=frozenset({"GET", "HEAD"}),
status_forcelist=(429, 503),
respect_retry_after_header=True,
)
def test_get_retries_429(self):
self.assertTrue(self.policy.is_retry("GET", 429))
def test_post_does_not_retry_429(self):
self.assertFalse(self.policy.is_retry("POST", 429))
def test_reads_retry_after_seconds(self):
response = HTTPResponse(
status=429,
headers={"Retry-After": "5"},
preload_content=False,
)
self.assertEqual(self.policy.get_retry_after(response), 5.0)
def test_bad_request_is_not_retried(self):
self.assertFalse(self.policy.is_retry("GET", 400))
if __name__ == "__main__":
unittest.main()
The expected result is four passing tests. These checks establish that the chosen methods, status codes, and numeric Retry-After behavior match the policy. They do not prove that your specific API accepts a request or that a live account can safely retry a write.
For an integration test, use a local HTTP server programmed to return a short sequence of 429 and 200 responses, then count requests. Add cases for missing Retry-After, an HTTP-date, and exhaustion of the attempt budget. Do not simulate production by sending dozens of rapid calls to a third-party service. A local test can reveal a looping bug without affecting another operator's quota.
Separate rate limits from network and permission failures
A 429 is a real HTTP response. A connection timeout is not. Requests distinguishes ConnectTimeout and ReadTimeout, and its documentation explicitly states that a ConnectTimeout is safe to retry. A ReadTimeout is more ambiguous for a method that changes server state because the request may already have reached the server.
Treat a 401 as an authentication issue until evidence shows otherwise. Treat a 403 as an access or policy problem rather than assuming an ordinary quota reset will fix it. A 400 often reflects invalid input. These categories should produce different recovery actions, even when they all appear in a generic error dashboard.
Log only what is necessary. A host and path can identify the affected endpoint without copying secrets from a query string. If the path itself contains user identifiers, redact that too.
from urllib.parse import urlsplit
def safe_event(method, url, status, retry_after=None):
parts = urlsplit(url)
return {
"method": method.upper(),
"host": parts.hostname,
"path": parts.path,
"status": status,
"retry_after": retry_after,
}
print(safe_event(
"GET",
"https://api.example.com/items?access_token=private",
429,
"30",
))
This sample removes query parameters from its diagnostic object. It is not a complete privacy policy for every application. In practice you may also need endpoint-specific field redaction, a correlation ID, and a rule for how long diagnostic events are retained.
Give the entire job a quota and a recovery point
Per-request backoff is only one layer. If a crawler schedules one hundred URLs and gives every URL three retries, a broad outage can create up to four hundred attempts. Add five worker processes and the total load becomes much harder to reason about. The fix cannot be limited to the sleep call in one function.
Set a concurrency cap, request budget, and application-level deadline. For paginated data, persist the last fully processed cursor. On resumption, start from a known state and deduplicate by stable record identifiers. When an API supports batch retrieval or conditional caching, use its documented features instead of repeatedly downloading unchanged content.
Be careful about what success means. If a collector fetched nine of ten pages and exhausted the last retry, do not mark the dataset as a complete refresh. Keep the partial result and record which page remains. Explicit incomplete status is more useful than a fresh timestamp attached to missing data.
Review the workload before tuning the retry mechanism. Sometimes the real cause is a duplicate scheduler, accidental loop, unnecessary polling interval, or several deployments that all call the same API. Lowering the initial request rate can resolve the incident without adding a complicated retry strategy.
The rule to keep
For HTTP 429, read the server's response, honor the supported wait instruction, use a finite attempt budget, and retry only operations whose semantics justify it. For uncertain writes, reconcile before repeating. Then test the policy locally and measure total requests rather than simply counting errors that disappeared.
The primary references are urllib3 Retry, Requests timeout behavior, RFC 6585, and RFC 9110. The important distinction is between a temporary server response and proof that an intended operation was never performed.
Originally published on Dispatch.
Top comments (0)