Boto3 Retries: Why Your AWS Scripts Fail Silently

Originally published on DevToolHub.

Boto3's default retry mode is legacy, not standard — and legacy mode only retries a narrow list of errors: ConnectionError, Throttling, ThrottlingException, and ProvisionedThroughputExceededException. If your script fails on a transient RequestTimeout or SlowDown error that standard mode would have retried automatically, that's why. You're on the older, narrower default until you explicitly ask for something else.

boto3 Retry Modes: Legacy, Standard, and Adaptive

There are three modes, and they are not equally capable.

Legacy (the default) retries 5 attempts total against a short error list: connection errors and a handful of throttling exception names. It uses exponential backoff with a base factor of 2.

Standard retries 3 attempts by default, adds circuit-breaking for sustained service outages, and covers a longer error list — RequestTimeout, TooManyRequestsException, TransactionInProgressException, RequestLimitExceeded, BandwidthLimitExceeded, SlowDown, plus the HTTP 500/502/503/504 status codes for nondescriptive transient errors. Backoff is capped at 20 seconds.

Adaptive builds on standard mode and adds client-side rate limiting through a token bucket that adjusts your call rate based on how the service is actually responding. AWS marks this one experimental.

Setting the Retry Mode: The Syntax That Trips People Up

The field names are different depending on where you configure retries, and mixing them up fails silently instead of raising an error.

In code, with a Config object:

from botocore.config import Config
import boto3

config = Config(
    retries={
        "total_max_attempts": 10,
        "mode": "standard",
    }
)

client = boto3.client("ec2", config=config)

In ~/.aws/config, the same settings use different key names:

[myConfigProfile]
region = us-east-1
max_attempts = 10
retry_mode = standard

Copy one format's keys into the other and boto3 won't error — it just ignores the keys it doesn't recognize. total_max_attempts counts the initial request too.

Environment variable:

export AWS_MAX_ATTEMPTS=10
export AWS_RETRY_MODE=standard

Catching ClientError the Right Way

Every service error in boto3 surfaces as botocore.exceptions.ClientError, with the code and message inside the response dictionary:

import botocore.exceptions

try:
    client.describe_stream(StreamName="myDataStream")
except botocore.exceptions.ClientError as error:
    error_code = error.response["Error"]["Code"]
    error_message = error.response["Error"]["Message"]

⚠️ Note: standard mode already backs off and retries throttling errors automatically. A hand-rolled time.sleep() retry loop on top of it is usually redundant.

Checking Whether a Retry Actually Happened

response = client.some_operation()
print(response["ResponseMetadata"]["RetryAttempts"])

Legacy mode logs through botocore.retryhandler; standard and adaptive log through botocore.retries.standard.

Why Third-Party Retry Wrappers Exist at All

If boto3's built-in retry modes were sufficient out of the box, packages like botobackoff and awsretry wouldn't exist. They do because plenty of scripts were written against the legacy default and never revisited.

Frequently Asked Questions

Q: What's the default boto3 retry mode?
A: legacy. You have to opt into standard or adaptive explicitly.

Q: Does standard mode retry throttling errors automatically?
A: Yes — several throttling exceptions are retried with backoff automatically.

Q: What's the difference between total_max_attempts and max_attempts?
A: Same concept, different key name — total_max_attempts in a Config object, max_attempts in ~/.aws/config.

Q: Should I use adaptive mode in production?
A: It's marked experimental by AWS. Standard mode is the stable choice for most production scripts.

Q: How do I check if a boto3 call actually retried?
A: Read response["ResponseMetadata"]["RetryAttempts"] after a call.

Quick Summary:

• boto3's default retry mode is legacy, not standard
• Config uses mode/total_max_attempts; the AWS config file uses retry_mode/max_attempts
• Standard mode already retries throttling errors — a manual retry loop on top is usually redundant
• Catch botocore.exceptions.ClientError and read error.response["Error"]["Code"]
• Check ResponseMetadata["RetryAttempts"] to confirm retries are happening