A Bedrock 429 means one of your account’s quotas said no. Which quota is the whole question, because the two candidates behave completely differently under load and only one of them is fixed by sending fewer requests.
The error
botocore.errorfactory.ThrottlingException: An error occurred
(ThrottlingException) when calling the Converse operation (reached max retries: 4):
Too many requests, please wait before trying again.
HTTP 429. AWS documents the cause as the request being denied due to exceeding the account quotas for Amazon Bedrock. Note the phrasing in the traceback above: botocore has already retried four times before you saw this, so by the time it surfaces you have been throttled five times, not once. That matters for latency budgets and it matters for your bill on any provider that charges for the attempt.
Which quota fired
Bedrock’s on-demand inference is governed per model by two separate limits — one counting requests and one counting tokens. In Service Quotas they appear per model, and the cross-region equivalents are documented by AWS as Cross-region model inference requests per minute for {Model} and Cross-region model inference tokens per minute for {Model}. The on-demand entries follow the same naming pattern without the cross-region prefix.
The response body does not tell you which one you hit. The traffic shape does:
- Requests per minute is what you hit when many small calls arrive at once — a fan-out over a list, a burst of user traffic, a retry storm. Symptom: throttling scales with request count and is unaffected by prompt length. Fix: concurrency limits and queueing.
- Tokens per minute is what you hit with few, large calls — long documents, big retrieved contexts, long generations. Symptom: a handful of concurrent requests throttles while a hundred small ones did not. Fix: shorter prompts, smaller
maxTokens, or more capacity. Throwing a semaphore at it will not help, because ten concurrent 100k-token requests and one sequential one consume the same minute’s tokens.
To find out which without guessing, sum the usage.totalTokens from your successful responses over a rolling minute and compare it against the tokens-per-minute quota value in the Service Quotas console, then do the same with request counts. The single most useful instrument here is a per-minute histogram of both; CloudWatch gives you Bedrock invocation and token metrics to build it from. Instrument both, because the quota you are near today is not necessarily the one you will hit after your next prompt change.
Quota names, values and adjustability all change, and adjustability varies by model — some per-model on-demand quotas are marked non-adjustable in Service Quotas, so the increase you want may only be available through your account team. Read the current entries in the Amazon Bedrock endpoints and quotas reference or in the Service Quotas console for your Region. AWS itself recommends the console over the table because there are so many.
Why this is not a 503
Bedrock returns two different errors that both mean “try later”, and conflating them sends you down the wrong path. AWS is explicit that a 503 ServiceUnavailable indicates the service is experiencing high demand or temporary capacity constraints and is not related to your account-level quotas or rate limits, which return 429 ThrottlingException.
So: 429 is about you and a quota increase can fix it. 503 is about the Region and no quota change will touch it — AWS’s own suggested remedies are retries, trying another Region, or cross-region inference. If your dashboards bucket both as “retryable” you will spend a week requesting a limit increase for a capacity event.
Two more 429s worth distinguishing. ModelNotReadyException also returns 429 and means the model is not ready to serve — AWS notes the SDK automatically retries it up to five times. And ModelTimeoutException is a 408, not a 429: the request took longer than the model timeout. Retrying a 408 with the same prompt reproduces it.
Retries, and where the SDK default fails you
AWS’s recommendation for a 429 is exponential backoff with jitter, and the SDKs implement it. The default configuration is nonetheless wrong for inference in two ways.
First, the default attempt count is low and the default backoff is tuned for control-plane calls that take milliseconds. A generation taking twenty seconds retried four times in quick succession is a minute of wall clock and four rejected requests inside the same throttled minute — you are backing off across a window shorter than the one the quota measures.
import boto3
from botocore.config import Config
config = Config(
retries={"max_attempts": 6, "mode": "adaptive"},
read_timeout=300,
connect_timeout=10,
tcp_keepalive=True,
)
client = boto3.client("bedrock-runtime", config=config)
adaptive mode adds client-side rate limiting that slows down before it is throttled rather than after, which is the behaviour you want when the alternative is a retry storm making the throttling worse.
Second, and unrelated to quotas but often mistaken for them: tcp_keepalive. AWS documents that NAT gateways, interface VPC endpoints and network load balancers have a fixed 350-second idle connection timeout, and that a pooled connection dropped silently shows up later as a request that hangs before the OS gives up. They are equally explicit that enabling it in the SDK is not enough on its own — Linux defaults net.ipv4.tcp_keepalive_time to 7200 seconds, far beyond 350, so you must also lower the kernel value (they suggest 45) via the pod or task securityContext, an init container, or /etc/sysctl.d/. A first call after idle taking seventy seconds is this, not throttling.
The four real fixes
- Move to an inference profile. Prefix the model id with
us.,eu.,apac.orglobal.and your traffic is measured against the separate cross-region quotas over a pooled multi-Region capacity. It is a one-string change plus an IAM statement people forget, and it is the cheapest thing on this list. - Move deferrable work off the synchronous path. Batch inference has its own quotas and a lower price. If a nightly enrichment job is competing with live user traffic for the same tokens-per-minute budget, it is both throttling you and costing you double.
- Request an increase. Some per-model quotas are adjustable in Service Quotas; where they are not, AWS directs you to your account manager or Support with your throughput requirements. Bring numbers — measured tokens per minute at peak, not a guess.
- Buy capacity. Provisioned Throughput removes the shared-pool question entirely, at a fixed hourly cost that only pays off at a sustained rate. It is the last resort for a steady high load and the wrong answer for a spiky one.
The structural fix for throttling on a single provider is somewhere to send the request instead, and that is harder than it sounds: a fallback from Bedrock to another provider means a second response shape, a second streaming event format, a second set of rate-limit semantics and a second key to rotate. Multigrid is a gateway that holds that logic — one request shape, with routing and fallback across providers — so a 429 becomes a routing decision rather than an incident.
Top comments (0)