Originally published at claudeguide.io/claude-api-production-architecture

Claude API Production Architecture: Patterns for Scalable Applications

A production Claude API architecture needs four layers: a request queue for rate limit management, a caching layer for repeated prompts, a cost monitor with circuit breakers, and a fallback chain for reliability. Each layer addresses a distinct failure mode — 429 errors, redundant API spend, runaway costs, and downstream outages. This guide covers the implementation patterns for all four, with Python code you can drop into an existing application.

---

Architecture overview

Before writing a single line of code, it helps to see how the layers connect. Every production Claude API application should pass requests through this sequence:

Client Request
     ↓
[Rate Limiter + Queue]  ← prevents 429s
     ↓
[Cache Check]           ← skip API for repeated prompts
     ↓ (cache miss)
[Model Router]          ← Haiku/Sonnet/Opus decision
     ↓
[Claude API]
     ↓
[Response Cache]        ← store for future cache hits
     ↓
[Cost Monitor]          ← track tokens, alert on spikes
     ↓
Client Response

Each layer is independently testable. The rate limiter does not need to know about the cache. The cost monitor does not need to know about the model router. This separation matters when you need to swap one component — for example, replacing an in-process queue with Redis without touching your caching logic.

---

Layer 1: Request queue with rate limit management

Claude API rate limits operate on two axes: requests per minute (RPM) and tokens per minute (TPM). Exceeding either returns a 429 RateLimitError. The naive fix is exponential backoff on the error, but that adds latency after the fact. A proper queue prevents the 429 from occurring.

The implementation below uses asyncio to queue requests and enforce both limits in-process. For multi-process or multi-host deployments, replace the in-memory deques with Redis sorted sets.

python
import asyncio
from collections import deque
import time

class RateLimitedQueue:
    def __init__(self, requests_per_minute: int = 50, tokens_per_minute: int = 40_000):
        self.rpm_limit = requests_per_minute
        self.tpm_limit = tokens_per_minute
        self.request_times = deque()
        self.token_counts = deque()
        self.semaphore = asyncio.Semaphore(10)  # max concurrent requests

    async def acquire(self, estimated_tokens: int = 1000):
        async with self.semaphore:
            now = time.time()
            # Clean old entries
            minute_ago = now - 60
            while self.request_times and self.request_times[0] < minute_ago:
                self.request_times.popleft()
                self.token_counts.popleft()

            # Wait if at limit
            while len(self.request_times) 

[→ Get the Agent SDK Cookbook — $49](https://shoutfirst.gumroad.com/l/ogxhmy?utm_source=claudeguide&utm_medium=article&utm_campaign=claude-api-production-architecture)

*30-day money-back guarantee. Instant download.*

---

## Related guides

- [Claude API Error Handling: Production Patterns](/claude-api-error-handling-guide)
- [Claude Prompt Caching: How to Cut API Costs by 90%](/claude-prompt-caching-guide)
- [Claude Agent Production Deployment: Fly.io, Vercel, and Lambda](/claude-agent-production-deploy)
- [Claude API Production Checklist: 25 Things Before You Ship](/claude-api-production-checklist) — Complete pre-launch checklist covering security, cost controls, reliability, and observability
- [Claude API Webhook Integration: Async Patterns](/claude-api-webhook-integration) — Event-driven patterns for Claude API with FastAPI and Node.js