DEV Community: Tufail Khan

FastAPI at 1M+ users: the patterns that actually matter

Tufail Khan — Tue, 21 Apr 2026 11:53:15 +0000

FastAPI is the default Python web framework in 2026 — 38% of Python teams ship on it, up from 29% a year ago. That means a lot of greenfield projects are making the same early mistakes.

This post is what I wish I'd known before scaling Savyour (Pakistan's first cashback platform, 1M+ users, 300+ merchant integrations) from 50 RPS to 3,000+ RPS on FastAPI.

Everything below is drawn from production. No "hello world" demos.

1. Know your async boundaries

FastAPI supports both def and async def endpoints. The framework is smart enough to run sync routes in a threadpool — but your code may not be.

The failure mode: an async def endpoint that calls a blocking library (say, requests instead of httpx). The sync call holds the event loop, everything queues behind it, and your p99 latency goes vertical.

Rule: if the function is async def, every IO operation inside it must be awaitable. Use httpx.AsyncClient, asyncpg, aioboto3, redis.asyncio.

When you must call a sync library, wrap it:

from fastapi.concurrency import run_in_threadpool

@app.get("/report")
async def generate_report():
    # sync pandas code — don't block the loop
    result = await run_in_threadpool(expensive_sync_function)
    return result

2. Connection pools are not optional

Naive async code opens a new database connection per request. At 500 RPS with a 50ms query, that's 25,000 connections fighting your Postgres instance. Postgres caps out around 200-500.

Fix: use a single pool per worker, with tuned sizing:

# database.py
from sqlalchemy.ext.asyncio import create_async_engine, AsyncSession
from sqlalchemy.orm import sessionmaker

engine = create_async_engine(
    DATABASE_URL,
    pool_size=20,           # steady-state per worker
    max_overflow=10,        # burst tolerance
    pool_pre_ping=True,     # detect dead connections
    pool_recycle=1800,      # rotate every 30min
)

AsyncSessionLocal = sessionmaker(engine, class_=AsyncSession, expire_on_commit=False)

async def get_db():
    async with AsyncSessionLocal() as session:
        yield session

For multi-worker deployments (Uvicorn --workers 4), multiply by worker count. If your Postgres caps at 200 connections, 4 workers × 30 max = 120 is safe. Monitor pg_stat_activity in prod.

3. Push heavy work to background queues

The endpoint that made Savyour go down in month two: a synchronous product-sync that iterated through 50K affiliate offers per merchant. Five merchants syncing at once = 250K records in-request = timeouts cascading.

The fix was simple but non-obvious to a team new to async: never do heavy work in the request cycle.

from arq import create_pool
from arq.connections import RedisSettings

@app.post("/sync/{merchant_id}")
async def trigger_sync(merchant_id: int, pool=Depends(get_arq_pool)):
    job = await pool.enqueue_job("sync_merchant", merchant_id)
    return {"job_id": job.job_id, "status": "queued"}

ARQ, Celery, or Dramatiq — pick one. The worker fleet scales independently of the API fleet. Requests return in milliseconds. Monitoring stays sane.

4. Pydantic v2 is 5-50× faster — use it

If you're still on Pydantic v1, migrate. The v2 rewrite in Rust dropped our request validation overhead from ~8ms to ~0.5ms per request. At 3,000 RPS that's a full CPU core back.

Gotchas we hit:

Config → model_config (nested dict)
.dict() → .model_dump()
validator → field_validator, root_validator → model_validator

Use bump-pydantic for the mechanical parts. The semantic changes (validator signatures) need human review.

5. Middleware for observability, not magic

We run three middleware layers in production. In order:

# main.py
app = FastAPI()

# 1. Request ID — every log line traces back
app.add_middleware(RequestIDMiddleware)

# 2. Timing — p50/p95/p99 per route
app.add_middleware(TimingMiddleware)

# 3. Structured logging — JSON out to CloudWatch
app.add_middleware(LoggingMiddleware)

# CORS goes OUTERMOST so OPTIONS requests skip everything
app.add_middleware(CORSMiddleware, allow_origins=FRONTEND_ORIGINS)

Avoid: auto-magic middleware that wraps your handlers with decorators you can't inspect. When things break at 3 AM, you need to grep the source and understand what's happening. Explicit > clever.

6. Health checks, liveness, readiness

Three distinct endpoints. Don't collapse them.

@app.get("/healthz")  # is the process up?
async def health():
    return {"status": "ok"}

@app.get("/readyz")  # can we serve traffic?
async def ready(db=Depends(get_db), redis=Depends(get_redis)):
    await db.execute("SELECT 1")
    await redis.ping()
    return {"status": "ready"}

@app.get("/livez")  # should kubelet restart us?
async def live():
    return {"status": "alive"}

Kubernetes (or ECS, or Fargate) uses these to make restart decisions. A failing dependency should make readyz fail so the LB stops sending traffic — but shouldn't make livez fail and trigger a restart loop.

7. One project structure to rule them all

After shipping a dozen FastAPI services, this is the structure I reach for:

app/
├── main.py            # FastAPI app, middleware, lifespan
├── config.py          # pydantic-settings, env-driven
├── db.py              # engine + session factory
├── dependencies.py    # shared Depends() providers
├── routers/
│   ├── customers.py
│   ├── orders.py
│   └── webhooks.py
├── schemas/           # pydantic request/response models
├── models/            # SQLAlchemy ORM
├── services/          # business logic, pure-ish
├── workers/           # ARQ/Celery task definitions
└── tests/

The key discipline: routers call services, services call models, models don't reach back up. Break that rule and tests get painful fast.

What I'd skip

Things I used to reach for that I don't anymore:

Starlette middleware for auth. Use FastAPI Depends() for auth — it composes cleanly with route permissions.
Custom exception handlers for every error. One global handler that maps exceptions → HTTP codes is enough for 95% of services.
Over-engineered response models for internal APIs. dict returns are fine for handlers only your own code calls.

The meta-point

FastAPI's documentation is aggressively good — better than most frameworks' books. Read it twice before inventing patterns. Most of the hard-won lessons above are implicit in the docs; I just didn't slow down enough to absorb them the first time.

Cutting our Claude API bill by 78% with prompt caching

Tufail Khan — Tue, 21 Apr 2026 11:41:20 +0000

In January 2026 our monthly Claude bill crossed $4,200, up from $600 six months earlier. We were serving a RAG-backed customer-support assistant that retrieved ~12K tokens of context per query, ran through an 800-token system prompt, and called Claude an average of 4.2 times per user session.

Rolling out Anthropic's prompt caching dropped that to $920/month — a 78% reduction — without touching any user-facing behavior.

This post is the exact playbook.

What prompt caching does

Claude's prompt caching stores prefix portions of your prompt in Anthropic's infrastructure. When a subsequent request reuses that same prefix, the cached portion costs 10% of the normal input-token price and is processed much faster.

The pricing in 2026:

Cache write: 1.25× input cost (on first use)
Cache read (hit): 0.1× input cost
TTL: 5 minutes by default, 1 hour available

Break-even is ~2 hits per cache write. In practice, a well-placed cache break point hits dozens to hundreds of times before it expires.

Where to cache — high, medium, low ROI

High ROI (always cache):

System prompts (usually stable across all requests)
Long tool-schema definitions
Retrieved context chunks reused within a session (RAG)
Few-shot example banks

Medium ROI:

User conversation history early in a session (caches grow as the conversation progresses)
Document chunks that appear frequently across queries

Low / anti-ROI:

Per-request user input
Anything that changes every call
Caches smaller than 1024 tokens (minimum cache block size for Claude Opus/Sonnet)

The anatomy of a cached prompt

In the Python SDK, you add cache_control markers to the content blocks you want cached. Everything before the marker gets cached as a prefix.

response = client.messages.create(
    model="claude-opus-4-7",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": LONG_SYSTEM_PROMPT,  # stable, reusable
            "cache_control": {"type": "ephemeral"},
        }
    ],
    tools=[
        {
            "name": "search_docs",
            "description": "...",
            "input_schema": {...},
            "cache_control": {"type": "ephemeral"},
        }
    ],
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": retrieved_context,  # session-scoped RAG chunks
                    "cache_control": {"type": "ephemeral"},
                },
                {
                    "type": "text",
                    "text": f"User question: {user_input}",
                    # no cache marker — this changes every request
                },
            ],
        }
    ],
)

# inspect cache metrics
print(response.usage.cache_creation_input_tokens)
print(response.usage.cache_read_input_tokens)

The key insight: up to 4 cache break points per request. We use all 4:

System prompt (changes ~monthly)
Tool schemas (changes ~monthly)
Retrieved RAG context (changes per session)
Conversation history (grows within session)

Metrics from real traffic

Before caching, on a representative 1,000-request sample:

Input tokens billed: 14.2M (≈ $42.60 at Opus 4.7 pricing)
Output tokens billed: 380K (≈ $28.50)
Total: $71.10

After caching, same workload:

Cache write input tokens: 1.8M ($6.75)
Cache read input tokens: 12.1M ($3.63)
Uncached input tokens: 300K ($0.90)
Output tokens: 380K ($28.50)
Total: $39.78 (−44%)

Output tokens dominate what's left. Short of switching models, the input side is essentially solved.

Watch out for: cache invalidation footguns

Cache hits match on exact byte-level prefix equality. Any variance busts the cache. Things that silently broke ours early on:

Whitespace drift in system-prompt templating (a stray \n from a template engine)
Dict-ordering when serializing tool schemas from a Python dict — always use json.dumps(..., sort_keys=True)
Timestamp injection into system prompts ("Today is {date}..." rebuilds the cache every day — move it to user content)
User-scoped data in system prompt — blows cache per user; move it down the prompt

Instrument cache_creation_input_tokens vs cache_read_input_tokens on every response and alert if the ratio drifts. A week of silent cache misses can cost you thousands.

The 1-hour cache tier

Anthropic added a 1-hour TTL option in mid-2025. It costs 2× the write price but lives 12× longer. For workloads with predictable hot paths — e.g. a support assistant where 80% of sessions hit the same product docs — the 1-hour tier amortizes beautifully.

"cache_control": {"type": "ephemeral", "ttl": "1h"}

Use it where cache hit rate is high. Don't use it for small cache blocks or unpredictable traffic — you'll pay the write premium without the hit volume.

The takeaway

Prompt caching is the highest-ROI single change I've made to a production Claude app in the last year. If you're running a RAG, agent, or long-context workload on Claude and not using prompt caching, the savings are almost certainly 40-80% sitting on the table.

The cost to implement: two afternoons, including the instrumentation. The cost to ignore: compounding every month you don't do it.

Why we replaced LangChain with the raw Anthropic SDK in production

Tufail Khan — Tue, 21 Apr 2026 11:40:59 +0000

LangChain was the right answer in 2023. It abstracted away a messy ecosystem of half-baked provider APIs, gave you a unified LLM interface, and let you stitch agents together with a few dozen lines of Python. We used it everywhere — including in production on Vettio, our AI recruitment platform.

In April 2026, we ripped it out.

This post is about why we made that call, what replaced it, and the metrics that justified the migration.

The symptoms

LangChain's abstractions started leaking the moment we went beyond happy-path demos. Three things kept biting us:

Stack traces from hell. A single AgentExecutor.invoke() call crossed 14 frames of LangChain internals before reaching our code. Debugging a malformed tool call felt like archaeology.
Version churn. Every minor bump renamed, relocated, or deprecated something we depended on. Our CI was pinned to a specific LangChain SHA for six months just to stay green.
Abstracted-away observability. We couldn't cleanly trace token usage, cache hits, or per-tool latencies without monkey-patching internal classes.

Meanwhile, Anthropic's native SDK was getting better. Native tool calling, prompt caching, extended thinking, streaming — all first-class and documented.

The refactor

The logic we were using LangChain for wasn't complicated:

Build a system prompt from templates
Call Claude with a list of tools
Route tool calls to our internal handlers
Return the result

We replaced ~800 lines of LangChain glue with this:

from anthropic import Anthropic

client = Anthropic()

def run_agent(user_input: str, tools: list[dict], tool_handlers: dict):
    messages = [{"role": "user", "content": user_input}]

    while True:
        response = client.messages.create(
            model="claude-opus-4-7",
            max_tokens=4096,
            tools=tools,
            messages=messages,
            system=SYSTEM_PROMPT,
        )

        if response.stop_reason == "end_turn":
            return response.content[0].text

        # Handle tool use
        tool_calls = [b for b in response.content if b.type == "tool_use"]
        messages.append({"role": "assistant", "content": response.content})

        tool_results = []
        for call in tool_calls:
            result = tool_handlers[call.name](**call.input)
            tool_results.append({
                "type": "tool_result",
                "tool_use_id": call.id,
                "content": str(result),
            })
        messages.append({"role": "user", "content": tool_results})

That's it. No AgentExecutor, no Callback, no ConversationBufferMemory. Just the model and our code.

The metrics

We ran the old and new paths side-by-side for two weeks on Vettio's interview-bot service. Results:

p50 latency: 2.1s → 1.4s (−33%)
p95 latency: 4.8s → 3.2s (−33%)
Error rate: 0.9% → 0.2%
Stack trace depth on errors: 14 → 4 frames
Lines of integration code: 812 → 187

The latency win came mostly from eliminating LangChain's implicit retry-and-retry-again behavior on tool-use mismatches. With direct SDK calls, a malformed tool schema fails loudly instead of silently retrying three times.

When LangChain still makes sense

This isn't a blanket "don't use LangChain" post. It still wins if you need:

Multi-provider abstraction. Swapping between Claude, GPT-4, and Gemini behind a stable interface.
LangGraph workflows for graph-based agent topologies you'd otherwise build from scratch.
LangSmith observability you don't want to rebuild.

For a team that's already committed to one provider (we're all-in on Claude) and wants full control over prompts, tool schemas, and observability — the native SDK is the right tool in 2026.

The lesson

Abstractions pay for themselves when the underlying APIs are bad. Anthropic's API isn't bad. It's clean, well-documented, and stable. The abstraction tax was real; the abstraction benefit had quietly evaporated.

If you're still on LangChain in a production Claude app, benchmark a direct-SDK rewrite of your hot path. You might be surprised.