DEV Community: agustin18

The 429 looked right. The limiter was wrong

agustin18 — Tue, 27 Oct 2026 18:30:43 +0000

A debugging story about layered limits in a FastAPI SSE demo.

I built a small browser demo for fastapi-stream-lease to show how a Redis lease stops one user from holding several long-lived streams at the same time. Two tabs, one session cookie, a synthetic 20-second SSE stream. The first tab streams. The second tab gets 429. Everything looked correct.

It wasn't.

The 429 was real, but it came from the wrong limiter. The second tab was rejected by a demo-specific per-IP guard before it reached the per-session Redis lease I wanted to show. The tests passed. The UI looked fine. The only thing wrong was the story the demo was telling.

This is how I found it, what I changed, and why I now test the layer that rejects a request, not just the status code.

What the demo does

The demo is local-only. It runs with:

docker compose up --build demo

It starts Redis and a small FastAPI app at http://localhost:8080. The app has two panels. Both share one signed HttpOnly session cookie. Each panel opens a synthetic SSE stream that yields tokens for 20 seconds.

The library provides per-session and global stream leases. The demo adds two extra layers:

an in-memory request rate limiter
a Redis-backed per-IP guard

The request path is:

rate limit -> per-IP guard -> per-session lease -> SSE

The per-session lease is the library. The other two are demo guardrails. That distinction matters.

The first limiter wins

Layered limits are useful, but they are also easy to misread. If the per-IP guard rejects first, the per-session lease never runs. The client still gets 429. A test that checks only status_code == 429 passes. The demo looks correct. But the feature you wanted to demonstrate was never exercised.

This is a small detail, but it changes completely the story. I was more focused in the 429 than in the layer that produced it.

The UI did not help. It said the second tab was rejected, which was true. It did not say which layer rejected it. I looked at the status code, saw 429, and moved on. That was the mistake.

How I found it

The response body had a code. The second tab returned:

{
  "code": "demo_ip_limit",
  "detail": "Too many active demo streams from this network."
}

The expected one was:

{
  "code": "stream_connection_limit",
  "reason": "user_limit",
  "detail": "User stream limit reached"
}

That was the clue. The default DEMO_MAX_PER_IP was 1, so the second tab from the same browser hit the IP guard before the session lease. The test config used max_per_ip=2, so the test took a different path and passed.

That is the part I missed: the test was not using the same config as the app.

The test gap

The test looked reasonable. It created a demo app, opened a stream, waited for the lease, and opened a second stream. It asserted 429 and reason == "user_limit".

The problem was in the config helper:

def make_config(**overrides):
    values = {
        ...
        "max_per_ip": 2,
        ...
    }

The app default was 1. The test created a world where the bug could not happen.

I removed the override so the test uses the real default. Then I added a second test that starts two sessions, fills the per-IP guard, and asserts the third one gets demo_ip_limit. The first test now asserts the layer:

assert rejected.json()["code"] == "stream_connection_limit"
assert rejected.json()["reason"] == "user_limit"

That is the test that would have caught the original problem.

The fix

I changed the demo default to DEMO_MAX_PER_IP=2, so two tabs from the same network can reach the per-session lease. The per-IP guard still exists. A third session from the same network hits it.

I also added max_per_ip and rate_limit_per_minute to /demo/stats, and the UI now shows the request path and the layer that rejected the stream. A blocked panel says per-session lease or per-IP guard, not just 429.

The library did not change. This was a demo bug.

The library call is still the same:

return await manager.stream(user_id, token_stream())

It takes any async token generator, acquires the lease before the stream starts, renews it while the response is open, and releases it on completion, error, or client disconnect.

What changed in v0.6.4

The per-IP guard default is now 2.
/demo/stats includes max_per_ip and rate_limit_per_minute.
The UI shows per-session and per-network limits, a guided "Try this" flow, and the request path.
The demo suite has 46 tests and 100% branch coverage for demo/app.py.
The library suite has 467 tests.
No runtime library changes.

If you are already on 0.6.3, this is a demo-only update.

What I would do differently

Three things:

Return a machine-readable reason, not only 429. A status code tells you that something rejected the request. It does not tell you what.
Test with the same config users get. A test-only override can hide the default path.
Test each layer separately. If you have a rate limit, a per-IP guard, and a per-session lease, each one needs its own assertion.

For streaming endpoints, I would also test the disconnect path. Close the client in the middle of the stream and check that the lease is released. That is where orphaned generations and socket leaks hide.

A 429 is not a test. The layer is.

Try it

git clone https://github.com/agustin18/fastapi-stream-lease.git
cd fastapi-stream-lease
docker compose up --build demo

Open http://localhost:8080. Start Tab A, then Tab B. Tab B gets 429 from the per-session lease. A third session from the same network hits the per-IP guard. No hosted service or API keys are required.

If you run streaming endpoints in production, what layer do you test first?

Why standard rate limiters fail for LLM & SSE streams in FastAPI

agustin18 — Mon, 19 Oct 2026 14:49:48 +0000

If you are building an AI chat assistant, real-time dashboard, or financial ticker in Python using FastAPI, chances are you are using Server-Sent Events (StreamingResponse) or WebSockets.

And if you care about uptime and cloud costs, you probably installed a rate-limiting middleware like slowapi or configured a Redis token bucket.

Here is the problem: standard rate limiters are designed for request frequency, not connection duration.

In this post, we'll break down why traditional rate limiters collapse under streaming workloads, and how to implement a distributed lease pattern using Redis to protect your workers.

The Root Cause: Frequency vs. Duration

Traditional HTTP endpoints are transactional:

Client sends request.
Server queries DB (50ms).
Server returns JSON response.
Connection closes.

In this model, limiting by frequency (10 requests per minute) works reliably.

What Happens with Streaming (SSE / LLM Tokens)?

An LLM stream (OpenAI, Claude, Ollama) stays open for 30 to 60 seconds while tokens are yielded chunk by chunk.

If you have a rate limit of 10 requests per minute:

A malicious or impatient user opens 10 browser tabs simultaneously.
Each tab requests an SSE stream.
The rate limiter inspects the counter: "10 requests in 60s? Allowed!"
The result: That single user now holds 10 open sockets concurrently for a full minute.

Standard Rate Limiting (Permissive Failure):
User Request 1 ──► [OK] ──► Holding Socket (60s) ──────┐
User Request 2 ──► [OK] ──► Holding Socket (60s) ──────┤
User Request 3 ──► [OK] ──► Holding Socket (60s) ──────┼──► ASGI Worker Pool Exhausted!
User Request 4 ──► [OK] ──► Holding Socket (60s) ──────┤    (HTTP 502 / Socket Starvation)
User Request 5 ──► [OK] ──► Holding Socket (60s) ──────┘

If your Uvicorn worker pool has 100 concurrent connection limits, just 10 users doing this simultaneously will starve the worker, causing HTTP 502/504 timeouts for all regular API endpoints.

The Solution: Connection Leases, Not Request Counts

To protect streaming endpoints, you need to answer a different question:

"How many active streams is this user holding RIGHT NOW?"

Instead of an incrementing counter with a TTL, you need a distributed lease mechanism:

Before streaming: Try to acquire an atomic slot for user_id.
If active_slots >= limit: Reject immediately with HTTP 429 Too Many Requests before triggering the LLM generation.
While streaming: Hold the lease and periodically refresh it.
On completion or client disconnect: Release the slot immediately.

Implementing Atomic Leases in Redis (Lua CAS)

A naive implementation using multiple Redis commands (GET then INCR) introduces race conditions. Two parallel requests could both read count=1 and both succeed, violating your concurrency limit.

To make lease acquisition atomic, we use Redis Sorted Sets (ZSET) where:

Key: stream_lease:{user_id}
Score: Current Unix epoch timestamp
Member: Unique lease_id (UUID)

Here is the atomic Lua script pattern ($O(\log N)$):

local key = KEYS[1]
local now = tonumber(ARGV[1])
local max_leases = tonumber(ARGV[2])
local ttl = tonumber(ARGV[3])
local lease_id = ARGV[4]

-- 1. Remove expired/stale leases
redis.call('ZREMRANGEBYSCORE', key, '-inf', now - ttl)

-- 2. Check current active lease count
local current_count = redis.call('ZCARD', key)
if current_count >= max_leases then
    return 0 -- Rejected (Concurrency limit reached)
end

-- 3. Acquire lease
redis.call('ZADD', key, now, lease_id)
redis.call('EXPIRE', key, ttl * 2)
return 1 -- Allowed

When the client disconnects or the generator completes, a cleanup script removes the lease_id:

redis.call('ZREM', KEYS[1], ARGV[1])

Handling the Edge Cases in Production

In real-world microservices, three critical failure modes occur:

1. Zombie Tasks on Abrupt Disconnections

If a user closes their laptop mid-stream, Starlette detects the socket disconnect via request.is_disconnected(). But if your generator doesn't actively poll this flag or catch the CancelledError, the background task keeps running, consuming downstream AI tokens and keeping the Redis lease alive until expiration.

2. Redis Sentinel Failover (`ReadOnlyError`)

When Redis switches from primary to replica, pending writes raise ReadOnlyError. If your lease manager doesn't catch this, your entire API crashes. You need an automated fail-open or fail-closed fallback policy.

Enter `fastapi-stream-lease`

Instead of writing custom Lua scripts, disconnect wrappers, and metrics hooks in every microservice, I packaged this into a small, dependency-light library:

👉 fastapi-stream-lease on GitHub (MIT, 100% test coverage)

1-Line Integration Example

from fastapi import FastAPI
from fastapi_stream_lease import StreamLeaseManager, LeaseConfig

app = FastAPI()
manager = StreamLeaseManager(redis_url="redis://localhost:6379")

# Register 1-line standard 429 exception handler
manager.register_exception_handlers(app)

@app.get("/stream")
async def chat_stream(user_id: str):
    async def token_generator():
        for i in range(10):
            yield f"Token {i} "

    # Acquires atomic Redis lease, wraps StreamingResponse,
    # detects disconnects, and releases slot on finish or error.
    return await manager.stream(
        user_id=user_id,
        content=token_generator(),
        media_type="text/event-stream",
        config=LeaseConfig(user_limit=2, ttl_seconds=30),
    )

What You Get Out of the Box:

1-Line Integration: Wrap any async generator in manager.stream().
Atomic Concurrency Bounds: Backed by Redis Lua CAS ($O(\log N)$).
Graceful Fallback: In-memory circuit breaker handles Redis failovers safely.
Production Observability: Native Prometheus and OpenTelemetry gauges (fastapi_stream_lease_active_streams).
Resilient Reconnection: Disconnect hooks clean up slots immediately.

Summary

When designing systems for LLM streaming and SSE, remember:

Rate limiters control how fast users knock on your door.
Lease managers control how many rooms they can occupy at once.

If you are running streaming endpoints in production, you need both.

Repo: github.com/agustin18/fastapi-stream-lease
PyPI: pypi.org/project/fastapi-stream-lease

(Note: English is not my native language; drafted with some AI assistance for translation and phrasing, but the architecture and code come directly from our production experience).

How are you currently handling streaming concurrency in your stack? Feedback and architecture critiques are welcome in the comments!