DEV Community

Cover image for The rate limiter that was "on" without Redis — and panicking our smoke tests
Kantemir Satibalov
Kantemir Satibalov

Posted on

The rate limiter that was "on" without Redis — and panicking our smoke tests

#devchallenge #bugsmash #redis #debugging

Summer Bug Smash: Smash Stories 🐛🛹

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

The rate limiter that was "on" without Redis — and panicking our smoke tests

The Setting

I build Grounded LLM — an open-source platform for cited, verified document assistants (Go orchestration + Python RAG). For the v0.4 enterprise-hardening push I added a sliding-window rate limiter that could use Redis across replicas, or fall back to an in-process map when Redis was not configured.

CI runs smoke-api / load-smoke against a Compose stack. Sometimes Redis is up for caching; sometimes a path boots without a live Redis client. The contract was simple: no Redis → memory limiter → no panic.

That contract quietly broke.

The First Sign

Green unit tests. Red CI.

Smoke jobs started failing on basic POST /session with a Go panic deep in the rate-limit path — not a polite 429, not a logged Redis timeout, a hard crash. Locally with Redis everything looked fine. In the failing job profile, Redis was effectively absent, yet logs hinted that the Redis backend had been enabled.

Classic heisenbug: works on my machine (Redis running), blows up in CI (typed-nil client handed to the limiter).

The Investigation

First suspects:

  1. Wrong env / missing REDIS_URL
  2. Race in the in-memory map
  3. Gin middleware order

None matched. The panic was on Redis command methods, and the limiter thought rdb != nil.

In Go that sentence is a trap.

I had something like:

func llmRedis() redis.Cmdable {
    return llm.Redis() // *redis.Client, often nil
}
Enter fullscreen mode Exit fullscreen mode

llm.Redis() returns *redis.Client. When Redis is not initialized, that pointer is nil. We returned it as redis.Cmdable (an interface).

In Go, an interface value is (type, value). A nil concrete pointer boxed into an interface is not a nil interface:

var c redis.Cmdable = (*redis.Client)(nil)
c == nil  // false !
Enter fullscreen mode Exit fullscreen mode

WithRedis(c) saw a non-nil interface, set rl.rdb = c, logged "Redis backend enabled", and on the first request called into methods on a nil *redis.Client → panic.

The memory fallback never ran: the code path was "Redis is configured", not "Redis call failed".

The Root Cause

Go's typed-nil / nil interface footgun, applied to github.com/redis/go-redis:

Step What happened
1 No Redis client created → *redis.Client == nil
2 Function returns that pointer as redis.Cmdable
3 Interface is non-nil → limiter enables Redis mode
4 First allow() hits Redis API on nil client → panic
5 CI smoke dies; local Redis hides the bug

The bug was not "Redis is flaky". The bug was lying about having Redis.

The Fix

Two layers of defense.

1. Don't box a typed nil at the source (server/internal/app/llm_bridge.go):

func llmRedis() redis.Cmdable {
    c := llm.Redis()
    if c == nil {
        return nil // true nil interface
    }
    return c
}
Enter fullscreen mode Exit fullscreen mode

2. Refuse typed-nil in WithRedis (server/internal/httpapi/ratelimit.go):

func (rl *RateLimiter) WithRedis(c redis.Cmdable) *RateLimiter {
    if rl == nil || c == nil {
        return rl
    }
    if rc, ok := c.(*redis.Client); ok && rc == nil {
        return rl
    }
    rl.rdb = c
    return rl
}
Enter fullscreen mode Exit fullscreen mode

Plus a regression test: box a typed-nil *redis.Client and assert the memory backend stays active (ratelimit_redis_nil_test.go).

Before: smoke-api red, panic on first rate-limited request without Redis.

After: smoke-api green; without Redis → memory limiter; with Redis → shared counters across replicas.

Merged as part of enterprise hardening in grounded-llm (v0.4.0 line).

What I'm proud of

CI caught it before prod. The fix is small but teaches a rule we now apply everywhere optional clients are passed as interfaces.

What I learned

  1. x == nil on an interface is not enough when x might hold a typed nil.
  2. Optional infrastructure needs an explicit enabled signal, or a concrete nil check before enabling the backend.
  3. Smoke tests without Redis are as valuable as tests with Redis — they exercise the fallback contract.

If you ship Go services with optional Redis, audit every func() SomeInterface { return concretePtr }. Your CI might already be one typed-nil away from a panic.

Top comments (0)