This article was originally published on Jo4 Blog.
We have a suggest-categories endpoint that calls Groq's API with Llama 4 Scout under the hood. Every call costs money. Not a lot per request, but a user hammering the endpoint 50 times in a minute adds up fast when you're an indie SaaS.
We needed rate limiting. Not the "use an API gateway" kind — the "I need per-user limits on one specific endpoint and I need it deployed by lunch" kind.
TL;DR: We built a sliding window rate limiter using a Redis sorted set and a 15-line Lua script. 5 requests per 60 seconds per user. The Lua script runs atomically in Redis — no race conditions, no distributed locks, no separate rate limit service. Fail-open on Redis unavailability, fail-closed on actual errors.
Why Not an Existing Library?
Fair question. Libraries like Bucket4j or Resilience4j exist. But we already had Redis, we already had a BidRateLimiterService using sorted sets for a different endpoint, and we wanted the same pattern for consistency. Rolling our own took less time than evaluating and configuring a library.
The existing BidRateLimiterService uses a dual-key approach (daily limit + per-brand limit). The category suggest endpoint only needs a single window. Different enough to warrant its own service, similar enough to follow the same Redis pattern.
The Lua Script
Here's the entire rate limiting logic. It runs atomically inside Redis — no round trips between your app and Redis during the check-and-record:
local key = KEYS[1]
local now = tonumber(ARGV[1])
local window = tonumber(ARGV[2])
local limit = tonumber(ARGV[3])
local member = ARGV[4]
-- Remove entries outside the window
redis.call('ZREMRANGEBYSCORE', key, 0, now - window)
-- Count current entries
local count = redis.call('ZCARD', key)
-- If at limit, return count without recording
if count >= limit then
return count
end
-- Record this request
redis.call('ZADD', key, now, member)
-- Set TTL for auto-cleanup (window + 1 second buffer)
redis.call('PEXPIRE', key, window + 1000)
return count
Fifteen lines. Let me walk through what each block does.
Step 1: Prune Expired Entries
redis.call('ZREMRANGEBYSCORE', key, 0, now - window)
Redis sorted sets store members with scores. We use the timestamp (in milliseconds) as the score. ZREMRANGEBYSCORE removes every entry older than now - window. If your window is 60 seconds, this removes everything older than 60 seconds ago.
This is why it's a sliding window. Every time the script runs, it cleans up old entries relative to the current time. No fixed buckets, no boundary alignment issues.
Step 2: Count and Decide
local count = redis.call('ZCARD', key)
if count >= limit then
return count
end
After pruning, ZCARD returns how many requests are in the current window. If we're at or over the limit, return the count immediately. We don't record the new request — we just reject it.
Returning the count (instead of a boolean) lets the caller log how far over the limit the user is. Useful for debugging.
Step 3: Record and Set TTL
redis.call('ZADD', key, now, member)
redis.call('PEXPIRE', key, window + 1000)
If we're under the limit, ZADD records the request with the current timestamp as the score. PEXPIRE sets a TTL on the entire key — if no new requests come in, Redis auto-deletes the key after the window expires. The extra 1-second buffer prevents the key from expiring between the prune and the check on a borderline request.
The Member Uniqueness Problem
The member value in ZADD needs to be unique. If two requests arrive in the same millisecond with the same member value, ZADD overwrites instead of adding — and your counter is wrong.
Our solution: timestamp:uuid-prefix.
String member = System.currentTimeMillis() + ":" + UUID.randomUUID().toString().substring(0, 8);
The timestamp makes it mostly unique. The 8-character UUID prefix handles the same-millisecond collision case. Could we use a full UUID? Sure. But sorted set members are stored in memory, and 8 characters of UUID gives us 2^32 combinations per millisecond. Good enough.
The Java Service
@Slf4j
@Service
public class CategorySuggestRateLimiterService {
private static final int MAX_REQUESTS = 5;
private static final int WINDOW_MILLIS = 60_000;
private final StringRedisTemplate redisTemplate;
private final DefaultRedisScript<Long> rateLimitScript;
public CategorySuggestRateLimiterService(StringRedisTemplate redisTemplate) {
this.redisTemplate = redisTemplate;
this.rateLimitScript = new DefaultRedisScript<>();
this.rateLimitScript.setScriptText(LUA_SCRIPT);
this.rateLimitScript.setResultType(Long.class);
}
public boolean isAllowed(String tenantId, String userId) {
String key = tenantId + ":ratelimit:category-suggest:" + userId;
String member = System.currentTimeMillis() + ":"
+ UUID.randomUUID().toString().substring(0, 8);
try {
Long count = redisTemplate.execute(
rateLimitScript,
List.of(key),
String.valueOf(System.currentTimeMillis()),
String.valueOf(WINDOW_MILLIS),
String.valueOf(MAX_REQUESTS),
member
);
if (count == null) {
// Redis returned null — fail open
log.warn("Rate limiter returned null for user {}, allowing request", userId);
return true;
}
return count < MAX_REQUESTS;
} catch (Exception e) {
// Hard Redis error — fail closed
log.error("Rate limiter error for user {}", userId, e);
throw e;
}
}
}
Static Final Script for Caching
private final DefaultRedisScript<Long> rateLimitScript;
DefaultRedisScript is initialized once in the constructor. Spring Data Redis caches the script's SHA1 hash after the first EVALSHA call. Subsequent executions use EVALSHA instead of EVAL, avoiding re-transmitting the script text on every request. If you created a new DefaultRedisScript per call, you'd re-register the SHA every time — pointless overhead.
Fail-Open vs. Fail-Closed
This was a deliberate design decision:
if (count == null) {
// Fail open: allow the request
return true;
}
If Redis returns null (network blip, script edge case), we allow the request. The user gets their category suggestion. We eat the LLM cost. This matches the pattern we use in BidRateLimiterService — rate limiting is a cost guard, not a security gate. Blocking a legitimate user because Redis hiccupped is worse than allowing an extra LLM call.
But a hard exception (Redis down, connection refused) propagates up:
catch (Exception e) {
throw e; // fail closed on actual errors
}
If Redis is genuinely unreachable, something is wrong with infrastructure, and we want the request to fail loudly rather than silently burning through our AI budget with zero rate limiting.
Tenant-Scoped Keys
String key = tenantId + ":ratelimit:category-suggest:" + userId;
Our app is multi-tenant. Keys are scoped to {tenant}:ratelimit:category-suggest:{userId}. Each tenant's users have independent rate limits. The key structure also makes it trivial to inspect or flush limits for a specific tenant:
# Check a user's current window
redis-cli ZRANGE "acme:ratelimit:category-suggest:user123" 0 -1 WITHSCORES
# Clear a user's limit (support request)
redis-cli DEL "acme:ratelimit:category-suggest:user123"
Using It in the Controller
@PostMapping("/suggest-categories")
public ResponseEntity<?> suggestCategories(
@AuthenticationPrincipal UserDetails user,
@RequestBody SuggestCategoriesRequest request) {
if (!rateLimiterService.isAllowed(user.getTenantId(), user.getUserId())) {
return ResponseEntity.status(429)
.body(Map.of("error", "Rate limit exceeded. Try again in 60 seconds."));
}
// Call Groq LLM
List<String> categories = categoryService.suggest(request);
return ResponseEntity.ok(categories);
}
Clean. The controller doesn't know about Redis, Lua, or sorted sets. It asks "is this allowed?" and gets a boolean.
Why Not a Token Bucket?
Token buckets are great for smoothing traffic. But for our use case — "has this user made more than 5 requests in the last 60 seconds?" — a sliding window is more intuitive and more precise.
Token bucket: refills at a fixed rate, allows bursts up to bucket size. A user could make 5 requests in 1 second, wait 59 seconds, and make 5 more.
Sliding window: counts actual requests in the last N seconds. At any point in time, the count reflects reality. No burst-then-wait gaming.
For protecting an AI budget, the sliding window gives us exactly the behavior we want: hard cap per user per minute, no exceptions.
What This Cost Us
- Implementation time: About 2 hours, including tests.
- Redis memory: Negligible. Each sorted set entry is ~50 bytes. 5 entries per user, auto-expiring. Even with 10,000 active users, that's 2.5 MB.
- Latency: The Lua script executes in microseconds inside Redis. One network round trip. Faster than the LLM call it's guarding by three orders of magnitude.
- AI budget saved: We caught a user running an automated script that was hitting suggest-categories 200+ times per hour. At our Groq pricing, that was adding up.
How do you rate-limit your AI endpoints? Curious whether people are using managed API gateways or rolling their own like us.
Building jo4.io — a URL shortener with analytics, bio pages, and an affiliate marketplace. Our AI features are now budget-friendly.
Top comments (0)