Token-Based Rate Limiting for AI APIs in Next.js (Production Guide)

#typescript #ai #webdev #nextjs

If you're building with Claude, GPT-4o, or any other LLM API, you need rate limiting. Without it, one viral moment -- or one buggy loop -- can burn through your entire month's API budget in hours.

Here's a production-grade rate limiting setup for Next.js AI routes, with real code you can drop in.

Why AI Routes Are Different

Standard rate limiting (by IP, by user) is well-understood. AI routes have a harder problem: token consumption varies wildly.

A user who sends "hi" costs you $0.0001. A user who sends a 10,000-token document costs you $0.03. If you rate limit by requests, you're not actually limiting cost.

You need to limit by tokens, not requests.

The Implementation

1. Install Upstash Redis

Upstash has a free tier and a Next.js SDK. Perfect for serverless.

npm install @upstash/redis @upstash/ratelimit

Add to .env.local:

UPSTASH_REDIS_REST_URL=your_url
UPSTASH_REDIS_REST_TOKEN=your_token

2. Create the Rate Limiter

// src/lib/rate-limit.ts
import { Ratelimit } from "@upstash/ratelimit"
import { Redis } from "@upstash/redis"

const redis = new Redis({
  url: process.env.UPSTASH_REDIS_REST_URL!,
  token: process.env.UPSTASH_REDIS_REST_TOKEN!,
})

// Request-based limit: 20 requests per minute per user
export const requestLimiter = new Ratelimit({
  redis,
  limiter: Ratelimit.slidingWindow(20, "1 m"),
  analytics: true,
  prefix: "ratelimit:requests",
})

// Token-based limit: 100k tokens per day per user
export const tokenLimiter = new Ratelimit({
  redis,
  limiter: Ratelimit.slidingWindow(100_000, "24 h"),
  analytics: true,
  prefix: "ratelimit:tokens",
})

3. Add to Your AI Route

// src/app/api/chat/route.ts
import { NextRequest, NextResponse } from "next/server"
import { getServerSession } from "next-auth"
import Anthropic from "@anthropic-ai/sdk"
import { requestLimiter, tokenLimiter } from "@/lib/rate-limit"
import { authOptions } from "@/lib/auth"

const client = new Anthropic()

export async function POST(req: NextRequest) {
  const session = await getServerSession(authOptions)
  if (!session?.user?.id) {
    return NextResponse.json({ error: "Unauthorized" }, { status: 401 })
  }

  const userId = session.user.id

  // Check request rate limit
  const { success: requestOk, remaining: requestsLeft } =
    await requestLimiter.limit(userId)

  if (!requestOk) {
    return NextResponse.json(
      { error: "Too many requests. Please wait a minute." },
      {
        status: 429,
        headers: { "Retry-After": "60" },
      }
    )
  }

  const { messages } = await req.json()

  // Estimate input tokens before calling the API
  const estimatedInputTokens = messages
    .reduce((sum: number, m: any) => sum + m.content.length / 4, 0)

  // Check token budget (rough estimate -- actual check happens after)
  const { success: tokenOk } = await tokenLimiter.limit(
    userId,
    { rate: estimatedInputTokens }
  )

  if (!tokenOk) {
    return NextResponse.json(
      { error: "Daily token limit reached. Resets at midnight UTC." },
      { status: 429 }
    )
  }

  // Call the API
  const response = await client.messages.create({
    model: "claude-sonnet-4-6",
    max_tokens: 1024,
    messages,
  })

  // Deduct actual tokens used
  const actualTokens = response.usage.input_tokens + response.usage.output_tokens
  // Adjust for estimation error
  const adjustment = actualTokens - estimatedInputTokens
  if (adjustment > 0) {
    await tokenLimiter.limit(userId, { rate: adjustment })
  }

  return NextResponse.json({
    content: response.content[0].type === "text" ? response.content[0].text : "",
    usage: response.usage,
    limits: {
      requestsRemaining: requestsLeft,
    },
  })
}

4. Show Limits to the User

Don't hide rate limits. Users who know they're close to their limit are less frustrated than users who hit a wall with no explanation.

// In your frontend
const { data } = await fetch("/api/chat", {
  method: "POST",
  body: JSON.stringify({ messages }),
}).then(r => r.json())

if (data.limits?.requestsRemaining < 5) {
  showToast(`${data.limits.requestsRemaining} requests remaining this minute`)
}

Tiered Limits by Plan

If you have free and paid tiers, make the limits reflect that:

// src/lib/rate-limit.ts
import { getServerSession } from "next-auth"

export async function getRateLimiters(userId: string, plan: "free" | "pro") {
  const limits = {
    free: { requests: 10, tokens: 50_000 },
    pro: { requests: 100, tokens: 500_000 },
  }

  const { requests, tokens } = limits[plan]

  return {
    requestLimiter: new Ratelimit({
      redis,
      limiter: Ratelimit.slidingWindow(requests, "1 m"),
      prefix: `ratelimit:${plan}:requests`,
    }),
    tokenLimiter: new Ratelimit({
      redis,
      limiter: Ratelimit.slidingWindow(tokens, "24 h"),
      prefix: `ratelimit:${plan}:tokens`,
    }),
  }
}

Handling the 429 Gracefully

Your frontend should handle rate limits without crashing the UX:

const sendMessage = async (content: string) => {
  try {
    const res = await fetch("/api/chat", {
      method: "POST",
      body: JSON.stringify({ messages: [...history, { role: "user", content }] }),
    })

    if (res.status === 429) {
      const retryAfter = res.headers.get("Retry-After")
      setError(
        retryAfter
          ? "Rate limit hit. Try again in " + retryAfter + " seconds."
          : "Daily limit reached. Resets at midnight UTC."
      )
      return
    }

    const data = await res.json()
    // handle success
  } catch (e) {
    setError("Something went wrong. Please try again.")
  }
}

Cost Monitoring

Rate limits protect you from spikes. Cost monitoring tells you where your money is going.

Log token usage per user to your database:

// After each API call
await db.aiUsage.create({
  data: {
    userId,
    inputTokens: response.usage.input_tokens,
    outputTokens: response.usage.output_tokens,
    model: "claude-sonnet-4-6",
    cost: calculateCost(response.usage),
    createdAt: new Date(),
  },
})

Then build a simple dashboard query to see your heaviest users. If one user is consuming 40% of your token budget, you know exactly who to reach out to.