DEV Community

Sathish
Sathish

Posted on

How to Rate Limit Next.js API Routes for Scrapers

#buildinpublic #nextjs #tutorial #webdev

I shipped my job board, opened up a couple API routes, and within days I saw traffic patterns that screamed “someone is hammering this endpoint.” It didn’t take my app down, but it did spike my Supabase usage and made me nervous about the scraper routes. Here’s the rate limiting setup I ended up using in Next.js 14 that actually works on Vercel.

Last month I was debugging a weird issue: my job board’s API was getting hit way more than my actual users.

I’m building PMHNP Hiring (a job board) with Next.js 14 + Supabase + Postgres. It scrapes 200+ jobs daily and serves 7,556+ jobs across 1,368+ companies.

The fun part: a job board is basically an invitation for bots.

So in this post, I’ll show you exactly how I rate limit Next.js API routes on Vercel.

TL;DR

  • I rate limit by IP + route using a tiny Upstash Redis script.
  • I keep limits different per endpoint (search is stricter than “job details”).
  • I return 429 with a clear Retry-After header.
  • I also block obvious abuse patterns (missing user-agent, weird paths) before Redis.

Context (why this matters)

When you build a job board, you end up with endpoints that are valuable to scrape: search, filters, “latest jobs”, maybe even your internal ingest endpoints.

I made the mistake of assuming “nobody will find this early.” They did.

Rate limiting is one of those boring features that pays rent. It protects your database, keeps your latency stable, and prevents one random script from eating your quota.

I’ll cover:

  • A simple rate limiter you can paste into a Next.js 14 App Router route
  • How I generate keys (IP + route)
  • How I tune limits per endpoint
  • Common mistakes I made (and how to avoid them)

Next, I’ll start with the exact libraries and project setup.

1) Use Upstash Redis (works well on Vercel)

I went with Upstash Redis because it’s serverless-friendly and fast enough for this use case.

Install:

npm i @upstash/redis @upstash/ratelimit
Enter fullscreen mode Exit fullscreen mode

Then set env vars:

  • UPSTASH_REDIS_REST_URL
  • UPSTASH_REDIS_REST_TOKEN

Now create a reusable rate limiter.

// lib/ratelimit.ts
import { Ratelimit } from "@upstash/ratelimit";
import { Redis } from "@upstash/redis";

// Upstash serverless Redis client
const redis = new Redis({
  url: process.env.UPSTASH_REDIS_REST_URL!,
  token: process.env.UPSTASH_REDIS_REST_TOKEN!,
});

// A default limiter (you can create more per endpoint)
export const defaultRatelimit = new Ratelimit({
  redis,
  limiter: Ratelimit.slidingWindow(60, "1 m"), // 60 requests/min
  analytics: true, // optional, but nice while tuning
});
Enter fullscreen mode Exit fullscreen mode

What this does:

  • Uses a sliding window limiter (more forgiving than fixed windows)
  • Gives you consistent behavior even under bursts

Pitfall I hit: I initially used in-memory rate limiting. It “worked” locally, then completely failed on Vercel because each request can land on a different instance.

Next, let’s generate good keys.

2) Build a stable key: IP + route + “bucket”

If you rate limit only by IP, users can get unfairly blocked on shared networks.

If you rate limit only by route, one bad actor ruins it for everyone.

I use a combined key: ip:pathname. It’s not perfect, but it’s a solid baseline.

// lib/request-key.ts
import type { NextRequest } from "next/server";

export function getClientIp(req: NextRequest) {
  // Vercel sets x-forwarded-for. First IP is usually the client.
  const forwardedFor = req.headers.get("x-forwarded-for");
  if (forwardedFor) return forwardedFor.split(",")[0].trim();

  // Fallback (rare on Vercel, but useful locally)
  return req.headers.get("x-real-ip") ?? "127.0.0.1";
}

export function getRatelimitKey(req: NextRequest) {
  const ip = getClientIp(req);
  const path = req.nextUrl.pathname;
  return `${ip}:${path}`;
}
Enter fullscreen mode Exit fullscreen mode

Why this matters:

  • Abusers usually hammer a specific endpoint (search/filter)
  • Normal users browse multiple pages and won’t trip this as easily

Pitfall: don’t include the full URL with query params in the key. Otherwise an attacker can bypass limits by changing ?q= each time.

Next, let’s actually enforce the limit in a route handler.

3) Add a rate limit guard to a Next.js route

Here’s a realistic example: a search endpoint that queries Postgres (via Supabase) and returns job listings.

This is one of the highest-value endpoints on my job board, so I keep it stricter.

// app/api/jobs/search/route.ts
import { NextRequest, NextResponse } from "next/server";
import { Ratelimit } from "@upstash/ratelimit";
import { Redis } from "@upstash/redis";
import { getRatelimitKey } from "@/lib/request-key";

const redis = new Redis({
  url: process.env.UPSTASH_REDIS_REST_URL!,
  token: process.env.UPSTASH_REDIS_REST_TOKEN!,
});

// Stricter limiter for search
const searchRatelimit = new Ratelimit({
  redis,
  limiter: Ratelimit.slidingWindow(20, "1 m"), // 20 req/min
});

export async function GET(req: NextRequest) {
  const key = getRatelimitKey(req);

  const { success, limit, remaining, reset } = await searchRatelimit.limit(key);

  if (!success) {
    // reset is a unix timestamp (ms). Convert to seconds for Retry-After.
    const retryAfterSeconds = Math.max(1, Math.ceil((reset - Date.now()) / 1000));

    return NextResponse.json(
      { error: "Too many requests", retryAfterSeconds },
      {
        status: 429,
        headers: {
          "Retry-After": retryAfterSeconds.toString(),
          "X-RateLimit-Limit": limit.toString(),
          "X-RateLimit-Remaining": remaining.toString(),
        },
      }
    );
  }

  // Your normal handler code would go here.
  // In my project, this calls Supabase/Postgres with filters + pg_trgm.
  return NextResponse.json({ ok: true, message: "Search results..." });
}
Enter fullscreen mode Exit fullscreen mode

What I like about this pattern:

  • It’s explicit (easy to reason about)
  • It returns useful headers (helps you debug)
  • It’s copy-pasteable across routes

Pitfall: don’t forget that bots will ignore your 429 and keep hammering. Rate limiting is still valuable because it protects your DB, but you may also want a WAF rule later.

Next, I’ll show how I tune limits per endpoint without duplicating code everywhere.

4) Centralize limits (different endpoints, different budgets)

Not all endpoints are equal.

On my job board:

  • Search/filter endpoints are expensive → stricter
  • “Job details by id” is cheaper → more lenient
  • Internal ingestion endpoints should be locked down → basically no public access

I keep a small helper that selects the limiter based on pathname.

// lib/ratelimit-guard.ts
import type { NextRequest } from "next/server";
import { NextResponse } from "next/server";
import { defaultRatelimit } from "@/lib/ratelimit";
import { getRatelimitKey } from "@/lib/request-key";

export async function enforceRatelimit(req: NextRequest, opts?: {
  limiter?: typeof defaultRatelimit;
}) {
  const limiter = opts?.limiter ?? defaultRatelimit;
  const key = getRatelimitKey(req);

  const result = await limiter.limit(key);
  if (result.success) return null; // allow

  const retryAfterSeconds = Math.max(1, Math.ceil((result.reset - Date.now()) / 1000));

  return NextResponse.json(
    { error: "Too many requests", retryAfterSeconds },
    {
      status: 429,
      headers: {
        "Retry-After": retryAfterSeconds.toString(),
        "X-RateLimit-Limit": result.limit.toString(),
        "X-RateLimit-Remaining": result.remaining.toString(),
      },
    }
  );
}
Enter fullscreen mode Exit fullscreen mode

Then in any route:

// app/api/jobs/latest/route.ts
import { NextRequest, NextResponse } from "next/server";
import { enforceRatelimit } from "@/lib/ratelimit-guard";

export async function GET(req: NextRequest) {
  const blocked = await enforceRatelimit(req);
  if (blocked) return blocked;

  // Fetch latest jobs (cheap query, cached on the client)
  return NextResponse.json({ jobs: [] });
}
Enter fullscreen mode Exit fullscreen mode

Pitfall: don’t share the same strict limiter for everything. I did that at first and I started rate limiting my own frontend during local testing because the app makes multiple calls on page load.

Next, let’s add one more layer: cheap “abuse checks” before Redis.

5) Add a fast pre-check (blocks obvious bad traffic)

Redis calls are fast, but they’re still work.

I added a couple quick checks that catch the laziest scrapers:

  • missing/empty user-agent
  • weird methods
  • hitting internal routes that should never be public 
// lib/basic-abuse-check.ts
import type { NextRequest } from "next/server";
import { NextResponse } from "next/server";

export function basicAbuseCheck(req: NextRequest) {
  const ua = req.headers.get("user-agent")?.trim() ?? "";

  // Block empty UA (lots of cheap scripts do this)
  if (!ua) {
    return NextResponse.json({ error: "Missing user-agent" }, { status: 400 });
  }

  // Example: block access to ingestion endpoints
  if (req.nextUrl.pathname.startsWith("/api/internal")) {
    return NextResponse.json({ error: "Not found" }, { status: 404 });
  }

  return null;
}
Enter fullscreen mode Exit fullscreen mode

Use it like this:

// app/api/jobs/search/route.ts
import { NextRequest } from "next/server";
import { basicAbuseCheck } from "@/lib/basic-abuse-check";
import { enforceRatelimit } from "@/lib/ratelimit-guard";

export async function GET(req: NextRequest) {
  const abuse = basicAbuseCheck(req);
  if (abuse) return abuse;

  const blocked = await enforceRatelimit(req);
  if (blocked) return blocked;

  return Response.json({ ok: true });
}
Enter fullscreen mode Exit fullscreen mode

Pitfall: don’t go overboard with user-agent blocking. Real users can have privacy-focused browsers, and some legitimate tools have minimal UAs. I keep this lightweight and rely on rate limiting for the real protection.

Next, I’ll share the actual outcome after shipping this.

Results / outcome

After adding rate limiting to my high-value endpoints (search, filters, “latest jobs”), my Supabase query spikes smoothed out immediately.

On PMHNP Hiring, I’m still scraping 200+ jobs daily, serving 7,556+ jobs, and my public API routes stay stable.

My average query time stayed around ~50ms, and the bigger win was predictability: one aggressive client can’t make my database feel slow for everyone else.

I also stopped worrying that a random script would burn through my monthly limits overnight.

Key takeaways

  • Rate limit per route, not just “globally.” Search deserves stricter limits.
  • Use a stable key like IP + pathname, but avoid query params.
  • Always return 429 with Retry-After so well-behaved clients can back off.
  • Add a tiny pre-check to block obviously bad requests before hitting Redis.
  • Start strict, watch logs, then loosen limits until real users stop getting blocked.

Closing (question + follow-up)

If you’ve built a job board (or any API that attracts bots), what endpoint got abused first: search, lists, or details?

Drop your setup in the comments—happy to help you tune limits. If people want it, I’ll write a follow-up showing how I combine rate limiting with caching (so the DB barely gets touched for repeated searches).

Top comments (0)