Next.js Bot Detection: Block AI Crawlers at the Edge

Your Vercel usage graph is climbing and your logs are full of names you did not invite: GPTBot, ClaudeBot, PerplexityBot, Bytespider. They hammer your App Router pages and quietly run up your bandwidth and compute bill.

The reflex is a ten-line user-agent block in middleware.ts. After you ship it the graph looks calmer for a day. Then it climbs again.

The ten-line block is not wrong. It is just the first of three layers, and on its own it catches only the crawlers honest enough to tell you who they are. This is a working guide to all three in a normal Next.js project: an edge gate on every request, honeypot routes that catch the bots that lie, and an origin fingerprint check for the signal the edge genuinely cannot see.

We will also be honest about that last part, because most tutorials are not.

The naive block, and exactly why it fails

Almost every Next.js bot-blocking guide ends here:

// middleware.ts
import { NextResponse } from 'next/server'
import type { NextRequest } from 'next/server'

const BLOCKED = /GPTBot|ClaudeBot|PerplexityBot|Bytespider|CCBot|Google-Extended|Meta-ExternalAgent|Amazonbot/i

export function middleware(req: NextRequest) {
  const ua = req.headers.get('user-agent') || ''
  if (BLOCKED.test(ua)) {
    return new NextResponse('Forbidden', { status: 403 })
  }
  return NextResponse.next()
}

export const config = {
  matcher: ['/((?!_next/static|_next/image|favicon.ico).*)'],
}

This works against a crawler that announces itself. GPTBot sends a user agent that says GPTBot, you match it, you return 403. Done.

The problem is that a user agent is a string the client chooses. Nothing forces a scraper to keep telling the truth, and the moment blocking becomes common, the well-funded scrapers stop. Perplexity was reported through 2025 to fetch pages with a generic Chrome user agent and rotating addresses after its declared bot was blocked. A scraper running headless Chrome or a plain HTTP client can set any user-agent header it likes in one line. Your regex never sees them.

So the honest framing: a user-agent block is a politeness filter. It removes the crawlers that respect your wishes, which is real and worth doing, and it does nothing to the ones that do not. The same logic applies to robots.txt, which is a request rather than a rule.

Layer one: a real edge middleware

Keep the user-agent gate, but stop treating it as the whole defense. A useful middleware does three jobs: cheaply block the honest crawlers, rate limit everyone else so a single client cannot flood you, and hand a signal to your origin so the deeper check knows where to look.

Where middleware lives, and what matcher does

middleware.ts sits at the root of your project, or inside src/. It runs on the Edge Runtime by default, before your routes and before cached output — exactly why it is the right place for a first gate. The request is stopped before it costs you a function invocation or a database hit.

The matcher is your most important performance setting. Without it, middleware runs on every asset, including static files Next.js already serves for free:

export const config = {
  matcher: [
    // run on everything except Next internals and static files
    '/((?!_next/static|_next/image|favicon.ico|robots.txt|sitemap.xml).*)',
  ],
}

Block, rate limit, or rewrite

NextResponse gives you three moves inside middleware:

return new NextResponse('Forbidden', { status: 403 })          // block
return new NextResponse('Too Many Requests', { status: 429 })  // rate limit
return NextResponse.rewrite(new URL('/tarpit', req.url))       // send to a decoy

You can hand-roll the gate from here, but it adds up fast: a regex of declared crawlers to maintain, plus a shared rate-limit store — an in-process counter (a plain Map) will not hold when edge invocations do not share memory, so you reach for Upstash Redis or Vercel KV.

That is ongoing work, and it is the work @webdecoy/nextjs exists to remove:

npm install @webdecoy/nextjs

// middleware.ts
import { withWebDecoy } from '@webdecoy/nextjs'
import { rateLimit } from '@webdecoy/node'

export default withWebDecoy({
  apiKey: process.env.WEBDECOY_API_KEY!,
  // Built-in rules engine: no separate counter store to stand up.
  rules: [rateLimit({ max: 100, window: 60 })], // 100 requests per 60s
  // Skip work on paths that never need protection.
  skipPaths: ['/_next', '/favicon.ico', '/robots.txt'],
})

export const config = {
  matcher: ['/((?!_next/static|_next/image|favicon.ico).*)'],
}

On every matched request this runs local analysis, applies your rules, and returns the right response on its own: a tripped rateLimit returns 429 with a Retry-After header, a deny rule returns 403, and an allowed request continues with an x-webdecoy-decision header so your routes can read the verdict downstream. onBlocked and onError let you override, and onError fails open by default, so a hiccup in detection never locks out real users.

This is a real improvement over the ten-line version. But notice what every signal so far has in common. User agent, headers, address, and request rate are all things the client controls or can rotate. To catch a bot that lies about all of them, you need a signal it does not get to set: its own TLS handshake.

The honest constraint nobody mentions

Here is the part most Next.js guides skip, and it holds whether you hand-roll the gate or use a package: you cannot compute a TLS fingerprint inside middleware.ts.

A JA3 or JA4 fingerprint is built from the raw ClientHello of the TLS handshake — the cipher suites, the extensions and their order, the supported groups, the way the client negotiates the connection. These are extremely hard to fake because they come from the client's TLS stack rather than from a header.

The catch on a platform like Vercel is that TLS terminates at the edge network before your middleware runs. By the time your code executes, the handshake is over and the ClientHello bytes are gone. The Edge Runtime has no socket access and no node:tls, so there is nothing to read. Recent Next.js versions let you move middleware to the Node.js runtime, which is useful for other reasons, but it still does not hand you the original handshake.

This is not a flaw in Next.js. It is just where the layers sit. Put the fingerprint check where the handshake is visible:

1. Your own origin, when you self-host Next.js behind your own TLS termination (next start behind nginx or Caddy), where the proxy reads the handshake and forwards it as headers.
2. A detection service that captures those handshake signals for you and returns a verdict your route handler can act on.

So the architecture becomes: gate cheaply at the edge, trap the liars with honeypots, run the fingerprint check at the origin where the signal lives.

Layer two: honeypot routes in the App Router

A honeypot exploits a simple asymmetry: a real visitor never touches it, so any hit is suspicious by definition. The classic version is a hidden form field. For a Next.js crawler problem, a honeypot route is a better fit, because crawlers follow links and probe paths humans never click.

First, plant a decoy link that humans cannot see but a link-following scraper will. Put it in your layout, and disallow the path in robots.txt so honest crawlers skip it — anything that fetches it has both ignored robots.txt and followed an invisible link, which is a strong signal:

// app/layout.tsx (excerpt)
export default function RootLayout({ children }: { children: React.ReactNode }) {
  return (
    <html lang="en">
      <body>
        {children}
        {/* Invisible to humans, irresistible to link-scraping bots. */}
        <a href="/api/trap" aria-hidden="true" tabIndex={-1}
           style={{ position: 'absolute', left: '-9999px' }}>
          Account archive
        </a>
      </body>
    </html>
  )
}

Then the trap itself — a route handler that records the hit and responds blandly so the bot does not learn it was caught:

// app/api/trap/route.ts
import { NextResponse } from 'next/server'
import type { NextRequest } from 'next/server'
import { flagClient } from '@/lib/threat'

export async function GET(req: NextRequest) {
  const ip = req.headers.get('x-forwarded-for')?.split(',')[0] ?? 'unknown'
  const ua = req.headers.get('user-agent') || ''

  // Record the hit. Anything reaching this route is presumed automated.
  await flagClient({ ip, ua, reason: 'honeypot:trap', score: 80 })

  // Respond like a boring empty resource. Do not reveal the trap.
  return new NextResponse(null, { status: 204 })
}

Now your middleware reads that stored flag and acts on it before serving real content. Rewrite flagged traffic to a tarpit instead of your actual route, which keeps the URL stable so the bot does not notice:

// inside middleware(), after the rate-limit check
import { isFlagged } from '@/lib/threat'

if (await isFlagged(ip)) {
  return NextResponse.rewrite(new URL('/tarpit', req.url))
}

The same pattern extends to fake API endpoints. A path like /api/v1/users/export that your real app never calls, but a scraper probing for data will, becomes a high-confidence trap.

Honeypots are powerful because they need no fingerprint and no machine learning. They exploit the gap between how a human and a script move through a site. But a careful scraper that only fetches linked, allowed pages at a human pace will avoid them. That is the gap layer three closes.

Layer three: origin fingerprinting

For the bot that spoofs its user agent, rotates its address, paces itself, and avoids your traps, you need the one thing it cannot rewrite: its TLS handshake. As covered above, that check has to run at the origin, in a Node runtime, not in edge middleware.

Next.js route handlers default to the Node.js runtime, which makes them the right home. The core SDK runs a two-tier check: a fast local pass on your server (suspicious headers, datacenter IP ranges, known bot user agents, missing client hints), and a deeper pass using JA3/JA4 fingerprinting to flag the case where a request claims to be Chrome but handshakes like curl.

npm install @webdecoy/node

// app/api/checkout/route.ts
export const runtime = 'nodejs' // the Edge Runtime cannot see the handshake

import { NextResponse } from 'next/server'
import type { NextRequest } from 'next/server'
import { WebDecoy } from '@webdecoy/node'

const webdecoy = new WebDecoy({
  apiKey: process.env.WEBDECOY_API_KEY!,
  enableTLSFingerprinting: true,
  threatScoreThreshold: 70, // block at a threat score of 70 or higher
})

export async function POST(req: NextRequest) {
  const result = await webdecoy.protect({
    method: req.method,
    path: new URL(req.url).pathname,
    ip: req.headers.get('x-forwarded-for')?.split(',')[0] ?? '0.0.0.0',
    user_agent: req.headers.get('user-agent') ?? '',
    headers: Object.fromEntries(req.headers),
    timestamp: Date.now(),
  })

  if (!result.allowed) {
    // result.detection carries decision, confidence (0 to 100), and bot_type.
    return NextResponse.json({ error: 'Request blocked' }, { status: 403 })
  }

  return NextResponse.json({ ok: true })
}

One honest caveat about where each tier can run. The local pass works anywhere, including a route handler on Vercel, because it only reads headers and the address. The JA3/JA4 pass needs the client's actual handshake, and your function only sees that when it has socket access. Self-hosting behind nginx or Caddy, you forward the handshake details as headers and the SDK gets the full fingerprint. On Vercel's managed edge, you lean on the local signals and put the deep fingerprint check on a self-hosted origin or proxy.

Still on the Pages Router? The same package gives you a handler wrapper:

// pages/api/checkout.ts
import { withBotProtection } from '@webdecoy/nextjs'
import type { NextApiRequest, NextApiResponse } from 'next'

async function handler(req: NextApiRequest, res: NextApiResponse) {
  res.json({ ok: true }) // req.webdecoy holds the detection result
}

export default withBotProtection(handler, {
  apiKey: process.env.WEBDECOY_API_KEY!,
  blockThreshold: 70,
})

One decision from three signals

The point of three layers is that they cover each other's blind spots. Combine them into a single verdict rather than three disconnected checks:

// app/lib/decide.ts
type Signals = {
  edgeScreened: boolean   // passed the edge user-agent and rate gate
  honeypotHit: boolean    // touched a trap at any point
  threatScore: number     // 0 to 100, from result.detection.confidence
}

export function decide(s: Signals): 'allow' | 'challenge' | 'block' {
  if (s.honeypotHit) return 'block'           // touched a trap: automated by definition
  if (s.threatScore >= 70) return 'block'     // handshake or local signals say automation
  if (s.threatScore >= 40) return 'challenge' // suspicious, verify before trusting
  return 'allow'
}

A naive HTTP scraper trips the edge gate. A link-following scraper that lies about its user agent trips a honeypot. A polished headless browser that avoids the traps trips the fingerprint. To get past all three, a bot has to be honest, careful, and use a real browser TLS stack at the same time — a much smaller and more expensive population than the flood you started with.

Vercel BotID versus a self-hosted stack

If you are on Vercel you have probably seen BotID, the invisible bot-detection product powered by Kasada. It is genuinely good, and worth knowing where it fits.

BotID is a managed black box. You enable it on the routes you want protected and it makes a verdict at the edge, with no signals to inspect and no logic to tune. That is the appeal and the limitation: strong detection with almost no code, in exchange for visibility into why a request was flagged, portability off Vercel, and the ability to combine the verdict with your own honeypots and scoring. It is also a paid feature once you scale.

The self-hosted stack in this guide is the opposite trade: more code and more moving parts, in return for portability to any host, transparency about every signal, and thresholds that are yours to tune. They are not mutually exclusive — some teams run BotID on checkout and login for the managed guarantee, and run the edge gate plus honeypots plus origin fingerprinting everywhere else for coverage and insight.

Production checklist

• Scope the matcher. Never run middleware on _next/static, images, or other assets. Wasted compute, and it can break caching.
• Do not hard-block on user agent alone. Treat it as the cheap first pass, then escalate. A single spoofed header should not be enough to ban a visitor.
• Allow the good bots on purpose. Verify Googlebot and Bingbot by reverse DNS rather than trusting the user-agent string, and decide deliberately which AI crawlers you keep. Some AI search engines send referral traffic worth having.
• Watch your false positive rate. Log every block and challenge with the reason, and review the challenge bucket. If real users land there, loosen the thresholds in decide().
• Fail open, not closed. If the fingerprint service is briefly unreachable, decide whether a timeout should allow or challenge. For most sites, allowing on timeout beats locking out real customers.
• Measure the bill. The whole point was bandwidth and compute. Watch the usage graph for a week after launch so you can prove the layers are paying for themselves.

Wrapping up

The shape that works in Next.js is layered: a cheap edge gate that screens and rate limits, honeypot routes that catch the bots that lie, and an origin fingerprint check for the signal the edge cannot see. None of it requires a separate WAF or DNS surgery, and the one real constraint — that TLS fingerprinting cannot happen in edge middleware — is a reason to move that check to the origin, not a reason to skip it.

What is your site seeing from AI crawlers lately? Curious whether others are blocking outright or rate limiting and keeping the referral traffic.

---

Originally published at webdecoy.com.

Related reading:

• Detect AI Scrapers: Block GPTBot, ClaudeBot & More
• JA4 Fingerprinting for AI Scraper Detection
• Honeypot Traps: Forms, Buttons & Endpoints