See which bots and AI crawlers visit your Next.js site

If you publish a blog, documentation, or a product catalog, it helps to know which automated clients are requesting those pages. Are they identifying themselves as AI crawlers? Which URLs do they visit? Are they fetching content or probing for files that should not be public?

This guide adds WebDecoy to a Next.js application in monitor mode. Requests continue to your site while you collect detections. You can review the traffic before deciding whether any of it needs blocking.

We build WebDecoy. The example below uses its Next.js SDK and cloud dashboard.

Put detection where crawler requests arrive

A browser script can only run when the client executes it. An HTTP crawler can download your HTML without running your React components or analytics script.

For this walkthrough, detection runs on the server, before the page response. That gives it access to requests from clients that do not execute JavaScript. It does not reveal the internal purpose of every visitor, and it does not prove who operates a client merely because its user agent contains a familiar name.

You need a Next.js application with a running server and a WebDecoy API key for dashboard reporting. A static export hosted as files needs collection at its hosting or CDN layer instead.

Try the complete example

Download the runnable Next.js example. Extract it, run npm ci, copy .env.example to .env.local, add your API key, and run npm run dev. It listens on http://127.0.0.1:3107.

The example was built and checked locally with Next.js 16.3.7 and @webdecoy/nextjs 0.18.0. Normal and reserved test requests both returned HTTP 200 in monitor mode. These local checks used no API key; dashboard delivery is a separate check you will perform below.

Install the adapter

The companion example pins the adapter to version 0.18.0:

npm install @webdecoy/nextjs@0.18.0

Create an API key in your WebDecoy dashboard and put it in .env.local:

WEBDECOY_API_KEY=your_api_key

Keep this variable server-side. Do not prefix it with NEXT_PUBLIC_ or place it in a React component. When you deploy, add it to your hosting environment as well.

Without a key, the SDK can run local checks, but those checks will not appear in your cloud dashboard.

Monitor page requests

Next.js 16 uses proxy.ts for this request hook. Place it beside app, or beside src/app when your application uses a src directory:

import { withWebDecoy } from '@webdecoy/nextjs';

export default withWebDecoy({
  apiKey: process.env.WEBDECOY_API_KEY,
  mode: 'monitor',
  // Assumes one trusted proxy in front of the application.
  trustProxy: 1,
});

export const config = {
  matcher: ['/((?!_next/static|_next/image|favicon.ico).*)'],
};

The matcher includes your homepage, articles, documentation, and other page routes. It excludes Next.js static assets, image optimization requests, and the favicon. Add exclusions for other assets if your site needs them, but keep the content pages you want to observe.

mode: 'monitor' lets the request continue even when WebDecoy would otherwise deny it. It does not disable protection implemented elsewhere in your application or hosting provider.

The proxy setting matters for attribution. This example assumes one trusted proxy that sanitizes forwarding headers. Configure it for your actual hosting chain; copying an arbitrary header into the client IP field can produce misleading results.

If you already have a proxy for authentication or redirects, integrate detection into that request flow rather than replacing the existing file. For an older Next.js application using the middleware convention, the adapter can be exported from middleware.ts; do not install both files.

Check that the installation reports a detection

Start your application, then send the reserved test request to a page covered by the matcher:

curl -i -A 'WebDecoy-Test/1.0' http://localhost:3000/

Use port 3107 if you are running the companion example.

Your page should still load. Open Detections in WebDecoy and look for the record labeled Test. The reserved test user agent checks the reporting pipeline; it is not evidence that a real AI crawler visited your site. The installation documentation describes this test and its dashboard label.

If nothing appears, check the API key, restart the development server after changing environment variables, and confirm that the request path matches your configuration. A successful page response alone does not establish that reporting worked. Check server errors as well.

Look at actual crawler activity

After deploying, return to Detections and the AI traffic views as real requests arrive. Start with a few records and inspect:

• The requested path. An article request and a request for /.env mean different things.
• The user agent and crawler classification. A recognizable name is useful context, but it is also a string a client can copy.
• The supporting signals and any identity verification shown. Keep a claimed identity separate from one supported by verification.
• The timing and repeated requests. A pattern across several paths can be more useful than a single score.

The SDK's detections are not a complete page-view log. Some requests can be allowed locally without being reported, and SDK caching can affect how often a decision reaches the service. Use access logs when you need a complete request count.

If an upstream CDN serves a cached response without reaching your Next.js application, this sensor will not see that request. To understand that traffic, collect at the CDN layer or use its access logs.

Decide what to do with what you find

You can leave the application in monitor mode while you learn which traffic is useful. A search crawler fetching public pages may be welcome. Repeated probes for secrets deserve a different response.

Start with the evidence in the records, then choose a policy for the specific behavior you want to change. You do not need to block every AI crawler to get value from knowing which ones are visiting.

The AI detection guide explains the available categories. Browser automation and browser-side AI signals require different evidence from a server request; this setup is not a claim that every kind of AI activity is identifiable.

Originally published on WebDecoy. This tutorial was prepared with AI assistance. The local checks performed and their limits are described above.