How to track AI crawlers in Next.js starts on the server, where the original request IP and user agent are available.
Clics provides a crawler package that checks both signals against supported providers.
Prepare the project
Create a crawler token in the production project.
Store the project ID and crawler token as server environment variables. Never expose the token in client code.
Install @clicsdev/crawler with npm, pnpm, Yarn, or Bun.
Add tracking
Use proxy.ts for Next.js 16 and middleware.ts for earlier versions.
Call trackClicsCrawlerRequest with the request, NextFetchEvent, project ID, and token.
The event lets the package use waitUntil so tracking can finish in the background.
Exclude API routes, Next.js static assets, optimized images, and the favicon. Keep crawler-facing content such as robots.txt, llms.txt, and sitemaps inside the matcher.
Verify identity
Deploy the middleware to production.
Open AI Analytics and select AI crawlers.
A copied crawler user agent from your own IP should not create a record. Clics requires both the source IP and user agent to match.
Use the result
Request-only tracking identifies the page.
Response-aware tracking can also record the returned HTTP status.
Review documentation, comparisons, product pages, crawler-facing files, redirects, missing pages, and server errors.
Keep the crawler report separate from AI referral traffic. A referral is a human click identified through the referrer. A crawler is an automated request.
Crawler data proves access to the page, not indexing, citation, answer usage, or training.
Top comments (0)