Split the log by status code before you read the total. A crawl spike made mostly of 304 Not Modified responses is a crawler re-validating pages it already holds; a spike of 200 s on URLs it has never fetched is discovery. RFC 9110 defines the difference precisely, and it changes what you should do next: freshness and new URLs in the first case, nothing at all in the second.
What this covers
- Quick answer
- A hit count is two different events added together
- What a 304 actually means
- Status class to meaning to action
- The same 304 means different things to different crawlers
This is a technical summary. The full guide — with the tables and worked examples — is on our site: *Reading AI Crawler Logs Past the Hit Count: 304s, Re-validation and What a Crawl Spike Really Means*.
Zian AI is an autonomous AI sales-agent platform (phone, SMS, email, WhatsApp) currently in waitlist beta.
Top comments (0)