I run a small multilingual tools site on Next.js and Vercel. About 116 calculators and converters, five languages, pages served with ISR. In late September I added programmatic subpages (one page per value, like a page for each military time), and the sitemap grew from 635 to about 1,185 URLs.
Two weeks later Vercel showed me this on the Hobby plan:
- ISR Reads: 768,741 of 1,000,000 for the rolling 30 days (October 3)
- daily reads went from 13 to 16K in early September to 30 to 84K in late September
Hobby does not bill you for overage. It pauses the feature until the 30-day window frees up. For a site built on ISR that means pages stop being served. So I had about a week.
Who was actually reading
Observability keeps 12 hours on Hobby. In those 12 hours there were 15K CDN requests. The top of the bot list:
| Bot | Requests | Served from cache |
|---|---|---|
| meta-externalagent | 4,000 | 79.9% |
| semrush-site-audit | 2,500 | 44% |
| google-inspectiontool | 429 | 85.8% |
| bingbot | 259 | 77.2% |
| googlebot | 95 | 63.2% |
Googlebot made 95 requests. Meta's crawler made 4,000.
The routes told the rest of the story. Regular pages had 85 to 97% cache hits. The new programmatic subpages were at 43 to 52%. A crawler walking through hundreds of rarely visited URLs keeps missing the regional CDN cache, and every miss falls back to the durable ISR cache, which is what you pay reads for.
One more thing I learned from the docs: each deployment gets its own cache. My site redeploys every night to refresh daily content, so every night the cache starts cold and the morning crawlers pay for it.
What I changed
A single Firewall rule: deny when the user agent contains meta-externalagent, Bytespider, CCBot, PetalBot or GPTBot. Search crawlers like Googlebot, Bingbot and OAI-SearchBot are not in it.
After that, on the programmatic routes:
| Route | Cache hits before | After |
|---|---|---|
| numerology pairs | 43% | 77% |
| military time values | 52% | 94% |
| pet age values | 51% | 69% |
Daily reads dropped to 23 to 29K. Better, but still above what the rolling window needed, because there were hourly spikes of 3 to 6K that I could not attribute to anyone within the 12-hour log window.
Later the Firewall traffic view showed one more source: a single cloud provider network (Tencent, AS132203) sending about 22% of all requests in a day. Blocking by AS number took care of that.
In the end I moved to Pro anyway. Over a 30-day window the site had 826K ISR reads, while Web Analytics counted about 900 visitors. That ratio is the part I keep thinking about.
The question
For those of you running programmatic pages on ISR: how do you keep the long tail cheap when most of the traffic is crawlers? Would you generate these pages fully static at build time instead, or is there a cache setup that works better for rarely visited URLs?
The pages in question look like this: https://www.utilixi.com/en/converter/military-time
Top comments (0)