77% of my Google crawl budget was spent re-downloading the same 222 files

Oleksandr | CalculatorAI — Wed, 05 Aug 2026 15:42:32 +0000

I run a Next.js site on Vercel with about 7,600 URLs in the sitemap. Google had indexed 2,700 and left 4,790 alone. Three thousand of those were sitting in Discovered – currently not indexed, which means Google found the URL and never actually fetched it.

That is the polite way of saying: you are publishing faster than we are crawling.

So I opened the Crawled – currently not indexed report to find out what the crawler was spending its time on instead.

The mistake I nearly shipped

Looking at the example URLs on screen, it looked obvious. Non-English locales everywhere. Clearly Google was skipping my translated pages and I needed to go rework hreflang.

I was about a day of work away from doing exactly that.

Then I exported the CSV instead.

Out of 1,000 URLs in that bucket:

971 were /_next/static/** or /favicon.ico
those 971 URLs pointed at only 222 distinct files
so ro ughly 77% were duplicate crawls of assets Google already had

The locale theory was never in danger. It just happened to be what the top of the list looked like.

Here is why that happens, and it will happen to you too: the examples Search Console shows in the UI are sorted by last crawl date. Your newest real pages sit on top. The bulk of the bucket — which is what you actually want to see — is below the fold and you never scroll to it. Eyeball that list and you will generalise from a biased sample every single time.

Export the CSV before you conclude anything.

What was actually happening

Vercel has a feature called Skew Protection. When you deploy while someone is mid-session, their browser may ask for a chunk that no longer exists in the new build. Skew Protection fixes that by appending ?dpl=<deploy-id> to every static asset URL, so a client keeps getting the build it started with.

It is a genuinely good feature and it solves a real problem.

It also means that every single deploy mints a complete new set of URLs for byte-identical files. And Googlebot does not know that main.js?dpl=aaa and main.js?dpl=bbb are the same thing. It sees two URLs, so it fetches two URLs.

By the time I looked, it had crawled 113 distinct dpl values. One font file, 39 times. The favicon, 37 times.

All of that came out of the same crawl budget those 3,005 real pages were queued in.

The fix, and the fix not to make

Turning Skew Protection off (Vercel dashboard → Settings → Advanced) stops new ones being minted. Asset filenames are already content-hashed, so caching is unaffected. The tradeoff you accept back is the original problem: a tab left open across a deploy can hit a chunk-load error.

For a content site where deploys are frequent and sessions are short, that is a trade worth making. For a long-session app it might not be.

What you should not do is "fix" it with Disallow: /_next/static in robots.txt. I have seen this suggested and it is worse than the disease:

Google needs your CSS and JavaScript to render the page and judge mobile usability. Block them and you are handing it a broken page to evaluate.
It would not drain the queue anyway. robots.txt stops the fetch, not the knowing. The URLs stay known, they just get relabelled as blocked, and everything already queued stays queued.

Fix it at the source.

What I would check on your own site

If you are on Vercel with Next.js and you have more than a few hundred URLs, go and look at how many distinct dpl values Google has crawled. That number is your deploy count wearing a different hat — and unlike most SEO problems, it is entirely under your control.

Which is the part I did not expect. This turned out to be the third independent reason to deploy less often, after ISR write costs and build minutes. I had been treating deploy frequency as a free variable. It is not.

(This came out of building CalculatorAI — a Next.js site with about 7,600 indexed URLs, which is how the numbers above got so specific.)

The honest ending

It has been a few days since I turned it off. The old URLs do not 404 — a query param on a static asset still serves the file — so they do not vanish, they just stop being referenced from any rendered page. My assumption is Google deprioritises them once they stop appearing.

But I am not going to claim the queue has drained, because I do not know yet. Ask me in a month.

DEV Community: Oleksandr | CalculatorAI

77% of my Google crawl budget was spent re-downloading the same 222 files