DEV Community: Bob

When Google Search Console Couldn’t Fetch My Sitemap, I Added an HTML Sitemap Fallback

Bob — Wed, 03 Jun 2026 09:35:08 +0000

Quick Summary: If GSC gives you a cryptic Could not fetch error on a technically valid XML sitemap, stop rewriting the XML too early. In this case, I controlled variables across Cloudflare Pages, Vercel, Cloudflare Workers/OpenNext, and domain changes, which pointed to a production-domain or GSC-state anomaly. The practical SEO fallback was to add a crawlable HTML sitemap.

I recently spent a lot of time debugging a sitemap problem that looked simple at first.

Google Search Console kept saying:

Could not fetch

The strange part was that the sitemap worked everywhere else.

It returned 200 OK. It had the correct application/xml content type. The XML was valid. robots.txt pointed to it. Browser access worked. Command-line checks worked.

But Google Search Console still refused to read it.

This post is a write-up of the debugging process: what I checked, what I ruled out, why the hosting platform was probably not the final root cause, and why I eventually added an HTML sitemap as a crawl-discovery fallback.

The Site Setup

The project is a Next.js tool site for browser-side video and image processing.

It has:

mostly static SEO/tool pages
localized routes
no login
no database
no payment system
client-side media processing
some heavier runtime assets served through the Cloudflare ecosystem

The production deployment was on Cloudflare Pages.

The XML sitemap was generated by the app and included localized URLs across supported languages.

At a high level, this should have been a boring sitemap setup.

It was not.

The Initial Symptoms

In Google Search Console, both the dynamic XML sitemap and a static XML sitemap showed the same failure:

Could not fetch

The discovered page count stayed at zero.

The frustrating part was that direct HTTP checks looked normal:

/sitemap.xml        -> 200 application/xml
/sitemap-static.xml -> 200 application/xml
/robots.txt         -> 200 text/plain

Both XML files passed validation.

The static sitemap was especially important. It was a plain file under public/, not a Next.js metadata route. If both the generated sitemap and a static XML file failed in GSC, then the issue was probably not limited to the Next.js sitemap route.

Checking robots.txt

The robots.txt file was simple:

User-Agent: *
Allow: /

Sitemap: /sitemap.xml

There was no disallow rule blocking the sitemap or the main pages.

So the obvious robots explanation did not fit.

Checking Cloudflare DNS and Custom Domains

Because the site was on Cloudflare Pages, I spent a lot of time checking Cloudflare configuration.

The production domain and www domain were active in Cloudflare Pages. SSL was enabled. The DNS records pointed to the Pages deployment and were proxied through Cloudflare.

I also checked old verification records, email records, custom domain status, and the basic DNS setup.

Nothing obvious looked broken.

The domain resolved. The site loaded. The sitemap returned 200. The custom domains were active.

So the basic Cloudflare Pages domain setup did not explain why GSC could not fetch the sitemap.

Checking Cloudflare Bot and Security Rules

The next suspicion was that Googlebot might be getting challenged or blocked by Cloudflare.

I checked Cloudflare AI Crawl Control and security events.

Googlebot was recognized as a search engine crawler. The relevant controls were not blocking it.

Cloudflare Security Events also showed requests to the sitemap path from verified crawlers, including Google-related user agents.

There was a custom rule for verified bots:

cf.client.bot

The rule skipped security products for verified bots.

The important part: I did not find evidence of:

WAF block
Managed Challenge
JS Challenge
Interactive Challenge
Googlebot being denied access

Googlebot-style requests could receive 200 responses.

That made a simple Cloudflare security-block explanation unlikely.

Checking HTTP Details

I also checked the sitemap response in several ways.

The sitemap returned:

200
application/xml

I checked normal requests, Googlebot-style user agents, HTTP/1.1 behavior, compressed responses, and XML validation.

The XML remained valid.

The dynamic sitemap had Next.js route headers, as expected. But the static sitemap also failed in GSC, which again suggested that the issue was not just the Next.js metadata route.

At this stage, the sitemap looked technically valid from the outside.

Google Search Console still disagreed.

Fixing Build and Deployment Noise

During the investigation, I also found unrelated build noise.

The Cloudflare build could fail because next/font/google tried to fetch fonts during the build. That was not directly the sitemap bug, but it made deployment verification less stable.

I removed the Google Fonts dependency and switched to a system font stack.

After that, both the normal Next build and the Cloudflare build completed successfully.

This mattered because I needed a stable deployment baseline before blaming Google, Cloudflare, or the domain.

The Vercel Diagnostic Test

At one point, I deployed the same project to Vercel as a diagnostic comparison.

The goal was not to move production to Vercel.

The goal was to answer a narrower question:

Is the sitemap XML/project output itself fundamentally broken?

The deployment itself worked.

But the key detail is this: when the production domain was used, Google Search Console still could not fetch the sitemap.

That meant simply changing the hosting platform was probably not enough.

Later, I tested the same project with a different temporary domain. Under that different domain, Google Search Console could fetch the sitemap successfully.

That changed the interpretation.

The issue was probably not just:

Cloudflare Pages vs Vercel

The stronger signal was:

The failure was likely tied to the production domain or Google/GSC state associated with that domain.

This distinction matters. Without it, it is easy to draw the wrong conclusion and think that a hosting migration alone would fix everything.

Here is the simplified control-variable table:

Deployment path	Domain used	GSC status	What it suggested
Cloudflare Pages	Production domain	Failed: `Could not fetch`	Not explained by basic Cloudflare DNS, SSL, WAF, or robots settings
Vercel diagnostic deployment	Production domain	Failed: `Could not fetch`	Moving the same project to Vercel did not fix the production-domain problem
Same project on a different temporary domain	Temporary domain	Success	The sitemap output was likely valid; the production domain or GSC state became the stronger suspect
Cloudflare Workers + OpenNext	Production domain	Failed: `Could not fetch`	Swapping backend infrastructure did not fix the production-domain problem

Trying Cloudflare Workers and OpenNext

Because the old Cloudflare Pages build chain used @cloudflare/next-on-pages, and that adapter is deprecated, I also tested a Workers/OpenNext path.

This was not a casual check. I actually went through the deployment path:

added OpenNext/Workers configuration
configured wrangler
tested a Workers custom domain
confirmed the Worker was serving traffic
saw the x-opennext response header
tested the homepage
tested robots.txt
tested the sitemap
tested the runtime routes needed by the app

At first, the Worker test domain worked.

Then I tried switching the production domain from Pages to Workers.

That required removing the production custom domains from Pages and adding them to the Worker, because Cloudflare would not allow the same hostname to be managed by both at once.

After the switch, the production domain did serve through Workers/OpenNext.

The key routes still returned valid responses:

/              -> 200
/sitemap.xml   -> 200 application/xml
/robots.txt    -> 200 text/plain

The response headers confirmed traffic was going through OpenNext Workers.

Then I submitted the sitemap again in Google Search Console.

It still failed.

I also tried a cache-busting sitemap URL with a query string.

That URL returned valid XML outside GSC.

GSC still said it could not fetch it.

That was an important result.

It meant the original hypothesis was not supported:

This was not simply a Cloudflare Pages or next-on-pages problem.

The same production domain still had the issue even after moving the delivery path to Workers/OpenNext.

The Strongest Conclusion

After all of these tests, the most likely root area was not the XML file itself.

It was also not clearly one hosting provider.

The strongest clue was the domain test:

same project
same kind of sitemap
different domain
GSC could fetch it

That points toward a domain-level or Google-side state issue.

Possible explanations include:

historical crawl state for the production domain
Google Search Console state for the domain property
DNS or routing history associated with the domain
Google-side host classification or cache

I cannot prove exactly which one it is.

But the evidence pointed away from endlessly rewriting the sitemap XML.

The Practical Problem

Even if the XML sitemap is valid, it is not very helpful if Google Search Console refuses to process it.

The site still needs its important pages discovered.

For a multilingual tools site, that matters.

So I stopped treating the XML sitemap as the only discovery mechanism.

I added an HTML sitemap.

Why an HTML Sitemap Helps

An HTML sitemap is just a normal page with internal links.

Googlebot can crawl it like any other page.

That gives the site another discovery path:

normal page -> footer link -> HTML sitemap -> localized tool pages

This does not fix the XML sitemap failure directly.

It reduces the risk of relying on only one discovery mechanism.

That was the practical goal.

How I Designed the HTML Sitemap

I kept the page intentionally simple.

The HTML sitemap:

returns plain HTML
is linked from the footer
uses index, follow
has a canonical URL
groups links by language
lists only the core tool pages
excludes privacy policy and terms pages
avoids duplicate homepage entries

The page is not trying to be a fancy user interface.

It is a reliable crawl hub.

What the HTML Sitemap Contains

The site has multiple languages and a fixed set of core tools.

The HTML sitemap lists the localized version of each core tool page.

In the current setup:

9 languages x 11 core tools = 99 tool links

For the homepage converter, the link is represented by each language's localized tool title, instead of repeating a generic brand link.

That keeps the sitemap focused and avoids unnecessary duplicates.

Why I Did Not Just Move Hosting

Moving hosting would have been the wrong lesson.

The production domain still failed even when the project was served through a different deployment path.

A different temporary domain worked.

That means the domain/GSC state was the more important signal.

Also, the product uses browser-side media processing and heavier runtime assets that fit well with the existing Cloudflare setup.

So the better solution was not to migrate everything just because one diagnostic deployment behaved differently.

The better solution was to add a second crawl-discovery path.

What I Learned

Sitemap debugging is not always about the sitemap file.

Sometimes:

the XML is valid
the headers are correct
the route is public
bots are not blocked
multiple hosting paths work technically

and Google Search Console still reports a fetch failure.

At that point, adding another discovery mechanism can be more useful than continuing to tweak a valid XML file.

For this case, the final strategy was:

keep the XML sitemap
keep monitoring GSC
add an HTML sitemap
link it from the footer
make the important localized pages discoverable through ordinary internal links

Final Setup

The live site discussed in this post is:

Main site: https://videosnap.cc/
HTML sitemap fallback: https://videosnap.cc/html-sitemap

The HTML sitemap is not a replacement for the XML sitemap.

It is a crawl-discovery fallback.

And in this case, that was the most practical solution.

[Fixed] How to Solve the 99% Hang in ffmpeg.wasm Apps

Bob — Fri, 08 May 2026 10:29:30 +0000

TL;DR: The hang is caused by memory overlap. Delete your input file before reading the output.

I’ve been building VideoSnap, a tool that processes video entirely in the browser using ffmpeg.wasm. For a long time, I was haunted by a specific, frustrating bug: the "99% Trap."

A user uploads a file, the progress bar climbs smoothly, hits 99%... and then everything just stops.

The UI becomes unresponsive. The fan starts spinning. It doesn't crash with an "Aw, Snap!" error, but it hangs there, sometimes for minutes. Then, suddenly, the download pops up as if nothing happened.

I realized that the 99% mark isn't where FFmpeg is working—it's where the browser is fighting for its life.

The 99% isn't FFmpeg—it's the Handover

In ffmpeg.wasm, the progress bar tracks the FFmpeg execution. When it hits 99%, the heavy lifting of transcoding is actually done.

The "hang" happens during the handover: when you call engine.readFile() to pull the processed video out of the WebAssembly virtual memory (MEMFS) and into the JavaScript heap.

The "Memory Overlap" Problem

WebAssembly (currently) has a hard 32-bit memory limit (effectively ~2GB). Imagine you are converting a 500MB video:

The Peak: At 99%, the WASM memory is holding your 500MB input file PLUS the newly generated 500MB output file. That’s 1GB of WASM memory occupied.
The Request: You call engine.readFile(). JavaScript now tries to allocate a new 500MB Uint8Array to copy that data.
The GC Storm: Your browser is now trying to manage nearly 1.5GB to 2GB of massive, contiguous memory blocks.

This triggers a "Stop-The-World" Garbage Collection (GC) event. The browser's Main Thread locks up completely. It is desperately trying to defragment memory to find a 500MB hole. This intense "GC thrashing" is why the UI freezes before the file finally breaks free.

The "Surgical" Fix: Breaking the Overlap

Once I understood that the stall was caused by the simultaneous existence of the input and output files in MEMFS, the fix became obvious.

I needed to clear the desk before trying to move the big box.

I implemented what I call Surgical Memory Management:

// The optimized handover logic:

// 1. FFmpeg is done. Before we even THINK about reading the output, 
// we must kill the input file to free up hundreds of MBs in WASM.
await engine.deleteFile('input.mp4'); 

// 2. Now that the WASM memory has breathing room, we read the result.
// The browser can allocate the JS buffer without a massive GC fight.
const data = await engine.readFile('output.mp4');

// 3. The millisecond we have the data in JS, we nuke the WASM output copy.
await engine.deleteFile('output.mp4');

// 4. Now WASM is empty, and we only hold the file in the JS heap.
const blob = new Blob([data.buffer], { type: 'video/mp4' });

By reordering these deletions, I eliminated the massive memory overlap at the exact moment the browser needs memory the most. The 99% hang doesn't magically vanish—it still takes time for the browser to allocate large JS buffers—but this surgical cleanup shaves off crucial seconds of GC thrashing. More portantly, it keeps the browser tab from quietly suffocating under heavy files.

Why I didn't use WORKERFS or OPFS

I explored other options, but they all had catch-22s:

WORKERFS: It mounts files without copying them, which sounds perfect. But it uses a synchronous I/O bridge that makes FFmpeg run significantly slower. I traded memory for a massive speed penalty. Not worth it.
OPFS (Origin Private File System): This is the future. It streams data directly to disk. But it requires a custom-built FFmpeg core with WASMFS support, which is a massive engineering undertaking that the official @ffmpeg/ffmpeg doesn't support out-of-the-box yet.

The Takeaway: Know Your Handovers

If you are building high-performance WebAssembly apps, remember: the most dangerous part of the pipeline is the data handover.

When you move large amounts of data between the WASM "world" and the JS "world," the browser doesn't see a file—it sees a massive, contiguous memory allocation request. If you don't clean up your internal state before you make that request, you're asking for a GC storm.

VideoSnap is now significantly more stable, not because I made the math faster, but because I managed the memory lifecycle with more precision.

I’m the builder of VideoSnap. I write about the messy reality of building high-performance tools in the browser. Follow for more deep dives.

Why Prompt-Only Moderation Failed in My AI Generation App

Bob — Fri, 10 Apr 2026 06:50:29 +0000

When I first added moderation to my AI generation app, I treated it as a text problem.

That seemed reasonable at the time. A user sends a prompt, I check the prompt, and if it looks unsafe, I block the request before it reaches the model.

That approach worked for a very short time.

It stopped working the moment I supported image inputs, reference images, and multiple generation flows. At that point, I realized something important: prompt-only moderation is not really moderation. It is just one partial check inside a much larger pipeline.

This post is about what changed in my backend once I accepted that.

The mistake: treating moderation as a wrapper

A lot of AI products start with moderation as a thin wrapper around generation:

receive a prompt
run a text safety check
call the model provider
return the result

The problem is that real generation workflows are rarely that simple.

Once users can upload source images, provide reference images, or switch between text-to-image and image-to-image generation flows, the prompt becomes just one component of the overall request. A completely harmless prompt can still be paired with problematic input images. If the backend only inspects the text, the system will inevitably have a blind spot.

That was the first issue I had to fix.

Moderation belongs inside the generation pipeline

I ended up moving moderation into the backend generation workflow itself instead of treating it as a separate utility.

Conceptually, the flow became:

validate the request
load the selected provider and model
inspect both prompt text and image inputs
block flagged requests before spending credits
create the generation task only if moderation passes

That decision helped for two reasons.

First, it kept moderation close to the actual business rules. I did not want unsafe requests to consume credits, create external jobs, or leave behind half-failed task records.

Second, it forced me to normalize the input shape. Instead of only thinking in terms of prompt, I had to define a moderation input that could include prompt text, image URLs, model context, and generation scene.

That made the system much easier to reason about.

Prompt checks are useful, but incomplete

Text moderation is still valuable. It catches a lot of obvious cases early, and it is usually cheaper and faster than processing images.

But text-only checks have two major limitations.

The first is obvious: users can submit problematic visual input even if the prompt itself looks harmless.

The second is less obvious: language coverage is uneven. Depending on the moderation provider, some languages are better supported than others. That means your confidence level should not be the same across all prompts.

In my case, that pushed me toward a more defensive design: if text checks are incomplete, the rest of the safety system has to acknowledge that limitation instead of pretending the problem is solved.

Images changed the design

The biggest improvement came from treating image inputs as first-class moderation targets.

That sounds straightforward, but it changed several implementation details:

the moderation step now had to collect image URLs from different request fields
the backend needed one normalized moderation interface, even if the underlying provider had different APIs for text and image checks
moderation results had to return structured categories and scores, not just a single boolean
failure behavior had to be explicit

That last point matters more than it seems.

If a moderation provider fails, what should happen?

You have to choose between two imperfect options:

fail-open: allow the request and accept some risk
fail-closed: block the request and accept some false positives or degraded UX

There is no universal correct answer. It depends on the kind of product you are building, your abuse tolerance, and how costly a bad generation is for you. But the important part is to make the decision deliberately. Silent fallback logic is where safety systems get weak.

Provider-specific APIs should not leak everywhere

Another lesson was that moderation providers should be isolated behind a small internal interface.

Not because provider abstraction is fashionable, but because safety logic tends to spread if you let it.

If one route handler knows how text moderation works, another knows how image moderation works, and a third knows how to interpret provider-specific category names, you do not have a moderation layer anymore. You have moderation fragments.

I found it much cleaner to keep a moderation manager in the backend and let the generation route ask one question: “Is this request safe enough to proceed?”

That does not remove complexity. It contains it.

The practical takeaway

The most useful shift in my thinking was this:

Moderation is not a feature attached to generation. It is part of generation.

Once I started treating it that way, the backend became easier to evolve. I could add checks for both prompt text and image inputs, make blocking decisions before credits were consumed, and keep provider-specific moderation details out of the rest of the app.

I am using this approach while building videoflux.video, where one workflow needs to support AI image and video generation without assuming that a prompt alone tells the full safety story.

Disclosure: I’m the builder of videoflux.video.