Google Indexed the Blog and Nothing Else on My Flutter Web Site

On 10 September, Search Console's verdict on this site was 18 pages indexed and
401 not. Four hundred of those sat in "Discovered — currently not indexed",
which is Google saying it knows the URL exists and hasn't bothered to fetch it.

The breakdown was stranger than the count. Of the 23 pages that had ever
appeared in a search result, 22 were blog posts and the 23rd was the home page.
Not one app, tool, game or package page had a single impression, and /apps,
the page that links to 96 of them, read "URL is unknown to Google".

"Discovered — currently not indexed" usually gets blamed on crawl budget or a
young domain. Neither explains why one kind of page gets in and every other
kind doesn't. Something about the blog was different.

Flutter web draws text Google can't read

The site is Flutter web with the CanvasKit renderer, which paints everything,
text included, onto a <canvas>. To a crawler, a canvas full of words is an
image with no alt text.

The usual answer is to ship the words separately. A build step
(tool/prerender.py here) writes a real HTML file for every route, with the
page's text and links inside the served document:

<div id="prerendered-content">
  <h1>Apps</h1>
  <p>…</p>
  <a href="/apps/helm">Helm</a> …
</div>

It's clipped to one pixel rather than hidden with display: none, so it stays
in the document for crawlers and screen readers without being seen. curl any
route and the content is there. That check passed on every page from the day
it shipped.

The part curl can't see

Google doesn't index the HTML the server sends. It runs the page's JavaScript
and indexes the DOM that results. For a Flutter app, that means main.dart.js
boots and paints a first frame.

And index.html had this, written alongside the prerendering:

window.addEventListener('flutter-first-frame', function () {
  // ...fade out the loading screen...
  var seo = document.getElementById('prerendered-content');
  if (seo) { seo.remove(); }
});

A comment in the stylesheet explained why: the crawler copy goes "the moment
Flutter paints the same content properly". That's true for a person looking at
the screen. It isn't true for the DOM. Under CanvasKit, "painted properly" means
pixels, so once the handler ran, the document held a canvas and nothing else:
zero words and zero links on every route.

Every route except blog posts. Posts carry a second copy: when a post opens,
the app adds its body to the document as a hidden <article>. The handler
removed one element by id, and that wasn't it. That is the whole difference
between the 22 pages Google knew and the rest.

It also explains "Discovered". Google had found the other pages through the
sitemap and nothing else. The hub pages that should have linked to them
rendered as empty dead ends, and a URL that only appears in a sitemap is about
the lowest crawl priority there is. So they waited.

Let the app keep the copy

The handler isn't wrong in general; the loading screen should go. The copy just
shouldn't go with it. So the app claims the block before its first frame by
renaming it, and the handler's getElementById finds nothing to remove:

// In main(), before runApp.
final el = web.document.querySelector('#prerendered-content');
if (el != null) {
  el.id = 'seo-prerendered';         // the first-frame handler now misses it
  el.setAttribute('style', clipped); // the clipping CSS was keyed on the old id
  el.setAttribute('aria-hidden', 'true');
}

Three details matter:

• Re-apply the clipping inline. The stylesheet rule that hid the block was keyed on the old id. Rename it without this and the whole crawler copy appears on screen.
• Hide it from screen readers. The app turns on Flutter's semantics tree, which is the accessible version of the page. With both exposed, a screen reader would read every page twice.
• Drop it on navigation. The block describes the route that was served. When the visitor moves to another route inside the app, it's removed, so the document never carries text for a page that isn't showing.

It's the same content the app paints, so this isn't a different page for
Google. The crawler just gets the text version of the page instead of a picture
of it.

What changed

Search Console's live test afterwards returned "URL is available to Google",
and the rendered HTML of /apps had 159 words and 96 links instead of none.

Then indexing was requested for eight pages: the home page, the six hubs
(/apps, /tools, /blog, /games, /work, /packages) and one app page.
Not the 400 leaves. Requests are capped at roughly ten a day, and one crawled
hub hands Google about 96 links, which is how leaves get discovered without
being requested one at a time.

By 14 September: 99 indexed, up from 18. It isn't a controlled experiment,
since the fix and the requests landed in the same week, but requesting an empty
hub would have handed Google nothing to follow. And 330 pages are still in
"Discovered — currently not indexed", so this is a slope, not a switch. The
queries now arriving are for tools and posts ("flutter treemap", "tone
generator"), the kinds of page that used to be invisible.

Check the rendered page, not the response

If you ship Flutter web with prerendered HTML, curl is not the test that
matters:

1. Search Console → URL Inspection → Test live URL → View tested page → HTML. That's the DOM Google indexes. Search it for a sentence from your page.
2. Or open the page, wait for it to paint, and run document.body.innerText.length in the console. If it's near zero on a page full of text, a crawler sees the same nothing.
3. Search index.html for anything that removes elements on flutter-first-frame, and ask what exactly it removes.

The served HTML was right the whole time. The page deleted it before the one
reader who mattered got there.

---

Originally published at devshakib.jumyn.com. I write about Flutter, Dart and the parts of shipping that are genuinely awkward — and publish the packages that came out of them at pub.dev/publishers/jumyn.com.