This series is written in the open, from a real production system. This chapter is the reach stack: prerendering, robots.txt, llms.txt, IndexNow, and measuring it all with access logs. [All chapters and diagrams live in the public repo.]
Somewhere between "launched" and "learned people exist," every product
discovers distribution. Ours arrived with a twist that defines this
chapter: half your future traffic now arrives via machines that never
render your app — search crawlers that mostly cope, and AI answer
engines (GPTBot, ClaudeBot, PerplexityBot) that flat-out don't run
JavaScript. If your site is a client-side SPA, those engines see a
blank page with a title. This chapter is everything we built so the
crawlable half of the internet can actually read the product — and how
we measure it.
The problem, precisely
The FoxyInvoice app is an Angular SPA: the server ships an empty shell
and JavaScript builds the page. Human browsers: fine. Googlebot:
tolerant, eventually. AI crawlers: blind. Fetch the marketing page
with curl — which is exactly what a crawler does — and you got:
<title>Invoicing</title>
<app-root></app-root>
One word and an empty div. To every machine reader, our entire free
invoice-template business was the word "Invoicing."
The fix stack, bottom to top
1. The allowlist robots philosophy
Our robots.txt doesn't list what's forbidden — it lists what's
allowed, then disallows everything else:
User-agent: *
Allow: /$ /pricing /privacy /terms /templates /templates/*
Disallow: /
Every future route — admin consoles, /upgrade, QA harnesses — is
non-indexable by default until deliberately made public. The same
policy is enforced server-side with X-Robots-Tag: noindex, nofollow
headers at the edge proxy, per URL: private pages carry it; public
pages don't. Belt and suspenders, because robots.txt disallow alone
never guarantees de-indexing.
2. Prerendering at build time (the pragmatic middle)
Full server-side rendering means an always-on SSR server — violating
our boringness constraint — for pages that are 95% app shell. Instead,
a post-build script generates static, crawler-ready copies of every
public route when the SPA compiles:
- Real
<title>and meta description per page ("Free Plumbing Invoice Template — FoxyInvoice") - Canonical URL, Open Graph tags, JSON-LD structured data
- Full article body copy inside the shell — line items, how-to steps, FAQ — because meta tags describe content; answer engines quote content
- The build fails if the sitemap and the template data drift apart (a one-line count check that caught a real bug on its first run)
The edge proxy's try_files serves these static files to anything that
fetches the URL; real browsers still get the app (Angular replaces the
static body on boot). No new server, no SSR framework, crawler-complete
pages.
3. llms.txt — a menu for answer engines
The emerging convention: a markdown file at the root telling AI systems
what the product is and linking its key pages, in their vocabulary.
Ours lists the product, every template page with a one-line
description, and contact points. Cheap, human-readable, and exactly
the artifact a "recommend an invoicing tool" query wants.
4. IndexNow: push instead of wait
Crawlers traditionally rediscover content on their own schedule.
IndexNow flips it: on every deploy, the pipeline POSTs all sitemap
URLs to the alliance endpoint (Bing-powered — which feeds several AI
answer engines) with a key file proving domain ownership. Our first
ping returned HTTP 202 — accepted — and the pages were in Bing's queue
the same evening.
5. Measurement: access logs are the truth
"You can't improve what you can't see," so the edge now logs every
request as JSON (rotated, bounded). The analysis is one script:
user-agents, paths, crawler classes. Within days we could see search
crawlers reading robots.txt and template pages — and, just as
valuable, probe-noise (stray /wp-admin scans) being correctly
absorbed. Pair this with Google Search Console + Bing Webmaster
registration (a five-minute runbook lives in the repo) for the
indexing-and-impressions view logs can't give you.
The funnel the surface feeds
Reach is only worth building if it lands somewhere. The template
gallery — now crawlable — is the top of a three-slice funnel we shipped
in order:
- Slice 1 — no-signup value + persistence: the generator works anonymously, autosaves locally (30 days), and offers resume on return.
- Slice 2 — the signup handoff: "Save & finish online" carries the exact filled invoice through signup into the new account as a real draft.
- Slice 3 — the soft capture: "Email me this invoice" stores the draft server-side against their address (with a nurture email); if they later sign up with that address, the dashboard converts it into a real invoice automatically.
Each slice was verified in a real browser before shipping — the funnel
is the one place where "should work" is banned.
Recap. Serve machines real HTML (prerendered at build), allowlist
what's public (default-closed), publish llms.txt, push updates via
IndexNow, and measure with access logs + Search Console. The era of
"GEO" — being the machine-readable answer — is just SEO where the
reader never renders your JavaScript.
Reading this and want to be found the way this series was? Create a free workspace at
foxyinvoice.com, then redeem founding code
U8B4Z8S87X on the Upgrade page — 6 months of Pro, free, no card. If anything
breaks, there's a feedback button in the app. I read every one.
Next: Chapter 12 — Engagement & automation: UI first, then automate.
Top comments (1)
Getting only "Invoicing" from a curl fetch makes the prerendering decision concrete. The sitemap/template count check is a useful guardrail; I'd also compare the generated page body with the Angular view in CI so crawlers don't get stale template instructions after the app changes. Access logs and an IndexNow 202 show that discovery is happening, but the stronger product signal is whether people land on a template, make an anonymous draft, and carry that exact invoice through signup.