Building FoxyInvoice — Chapter 11: Reach — SEO, AI crawlers, and being the machine-readable answer

This series is written in the open, from a real production system. This chapter is the reach stack: prerendering, robots.txt, llms.txt, IndexNow, and measuring it all with access logs. [All chapters and diagrams live in the public repo.]

Somewhere between "launched" and "learned people exist," every product
discovers distribution. Ours arrived with a twist that defines this
chapter: half your future traffic now arrives via machines that never
render your app — search crawlers that mostly cope, and AI answer
engines (GPTBot, ClaudeBot, PerplexityBot) that flat-out don't run
JavaScript. If your site is a client-side SPA, those engines see a
blank page with a title. This chapter is everything we built so the
crawlable half of the internet can actually read the product — and how
we measure it.

The problem, precisely

The FoxyInvoice app is an Angular SPA: the server ships an empty shell
and JavaScript builds the page. Human browsers: fine. Googlebot:
tolerant, eventually. AI crawlers: blind. Fetch the marketing page
with curl — which is exactly what a crawler does — and you got:

<title>Invoicing</title>
<app-root></app-root>

One word and an empty div. To every machine reader, our entire free
invoice-template business was the word "Invoicing."

The fix stack, bottom to top

1. The allowlist robots philosophy

Our robots.txt doesn't list what's forbidden — it lists what's
allowed, then disallows everything else:

User-agent: *
Allow: /$ /pricing /privacy /terms /templates /templates/*
Disallow: /

Every future route — admin consoles, /upgrade, QA harnesses — is
non-indexable by default until deliberately made public. The same
policy is enforced server-side with X-Robots-Tag: noindex, nofollow
headers at the edge proxy, per URL: private pages carry it; public
pages don't. Belt and suspenders, because robots.txt disallow alone
never guarantees de-indexing.

2. Prerendering at build time (the pragmatic middle)

Full server-side rendering means an always-on SSR server — violating
our boringness constraint — for pages that are 95% app shell. Instead,
a post-build script generates static, crawler-ready copies of every
public route when the SPA compiles:

• Real <title> and meta description per page ("Free Plumbing Invoice Template — FoxyInvoice")
• Canonical URL, Open Graph tags, JSON-LD structured data
• Full article body copy inside the shell — line items, how-to steps, FAQ — because meta tags describe content; answer engines quote content
• The build fails if the sitemap and the template data drift apart (a one-line count check that caught a real bug on its first run)

The edge proxy's try_files serves these static files to anything that
fetches the URL; real browsers still get the app (Angular replaces the
static body on boot). No new server, no SSR framework, crawler-complete
pages.

3. llms.txt — a menu for answer engines

The emerging convention: a markdown file at the root telling AI systems
what the product is and linking its key pages, in their vocabulary.
Ours lists the product, every template page with a one-line
description, and contact points. Cheap, human-readable, and exactly
the artifact a "recommend an invoicing tool" query wants.

4. IndexNow: push instead of wait

Crawlers traditionally rediscover content on their own schedule.
IndexNow flips it: on every deploy, the pipeline POSTs all sitemap
URLs to the alliance endpoint (Bing-powered — which feeds several AI
answer engines) with a key file proving domain ownership. Our first
ping returned HTTP 202 — accepted — and the pages were in Bing's queue
the same evening.

5. Measurement: access logs are the truth

"You can't improve what you can't see," so the edge now logs every
request as JSON (rotated, bounded). The analysis is one script:
user-agents, paths, crawler classes. Within days we could see search
crawlers reading robots.txt and template pages — and, just as
valuable, probe-noise (stray /wp-admin scans) being correctly
absorbed. Pair this with Google Search Console + Bing Webmaster
registration (a five-minute runbook lives in the repo) for the
indexing-and-impressions view logs can't give you.

The funnel the surface feeds

Reach is only worth building if it lands somewhere. The template
gallery — now crawlable — is the top of a three-slice funnel we shipped
in order:

1. Slice 1 — no-signup value + persistence: the generator works anonymously, autosaves locally (30 days), and offers resume on return.
2. Slice 2 — the signup handoff: "Save & finish online" carries the exact filled invoice through signup into the new account as a real draft.
3. Slice 3 — the soft capture: "Email me this invoice" stores the draft server-side against their address (with a nurture email); if they later sign up with that address, the dashboard converts it into a real invoice automatically.

Each slice was verified in a real browser before shipping — the funnel
is the one place where "should work" is banned.

---

Recap. Serve machines real HTML (prerendered at build), allowlist
what's public (default-closed), publish llms.txt, push updates via
IndexNow, and measure with access logs + Search Console. The era of
"GEO" — being the machine-readable answer — is just SEO where the
reader never renders your JavaScript.

---

Reading this and want to be found the way this series was? Create a free workspace at
foxyinvoice.com, then redeem founding code
U8B4Z8S87X on the Upgrade page — 6 months of Pro, free, no card. If anything
breaks, there's a feedback button in the app. I read every one.

---

Next: Chapter 12 — Engagement & automation: UI first, then automate.