I run TapArcade, a free browser-game arcade. It's a
PHP/MySQL app behind Caddy in Docker, fronted by Cloudflare, on the smallest VPS
tier I could find. At 2,700+ game pages, the things that broke were not the things
I expected.
The setup
- PHP app served by Caddy in a container
- Caddy reverse-proxies to the app container on port 80
- Cloudflare in front, origin only speaks HTTP
- Sitemap generated on demand: home + 19 category pages + 2,700 game pages = 2,721 URLs
Nothing exotic. The interesting part is what happens at that page count.
1. Static files in the web root are load-bearing
Search engines verify site ownership by asking you to drop a file in the web root
and then fetching it. google<hash>.html, BingSiteAuth.xml, whatever.
My deploy strategy was "build into a staging directory, then rsync over the live
directory". That works right up until the day someone adds --delete and wipes
every verification file in the root, because they were never in the build output.
Two verification files went 404 and I spent an evening debugging a sitemap problem
that was actually an ownership-verification problem. The fix was boring: exclude
google*.html and BingSiteAuth.xml from the delete, and assert they still exist
in the staging directory before the swap.
Lesson: if a file's only job is to sit in the web root and be fetched by a bot,
it belongs in your deploy whitelist, not your build output.
2. A sitemap that is served correctly can still never be fetched
The sitemap returned 200, application/xml, correct byte count, valid XML, 2,721
unique URLs, no illegal characters. I checked it fifteen times through Cloudflare
and it never once failed.
The search console still said "couldn't fetch".
Server logs were the answer: the crawler had requested robots.txt, /, and
favicon.ico — and never asked for the sitemap at all. It wasn't a serving
problem. The property was never verified, so nothing downstream mattered.
Lesson: "couldn't fetch" in a search console is not proof that a fetch was
attempted. Check the access log for the sitemap path before you touch your server
config.
3. Category pages need to be different from each other
At 2,700 games, category pages are the only realistic entry points — nobody
lands on game #2,431 from search. Early on, every category page was the same
template with a different grid, which is a thin-content problem at scale.
What helped: each category page got a short editorial intro explaining what the
category actually is and who it's for, plus the grid. Same template, different
copy per category. It doesn't sound like much. It's the difference between 19
duplicate pages and 19 real pages.
4. 2,700 pages is where "just add a CDN" stops helping
Cloudflare fixes latency. It does not fix crawl budget, and it does not fix the
fact that a catalog page with no unique text is a catalog page with no unique text.
The wins at this size came from:
- one sitemap, regenerated when the catalog changes
- category intros that are actually about the category
- per-game pages that say something specific (controls, platform, what it is)
- not shipping a page until it has content worth indexing
Wrap-up
None of this is clever. It's the unglamorous part of running a catalog site: make
the verification files survive deploys, make sure the sitemap is actually being
requested, and don't ship 19 pages that are the same page.
If you want to see the result: TapArcade — 2,700+
free browser games across 19 categories, no download, no account.
Top comments (0)