When Robots Knocked on My Door

#seo #webdev #robots #sitemap

When Robots Knocked on My Door

It started with a simple thought:

“Do the robots even know where to find me?”

I wasn’t thinking about sci-fi cyborgs or apocalyptic machines — just the quiet, tireless web crawlers that roam the internet day and night, indexing everything they see. Googlebot, Bingbot, DuckDuckGo’s little spiders… the whole buzzing colony of digital bees.

So I opened my browser, typed:

https://fazaeli.dev/robots.txt

and hit Enter.

What greeted me wasn’t mine. It was Cloudflare’s. A whole manifesto about not scraping data, followed by a long blacklist of bots I didn’t even know existed. Amazonbot, ClaudeBot, GPTBot — all shut out with a single Disallow: /.

For a second, it felt like someone else was answering the door to my house, telling visitors what they could and couldn’t see.

That’s when it hit me: if I don’t give the crawlers a map, they’ll wander my site blind. My blog posts, my projects, my little corner of the web might stay hidden in the dark corners of the internet.

So I rolled up my sleeves.

Inside ExpressionEngine, I created a template called seo/robot and gave it the simplest set of lines you can imagine:

User-agent: *
Disallow:

Sitemap: https://fazaeli.dev/sitemap.xml

Then I mapped the template to /robots.txt.

At first, Cloudflare fought me, insisting on injecting its own text. But after some tweaking, bypass rules, and a little stubbornness, my voice came through. The crawlers could finally hear me.

The last piece was the sitemap — a quiet XML file at the root of my site. It doesn’t look like much, just lines of <url> and <lastmod>, but it’s the treasure map. It tells Google where every page lives, when I last dusted it off, and how important it is in the grand scheme of things.

When I finally curled it in the terminal with:

curl -A "Googlebot" https://fazaeli.dev/sitemap.xml

and saw those clean XML lines, it felt like unlocking a door.

Now, if you drop by my robots.txt, you’ll see my own words, my own directions. The crawlers know where to go. And soon, the posts and projects I’ve been working on in the quiet will be easier for the world to find.

Sometimes, building a website isn’t just about design or features — it’s about remembering that even the robots need a little guidance.

And if you don’t tell them where the map is, they’ll never know where to look.