When Robots Knocked on My Door

Javid Fazaeli — Thu, 11 Sep 2025 05:28:00 +0000

When Robots Knocked on My Door

It started with a simple thought:

“Do the robots even know where to find me?”

I wasn’t thinking about sci-fi cyborgs or apocalyptic machines — just the quiet, tireless web crawlers that roam the internet day and night, indexing everything they see. Googlebot, Bingbot, DuckDuckGo’s little spiders… the whole buzzing colony of digital bees.

So I opened my browser, typed:

https://fazaeli.dev/robots.txt

and hit Enter.

What greeted me wasn’t mine. It was Cloudflare’s. A whole manifesto about not scraping data, followed by a long blacklist of bots I didn’t even know existed. Amazonbot, ClaudeBot, GPTBot — all shut out with a single Disallow: /.

For a second, it felt like someone else was answering the door to my house, telling visitors what they could and couldn’t see.

That’s when it hit me: if I don’t give the crawlers a map, they’ll wander my site blind. My blog posts, my projects, my little corner of the web might stay hidden in the dark corners of the internet.

So I rolled up my sleeves.

Inside ExpressionEngine, I created a template called seo/robot and gave it the simplest set of lines you can imagine:

User-agent: *
Disallow:

Sitemap: https://fazaeli.dev/sitemap.xml

Then I mapped the template to /robots.txt.

At first, Cloudflare fought me, insisting on injecting its own text. But after some tweaking, bypass rules, and a little stubbornness, my voice came through. The crawlers could finally hear me.

The last piece was the sitemap — a quiet XML file at the root of my site. It doesn’t look like much, just lines of <url> and <lastmod>, but it’s the treasure map. It tells Google where every page lives, when I last dusted it off, and how important it is in the grand scheme of things.

When I finally curled it in the terminal with:

curl -A "Googlebot" https://fazaeli.dev/sitemap.xml

and saw those clean XML lines, it felt like unlocking a door.

Now, if you drop by my robots.txt, you’ll see my own words, my own directions. The crawlers know where to go. And soon, the posts and projects I’ve been working on in the quiet will be easier for the world to find.

Sometimes, building a website isn’t just about design or features — it’s about remembering that even the robots need a little guidance.

And if you don’t tell them where the map is, they’ll never know where to look.

How I Deployed My GitHub Project to Vercel in Minutes

Javid Fazaeli — Sat, 06 Sep 2025 18:07:00 +0000

When you finish a side project, the next step is getting it online. Vercel makes this almost frictionless — especially if you’re already using GitHub.

This is exactly how I deployed my Innoweg repo → innoweg.vercel.app.

1) Initialize Git + Push to GitHub

If your project isn’t versioned yet:

git init
git add .
git commit -m "Initial commit"
git branch -M main
git remote add origin https://github.com/username/repo-name.git
git push -u origin main

Confirm it’s live at: https://github.com/username/repo-name

2) Sign in to Vercel

Go to vercel.com
Sign in with GitHub (recommended)
Authorize Vercel to access your repos

3) Import the GitHub Repo

Click New Project
Select your GitHub repository
Vercel auto-detects your framework (React, Next.js, Vue, static, etc.)

For a Vite/React project, defaults typically are:

Build command: vite build
Output directory: dist/

4) Deploy 🚀

Click Deploy. Within ~1 minute, you’ll get a live URL like:

https://your-project.vercel.app

5) Continuous Deployment

Every git push triggers an automatic redeploy. Every PR gets a preview URL.

git add .
git commit -m "Update UI"
git push origin main

Gotchas & Notes

Environment variables: Set them in Settings → Environment Variables. They’re available at build/runtime.
Branch config: By default, Vercel deploys main. Adjust under Project Settings → Git if you use develop or staging
Framework detection: If auto-detect fails, add a minimal vercel.json (example for Node):
CI/CD contexts: Production (pushes to main), Preview (PRs), Development (vercel dev locally).

Final Result

Repo: github.com/jfaza/innow
Live demo: innoweg.vercel.app

DEV Community: Javid Fazaeli

When Robots Knocked on My Door