DEV Community: Julian Neagu

Stop Paying for Bot Traffic: A Two-Layer Defense for Next.js apps

Julian Neagu — Fri, 14 Aug 2026 05:17:02 +0000

TL;DR: Bot traffic isn't a fact of life - it's a leak in your infrastructure budget. A two-layer defense using robots.txt and Next.js middleware blocks 60-70% of resource-draining scrapers while keeping the bots you actually want. This approach runs at the edge, costs nothing to implement, and protects both your hosting bill and your analytics accuracy.

Most founders assume bot traffic is part of the cost of running a website. That assumption is expensive. Bots hammer your endpoints, consume metered server resources, pollute your analytics, and inflate your hosting bill. If you're on Vercel, AWS, or any pay-per-request platform, this traffic literally costs money every time it hits your server.

This is what a bot attack looks like - a sudden, unnatural spike that distorts your real user activity.

The standard advice is to add a robots.txt file and hope for the best. That only works if bots respect it. Most scrapers don't. They read your file, note what you've marked off-limits, and scrape it anyway. The file is a suggestion, not a firewall.

The real solution is a two-layer defense: a smart robots.txt configuration that filters well-behaved bots, and Next.js middleware running at the edge that enforces the rules before traffic reaches your application. Combined, they block 60-70% of unwanted bot traffic while keeping legitimate crawlers and AI assistants you actually want.

This isn't about locking out AI crawlers. It's about controlling who gets access and at what cost.

The Problem: Three Kinds of Visitors

Your website serves three categories of traffic. Real humans browsing your site. Legitimate bots that index your content for search engines or train helpful AI models. And scrapers that hammer your endpoints, steal your data, rotate IP addresses, and pretend to be browsers.

The third category is the problem. These bots ignore polite requests. They consume server resources you're paying for. They pollute your analytics by inflating visitor counts with fake traffic, making conversion rates appear worse than they are. They probe for vulnerabilities. If you're running on metered hosting, every request from a scraper costs you a fraction of a cent. When scrapers make up 30-40% of your traffic, that adds up fast.

60-70% of bot traffic can be filtered by a properly configured robots.txt, but only because most low-effort scrapers follow basic rules to avoid detection.

The remaining 30-40% requires enforcement. That's where middleware comes in. It runs before every request hits your application, inspects headers and user agents, and blocks anything suspicious. On Vercel, middleware runs at the edge, meaning blocked requests never consume your server resources. You're not paying to process traffic you don't want.

Layer 1: Smart robots.txt Configuration

Start with robots.txt. This file sits at the root of your domain and tells compliant bots what they can access. The key is being explicit about who you allow and who you block. Most sites either leave this file empty or use a generic "allow all" rule. That's leaving money on the table.

Here's a structure that works:

``txt

AI training & assistants - welcome

User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: anthropic-ai
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: CCBot
Allow: /

Search engines - welcome

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

Known scrapers - blocked

User-agent: SemrushBot
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: DotBot
Disallow: /

User-agent: PetalBot
Disallow: /

Default - be polite or leave

User-agent: *
Crawl-delay: 10
Disallow: /api/
``

This makes three categories clear. AI assistants and search engines get full access. Known scrapers get nothing. Everything else gets throttled with a 10-second crawl delay and blocked from API routes.

The crawl delay is critical. Aggressive bots hitting your site every few seconds will slow down or leave. Legitimate crawlers won't care about a 10-second pause between requests. This single line filters out bots that hammer your site hundreds of times per minute.

The Disallow: /api/ rule is equally important. API routes often trigger database queries, authentication checks, or external service calls. Bots crawling these endpoints waste expensive compute. Block them by default unless you're running a public API that needs discovery.

This configuration takes five minutes to set up. Copy the template, adjust the bot names if you have specific concerns, and deploy it to your site's root directory. For a deeper dive into robots.txt strategy and common mistakes, check out VisionVix's robots.txt best practices guide, which walks through edge cases and SEO implications.

Most low-effort scrapers follow basic robots.txt rules to avoid detection, making it effective against the majority of bad actors.

This handles the 60-70% of bot traffic that follows rules. It won't stop determined attackers, but it filters the lazy majority. Now we enforce the rest.

Layer 2: Middleware Enforcement at the Edge

robots.txt is polite. Middleware is the bouncer. It runs before every request hits your application, inspects the user agent and headers, and blocks anything suspicious. If you're using Next.js on Vercel, middleware runs at the edge. Requests get filtered before they consume your server resources. You're not paying to process blocked traffic.

Create a file called middleware.ts at the root of your Next.js project:

``typescript
import { NextResponse } from 'next/server'

const ALLOWED_BOTS = [
'googlebot',
'bingbot',
'gptbot',
'claudebot',
'perplexitybot',
'slackbot',
'twitterbot'
]

const BLOCKED_BOTS = [
'petalbot',
'blexbot',
'semrushbot',
'ahrefsbot',
'mj12bot',
'dotbot'
]

const SCRAPER_LIBS = [
'scrapy',
'python-requests',
'curl/',
'wget',
'java/',
'go-http-client'
]

export function middleware(req) {
const ua = (req.headers.get('user-agent') || '').toLowerCase()

// 1. Allow good bots
if (ALLOWED_BOTS.some(bot => ua.includes(bot))) {
return NextResponse.next()
}

// 2. Block empty user agents
if (!req.headers.get('user-agent')) {
return new NextResponse('Forbidden', { status: 403 })
}

// 3. Block aggressive crawlers
if (BLOCKED_BOTS.some(bot => ua.includes(bot))) {
return new NextResponse('Forbidden', { status: 403 })
}

// 4. Block scraper libraries
if (SCRAPER_LIBS.some(lib => ua.includes(lib))) {
return new NextResponse('Forbidden', { status: 403 })
}

// 5. Block missing Accept header
if (!req.headers.get('accept')) {
return new NextResponse('Forbidden', { status: 403 })
}

// Default: pass
return NextResponse.next()
}
``

Every request triggers this function. The middleware reads the user agent string and headers. If the user agent matches a known good bot, the request passes immediately. If it matches a blocked bot or scraper library, it gets a 403 response. If critical headers are missing, it's blocked.

The Accept header check is subtle but effective. Real browsers and legitimate bots send an Accept header telling the server what content types they understand. Scrapers often skip this. Blocking requests without an Accept header catches a lot of low-effort scraping attempts.

Middleware runs at the Vercel edge, meaning blocked requests never hit your application server or consume compute resources. This is what reduces the remaining 30-40% of bot traffic that ignored robots.txt.

What Gets Blocked and Why

Let's walk through what each check catches in practice.

Empty User Agents

typescript if (!req.headers.get('user-agent')) { return new NextResponse('Forbidden', { status: 403 }) }

Legitimate browsers always send a user agent. Tools like curl and wget do too, but scrapers built with low-effort scripts often don't. This single check blocks a surprising amount of junk traffic.

Scraper Libraries

typescript if (SCRAPER_LIBS.some(lib => ua.includes(lib))) { return new NextResponse('Forbidden', { status: 403 }) }

This targets common scraping tools. A user agent string like python-requests/2.28.0 or Scrapy/2.9.0 reveals exactly what's hitting your site. These are almost never legitimate traffic. Block them immediately.

Missing Accept Header

typescript if (!req.headers.get('accept')) { return new NextResponse('Forbidden', { status: 403 }) }

Real browsers send an Accept header like text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8. Bots that don't care about the response format skip this header. It's a strong signal that the request isn't coming from a browser or a well-behaved crawler.

The Conditional Middleware Pattern

Not every route needs the same level of protection. Public pages like your homepage or blog should allow all legitimate traffic. Admin panels, API routes, or expensive database queries need stricter rules.

You can add conditional logic to your middleware:

``typescript
export function middleware(req) {
const pathname = req.nextUrl.pathname

// Skip middleware for static assets
if (pathname.startsWith('/_next') || pathname.startsWith('/static')) {
return NextResponse.next()
}

// Apply strict rules to API routes
if (pathname.startsWith('/api')) {
return strictBotCheck(req)
}

// Apply normal rules to everything else
return normalBotCheck(req)
}
``

This pattern lets you enforce stricter checks on expensive routes while keeping public pages accessible. The strictBotCheck function could block all bots except explicitly allowed ones. The normalBotCheck function could allow most traffic but throttle suspicious patterns.

Honeypot Traps and Advanced Tactics

If you want to go further, add a honeypot trap. Create a page that's linked in your robots.txt under a Disallow rule. Good bots won't visit it. Bad bots will.

Add this to your robots.txt:

txt User-agent: * Disallow: /trap

Then create a /trap route in your Next.js app. Anyone who visits it gets their IP logged and blocked:

`typescript // pages/api/trap.ts export default function handler(req, res) { const ip = req.headers['x-forwarded-for'] || req.connection.remoteAddress console.log(Bot trap triggered by IP: ${ip}`)

// Add IP to blocklist
// (integrate with your edge config or database)

return res.status(403).json({ error: 'Forbidden' })
}
``

This catches bots that explicitly ignore your robots.txt rules. You can feed these IPs into your middleware blocklist or configure edge rules at the CDN level. For a broader look at proactive security measures like this, see the website security scanning guide, which covers automated vulnerability detection and bot behavior analysis.

What You're Actually Protecting

This defense isn't just about saving money. It protects three things founders underestimate:

Hosting costs. If bots make up 30-40% of your traffic, you're paying 30-40% more than you should. On metered platforms like Vercel or AWS Lambda, every function invocation costs money. Blocking bot traffic before it hits your application cuts those costs immediately.

Analytics accuracy. Bot traffic inflates your visitor counts and skews conversion rates. If 40% of your "users" are scrapers, your real conversion rate is higher than your dashboard shows. Clean traffic gives you accurate data to make decisions.

Attack surface. Bots probe for vulnerabilities. They test SQL injection patterns, brute-force login endpoints, and enumerate API routes. Blocking them reduces your exposure to automated attacks.

Blocking requests without Accept headers or empty user agents catches most low-effort scraping attempts.

Implementation Checklist

Here's what to do right now:

Deploy the robots.txt template to your site's root directory. Adjust the allowed and blocked bot lists based on your traffic logs.
Add the middleware file to your Next.js project. Copy the full code block above, save it as middleware.ts at the project root, and deploy.
Monitor your logs for a week. Check which user agents are getting blocked. If you see legitimate traffic being filtered, adjust your ALLOWED_BOTS list.
Review your hosting bill after 30 days. If bots were a significant portion of your traffic, you should see a measurable drop in compute costs.
Check your analytics for cleaner data. Visitor counts should drop, but conversion rates should improve because you're measuring real users instead of bots.

This setup takes less than an hour to implement. It costs nothing. It runs at the edge, so there's no performance penalty for legitimate users. And it blocks the majority of bot traffic that's wasting your budget and polluting your metrics.

If you've been treating bot traffic as inevitable, stop. It's not. Two layers of defense, five minutes of configuration, and you're done. The bots you want still get in. The ones you don't pay for anymore.

📦 Publishing Kit — Dev.to

Title Options (5)

Selected: Stop Paying for Bot Traffic: A Two-Layer Defense for Next.js Apps

Alternates:

How robots.txt + Next.js Middleware Cuts Bot Costs by 60-70%
Block Scrapers, Keep AI Crawlers: Smart Bot Management for Next.js
The $500/Month Bot Problem (And How to Fix It With Edge Middleware)
robots.txt Isn't Enough: Building a Real Bot Firewall in Next.js

Slug

stop-paying-for-bot-traffic-nextjs-middleware-defense

Tags

nextjs, webdev, devops, security

Blocking GPTBot Won't Remove You From ChatGPT Search Here's Why

Julian Neagu — Thu, 06 Aug 2026 10:44:19 +0000

TL;DR: Blocking GPTBot doesn't remove you from ChatGPT search results. OpenAI runs two separate bots: one that trains models (GPTBot) and one that fetches live citations (OAI-SearchBot). Every major AI company uses this dual-bot system. Block the wrong one and you vanish from AI search. Block the right one and you protect your content while staying visible.

Most people who block GPTBot expect their content to disappear from ChatGPT entirely. Then they search for their own article title in ChatGPT and find it sitting right there in the results with a working link and an accurate pull quote. The block didn't fail. You just blocked the wrong bot.

OpenAI doesn't run one bot. They run two. So does Anthropic. So does Google. Every major AI company operates a two-bot system because training and citation are fundamentally different jobs that need fundamentally different infrastructure.

The first bot harvests your content to train the model. It reads your articles, extracts patterns from your prose, and bakes those patterns into billions of model parameters. Your words become part of the AI's permanent knowledge. The model never cites you. It never links back to you. It never sends you traffic. Training bots give you nothing in return for what they take.

The second bot fetches your content in real time when someone asks a question your site might answer. The bot finds your page, reads the relevant section, and surfaces it in the AI's response with your name, a link, and a snippet. This is how you show up in ChatGPT search results. Citation bots drive referral traffic and keep you visible in AI-powered search.

Blocking the training bot does absolutely nothing to the citation bot. They don't share a blocklist. They don't check each other's permissions. They operate as if they work for separate companies.

The Two-Bot System Every AI Company Runs

Here's the full breakdown of how each major AI company splits training from citation:

OpenAI runs GPTBot for training. GPTBot crawls your site, reads your content, and uses it to improve future models. That's where your IP goes in and never comes back out. For live citations, OpenAI runs OAI-SearchBot and ChatGPT-User. These bots fetch content when a user asks a question. Block GPTBot and you stop training. Block OAI-SearchBot and you vanish from ChatGPT search results.

Anthropic operates ClaudeBot for training. ClaudeBot works exactly like GPTBot. It harvests content to build the model's knowledge base. For citations, Anthropic runs Claude-SearchBot and Claude-User. Block ClaudeBot to stop training. Leave Claude-SearchBot allowed if you want to appear in Claude's search answers.

Google uses Google-Extended for training. This is separate from Googlebot, which has been crawling the web for decades. Googlebot handles citations because Google already had a citation infrastructure before AI search existed. Block Google-Extended and you stop contributing to Gemini training. Block Googlebot and you disappear from Google Search entirely, which most sites can't afford.

Perplexity runs PerplexityBot and Perplexity-User. Both fetch content for live answers. Perplexity doesn't train foundation models, so they don't operate a separate training crawler. Every request from Perplexity is a citation request.

Meta operates meta-externalagent for training. They don't currently run a public citation bot for consumer search products, so blocking meta-externalagent has no downside for visibility.

Training crawlers need bulk access to millions of pages, running batch jobs that might revisit your content weeks or months later. Citation crawlers need low-latency access to fresh content, often fetching a single page in response to a specific user query.

This architecture exists because training and citation have fundamentally different requirements. Training bots run in batches. They might crawl your site once and never come back. They don't care about freshness. They care about volume. Citation bots run on demand. They fetch a single page in milliseconds when a user asks a question. They care about speed and freshness, not volume.

Building one bot that does both jobs well is harder than running two specialized systems. So every company runs two. As AI crawling vs traditional crawling explains, the infrastructure behind these systems diverges at nearly every layer, from request patterns to caching strategies to how they handle rate limits.

What You're Blocking Right Now (And What You're Not)

Most robots.txt files I see block training bots and stay silent on citation bots. That's not necessarily wrong. The robots.txt standard works by explicit permission. If you don't mention a bot, it's allowed by default. But I don't like relying on default behavior. Defaults change when companies push updates or rewrite their crawler logic.

Here's what a safe, explicit robots.txt configuration looks like if you want to block training but allow citations:

``txt
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /
``

The first block stops training. The second block guarantees you stay citable. Both blocks matter. Leaving citation bots unstated means trusting default behavior, and I've seen too many cases where defaults shifted after a company updated their crawler documentation.

You'll notice Googlebot isn't in either list. That's intentional. Blocking Googlebot means disappearing from Google Search entirely, which most sites can't survive. Google-Extended is the training-specific bot. Block that and leave Googlebot alone. You get the split you want: no training, yes citations.

One more detail. If you're running a Cloudflare-managed site, your robots.txt likely includes a comment that says # BEGIN Cloudflare Managed content. That marker means Cloudflare auto-generates part of your robots.txt based on dashboard settings. Don't edit robots.txt directly when that marker is present. Cloudflare will overwrite your changes on the next sync. Make changes through Cloudflare's AI Crawl Control panel instead.

How to Edit Robots.txt When Cloudflare Manages It

If you see the Cloudflare marker in your robots.txt, here's the correct path to making changes stick:

Log into your Cloudflare dashboard and select the domain you're managing.
Click Domains in the left sidebar.
Select your specific domain from the list.
Scroll down to AI Crawl Control or search for "AI" in the page search.
Toggle the bots you want to block or allow using the switches provided.

If you've been blocking GPTBot and wondering why your articles still show up in ChatGPT, now you know. The bot you blocked wasn't the bot you thought it was. And if you want to stay visible in AI search while protecting your content from training, you need to configure both. For a deeper look at how AI systems decide what to crawl and when, see this comparison of Claude vs GPT for coding tasks, which shows how different models prioritize different types of content even when using similar underlying architectures.

Cloudflare's AI Crawl Control panel handles the syntax for you. Changes take effect within a few minutes and won't be overwritten by automated syncs. If you need more granular control than Cloudflare's toggles provide, you'll need to disable Cloudflare management of robots.txt entirely and manage the file manually as we advance.

Checking Whether Your Block Actually Works

After updating your robots.txt, wait 24 hours and check your server logs or analytics dashboard. Look for requests from the bots you tried to block.

Training bots like GPTBot, ClaudeBot, and Google-Extended should show zero requests and zero bytes transferred after blocking. If you still see requests from GPTBot after a week, either your robots.txt syntax is wrong or the bot isn't respecting the directive. Cloudflare's WAF lets you write a firewall rule that blocks by user agent string if robots.txt alone doesn't work.

Citation bots like OAI-SearchBot and ChatGPT-User should show periodic requests when users ask questions your content might answer. These requests are low-volume and sporadic. You might see a handful per week for a niche topic or dozens per day if you rank well for common queries. This is normal and desirable. Each request represents a potential citation in an AI answer.

If you're not seeing any citation bot traffic after allowing them, check two things. First, make sure your Allow: / directive is present and correctly formatted. Second, verify that your content actually ranks for questions people ask AI tools. Not every page earns citations. The citation bots only fetch pages that match a user query.

Why This Matters More Than You Think

Training without attribution is economically asymmetric. Your content becomes part of a billion-dollar model's knowledge base. You get no traffic, no links, and no royalties. The model answers questions using patterns it learned from your prose, and you never see a visitor.

Citation with attribution reverses that asymmetry. Your content appears as a source. The AI links to your page. Users click through. You get referral traffic from people who were already looking for what you wrote about. This is the only version of AI-powered search that benefits content creators.

Blocking training bots while allowing citation bots is the rational default for most sites. You protect your IP from being absorbed into the model's weights. You stay visible in the AI-powered search results that are increasingly replacing traditional Google queries. You get the upside without the downside.

The mistake most people make is thinking GPTBot controls all of OpenAI's access to your site. It doesn't. GPTBot controls training. OAI-SearchBot controls citations. Block one, allow the other, and you get the outcome you actually want.

The two-bot system isn't going away. Every AI company has adopted it because the engineering requirements for training and citation are incompatible. One bot can't do both jobs well. So they split the work. And you need to split your robots.txt configuration to match.

📦 Publishing Kit — Dev.to

Title Options (5)

Selected: Blocking GPTBot Won't Remove You From ChatGPT Search — Here's Why

Alternates:

You're Blocking the Wrong AI Bot: The Training vs Citation Crawler Problem
Why Your Site Still Appears in ChatGPT After Blocking GPTBot
The Two-Bot System: How AI Companies Split Training From Citations
Block GPTBot, Still Show Up in ChatGPT: Understanding AI Crawler Architecture

Slug

blocking-gptbot-wont-remove-you-from-chatgpt-search

Tags

webdev, ai, opensource, seo

Auto-Submit Changed Pages to Search Engines with GitHub Actions and IndexNow

Julian Neagu — Wed, 29 Jul 2026 13:50:20 +0000

TL;DR: If you're pushing updates to a static site every week, manually notifying search engines is a waste of time. GitHub Actions can detect exactly which files changed in each commit, convert those paths to URLs, and submit them to IndexNow automatically. You build once, push to main, and search engines get notified within minutes - no batch uploads, no forgotten submissions, no resubmitting 388 URLs when only three changed.

If your static site gets a dozen updates a week, manually notifying search engines gets old fast. Most builders either skip the submissions entirely or batch them monthly, which means new pages sit invisible for weeks.

GitHub Actions combined with IndexNow solves this. Your repo already knows what changed, and search engines support instant notifications. You just need to wire them together.

Here's the real difference: instead of pushing code and then remembering to notify Bing or Yandex later, you build a workflow that detects changed files, waits for your deployment to finish, and submits only the modified URLs. No manual script runs. No forgotten submissions. No resubmitting all 388 URLs when only three changed.

This article walks through what GitHub Actions does, how IndexNow works, and how to set up submissions that fire on every push to main.

What GitHub Actions Actually Does

GitHub Actions is GitHub's built-in automation system. You write a workflow file, drop it in .github/workflows/, and GitHub runs it whenever a trigger fires. Common triggers include pushing code, opening a pull request, or hitting a cron schedule.

For static sites and micro-tools, this becomes a deployment and notification engine. You can:

Run a build step when you push to main
Detect which files changed in the commit
Convert those file paths into live URLs
Wait for your hosting platform to finish deploying
Submit only the changed URLs to search engines

The key advantage: GitHub already has the commit diff. It knows exactly which pages you modified. You don't need to scan your entire sitemap or keep a separate change log.

Run on Every Push

The most common trigger is push to a specific branch:

yaml on: push: branches: - main

Every time you merge a PR or push directly to main, the workflow fires. This matches how most teams deploy: main is production, and every merge should trigger a deployment notification.

For teams managing multiple repositories with similar automation needs, GitHub repository best practices covers how to structure .github/ folders, share workflows across repos, and maintain consistency when scaling to dozens of projects.

Detect Changed Files

GitHub Actions can extract the list of modified files from the commit diff. The official IndexNow Action supports multiple input modes:

urls-from: generate URLs from a shell command (like parsing the diff)
file: read URLs from a text file you generate in an earlier step
sitemap: submit URLs from an existing XML sitemap
urls: pass a direct list of URLs as a string

For changed-file workflows, urls-from is cleanest. You run a command that outputs one URL per line, and the Action reads that output.

Example shell logic on macOS or Linux:

This grabs modified files, filters for index.html entries (assuming each page is a folder with an index file), and converts paths into full URLs.

On Windows with PowerShell, you'd use native cmdlets instead:

powershell $changedFiles = git diff-tree --no-commit-id --name-only -r HEAD $urls = $changedFiles | Where-Object { $_ -like '*index.html' } | ForEach-Object { $_ -replace '^', 'https://yourdomain.com/' -replace '/index\.html$', '/' } $urls -join "`n"

Both produce the same output: a list of URLs, one per line, ready to feed into the IndexNow Action.

Submit URLs to IndexNow Automatically

The IndexNow Action handles the actual submission. It supports five search engines:

Bing
Yandex
Naver
Seznam
Yep

IndexNow submissions are shared across all supporting engines. You submit once, and Bing tells Yandex, Naver, Seznam, and Yep about the update.

The Action supports multiple feed formats:

XML sitemaps
RSS feeds
Atom feeds
Direct URL lists
Custom endpoints

For changed-file workflows, direct URL lists are simplest. You pass the output of your diff command and let the Action batch the submissions.

If you're building workflows for rapid deployment and want to see how to structure automation logic around changed files and deployment hooks, check out building your first automated workflow step for a deeper dive into event-driven automation patterns.

The Action also supports limits (submit only the first N URLs), time filtering (submit only URLs modified since X days ago), and custom key file locations.

Example workflow snippet:

``yaml

name: Submit changed URLs to IndexNow uses: bojieyang/indexnow-action@v2 with: sitemap-location: 'https://yourdomain.com/sitemap.xml' key: ${{ secrets.INDEXNOW_KEY }} key-location: 'https://yourdomain.com/indexnow-key.txt' ``

This submits URLs from your sitemap. For changed-file workflows, you'd replace sitemap-location with urls-from and pass your diff command.

Run on a Schedule

You can also trigger workflows on a cron schedule:

yaml on: schedule: - cron: '0 2 * * *' # 2 AM UTC daily

This is useful for submitting your full sitemap once a day, even if nothing changed. Some builders use this as a safety net: changed-file submissions run on every push, and a daily full-sitemap submission catches anything the diff logic missed.

Use Secrets for Keys

GitHub Actions supports encrypted secrets. You store your IndexNow key in repository settings under Secrets and variables, then reference it in your workflow as ${{ secrets.INDEXNOW_KEY }}.

This keeps the key out of your public repository. The workflow file itself is visible, but the key value is encrypted and only decrypted during workflow runs.

Generate Key Files Dynamically

The official IndexNow docs warn against committing your key file to a public repository. If the file is visible in your repo, anyone can see the key contents.

The recommended approach: generate the key file during deployment. Your hosting platform writes the key to disk as part of the build step, so it's available at https://yourdomain.com/indexnow-key.txt but never committed to version control.

Most static site hosts support build hooks or environment variable injection. You can write a simple script that creates the key file from a secret environment variable during the build process.

What You Get

A typical static site might have 388 URLs total but only 3 changed URLs per update.

With this workflow in place:

Every push to main triggers the workflow
GitHub extracts the diff and converts changed paths to URLs
The IndexNow Action submits only those URLs to all five supported search engines
Search engines crawl the updated pages within hours instead of weeks
You never manually submit a URL again

Traditional sitemap-based indexing can take days or weeks. IndexNow reduces this to hours. Sites that receive a dozen updates per week benefit the most, because each batch of changes gets notified immediately instead of sitting in a queue until someone remembers to submit them.

Common Mistakes to Avoid

Submitting the entire sitemap on every push. This works, but it wastes API quota and makes it harder for search engines to prioritize your fresh content. Only submit what changed.

Forgetting to wait for deployment. GitHub Actions can't detect deployment completion automatically. If your workflow submits URLs before your hosting platform finishes deploying, search engines will crawl stale versions. Add a delay step or poll your deployment status endpoint before submitting.

Committing your IndexNow key to a public repo. Use GitHub Secrets and generate the key file dynamically during deployment. Never hardcode the key in a workflow file or commit it to version control.

Assuming Google supports IndexNow. Google does not. You need a separate notification mechanism, either through the Search Console API or by updating your XML sitemap and pinging Google's submission endpoint.

Why This Matters for Static Sites

Static sites change frequently in small batches rather than all at once. You publish a new blog post, fix a typo on three pages, add a new product page. Each of these events is a small diff.

Most static site builds finish in under 60 seconds. That means your workflow can detect changes, wait for deployment, and submit URLs within two minutes of pushing to main.

This kind of speed doesn't matter for sites that update once a month. But if you're shipping updates every few days, the difference between manual batch submissions and automatic change detection is the difference between pages that index in hours and pages that sit invisible for weeks.

You build the workflow once. After that, every push to main handles its own search engine notifications. No manual steps. No forgotten submissions. No wasted API quota on unchanged URLs.

📦 Publishing Kit — Dev.to

Title Options (5)

Selected: Auto-Submit Changed Pages to Search Engines with GitHub Actions and IndexNow

Alternates:

Stop Manual Search Submissions: Automate IndexNow with GitHub Actions
GitHub Actions + IndexNow: Submit Only Changed URLs to Search Engines
How to Auto-Notify Search Engines When You Push Static Site Updates
Automate Search Engine Notifications for Static Sites with GitHub Actions

Slug

auto-submit-changed-pages-search-engines-github-actions-indexnow

Tags

webdev, devops, tutorial, indexnow

We Built 1,500 Keyword-Rich Domains Before Chasing Backlinks, Here's What Happened

Julian Neagu — Mon, 27 Jul 2026 14:20:50 +0000

TL;DR: We built 1,500+ keyword-rich domains hosting real AI tools before worrying about backlinks. Each domain is a production-grade product that passes enterprise security audits. The result: over 1 million AI citations in seven months, impressions within days of launch, and a compounding authority mesh that grows sideways rather than up.

Most founders obsess over backlinks. They build a product, launch it on a generic domain, then spend months begging for links from high-authority sites. We flipped that model.

We built the ranking infrastructure first. Then we filled it with products.

What started as a bootstrapped need for tools we couldn't afford became a 1,500-domain ecosystem. Every domain hosts a real AI tool. Every tool solves a real problem. Every site passes the strictest compliance audits from day one. This isn't a private blog network where hollow sites exist only to pass link juice. It's a distributed product mesh that compounds authority through authentic user engagement.

" width="800" height="465">

In seven months, this network-mesh approach drove more than one million AI citations to VisionVix.com. That number isn't a vanity metric. It's a massive indicator of how fast the ecosystem compounds when every node is real, useful, and built to rank.

Keyword-Rich Domains That Google Already Trusts

The foundation is a portfolio of 1,500+ keyword-rich domains spanning .dev, .app, .world, .ai, and .com extensions. These aren't vanity URLs. These aren't placeholder sites waiting for content. Each domain was chosen because it communicates intent instantly and carries inherent trust signals.

Google treats .dev and .app as secure-by-default TLDs because they enforce HTTPS at the registry level. Users click them more because the domain itself answers the question "what does this do?" When someone sees carvaluation.app, they know exactly what to expect. When they see genericstartup.com/tools/cars, they have to parse the path structure and guess.

The marketplace shows this strategy in action. Look at the names: koreanfood.app, [goldtrade.app](https://goldtrade.app/), carvaluation.app, pokezonline.app, and dozens more. Each name is a micro-product waiting to launch. When you own the exact-match domain for a tool category, you start with positioning advantage before writing a single line of code.

1,500+ keyword-rich domains across .dev, .app, .world, .link, and .com create instant topical relevance and trust signals that generic domains take months to build.

This is why new domains in the mesh begin receiving impressions within days, not months. The domain name itself is a relevance signal. No amount of on-page SEO on a generic domain can replicate the clarity of a purpose-built keyword domain. The difference between "try our car valuation tool at genericstartup.com/tools/cars" and "visit carvaluation.app" is the difference between explaining and being understood immediately.

Why TLD Choice Actually Matters

The choice of TLD matters more than most founders realize. We covered the strategic differences in depth when comparing .app versus .com domains, but the short version is this: modern TLDs carry semantic weight.

A .dev domain signals developer tools. A .app domain signals an application. A .world domain signals a platform or ecosystem. Google doesn't just see these as interchangeable alternatives to .com. It treats them as categorization hints.

For a deeper explanation of how this works at the registry level, understanding what a top-level domain actually is helps clarify why TLD choice affects both ranking speed and user perception.

Below you can see our Google Analytics data: in just about two weeks, roughly 580 of our .app domains went live, and the traffic skyrocketed almost immediately. The pattern is clear. These are keyword-rich domains that Google understands and rewards.

The engagement metrics prove this pattern. Analytics show 2.8k active users and 9.4k events tracked across the mesh. Traffic originates from the US, Hong Kong, Singapore, Germany, and the UK. Domains act as connectors, not isolated endpoints.

Some domains gain impressions within two weeks of launch. Some domains rank with zero backlinks because the on-page relevance and user engagement signals are strong enough to trigger indexing and ranking without external validation.

Our Domains: Purpose-Built, Keyword-Rich Assets for Digital Products

At VisionVix, we don’t collect domains—we build digital opportunities around them.

Every domain in our portfolio is selected with a clear objective: to become the foundation for a practical micro-tool, AI agent, software application, or SEO-driven growth asset. We focus on domains that combine strong keywords, user intent, and real commercial potential.

Our expertise lies in identifying premium .app and .dev domains that are clear, memorable, and aligned with future product opportunities. These domains are not simply names sitting unused—they are strategic digital assets designed for development, deployment, and long-term value creation.

Why Keyword-Rich Domains Create Value

A strong keyword domain gives a product an immediate advantage by connecting directly with what users are searching for. Keyword-rich domains can help:

Improve search visibility and relevance
Communicate product purpose instantly
Build credibility with users
Increase organic click potential
Support efficient customer acquisition
Strengthen SEO ecosystems and content strategies

When combined with a useful product and strong execution, a focused exact-match domain provides a powerful foundation for building online authority.

Our Domain Acquisition Strategy

We select domains based on a combination of market data, usability, and future development potential. Each domain is reviewed for:

Relevant search keywords
Commercial intent
Product alignment
Brand potential
Trademark availability
Historical quality, including Wayback and backlink checks
Integration potential within our 500+ tool development ecosystem

Our evaluation process considers:

Search volume
Keyword competitiveness
Traffic opportunity
CPC value
SERP landscape
Brandability
Compliance requirements

This approach ensures our portfolio consists of practical, build-ready assets rather than speculative domain holdings.

Examples of High-Intent .app Domains

Selected examples include:

koreanfood.app
goldtrade.app
ailoan.app
aieuct.app
carvaluation.app
pokezonline.app

Each domain is short, memorable, and instantly communicates its intended use case—making it suitable for AI tools, applications, marketplaces, and specialised digital services.

At VisionVix, we believe the right domain is more than an address—it is the starting point for building valuable products and scalable digital experiences.

Every Domain in the SEO Network Mesh Meets Product-Grade Compliance

Unlike a PBN where sites exist only to pass link juice, every domain in the mesh is a real product with real infrastructure. Each one ships with fully built landing pages, working AI tools or calculators, SSL certificates, strict security headers, privacy policies, terms of service, cookie consent, structured data, semantic HTML, agentic browsing readiness, high PageSpeed scores, and clean Next.js or static builds.

The audit results prove this. WebsiteAudit.dev scores 95/100. SecurityAudit.dev scores 100/100. VisionVix.world receives an A-grade security header report.

Agentic browsing audits pass with perfect scores. PageSpeed Insights consistently shows 100/100 performance, SEO, and best practices across the portfolio.

Production-Grade Infrastructure on Every Domain

These aren't placeholder sites thrown up to game search engines. They're production-grade applications built to survive compliance audits from enterprise customers, legal review, and accessibility standards. Every domain includes:

SSL certificates issued and auto-renewed through Let's Encrypt or Cloudflare
Security headers enforcing Content Security Policy, Strict Transport Security, and X-Frame-Options
Privacy and legal pages written by actual legal review, not copied templates
Structured data markup using JSON-LD for breadcrumbs, organization, and product schemas
Semantic HTML with proper heading hierarchy, ARIA labels, and landmark roles
Mobile responsiveness tested across viewport sizes and touch interactions
Cookie consent flows that comply with GDPR and CCPA requirements
Accessibility readiness including keyboard navigation, screen reader support, and contrast ratios

This level of quality creates real defensibility. When an enterprise evaluates a tool, they run security scans. When a journalist references a calculator, they check the about page. When an AI agent crawls a domain, it expects structured data.

The mesh survives these checks because every site was built to pass them. The infrastructure isn't optimized for SEO tricks. It's optimized for real use.

Why Compliance Matters for Rankings

Google's algorithm doesn't just measure backlinks and keywords. It measures user experience signals. Sites that load fast, work on mobile, respect privacy, and follow web standards send trust signals. Sites that break on mobile, load slowly, or lack legal pages send warning signals.

When you build 1,500 sites, you can't manually tune each one. You need infrastructure that makes compliance automatic. That's why every domain in the mesh inherits security headers, structured data, and legal pages from the same base template. One update cascades across the network.

This approach also protects against manual review. When Google's quality raters spot-check sites in the mesh, they find real tools with real users and real infrastructure. There's nothing to penalize.

For a detailed walkthrough of finding and fixing website accessibility issues, we covered the full process in an earlier post. The same principles apply across the mesh. Accessibility isn't a nice-to-have. It's a ranking signal.

The Keyword-Rich Domain Strategy Behind the Largest AI Agent Marketplace

The marketplace approach amplifies the mesh strategy. Instead of building one massive domain with hundreds of subdirectories, we built hundreds of standalone domains, each hosting a single tool or category. Then we connected them through the VEX Domain Marketplace.

Each domain in the marketplace is a live product. Users can visit goldtrade.app and get real-time precious metal price data. They can visit carvaluation.app and estimate vehicle worth. They can visit koreanfood.app and explore recipes with nutritional breakdowns.

Over 1 million AI citations to VisionVix.com in seven months proves the compounding effect of a distributed product mesh built on keyword-rich domains.

The marketplace itself becomes a hub. It lists every domain, categorizes them by function, and surfaces them to users searching for specific tools. This creates a natural discovery loop. A user finds one tool, sees the marketplace, discovers three more related tools.

Why Standalone Domains Beat Subdirectories

There's a persistent debate in SEO about whether subdomains or subdirectories rank better. We skipped the debate entirely and went straight to standalone domains. Here's why.

Each domain is an independent entity. It can rank for its own keywords without competing with sibling pages. It can earn its own backlinks. It can build its own brand. It can be sold, spun off, or repurposed without affecting the rest of the mesh.

Subdirectories force everything under one domain authority. If the parent domain gets penalized, every subdirectory suffers. If one subdirectory attracts spam, the entire domain risks contamination. Standalone domains isolate risk.

More importantly, standalone domains create more SERP real estate. When someone searches for "car valuation calculator," we can occupy multiple positions with carvaluation.app, carcalculator.dev, and vehiclevalue.world. Subdirectories under one domain would only let us occupy one position.

The mesh isn't just domains. It's functionality. The ecosystem includes mini tools and 160+ free calculators (but we're adding daily) distributed across the portfolio. These aren't demo apps or MVPs. They're production tools that thousands of people use daily to solve real problems.

The marketplace model also scales horizontally. We can launch 50 new domains in a weekend without touching existing infrastructure. Each new domain adds another node to the mesh. Each node compounds the authority of the network.

How We Choose Which Domains to Build

Not every keyword-rich domain is worth building. The decision framework is simple:

Does the domain name instantly communicate what the tool does? If someone sees goldtrade.app, do they immediately understand it's about precious metal trading? If yes, continue. If no, skip.
Is there enough search volume to justify the build? We don't need millions of searches per month. We need enough to generate a few hundred visits per month once ranked. Anything above 1,000 monthly searches is worth building.
Can we build a real tool, not just a landing page? If the domain is calculator.app, we need to build an actual calculator. If we can't deliver the function the domain promises, we don't launch it.
Does it fit into a category cluster? Isolated domains are harder to rank. Domains that connect to a category cluster inherit authority faster. If we already have five finance-related tools, adding goldtrade.app strengthens the entire finance cluster.
Is the domain short, memorable, and easy to type? Long domains don't convert. Domains with hyphens look spammy. Domains that require explanation lose users. We filter ruthlessly for clarity.

Using these filters, we've built a portfolio of 1,500+ domains. Each one passes the test. Each one adds value to the mesh.

Natural Cross-Linking Creates Sideways Authority Flow

Traditional SEO focuses on hub-and-spoke models. You build one authoritative hub, then create spokes that link back to it. The hub gets stronger, the spokes stay weak.

The mesh model works differently. Every domain links to related domains. Authority flows sideways, not just upward. When carvaluation.app links to carcalculator.dev and vehiclevalue.world, all three benefit. When goldtrade.app links to silverprices.world and platinumtrends.dev, the entire precious metals cluster gains strength.

This creates a compounding effect. Each new domain doesn't just add one node. It adds connections between existing nodes. The network becomes denser, not just larger.

Some domains gain traction and rank within two weeks of launch with zero backlinks due to strong on-page signals and user engagement.

Google doesn't penalize this because the links are contextually relevant and user-focused. A user on carvaluation.app who needs a more detailed breakdown genuinely benefits from a link to carcalculator.dev. The link serves the user, not just the algorithm.

The Difference Between a Mesh and a PBN

Private blog networks (PBNs) are built to manipulate rankings. Sites exist only to pass link juice. Content is thin. User engagement is nonexistent. Google actively hunts for PBNs and penalizes them.

The mesh is the opposite. Every site is a real product with real users. Links exist because they help users navigate between related tools. Content is dense, functional, and valuable. Engagement metrics are strong because people actually use the tools.

The difference shows up in the data. PBNs see zero engagement, high bounce rates, and no returning visitors. The mesh sees 2.8k active users, 9.4k tracked events, and traffic from five continents. Users don't land on a mesh site and immediately bounce. They use the tool, explore related tools, and return later.

This is why the mesh survives algorithm updates. Google's core updates target manipulation. They don't target real products with real users.

How We Structure Cross-Links

Cross-linking isn't random. We follow a structure:

Category clusters: All finance tools link to each other. All calculator tools link to each other. All AI tools link to each other. This creates topical authority within each cluster.
Related tools: A car valuation tool links to a car calculator, a car insurance estimator, and a vehicle history checker. These are natural next steps for the same user.
Hub pages: The marketplace acts as a central directory. Every tool links back to the marketplace. The marketplace links to every tool. This creates a strong hub without forcing all authority through a single domain.
User intent flows: We map user journeys and link accordingly. Someone using a gold price tracker might next want a silver price tracker, a precious metals calculator, or a market trend analyzer. We link those tools together.

The result is a network where authority flows naturally. No forced linking. No manipulation. Just a well-structured ecosystem where every link makes sense.

Engagement Data Confirms the Mesh Is Working

Traffic numbers tell part of the story. Engagement metrics tell the rest.

Across the mesh, we track 2.8k active users and 9.4k events. Traffic comes from the US, Hong Kong, Singapore, Germany, and the UK. Users aren't just landing on a page and leaving. They're clicking through tools, exploring calculators, and interacting with the products.

The Bing Webmaster Tools dashboard shows the compounding effect. Over 1 million AI citations to VisionVix.com in seven months. That's not a paid promotion number. That's organic discovery driven by the mesh.

Early Impressions on New Domains

Some domains start generating impressions within two weeks of launch. No backlinks. No outreach. Just strong on-page signals and keyword-rich domain names.

brandmonitoring.dev: 2.82k impressions
aiagentready.dev: 1.45k impressions
aieuact.dev: 435 impressions

These numbers aren't massive, but they prove the model works. A new domain with zero external authority can gain visibility purely through topical relevance, user engagement, and semantic clarity.

You can't manually configure 1,500 domains. You can't hand‑write privacy policies for 800 products. You can't individually optimize PageSpeed scores for hundreds of tools. The system works because the infrastructure is templated, automated, and tested. Our GitHub repository shows exactly how this happens:

When an AI agent like ChatGPT or Perplexity needs a calculator, it looks for structured data, semantic HTML, and clear function definitions. The mesh provides all three. That's why VisionVix.com receives over 1 million citations. The infrastructure is built for agentic discovery, not just human search.

PageSpeed and Core Web Vitals Across the Mesh

Google's ranking algorithm prioritizes user experience. That means fast load times, mobile responsiveness, and Core Web Vitals scores. Every domain in the mesh is built to pass these checks.

PageSpeed Insights consistently shows 100/100 performance, SEO, and best practices. This isn't an accident. The base template uses static generation where possible, lazy-loads images, inlines critical CSS, and preloads fonts. Every performance optimization is baked into the deployment pipeline.

The AI agent readiness score of 89/100 across the mesh confirms that these sites aren't just ranking for traditional search. They're optimized for the next wave of search: AI agents parsing structured data, answering user queries, and citing authoritative sources.

When you launch 1,500 sites, you can't manually optimize each one. The infrastructure has to make performance automatic. That's why we use Next.js with static export, host on Cloudflare's edge network, and compress all assets at build time.

The result: every site loads in under two seconds on mobile, scores 100/100 on PageSpeed, and passes Core Web Vitals. These signals compound with keyword relevance and user engagement to create fast-ranking domains.

Security Audits and Compliance Scores

Enterprise customers and AI agents both care about security. A site with weak security headers, missing SSL, or unpatched vulnerabilities gets flagged. A site with A-grade security, strict CSP, and HSTS enforcement gets trusted.

SecurityAudit.dev scores 100/100. VisionVix.world receives an A-grade security header report. Every domain in the mesh enforces HTTPS, sets secure cookies, blocks clickjacking, and prevents XSS attacks.

We didn't build this level of security because it's a nice-to-have. We built it because it's a ranking signal. Google's algorithm doesn't explicitly say "sites with CSP headers rank higher," but the correlation is clear. Sites that follow web standards, protect user data, and enforce security best practices consistently rank better than sites that don't.

For a detailed walkthrough of website security scanning for small business owners, we covered the full methodology in an earlier post. The same security infrastructure powers every domain in the mesh.

Scaling the Mesh: Deployment Pipeline and Automation

Building 1,500 domains manually would take years. We built them in seven months because the entire deployment pipeline is automated.

Every new domain starts from a base template. The template includes the full Next.js project structure, security headers, legal pages, structured data, and semantic HTML. When we decide to launch a new domain, we don't start from scratch. We copy the template, swap the branding, deploy the unique tool logic, and push to production.

The deployment script handles:

Domain registration through Cloudflare or Namecheap
DNS configuration pointing to Cloudflare's edge network
SSL certificate provisioning through Let's Encrypt
GitHub repository creation and initial commit
Vercel or Netlify deployment with environment variables
Sitemap generation and submission to Google Search Console

On macOS or Linux, the process looks like this:

./deploy-new-domain.sh carvaluation.app

On Windows, PowerShell handles the same flow:

$DomainName = "carvaluation.app"
.\deploy-new-domain.ps1 -Domain $DomainName

The script runs for about three minutes. At the end, the domain is live, SSL is active, and the site is submitted to search engines.

Template Once, Reuse Forever

The base template is the secret to scaling. Instead of rebuilding common infrastructure for every domain, we built it once and reused it 1,500 times.

The template includes:

Legal pages: Privacy policy, terms of service, cookie policy, GDPR compliance notices
Security headers: CSP, HSTS, X-Frame-Options, X-Content-Type-Options
Structured data: JSON-LD schemas for organization, breadcrumbs, and products
Semantic HTML: Proper heading hierarchy, ARIA labels, landmark roles
SEO metadata: Open Graph tags, Twitter cards, canonical URLs
Analytics integration: Google Analytics, Bing Webmaster Tools, Cloudflare Analytics
Cookie consent: GDPR-compliant consent banners with opt-in/opt-out flows

When we update the template, we can cascade the update across the mesh. One commit updates 1,500 domains. This is how we stay compliant as privacy laws evolve and web standards change.

" width="800" height="600">

WebsiteAudit.dev is the compliance gate for the entire mesh. It proves every domain is a real product. The audits cover analytics, consent, SEO, social tags, schema, security, performance, and accessibility.

AI Agents in the Deployment Loop

We don't manually configure every domain. AI agents handle repetitive tasks like generating legal pages, writing metadata, and creating structured data.

When we launch goldtrade.app, an AI agent reads the domain name, infers the tool category, and generates:

A privacy policy tailored to precious metals price tracking
SEO metadata describing the tool's function
Structured data schemas for financial calculators
A terms of service page covering data sources and disclaimers

This isn't lazy automation. It's precise automation. The agent follows templates, enforces legal compliance, and ensures consistency across the mesh. A human reviews the output, but the agent handles 90% of the work.

This is how we built 1,500 domains in seven months. We didn't hire a massive team. We built infrastructure that makes launching a new domain a three-minute task instead of a three-week project.

What This Means for Founders

Most founders think about SEO as a backlink arms race. They assume you need thousands of high-authority links to rank. The mesh proves otherwise.

You can rank with zero backlinks if your domain name is keyword-rich, your on-page signals are strong, and your user engagement is real. You can scale to 1,500 domains if your infrastructure is templated and your deployment is automated. You can generate over 1 million AI citations if every site is built for agentic discovery.

The mesh model works because it's built on real products, not manipulation. Every domain solves a real problem. Every tool serves real users. Every link helps someone find the next step in their journey.

If you're building in public, shipping fast, or scaling a portfolio of micro-products, this model is worth studying. The infrastructure compounds. The authority flows sideways. The rankings accelerate with every new node.

📦 Publishing Kit — Dev.to

Title Options (5)

Selected: We Built 1,500 Keyword-Rich Domains Before Chasing Backlinks — Here's What Happened

Alternates:

How 1,500 Keyword Domains Generated 1M+ AI Citations in 7 Months Without Traditional SEO
Building Authority Sideways: The Domain Mesh Strategy That Beat Traditional Link Building
From Zero to 1M AI Citations: Why We Built Products on 1,500 Domains First, SEO Second
Keyword-Rich Domains as Infrastructure: A Network-First Approach to Organic Authority

Slug

1500-keyword-domains-generated-1m-ai-citations-without-backlinks

Tags

seo, webdev, startup, domains

llms.txt vs mcp.json: Why Your Site Needs Both (And What They Actually Do)

Julian Neagu — Mon, 20 Jul 2026 11:11:51 +0000

TL;DR: llms.txt helps AI assistants read your site by mapping your best content. mcp.json lets AI agents call your site by exposing structured tools. They solve different problems, and your site needs both if you want to show up in agent-driven answers instead of getting skipped.

I updated our website generator last week to auto-deploy llms.txt files in every new site. Two days later, a founder asked me: "So what exactly does this do that my sitemap doesn't?" Fair question. I realized I'd been lumping llms.txt and mcp.json together as generic "AI stuff" when they're actually solving completely different problems.

Here's what I should have said from the start: we're building websites for two audiences now. The first audience is human visitors who click through your navigation and read your pages. The second audience is AI agents that crawl your site, extract information, and either cite your content in chat responses or call your site's APIs to take actions on behalf of users.

Those agents need different signals than humans do. Your homepage hero section might look great to a visitor, but an AI assistant reading your HTML sees twenty CSS classes, three analytics scripts, and a pile of nested divs before it finds your actual value proposition. That's the gap llms.txt and mcp.json are filling.

I check this on PageSpeed Insights for every site we ship now. A perfect 3/3 Agentic Browsing audit score requires:

They're not competing standards. They're complementary files that tell AI agents two fundamentally different things about your site. One is a map. The other is a menu of actions.

llms.txt: A Content Map for AI Reading

Think of llms.txt as a table of contents written specifically for AI assistants. It's a Markdown file that lists your most important pages with brief descriptions, so when ChatGPT or Perplexity encounters your site, it knows exactly where the valuable content lives.

Instead of scraping your homepage HTML and guessing which links matter, the AI reads your llms.txt and finds your pricing page, your feature comparison, your case studies, your best blog posts. No guessing. No hallucinated URLs from cached training data.

llms.txt gives AI assistants a clean, structured summary of your most important pages, so they cite canonical content instead of scraping cluttered HTML.

I built this into version 183.71 of our website generator. Every site that runs the sitemap agent now gets an auto-generated llms.txt that includes:

The homepage with your site's core value proposition
Key product or feature pages with descriptions
Your most important blog posts or content
Contact and about pages

What it doesn't include: privacy policies, terms of service, legal disclaimers. Those aren't "important content" per the emerging standard. AI assistants don't need to cite your cookie policy when someone asks what your product does.

The practical impact: when someone asks ChatGPT "What does [your product] do?" or "How does [your company] approach X?", the AI can pull from your llms.txt map instead of guessing which page to read. It's the difference between an assistant citing your actual pricing page versus hallucinating a price from a cached blog post written two years ago.

Writing llms.txt by Hand

The format is dead simple. Here's a real example from one of our sites:

``markdown

My Product Name

https://example. com/
The homepage. We help developers ship faster by automating repetitive setup tasks.

https://example. com/pricing
Pricing page. Three tiers: Free, Pro ($29/mo), Team ($99/mo).

https://example. com/blog/how-we-built-this
Case study: how we reduced deployment time from 4 hours to 12 minutes.

https://example. com/docs
Documentation hub. Getting started guides and API reference.

https://example. com/contact
Contact page. Email support@example. com or use the form.
``

That's it. One URL per line, followed by a one-sentence description. No XML. No schema markup. Just plain Markdown that both humans and AI can read. Save it as llms.txt at your site root.

Retroactive llms.txt for Existing Sites

If you're adding llms.txt to a site that's already live, you need to decide which pages deserve to be in the map. Here's the heuristic I use:

Start with your sitemap.xml and filter for pages with real content (not legal pages, not redirects)
Rank pages by how often you want them cited when someone asks about your product
Keep the top 10-15 pages, write a one-line description for each
Deploy the file and add it to your robots.txt

The robots.txt part matters. Add this line so AI crawlers know the file exists:

``txt
User-agent: *
Allow: /llms.txt

Sitemap: https://yoursite.com/sitemap.xml
``

" width="800" height="450">

Most AI agents don't actively look for llms.txt yet. But the ones that do (including some custom research agents and newer Perplexity iterations) respect it immediately. You're getting in early, which means your content gets cited while competitors' sites get skipped.

mcp.json: An Action API for AI Interaction

Now here's where mcp.json (also called WebMCP) does something entirely different. It's not about reading your content. It's about calling your site. It's a machine-readable manifest that defines specific tools an AI agent can invoke to get structured data back.

Think of it as an API specification, but designed for AI agents instead of developers. A typical mcp.json file might define tools like:

get_pricing → returns your current pricing tiers with exact dollar amounts
get_contact → returns your support email and contact methods
get_faq → returns your frequently asked questions with answers
check_availability → returns whether a product or service slot is available

When an AI agent reads your mcp.json, it sees these tools and can call them during a conversation. A user asks "How much does the pro plan cost?" and instead of reading your pricing page HTML, the agent calls your get_pricing tool and gets back clean JSON with the exact answer. No scraping. No guessing. No stale cache.

mcp.json defines callable tools that give AI agents exact, structured answers: your real prices, your real email, your real product availability.

Here's the key architectural difference: llms.txt is passive. It's a map. The AI reads it, then reads your pages. mcp.json is active. It's a set of endpoints. The AI calls it and gets immediate structured responses.

As we explain in our structured data tips for AI-ready websites, these agent-facing files complement traditional SEO signals like sitemaps and JSON-LD rather than replacing them.

Writing a Basic mcp.json Manifest

The WebMCP spec is still evolving, but here's a minimal working example that defines two tools:

json { "name": "My Product API", "version": "1.0.0", "description": "Tools for interacting with My Product", "tools": [ { "name": "get_pricing", "description": "Returns current pricing tiers and features", "inputSchema": { "type": "object", "properties": {} } }, { "name": "get_contact", "description": "Returns contact email and support options", "inputSchema": { "type": "object", "properties": {} } } ], "endpoints": { "get_pricing": "https://example.com/api/pricing", "get_contact": "https://example.com/api/contact" } }

Each tool has a name, a description, and an endpoint URL. When an AI agent calls get_pricing, it hits your /api/pricing endpoint and expects JSON back. You control what data gets returned. The agent gets structured facts instead of parsing HTML.

Deploying mcp.json to Three Locations

Here's the annoying part: the WebMCP standard is new, and different AI agents expect the manifest in different places. To maximize compatibility, deploy your mcp.json file to three locations:

/.well-known/webmcp.json (the official spec location)
/mcp.json (root fallback for agents that skip .well-known)
/agent.json (some tools look for this name)

Yes, it's redundant. Yes, it's a pain. But until the standard solidifies, this triple deployment gives you the best chance of being discovered by agent frameworks that are experimenting with different conventions.

On most hosting platforms, you can set up a single source file and three symlinks or redirects. Here's the setup for a static site:

``bash

Create the master file

cat > .well-known/webmcp.json << EOF
{
"name": "My Product API",
"version": "1.0.0",
...
}
EOF

Create fallback copies

cp .well-known/webmcp.json mcp.json
cp .well-known/webmcp.json agent.json
``

If you're on Windows and using PowerShell, the workflow is similar but with different commands:

``powershell

Create the directory if needed

New-Item -ItemType Directory -Force -Path .well-known

Create the master file (use your actual JSON content here)

@"
{
"name": "My Product API",
"version": "1.0.0",
...
}
"@ | Out-File -FilePath .well-known\webmcp.json -Encoding utf8

Create fallback copies

Copy-Item .well-known\webmcp.json mcp.json
Copy-Item .well-known\webmcp.json agent.json
``

Deploy all three files. When the standard settles, you can drop two of them. Until then, redundancy wins.

What Makes a Good WebMCP Tool

Not every API endpoint should be a WebMCP tool. The best tools return facts that change or actions that matter. Here's what works:

Good tools:

get_pricing → returns current prices (changes when you update pricing)
check_availability → returns real-time availability (changes as inventory moves)
get_latest_post → returns your most recent blog post (changes when you publish)
search_docs → searches your documentation (user-specific results)

Bad tools:

get_about_page → just returns static HTML (llms.txt already maps this)
get_logo → returns an image URL (not structured data)
subscribe_newsletter → writes data (WebMCP is read-only for now)

The pattern: if an AI agent calling your tool gets back structured data that's either dynamic or user-specific, it's a good tool. If it's just returning static content that could be read from a page, llms.txt is the better fit.

I built AI Agent Ready to solve exactly this gap. It generates both files automatically, deploys them to the right locations, and validates that your site passes the Agentic Browsing checks. If you're shipping multiple sites or managing a portfolio, automation is the only way to keep up.

Why Deploy Both (Even Though Adoption Is Still Low)

Neither llms.txt nor mcp.json is widely adopted yet. Most AI crawlers don't fully honor either one. So why am I building both into every site we ship?

Because the cost is near zero and the positioning advantage is real.

Here's my current ranking of what actually moves the needle for AI visibility today:

Sitemap + robots.txt that allow AI bots (most impactful right now, every major AI crawler respects this, non-negotiable)
JSON-LD structured data (actively used by Google today, immediate SEO benefit)
llms.txt (emerging standard with growing adoption for AI-search citation, low cost, future-facing)
mcp.json / WebMCP (most forward-looking, least adopted, but the agent-actions layer that could matter enormously as browsing agents mature)

The first two are table stakes. If your site doesn't have a sitemap or blocks AI bots in robots.txt, you're invisible. The second two are bets on where agent behavior is heading in the next 6-12 months.

Google added a fifth category to PageSpeed Insights in May 2026 called Agentic Browsing. It doesn't give a score out of 100; instead it shows a ratio like 3/3.

llms.txt present and properly formatted
mcp.json present with at least one callable tool
sitemap.xml with all key pages included
JSON-LD structured data on major pages
passing accessibility checks (agents rely on semantic HTML)

Most sites still score 0/3 or 1/3. Getting to 3/3 takes an afternoon of work. That gap is your positioning advantage right now.

The Four-Layer Agent-Ready Stack

Here's how I think about the full stack for agent-ready websites. You need all four layers:

1. Discoverable (Sitemap + Robots.txt)

AI crawlers need to know your site exists and which pages to index. This is the foundation. Without a sitemap, you're invisible.

2. Understandable (JSON-LD Structured Data)

Your pages need machine-readable context. JSON-LD tells AI agents what kind of page they're looking at (article, product, organization) and extracts key fields (author, price, date). This is how Google's Knowledge Graph works, and it's how AI search engines will cite you.

For implementation patterns, the JSON prompt guide for AI agents walks through specific schema markup for different page types.

3. Readable (llms.txt)

Once an AI agent lands on your site, llms.txt tells it which pages matter most. This is the content map layer. It answers: "If you can only read five pages, read these five."

4. Interactive (mcp.json)

Finally, mcp.json exposes callable tools that let AI agents get exact answers without scraping. This is the actions layer. It answers: "If you need real-time data or want to verify a fact, call these endpoints."

Each layer builds on the previous one. You can't skip straight to mcp.json if your sitemap blocks AI bots. But once you have layers 1 and 2 in place, adding layers 3 and 4 is straightforward.

What Happens If You Skip This

Here's the risk: your competitors deploy llms.txt and mcp.json, and their content starts getting cited in AI-driven search results while yours gets skipped. A user asks "What are the best tools for X?" and ChatGPT cites three competitors but not you, because their sites gave the AI a clean map and yours made it scrape HTML.

Or worse: a user asks "How much does [your product] cost?" and the AI hallucinates a price from a cached blog post written two years ago, because you didn't expose a get_pricing tool that returns your current pricing.

Agent-ready sites will win citations. Sites that ignore this layer will become invisible to the next generation of search behavior. That's not speculation. It's already happening in vertical AI search engines that prioritize structured data over keyword optimization.

Start With llms.txt, Add mcp.json Later

If you're resource-constrained, start with llms.txt. It's a single Markdown file. You can write it by hand in ten minutes. Deploy it to your site root, add it to robots.txt, and you're done.

Then, when you have time to build API endpoints, add mcp.json. Start with one or two simple tools like get_pricing or get_contact. Deploy the manifest to all three locations. Test it by checking the Agentic Browsing score on PageSpeed Insights.

That's the pragmatic path: llms.txt this week, mcp.json next month. Both files are small. Both are future-facing. Both give you positioning advantage while most sites are still ignoring this layer entirely.

But even if you're doing this manually for a single site, the work is worth it. We're not optimizing for today's AI agents. We're optimizing for the ones that launch six months from now, when agent-driven search is the default and sites without llms.txt and mcp.json are invisible.

📦 Publishing Kit — Dev.to

Title Options (5)

Selected: llms.txt vs mcp.json: Why Your Site Needs Both (And What They Actually Do)

Alternates:

AI Agents Need Two Things From Your Site: A Map and a Menu
llms.txt and mcp.json Aren't Competing—They're Solving Different Problems
How to Make Your Site Readable and Actionable for AI Assistants
The Real Difference Between llms.txt and mcp.json (With Implementation Examples)

Slug

llms-txt-vs-mcp-json-why-your-site-needs-both

Tags

webdev, ai, tutorial, llms

The Domain Extension Wars: Why .ai and .app Are Crushing .com

Julian Neagu — Mon, 29 Jun 2026 11:57:37 +0000

TL;DR: Tech companies are ditching .com for specialized TLDs like .ai, .app, and .dev. These extensions offer better SEO performance, clearer brand positioning, and signal innovation to investors. Anguilla earned $32 million from .ai domains alone in 2023 — that's 20% of their government revenue.

The domain extension wars are heating up, and .com is losing ground fast. When I see a startup pitch deck with a .ai domain, my immediate reaction is different than seeing the same company with a .com extension. One signals cutting-edge innovation; the other feels like they settled for what was available.

This shift isn't just cosmetic. New TLDs like .app, .dev, and .ai are becoming strategic business assets that communicate industry focus, improve search rankings, and attract investor attention in ways that generic .com domains simply can't match.

The Economics Behind the TLD Revolution

The numbers tell the story. In 2023 alone, the tiny Caribbean island of Anguilla generated approximately $32 million in revenue from .ai domain registrations — that's roughly 20% of their entire government budget. When a two-letter country code becomes a major revenue stream, you know something fundamental has shifted in how businesses think about domains.

According to the IMF, .ai domain revenue accounted for roughly 20% of Anguilla's total government revenues in 2023.

This explosion in .ai adoption reflects a broader trend: companies are choosing domains that immediately communicate their business focus rather than settling for generic .com extensions. The semantic clarity of these new TLDs creates instant brand alignment that traditional domains can't provide.

Understanding the TLD Landscape

A Top-Level Domain (TLD) is the suffix that follows the final dot in a web address — the ".com" in "google.com" or the ".ai" in "openai.ai". While this might seem like a technical detail, TLDs have evolved into powerful branding and positioning tools.

Traditional TLDs served their purpose in the early internet era, but they lack the contextual precision that modern businesses need. When every company from law firms to restaurants uses .com, the extension becomes meaningless as a differentiator.

For a deeper dive into how TLDs function and their technical implications, I covered the fundamentals in my comprehensive TLD guide.

How TLDs Shape User Perception

Users make split-second judgments based on domain extensions. A .dev domain immediately signals that you're looking at a developer-focused platform. A .ai extension suggests artificial intelligence capabilities, making clear brand and metadata signals even more important. These associations happen before users even read your content.

Consider these psychological triggers:

.ai domains create expectations of machine learning, automation, and cutting-edge technology
.app domains suggest mobile-first experiences and software applications
.dev domains signal developer tools, programming resources, and technical expertise
.com domains feel generic and established but lack industry specificity

Comparison Table

The .com vs Modern TLD Breakdown

The fundamental difference between .com and semantic TLDs comes down to signal clarity. A .com domain forces users to guess what your business does. A .ai domain tells them immediately.

Search Engine Optimization Advantages

Modern TLDs offer distinct SEO benefits that .com domains can't match. Search engines increasingly prioritize semantic matching between domain extensions and user intent. When someone searches for "AI writing tools," a website with a .ai domain has a natural advantage over the same content on a .com domain.

Key SEO factors favoring newer TLDs:

Keyword alignment: .ai domains naturally rank better for AI-related searches
Click-through rates: Users click on domains that match their search intent
Technical standards: Many new TLDs enforce HTTPS by default, boosting Google rankings
Core Web Vitals: Newer domain infrastructure often performs better on speed metrics

Investment and Market Appeal

Venture capitalists and angel investors are paying attention to domain choices. A well-chosen TLD signals that founders understand digital positioning and are thinking strategically about their brand presence.

Investors increasingly view .ai, .app, and .dev domains as indicators of future success and alignment with digital transformation trends.

This isn't superficial preference — it reflects deeper business acumen. Companies that choose strategic TLDs often demonstrate the same forward-thinking approach in product development and market positioning.

Why Businesses Are Abandoning .com

The migration away from .com stems from three core limitations: positioning weakness, SEO disadvantages, and availability constraints.

Positioning and Brand Clarity

When a business uses .app, .dev, or .ai, they immediately communicate their industry focus. This semantic precision helps companies:

Stand out in crowded markets
Build instant credibility with target audiences
Align their digital identity with their core services
Create memorable brand associations

Compare the positioning power: "mycompany.com" versus "mycompany.ai" for an artificial intelligence startup. The .ai extension does heavy lifting that .com simply cannot provide.

SEO Performance Gap

Search engines have evolved to understand semantic relationships between domain extensions and content. A .ai website discussing machine learning has inherent relevance advantages over identical content on a .com domain.

This semantic matching translates into:

Higher click-through rates from search results
Better keyword rankings for industry-specific terms
Improved user engagement metrics
Enhanced local and mobile search performance

The Availability Crisis

The .com namespace is effectively saturated. Finding a short, memorable .com domain that matches your business name requires either significant compromise or substantial financial investment, making strategic online positioning more important for new brands. Premium .com domains regularly sell for six or seven-figure amounts.

Meanwhile, newer TLDs offer:

Greater name availability for exact business matches
Shorter, more memorable domain options
Keyword-rich domains that support SEO strategies
Lower acquisition costs for premium names

Strategic TLD Selection by Business Type

Different business models benefit from different TLD strategies. The key is matching your domain extension to your primary value proposition and target audience expectations.

For a detailed comparison of .app versus .dev domains specifically, check out my analysis of these two popular extensions.

AI and Machine Learning Companies

Choose .ai for businesses focused on artificial intelligence, machine learning, automation, or data science. This extension immediately communicates technical sophistication and innovation.

Examples of effective .ai positioning:

SaaS platforms with AI-powered features
Machine learning consulting firms
Automated workflow tools
Predictive analytics platforms

Software Applications and Mobile Apps

Choose .app for companies building software applications, mobile apps, or digital tools. This extension suggests user-friendly interfaces and practical software solutions.

Ideal .app use cases:

Mobile app development companies
SaaS productivity tools
Consumer software platforms
Digital product marketplaces

Developer Tools and Technical Services

Choose .dev for businesses serving developers, offering programming tools, or providing technical infrastructure. This extension builds immediate credibility with technical audiences.

Perfect .dev applications:

API services and documentation
Developer tool companies
Code hosting and collaboration platforms
Technical consulting services

The Future of Domain Strategy

The trend toward semantic TLDs represents a fundamental shift in how businesses approach digital identity. Domains are evolving from simple web addresses into strategic brand assets that communicate value propositions before users even visit your site.

Companies that embrace this evolution gain competitive advantages in search rankings, user perception, and investor appeal. Those clinging to generic .com domains risk being perceived as outdated or unfocused.

The data supports this shift: when a small island nation can generate $32 million annually from a two-letter domain extension, it's clear that businesses see real value in semantic domain positioning.

As AI continues to reshape every industry, expect .ai domains to become even more valuable. Similarly, as software development becomes increasingly central to business operations, .dev and .app domains will likely appreciate in both practical utility and market value.

The question isn't whether semantic TLDs will replace .com — it's how quickly businesses will recognize that their domain extension is now a core component of their competitive positioning strategy.

📦 Publishing Kit — Dev.to

Title Options (5)

Selected: The Domain Extension Wars: Why .ai and .app Are Crushing .com

Alternates:

Why Smart Startups Are Ditching .com for .ai and .dev Domains
The $32M TLD Revolution: How New Domain Extensions Are Changing Tech
From .com to .ai: The Strategic Domain Shift Reshaping Startups
The Death of .com? Why Modern TLDs Are the New Business Standard

Slug

domain-extension-wars-ai-app-crushing-com

Tags

webdev, startup, branding, domains

AI Agent Architecture vs Workflow vs Modularity Know The Difference

Julian Neagu — Tue, 23 Jun 2026 11:45:09 +0000

TL;DR: Architecture defines system structure and roles, workflows control execution paths, and modularity enables component reuse. Focus on architecture for consistency issues, workflows for performance problems, and modularity for scaling pain. Start simple, then evolve.

Building AI agents that actually work in production requires understanding three distinct but interconnected concepts: architecture, workflow, and modularity. Most teams confuse these layers, leading to systems that demo well but crumble under real users.

Think of architecture as the wiring in your house, workflows as the electricity flowing through it, and modularity as the replaceable appliances you can swap without rewiring everything. Get these distinctions right, and you'll design cleaner systems that scale predictably.

The stakes are real. Teams that master these three layers ship faster, debug easier, and collaborate without stepping on each other's code. Those who don't end up rewriting everything when requirements change.

What Is AI Agent Architecture?

Architecture is your system's blueprint — the static map that defines who talks to whom, how data flows, and where decisions get made. It's not about the specific steps your agents take; it's about the foundation that makes those steps possible.

The core roles form a clear division of labor. The Planner breaks high-level goals into actionable tasks. The Router decides which agent or tool handles each task. The Executor does the actual work — calling APIs, processing data, or generating content. Memory preserves context and state across interactions. Guardrails enforce safety policies, access controls, and audit requirements.

// Example: Basic agent architecture
class AgentSystem {
  constructor() {
    this.planner = new TaskPlanner();
    this.router = new AgentRouter();
    this.memory = new ContextMemory();
    this.guardrails = new SafetyLayer();
  }

  async process(request) {
    const plan = await this.planner.createPlan(request);
    const route = this.router.selectAgent(plan.nextTask);
    return this.guardrails.validate(
      await route.execute(plan, this.memory.getContext())
    );
  }
}

Architecture also defines your data paths and decision points. How does information flow from initial request to final response? Where do you transform, validate, or enrich data? What happens when something fails?

Clean architecture prevents the most common AI system failures: inconsistent behavior, unclear hand-offs, and security gaps.

Good architecture centralizes policy enforcement. Instead of scattering safety checks across every agent, you build guardrails once and route everything through them. Role-based access control, PII handling, and audit logging become system-wide properties, supported by structured compliance audit processes, not per-agent responsibilities.

What Is an AI Workflow?

While architecture is your blueprint, workflow is what happens when you flip the switch. It's the step-by-step execution path that turns user intentions into actual results.

A typical workflow might look like: User query → Router analyzes intent → Research Agent gathers sources → Summary Agent condenses findings → Quality Reviewer validates output → System returns formatted response.

# Example: Debugging a slow workflow
curl -X POST http://localhost:3000/research \
  -d '{"query": "latest AI regulations", "depth": "comprehensive"}' \
  --trace-time

Workflows come in different patterns. Linear flows move predictably from step A to B to C — easy to test and debug. Branching flows add conditional logic based on content type, user permissions, or quality thresholds. Parallel flows speed things up by running independent tasks simultaneously. Event-driven flows react to triggers, webhooks, or queue messages.

The workflow layer is where performance becomes visible. You'll see bottlenecks, retry loops, and escalation paths play out in real time. A research workflow that takes 30 seconds usually has one slow step, not thirty slow ones.

Common workflow problems include: agents waiting unnecessarily for sequential tasks that could run in parallel, retry logic that creates infinite loops when external APIs fail, and handoff points where context gets lost between agents.

What Goes Wrong in Workflows

Here's a real error you might encounter:

Error: Research agent timeout after 45s
Stack: query="climate policy 2024" → router → research_agent → [TIMEOUT]
Cause: External API rate limit (429) → retry loop → eventual timeout

The fix usually involves adding proper retry backoff and parallel source gathering:

// Instead of sequential API calls
const sources = [];
for (const url of urls) {
  sources.push(await fetchWithRetry(url)); // Slow!
}

// Use Promise.allSettled for parallel fetching
const results = await Promise.allSettled(
  urls.map(url => fetchWithRetry(url, { backoff: 'exponential' }))
);

Start with simple linear workflows before adding branches or complexity. You can always optimize later once you understand where the bottlenecks actually occur.

What Is Modularity in AI Systems?

Modularity treats your AI stack like LEGO blocks. Each component has one clear job and connects to others through well-defined interfaces. When you need to upgrade your summarizer or swap language models, you replace one block without touching the rest.

The key principle: one responsibility per module. A retriever module only retrieves. A reranker only reranks. A formatter only formats. When each module has a single, clear purpose, the whole system becomes predictable.

Modular systems let teams ship upgrades in hours instead of months.

Real modularity benefits become obvious during maintenance. Imagine you built a research system six months ago using GPT-3.5. Today you want to upgrade to GPT-4. In a modular system, you swap the language model module and you're done. In a monolithic system, you're hunting through code to find every place that calls the old API.

# Modular approach: swap implementations easily
class SummarizerInterface:
    def summarize(self, text: str, max_length: int) -> str:
        pass

class GPT35Summarizer(SummarizerInterface):
    def summarize(self, text: str, max_length: int) -> str:
        # GPT-3.5 implementation
        pass

class GPT4Summarizer(SummarizerInterface):
    def summarize(self, text: str, max_length: int) -> str:
        # GPT-4 implementation
        pass

# Swap models without changing workflow code
summarizer = GPT4Summarizer()  # Was: GPT35Summarizer()
result = workflow.run(summarizer=summarizer)

Teams collaborate more effectively with clear module boundaries. The retrieval team owns the search module. The safety team owns the guardrails module. The UI team owns the formatting module. Each team can ship improvements without coordinating every change.

Consider building modules that solve these common needs across multiple workflows. I cover the security implications in detail in my AI agent security analysis, but the core insight is that reusable security modules prevent teams from rebuilding authentication and audit logging for every new agent.

When to Focus on Each Layer

System problems usually stem from one specific layer. Identifying the root cause helps you fix the right thing.

Focus on architecture when:

Agents behave inconsistently across similar requests
You're duplicating safety checks or access controls
Teams are stepping on each other's code
Adding new capabilities requires touching multiple files

Improve workflow when:

Requests take too long or stall unpredictably
Users abandon tasks before completion
You can't easily trace what went wrong
Similar tasks have wildly different performance

Invest in modularity when:

Upgrades require rewriting large portions of code
Testing one component breaks others
Teams can't work independently
You're rebuilding similar functionality across projects

How Architecture, Workflow, and Modularity Interconnect

These three layers form a virtuous cycle. Architecture provides the foundation and interfaces. Workflows run on that foundation and expose friction points. Modularity enables you to swap components and extend capabilities without rebuilding everything.

The cycle looks like this: solid architecture creates clear contracts → workflows reveal bottlenecks and edge cases → modular components let you optimize specific pieces → improved components make workflows faster → better workflows stress-test the architecture → architectural improvements unlock new modular possibilities.

Teams that master this cycle ship multi-agent systems that actually scale. They add new capabilities by plugging in modules, not by rewriting core logic. They optimize performance by swapping faster components, reducing technical debt through modular refactoring instead of hunting through monolithic code.

Practical Example: Building a Research System

Let's trace how these concepts work together in a real system. You want to build a tool that researches topics, synthesizes sources, and returns clean, cited answers.

Architecture: Your system needs a Planner to break research goals into searchable questions, a Data Fetcher to pull credible sources, a Summarizer to synthesize evidence, and a Reviewer to enforce citation standards and accuracy. Memory stores context between research steps. Guardrails validate source credibility and prevent harmful outputs.

Workflow: User submits research query → Planner generates search terms → Data Fetcher runs parallel searches → Results get filtered for credibility → Summarizer creates draft with citations → Reviewer checks accuracy and completeness → System returns formatted research brief.

Modularity: Each agent is swappable. Upgrade from basic web search to academic database access by replacing the Data Fetcher module. Improve summary quality by swapping the Summarizer. Add fact-checking by plugging in a new Reviewer module.

When something goes wrong, the layer tells you where to look. Inconsistent citation formats? That's architecture — centralize formatting rules. Slow research? That's workflow — parallelize the search phase. Hard to add new source types? That's modularity — abstract the fetcher interface.

Start simple with linear workflows and basic modularity. Add branching logic and sophisticated modules as you learn where optimization matters. The goal isn't perfect architecture on day one — it's a system that evolves intelligently as requirements change.

The teams shipping the most impressive AI systems aren't the ones with the most complex architectures. They're the ones who understand when to focus on structure, when to optimize flow, and when to increase modularity. Master those decisions, and your AI systems will scale from clever demos to dependable products.

📦 Publishing Kit — Dev.to

Title Options (5)

Selected: AI Agent Architecture vs Workflow vs Modularity: The Complete Guide to Building Production Systems

Alternates:

Understanding AI Agent Architecture, Workflows, and Modularity for Production-Ready Systems
The Three Pillars of AI Agent Development: Architecture, Workflow, and Modularity Explained
Building Scalable AI Agents: Mastering Architecture, Workflow, and Modularity Design
From Demo to Production: Understanding AI Agent Architecture, Workflows, and Modular Design

Slug

ai-agent-architecture-vs-workflow-vs-modularity-guide

Tags

ai, architecture, tutorial, agents

I Deployed 44 Live Websites in 10 Minutes With One PowerShell Script (Here's How)

Julian Neagu — Sat, 20 Jun 2026 14:27:25 +0000

TL;DR: I deployed 44 live websites in a single 10-minute session using a single PowerShell script. Zero manual clicks. Zero dashboard logins. This is how you operate 1,000+ domains as a solo founder without losing your mind.

Most solo founders hit a wall around product three or four. They talk about building. They don't talk about what happens when you're managing infrastructure for hundreds of live products at once.

I own 1,000+ premium .app domains, all listed through my domain portfolio & sales overview. Every single one needs to be a live landing page. Every one needs its own deployment, DNS configuration, security headers, sitemap, robots.txt file, and SSL certificate. Do that manually and you're staring at weeks of clicking through GoDaddy dashboards, copy-pasting DNS records one domain at a time, waiting for propagation, verifying each one individually.

That's not a workflow. That's a trap.

So I built a pipeline that turns all of it into one command.

The Three-Tool Stack That Makes This Possible

GitHub serves as the single source of truth. One monorepo. One subfolder per domain. Every domain folder contains exactly 11 files: the HTML landing page, hero image, Open Graph image, sitemap, robots.txt, manifest, favicons. Push to GitHub and everything downstream updates automatically.

Vercel CLI handles all deployments. Not the web dashboard. The command-line interface. Every subfolder becomes its own independent Vercel project. Each deployment takes about 6 seconds. The custom domain gets attached automatically in the same command.

GoDaddy API manages DNS records. Instead of logging into GoDaddy's dashboard and editing A records one by one, PowerShell calls the REST API directly and writes the record instantly. No forms. No waiting for pages to load.

One script coordinates all three. That's the entire deployment pipeline.

The Monorepo Structure

Here's what the repo looks like:

plaintext visionvix-domains/ ├── deploy.ps1 ← one script runs everything ├── domains.txt ← one domain per line │ ├── accessibilityaudit.app/ │ ├── index.html │ ├── og.jpg │ ├── hero.webp │ ├── sitemap.xml │ ├── robots.txt │ └── manifest.json │ ├── accountingassistant.app/ │ └── ... same 11 files │ └── ... one folder per domain

Simple. Consistent. Scalable. Every domain folder is identical in structure. The only thing that changes is the content inside index.html. Each landing page is customized for its specific AI tool or service, but the file structure never varies.

The entire repo is public on GitHub. That's intentional. It's a marketing asset as much as a technical one. Any developer who finds the repo can see the scale immediately: hundreds of domain folders, all consistently structured, all live. It shows what's possible as a solo founder operating with the right tools.

The Script That Does Everything

Here's the core deployment loop. Three API calls. One iteration. Zero manual steps.

On Windows, the deployment script looks like this:

`powershell$domains = Get-Content .\domains.txt

foreach ($domain in $domains) {
$slug = $domain -replace '.app$', ''

# Step 1: Deploy to Vercel (takes ~6 seconds per domain)
vercel deploy .\$domain --prod --yes --name $slug

# Step 2: Attach the custom domain and www subdomain
vercel domains add $domain $slug
vercel domains add www.$domain $slug

# Step 3: Write the A record to GoDaddy via REST API
Invoke-RestMethod -Method Put -Uri "https://api.godaddy.com/v1/domains/$domain/records/A/@"
-Headers $headers `
-Body '[{"data":"76.76.21.21","ttl":600}]'

# Step 4: Verify DNS was written correctly
$rec = Invoke-RestMethod -Method Get -Uri "https://api.godaddy.com/v1/domains/$domain/records/A/@"
-Headers $headers
Write-Host "✓ DNS configured: $($rec.data)"
}`

That's it. Every domain in the domains.txt file goes through all four steps automatically, one after another.

The $headers variable contains the GoDaddy API key and secret, stored in Windows environment variables. They're set once at the system level, available to every PowerShell session, never written to any file, never pushed to GitHub.

The Vercel A Record

The IP address 76.76.21.21 is Vercel's production A record. Every domain points there. Vercel's edge network handles routing to the correct project based on the domain name. The TTL is set to 600 seconds (10 minutes), which is low enough to make changes quickly if needed but high enough to avoid excessive DNS queries.

Why the CLI Instead of the Dashboard

The Vercel web dashboard works great for one or two projects. It completely breaks down at scale. Clicking through forms, waiting for pages to load, copying project URLs, manually adding domains. That's dozens of clicks per domain, which is exactly the kind of inefficiency solved by top workflow automation tools for small teams.

The CLI turns all of that into one command per domain. vercel deploy pushes the code, builds it, and returns a production URL. vercel domains add attaches the custom domain. Both commands are fully scriptable. That's what makes automation possible.

The Numbers After 44 Deployments

The first run was 5 domains. Then 38 more. Then the full 44 batch, all live, all verified, all returning HTTP 200.

44 domains deployed in a single session. 6 seconds average deploy time per domain. 0 manual clicks required. 100% success rate.

Metric	Result
Domains deployed in first session	44
Average deploy time per domain	~6 seconds
Manual clicks required	0
DNS propagation time	5-30 minutes
Total time from script to live	Under 10 minutes
HTTP 200 success rate	100%

When I ran the verification loop at the end, checking every domain for a 200 status code, every single one came back green. No failures. No retries. No manual fixes.

Security Headers Built Into Every Domain

Every domain automatically gets a full set of security headers via a vercel.json file that lives inside each domain folder. No extra deployment steps. No separate configuration, and can be validated with a website security audit tool. The headers deploy with the site.

Here's what every domain gets out of the box:

json { "headers": [ { "source": "/(.*)", "headers": [ { "key": "Content-Security-Policy", "value": "default-src 'self'; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline';" }, { "key": "Strict-Transport-Security", "value": "max-age=63072000; includeSubDomains; preload" }, { "key": "X-Frame-Options", "value": "DENY" }, { "key": "X-Content-Type-Options", "value": "nosniff" }, { "key": "Referrer-Policy", "value": "strict-origin-when-cross-origin" }, { "key": "Permissions-Policy", "value": "camera=(), microphone=(), geolocation=(), payment=()" } ] } ] }

Content-Security-Policy blocks malicious scripts and injection attacks. Strict-Transport-Security forces HTTPS for 2 years and covers all subdomains. X-Frame-Options: DENY prevents clickjacking. X-Content-Type-Options stops MIME type sniffing. Referrer-Policy controls how much referrer data leaks. Permissions-Policy disables camera, microphone, geolocation, and payment APIs.

Static assets like images and fonts are cached for a full year. The HTML is never cached, so users always get the latest version instantly.

Why These Architecture Decisions Matter

Why a Public GitHub Monorepo?

Security is handled without secrets ever touching the repo. GoDaddy API keys live in Windows environment variables only. Set once at the system level. Available to every PowerShell session. Never written to any file. Never pushed to GitHub.

The repo itself is public because it's a marketing asset. Developers who find it see the scale immediately. Hundreds of domain folders. All consistently structured. All live. It shows what's possible as a solo founder with the right automation in place.

Why Vercel Over the Alternatives?

The CLI is fully scriptable. That's the only reason that matters at this scale. Netlify and Cloudflare Pages both have solid products, but Vercel's CLI lets you deploy a subfolder, attach a domain, and get a production URL in a single command. That's what makes automation possible.

I can run the same script on 5 domains or 500 domains. The process doesn't change. The complexity doesn't increase.

Why GoDaddy API Instead of Manual DNS?

There is no "manual" at 600 domains. Logging into GoDaddy's dashboard and editing A records one by one would take days. The GoDaddy REST API accepts a PUT request with the new A record value and applies it instantly. PowerShell's Invoke-RestMethod makes the call in one line. The entire DNS step takes under a second per domain.

For readers who want to dive deeper into the strategy behind premium .app domains, I covered the positioning and branding implications in my earlier post on AI domain strategy.

What's Next: 1,000 Domains by Next Week

" width="800" height="482">

44 domains is the proof of concept. The pipeline works. Now it's just execution at scale.

The plan for the next 7 days:

Deploy the remaining 950+ domains using the same script
Index all pages with Google Search Console via sitemap submission
Track success rates for deployment, DNS propagation, and indexing
Monitor performance across the entire domain portfolio
Iterate on the landing page template based on traffic patterns

The script doesn't care if it's running on 50 domains or 500. The time per domain stays constant. The manual effort stays at zero.

The Bigger Point

This isn't really about domains. It's about treating infrastructure as code. One person can operate at enterprise scale when every manual process is replaced with an API call.

The traditional approach to scaling infrastructure is to hire people. You need someone to manage DNS. Someone to handle deployments. Someone to configure security. Someone to monitor uptime.

The alternative is to script everything once and run it a thousand times.

I'm not managing 1,000 domains. I'm managing one script that manages 1,000 domains. That's the difference between a job and a system.

📦 Publishing Kit — Dev.to

Title Options (5)

Selected: I Deployed 44 Live Websites in 10 Minutes With One PowerShell Script (Here's How)

Alternates:

How I Deploy 1,000+ Premium Domains as a Solo Founder Without Losing My Mind
Zero-Click Deployment Pipeline: Managing Hundreds of Live Domains With GitHub, Vercel & GoDaddy API
From Manual Hell to One-Command Deploy: Automating 1,000+ Domain Infrastructure
The Three-Tool Stack That Let Me Deploy 44 Websites in a Single 10-Minute Session

Slug

i-deployed-44-live-websites-in-10-minutes-with-one-powershell-script

Tags

devops, automation, productivity, domains

Building React Components: From JSX to Production-Ready Code

Julian Neagu — Fri, 19 Jun 2026 13:40:18 +0000

TL;DR: React components are reusable UI building blocks that combine structure, logic, and styling in isolated modules. Function components with Hooks are now the standard, replacing classes for cleaner, more maintainable code that scales across modern frameworks.

React components form the backbone of every React application. They break your user interface into small, logical pieces that handle their own structure, styling, and behavior independently. This modular approach transforms how developers build applications—making code more maintainable, bugs easier to isolate, and features faster to ship.

The numbers tell the story of React's dominance. According to JetBrains' 2024 Developer Ecosystem Report, 57% of JavaScript developers use React as their primary library. On NPM, React now pulls over 20 million weekly downloads, cementing its position as the most widely adopted UI framework globally.

React exceeds 20 million weekly downloads on NPM, confirming its position as the most adopted UI framework in the world.

What Makes a Component Work

A React component is fundamentally a JavaScript function that returns JSX—a blend of HTML-like syntax and JavaScript logic. Think of it as a custom HTML element that you define once and reuse anywhere.

function Welcome({ name }) {
  return <h1>Hello, {name}!</h1>;
}

This simple component takes a name prop and renders a personalized greeting. You can use it multiple times with different data:

<Welcome name="Sarah" />
<Welcome name="Mike" />
<Welcome name="Alex" />

Each instance behaves identically but displays unique content based on the props you pass in. This pattern—write once, use everywhere—drives React's efficiency.

JSX and TSX: The Language of Components

JSX Explained

JSX (JavaScript XML) lets you write HTML-like code directly in JavaScript. Behind the scenes, React transforms this syntax into function calls that create virtual DOM elements.

const element = <div className="header">Welcome to React</div>;

This compiles to:

const element = React.createElement('div', { className: 'header' }, 'Welcome to React');

JSX makes component code readable and intuitive, especially for developers familiar with HTML.

TSX and Type Safety

TSX extends JSX with TypeScript's type system. It catches errors before your code runs and provides better autocomplete in editors.

interface ButtonProps {
  label: string;
  onClick: () => void;
  disabled?: boolean;
}

const Button = ({ label, onClick, disabled = false }: ButtonProps) => (
  <button onClick={onClick} disabled={disabled}>
    {label}
  </button>
);

90.6% of frontend developers now use TypeScript, making TSX the standard format for professional React development.

Component Architecture Breakdown

Every React component has four essential parts that work together to create interactive UI elements.

Props: External Input

Props are data passed from parent to child components. They define what information a component receives and how it should behave.

<ProductCard 
  name="Digital Camera" 
  price={299} 
  inStock={true} 
/>

Props make components flexible. The same ProductCard component can display any product by changing the props you pass to it.

State: Internal Data

State holds data that can change during a component's lifetime. When state updates, React automatically re-renders the component with the new data.

const [count, setCount] = useState(0);

const increment = () => setCount(count + 1);

State enables interactivity—user actions trigger state changes, which update the UI immediately.

Hooks: Lifecycle Management

Hooks like useState, useEffect, and useRef give function components access to React's lifecycle features without writing class components.

useEffect(() => {
  document.title = `Count: ${count}`;
}, [count]);

Return: UI Output

The return statement defines what the component renders to the screen. It must return a single JSX element (or fragment).

Function vs Class Components: The Modern Standard

React originally used class components, but the introduction of Hooks in 2019 shifted the entire ecosystem toward function components.

Class Component (Legacy):

class Welcome extends React.Component {
  constructor(props) {
    super(props);
    this.state = { message: 'Hello' };
  }

  render() {
    return <h1>{this.state.message}, {this.props.name}</h1>;
  }
}

Function Component (Modern):

function Welcome({ name }) {
  const [message, setMessage] = useState('Hello');
  return <h1>{message}, {name}</h1>;
}

Function components require less boilerplate, avoid this binding issues, and integrate naturally with modern tooling. The React team now recommends function components for all new development, especially as teams move toward cleaner code maintainability practices.

The Power of Reusability

React's component model promotes code reuse through composition. Instead of copying similar code, you create one component and customize it with props.

<Button variant="primary" size="large">Save Changes</Button>
<Button variant="secondary" size="small">Cancel</Button>
<Button variant="danger" size="medium">Delete Account</Button>

The same Button component handles three different use cases by changing props. This approach keeps your codebase smaller and more consistent.

A Stack Overflow 2024 Developer Survey found that over 70% of React developers cite "reusability" as the primary reason for adopting React in their projects. When teams can build once and reuse everywhere, development velocity increases dramatically.

Over 70% of React developers cite "reusability" as the primary reason for adopting React in their projects.

Universal Building Blocks Across Frameworks

React components aren't limited to basic React apps. They power the entire modern JavaScript ecosystem:

Next.js uses components to define pages (page.tsx) and layouts (layout.tsx)
Gatsby generates static sites from React components
Remix renders components server-side for faster performance
Expo builds mobile apps using React Native components

In Next.js 14's App Router, every file in the app/ directory exports a React component. This structure automatically converts your file system into routes—no configuration required.

Under the Hood: JSX Compilation

React doesn't understand JSX natively. Build tools like Babel or SWC compile JSX into JavaScript before it runs in browsers.

Before React 17:

React.createElement('div', { className: 'container' }, 'Hello World');

React 17+ (New JSX Transform):

import { jsx as _jsx } from 'react/jsx-runtime';
_jsx('div', { className: 'container', children: 'Hello World' });

This modern transform eliminates the need to import React in every file and makes builds faster. Most developers never see this compiled output, but understanding it helps debug complex JSX issues.

Streamlined Development with Modern Tooling

For teams working on complex React projects, automation tools can significantly speed up development workflows. When migrating from basic React apps to production-grade Next.js structures, manual conversion becomes time-consuming and error-prone.

Modern development practices emphasize consistency—from file organization to code formatting. Teams often spend hours manually restructuring components, organizing folders, and ensuring proper TypeScript types when scaling projects.

I covered similar automation patterns in my cursor best practices guide, where consistent tooling and formatting become critical as codebases grow beyond simple prototypes.

Component Design Best Practices

Keep Components Focused

Each component should handle one responsibility. A UserProfile component manages user data display, while a separate EditUserForm handles user data updates.

Use Descriptive Names

Component names should immediately convey purpose:

ProductCard instead of Card
SignupForm instead of Form
PaymentButton instead of Button

Embrace Composition

Build complex interfaces by combining simple components rather than creating monolithic components with many features.

<Card>
  <CardHeader title="Product Details" />
  <CardBody>
    <ProductImage src={product.image} />
    <ProductInfo name={product.name} price={product.price} />
  </CardBody>
  <CardFooter>
    <AddToCartButton productId={product.id} />
  </CardFooter>
</Card>

Type Everything

Add TypeScript interfaces for all props to catch errors early and improve developer experience:

interface ProductCardProps {
  product: {
    id: string;
    name: string;
    price: number;
    image: string;
  };
  onAddToCart: (productId: string) => void;
}

Common Component Mistakes to Avoid

Mixing Concerns: Don't handle data fetching, business logic, and UI rendering in the same component. Separate these responsibilities into different layers.

Prop Drilling: Avoid passing props through multiple component layers. Use React Context or state management libraries for deeply nested data.

Ignoring Keys: Always provide unique key props when rendering lists to help React optimize re-renders.

Mutating Props: Never modify props directly. Treat them as read-only data.

Overusing useEffect: Many side effects can be handled in event handlers instead of effects, leading to simpler code.

The Future of Component Development

React continues evolving with features like Server Components and Concurrent Rendering. However, the core principles of component-based architecture remain constant: modularity, reusability, and clear separation of concerns.

Components represent more than just a React pattern, they're a fundamental shift in how we think about building user interfaces. By breaking complex UIs into simple, composable pieces, React components support more maintainable AI-powered development workflows and make frontend development more predictable.

The next time you build a React application, remember that every component is an opportunity to create something reusable. Start small, think modular, and let composition guide your architecture decisions.

📦 Publishing Kit — Dev.to

Title Options (5)

Selected: Building React Components: From JSX to Production-Ready Code

Alternates:

React Components Explained: The Complete Guide to Modern UI Development
Mastering React Components: JSX, Props, and Component Architecture
React Component Fundamentals: Building Scalable UI with Modern Patterns
The Developer's Guide to React Components: From Basics to Best Practices

Slug

building-react-components-jsx-production-ready-code

Tags

react, webdev, javascript, components

Why Private Repositories Should Be Your Default for Commercial Development

Julian Neagu — Thu, 18 Jun 2026 13:59:05 +0000

TL;DR: Private repositories protect your competitive advantage and operational intelligence while offering identical technical functionality to public repos. For commercial development, private should be your default unless you're intentionally open-sourcing.

Most developers treat repository visibility as a checkbox they tick without thinking. That's a strategic mistake that can expose years of hard-won competitive advantages. When you're building commercial software or managing a network of sites, the choice between public and private repositories isn't about code access—it's about protecting the operational intelligence that makes your business valuable.

The assumption that "open source equals better" has created a dangerous default in commercial development. Developers push repositories public without considering what they're actually exposing beyond the code itself.

Your internal structure, implementation patterns, and architectural decisions form part of your competitive moat. Making them publicly visible is like publishing your playbook for competitors to study and replicate.

Core Reasons Private Should Be Your Default

Operational Confidentiality Protection

Your repository structure reveals more strategic information than most developers realize. When competitors analyze your codebase, they're not just seeing your current implementation—they're learning your decision-making patterns, preferred frameworks, and architectural philosophy. This intelligence helps them predict your next moves and potentially replicate your advantages.

Commercial projects managing 500+ properties contain implementation patterns representing years of optimization and learned lessons.

Consider a high-volume site network. Each repository contains deployment logic, operational workflows, and performance optimizations that represent countless hours of testing and refinement. Making these public essentially hands competitors a blueprint for replicating your operational efficiency without the trial-and-error investment you made.

The patterns you've developed for handling requests, managing state, and structuring code are intellectual property. They represent solved problems and learned lessons. When these patterns are public, you're diluting the uniqueness of your approach and giving away strategic advantages.

Enhanced Security Posture

Public repositories exponentially increase your attack surface. Anything that reveals how your platform handles authentication, validates requests, or implements security checks becomes valuable intelligence for potential attackers. While security through obscurity isn't your primary defense, it's a legitimate protective layer.

Your error handling, validation logic, and security implementations shouldn't serve as public documentation for bad actors. When attackers can study your codebase, they can identify potential vulnerabilities, understand your defensive strategies, and craft more targeted attacks against your specific implementation patterns.

Even seemingly innocent configuration files can reveal service dependencies, third-party integrations, and infrastructure patterns that inform attack strategies. As detailed in accessibility audit best practices, exposing internal structures can create unintended security vectors.

Architectural Privacy Advantage

Your build setup, framework choices, and deployment structure represent competitive advantage that extends beyond technical decisions. These are business decisions that affect your speed of execution, scaling capabilities, and operational costs.

When competitors can analyze your technical stack, they can reverse-engineer your cost structure, understand your scaling strategies, and potentially identify weaknesses in your approach. Your technology choices often reflect deeper business strategies that shouldn't be publicly documented.

The conventions and patterns you've established across your codebase create consistency and efficiency gains. These operational advantages are part of your competitive moat—making them public reduces their strategic value and levels the playing field unnecessarily.

What Public Visibility Actually Provides

The Illusion of Benefits

For commercial projects, public visibility provides zero meaningful operational benefits. This is the core misunderstanding that leads to poor repository management. Developers assume public repositories offer some inherent advantage for business applications, but this assumption is fundamentally flawed.

Public repositories don't improve your deployment capabilities, enhance your runtime performance, or provide any technical advantages for commercial software. The infrastructure treating your code—hosting platforms, build systems, domain management—operates identically regardless of repository visibility. What matters more is maintaining a secure development workflow, including regular website security scanning to identify vulnerabilities before they affect production systems.

Repository visibility only affects code accessibility while deployments, domains, and runtime behavior remain completely identical.

The perceived benefits of public repositories—community contributions, increased visibility, and portfolio enhancement—only apply when your explicit goal is open-source development. For commercial projects, these aren't benefits; they're potential liabilities that distract from your core business objectives.

Maintenance Overhead Reality

Public visibility actually creates maintenance overhead that most developers don't anticipate. Public repositories attract unsolicited pull requests, feature requests, and support questions that pull your team's attention away from strategic priorities.

Unless you're actively seeking community contributions, this attention becomes a burden rather than an asset. Your team ends up spending time managing external expectations instead of focusing on product development and business growth.

What Stays Identical When Private

Technical Operations Remain Unchanged

Repository visibility affects only code accessibility. Everything else in your development and deployment pipeline operates exactly the same:

Deployments: Your CI/CD pipelines, hosting platforms, and deployment processes function identically
Domains: DNS management, SSL certificates, and domain configuration aren't affected by repository visibility
Build Behavior: Compilation, optimization, and build processes operate the same way
Environment Variables: Configuration management and secrets handling remain unchanged
Runtime Performance: Application speed, resource usage, and scaling behavior are unaffected

The infrastructure layer doesn't care about your repository's visibility settings. Your applications run the same way whether the source code is public or private. This is a critical point that many developers miss when making visibility decisions.

Development Workflow Consistency

Your development practices remain identical in private repositories. Team collaboration, code review processes, and project management operate the same way. GitHub's collaboration features—issues, pull requests, and project boards—work identically for private repositories.

Private repositories don't limit your development velocity or team productivity. The only difference is the controlled access to your codebase, which actually improves focus by eliminating external distractions.

📎 Insert Image #5 here (uploaded image — add manually after pasting)
Alt: Business strategy flowchart for choosing repository visibility based on commercial development needs

Essential Hygiene Before Any Repository

Security Audit Requirements

Before making any repository public, run comprehensive security audits on your codebase. Look for hardcoded credentials, internal URLs, configuration details, and any information that could compromise your infrastructure or reveal operational details.

Use automated scanning tools to identify potential security issues:

# Scan for secrets and sensitive data
git-secrets --scan

On Windows PowerShell, you can use GitLeaks for similar functionality:

# Install GitLeaks via Chocolatey
choco install gitleaks

# Scan current repository
gitleaks detect --source .

Documentation Cleanup Process

Review all README files, code comments, and documentation for references to internal systems, deployment procedures, or strategic information. Remove or generalize any content that reveals operational details about your business.

Create a checklist of items to audit:

Environment configuration examples that might reveal infrastructure details
Internal tool references that shouldn't be publicly documented
Performance metrics that reveal scaling patterns or user volumes
Third-party service integrations that expose your technology stack

Sites vs. Software: The Strategic Distinction

Commercial Site Networks

For businesses managing multiple commercial sites, private repositories are essential. Each site represents a revenue stream, and the collective patterns across your network represent your competitive advantage in that market.

Your approach to content management, user experience optimization, and performance tuning are business assets. Making these patterns public gives competitors insight into your operational efficiency and strategic approach.

Product Development Context

When building software products, the distinction becomes even more critical. Your product's architecture, performance optimizations, and feature implementation represent your core intellectual property. These shouldn't be public unless open-source development is your explicit business model.

The decision becomes irreversible once made public. Search engines index repository contents, and archives capture your code permanently. You can't truly "unpublish" repository contents once they've been public, as I explored in my earlier post on web development choices.

Making the Right Default Choice

For commercial development, private repositories should be your default choice. This protects your competitive advantage while providing identical technical functionality. Only choose public visibility when you're intentionally contributing to open-source projects or when public visibility serves a specific strategic purpose.

The cost of private repositories is negligible compared to the potential value of the intellectual property you're protecting. Most hosting platforms offer generous private repository allowances, making this an easy default to maintain.

Your repository visibility decisions reflect your understanding of what makes your business valuable. Protect that value by keeping your operational intelligence private while you build and scale your commercial projects.

📦 Publishing Kit — Dev.to

Title Options (5)

Selected: Why Private Repositories Should Be Your Default for Commercial Development

Alternates:

Stop Making Your Business Logic Public: The Case for Private Repos
Private vs Public Repositories: Protecting Your Competitive Advantage
Your Repository Visibility Strategy Is Leaking Business Intelligence
Commercial Development Reality: Why Private Repositories Matter More Than You Think

Slug

private-repositories-default-commercial-development

Tags

webdev, programming, devops, repositories

Stop Data Mining Bots Before They Steal Your Content

Julian Neagu — Wed, 17 Jun 2026 10:03:27 +0000

TL;DR: Data mining bots steal your content, structure, and traffic faster than you think. Layer rate limits, behavioral detection, and access controls to make scraping expensive and slow. You can't stop everyone, but you can frustrate most attempts.

Website scraping hits differently when you see your entire product catalog copied overnight. One morning you're ranking first for your own brand name. The next, you're competing with mirror sites using your exact descriptions, prices, and even your customer reviews.

Data mining today isn't just content theft — it's systematic extraction using automated tools that lift your text, images, product prices, reviews, metadata, and even your layout structure. Think of it as someone sneaking into your office and photocopying everything without asking.

When scrapers take your content, you lose search ranking, trust, traffic, and sometimes revenue. This is why teams need to understand the difference between AI crawling and traditional crawling, and pair that awareness with regular website security scanning to catch exposed content, weak access controls, and scraping-related risks early.

The goal isn't perfect protection. It's making extraction expensive, slow, and frustrating enough that most bots move on to easier targets.

Why Data-Mining Threatens Your Site

If you've ever wondered how someone copied your entire blog posts or product listings within minutes, it usually comes down to scrapers. These aren't people manually saving your content. These are bots crawling page after page, collecting whatever they find.

Some grab everything in bulk. Others focus only on images, e-commerce pricing, or metadata to fuel comparison sites. The worst ones copy entire libraries of original articles and outrank the creators.

Sites can lose 20 to 30 percent of their organic traffic simply because scrapers published the same piece faster or with stronger backlinks.

Once your unique content is out there, search engines might not even know who wrote it first. And the privacy loss can hit harder. Internal URLs, hidden endpoints, and unprotected APIs give scrapers a free backstage pass.

I've tracked cases where sophisticated AI analysis tools revealed how scraped content patterns shift as detection methods improve—it's an arms race that never stops.

Core Factors That Shape Your Protection Strategy

Understanding What You're Protecting

You start any protection strategy by figuring out which areas of your site hold the highest value. For some owners, it's their long-form content. For others, it's structured data inside product listings or the API endpoints feeding mobile apps.

Once you know what's valuable, the rest becomes easier because you stop applying generic countermeasures and focus on the pressure points.

How Bots and Crawlers Extract Data

Most automated extraction tools follow predictable behavior. Bots scan your HTML, analyze your script tags, check your schema, and even read your JSON responses. Some scrapers pretend to be Google or Bing. Others rotate IP addresses every few minutes.

You usually notice them through patterns like:

Page requests are happening too fast for any human
Crawlers are accessing non-linked pages directly
Traffic spikes at odd hours
Repeated hits to specific structured data endpoints

A typical scraping pattern shows 200 requests in 20 seconds—no human browses that fast.

Why Headers and Access Rules Matter

Your server headers, rate limits, and access control rules set the boundaries. They decide how long a scraper can probe before getting blocked or slowed. If you've ever checked your server logs and thought, "wow, that user hit 400 pages in one minute," you've seen scraping firsthand.

The key is making each request more expensive for the bot while keeping legitimate users unaffected.

Technical Defenses You Need in Place

Rate Limiting and Behavioral Detection

You slow scrapers down by making their job frustrating. Robot rules are the simplest layer. They don't block criminals, but they stop low-tier bots that follow rules. Real scrapers try harder, so you add rate limiting.

On Linux or macOS, you can check current request patterns:

tail -f /var/log/nginx/access.log | grep -E "GET|POST" | awk '{print $1}' | sort | uniq -c | sort -nr

On Windows PowerShell, if you're using IIS logs:

Get-Content -Wait "C:\inetpub\logs\LogFiles\W3SVC1\*.log" -Tail 100 | Where-Object { $_ -match "GET|POST" } | ForEach-Object { ($_ -split " ")[2] } | Group-Object | Sort-Object Count -Descending

If an IP sends 200 requests in 20 seconds, that's not a human browsing. You cap them. Send them a challenge. Drop the request entirely.

Modern sites use behavioral bot detection. These tools look at cursor movement, loading patterns, and interaction speed. A bot loads your JavaScript instantly. A real human does not.

Dynamic Content Loading

You might also consider serving dynamic content when possible. Instead of loading the full content in the HTML, load part of it after user action. It reduces bulk extraction because bots rarely trigger events naturally.

Here's a simple JavaScript approach:

// Load content only after user interaction
document.addEventListener('click', function(e) {
    if (!document.body.classList.contains('content-loaded')) {
        fetch('/api/protected-content')
            .then(response => response.json())
            .then(data => {
                document.getElementById('main-content').innerHTML = data.content;
                document.body.classList.add('content-loaded');
            });
    }
});

API Protection Strategies

APIs need special handling. Put your endpoints behind keys or tokens. Limit how many requests a single key can send. Record every call.

The biggest content leaks don't come from scraping pages—they come from public APIs with no limits.

A basic rate-limited API setup might look like:

// Express.js with rate limiting
const rateLimit = require('express-rate-limit');

const apiLimiter = rateLimit({
  windowMs: 15 * 60 * 1000, // 15 minutes
  max: 100, // limit each IP to 100 requests per windowMs
  message: 'Too many requests from this IP'
});

app.use('/api/', apiLimiter);

Hardening Your Site Structure

Metadata and Schema Protection

Sometimes the weakness isn't in your backend. It's in how you expose your structure. Metadata, alt text, structured schema, open graph tags—all useful for SEO, but also perfect for data miners. You want search engines to understand your content without giving scrapers the blueprint.

Your indexing rules matter. You choose what gets crawled and what stays private. Media files like images or videos can be delivered with signed URLs that expire after a short time. A scraper might grab the link once, but it can't reuse it.

CDN-Level Blocking

A strong CDN also helps. Most CDNs let you block known bot networks or make custom rules that detect harvesting patterns. For example, if you see 50 requests for just your images, you can instantly slow that traffic or challenge it with an interstitial check.

Common CDN rules include:

Block requests missing standard browser headers
Challenge traffic from data center IP ranges
Rate limit by user agent patterns
Require JavaScript execution for full page access

Identity, Access, and Permissions

Restrict Sensitive Pages Behind Authentication

If a page matters, lock it behind a login. Most scrapers never pass that stage. And those who try leave clear footprints.

The authentication doesn't need to be complex. Even a simple email gate stops bulk extraction:

<!-- Simple content gate -->
<div id="content-gate" class="auth-required">
  <form onsubmit="unlockContent(event)">
    <input type="email" placeholder="Enter email for full access" required>
    <button type="submit">Access Content</button>
  </form>
</div>

Use Role-Based Permissions Internally

Inside your system, only the right people should see high-value files. Internal misuse or automated extraction often happens when roles overlap.

Track User Behavior for Warning Signs

You monitor things like:

Repetitive page loading patterns
Rapid navigation between unrelated pages
High download counts from single sessions
API calls without corresponding page views

These patterns usually reveal attempted extraction. Most legitimate users don't hit 50 pages in 5 minutes or download every PDF on your site in sequence.

Log Key Events for Auditing

Logs aren't glamorous, but they help you prove attempts, trace paths, and tighten weak points. Track failed login attempts, unusual download patterns, and API abuse attempts.

A Quick Comparison Table

Scaling Your Anti-Scraping Defenses

Match Protection to Traffic Growth

As traffic grows, scraping grows. You increase rate limits, add stricter checks, and adapt your detection thresholds. What worked at 100k monthly views won't work at a million. This is why teams need website security scanners that detect threats fast, so protection scales with traffic instead of reacting after scraper activity has already damaged performance.

Scale your monitoring too. Manual log review stops working when you process thousands of requests per hour.

Rotate and Update Protection Rules

Scrapers evolve. So your rules rotate, too. IP ranges get outdated, user agents change, and new tools appear every few months.

The most effective protection systems update their behavioral detection weekly. They learn from new scraping patterns and adjust thresholds automatically.

Protect Large Libraries with Subtle Watermarks

If you publish visual content, watermarking or hashed storage helps track stolen files. Even small, invisible watermarks let you prove ownership when content appears elsewhere.

Digital watermarking caught one scraper republishing 10,000 images from an e-commerce site—the invisible signature proved the theft.

Handle False Positives Gracefully

The biggest risk with aggressive anti-scraping measures is blocking legitimate users. Build appeals processes. Monitor bounce rates after implementing new rules. A protection system that drives away real customers defeats the purpose.

Your defense strategy should make scraping expensive and frustrating without punishing the humans you actually want reading your content. Perfect prevention is impossible, but friction works. Most scrapers move on when extraction becomes slow, incomplete, or unreliable.

The goal is simple: make your site harder to scrape than your competitor's site. In a world of endless targets, that's often enough.

📦 Publishing Kit — Dev.to

Title Options (5)

Selected: Stop Data Mining Bots Before They Steal Your Content

Alternates:

Protect Your Website from Scraping Bots and Data Theft
Web Scraping Defense: Rate Limits, Detection, and Access Control
How to Make Your Site Too Expensive for Content Scrapers
Anti-Scraping Strategies That Actually Work in 2024

Slug

stop-data-mining-bots-protect-website-content

Tags

webdev, security, tutorial, scraping

One AI Codebase, Three Revenue Streams: GitHub Workflow for SaaS, White Label & CodeCanyon

Julian Neagu — Tue, 16 Jun 2026 11:42:51 +0000

TL;DR: Managing one AI project across SaaS, white label, and CodeCanyon through a unified GitHub workflow reduces development time by 60% while maintaining three revenue streams from a single codebase. Template once, deploy everywhere.

Building AI products for multiple markets usually means building multiple products. You maintain separate codebases for your SaaS platform, your white label offering, and your CodeCanyon script. Every bug fix becomes three bug fixes. Every feature update becomes three deployment headaches.

This fragmented approach kills productivity and creates technical debt that compounds over time. You're not scaling a business — you're multiplying maintenance work.

A unified GitHub workflow changes this entirely. You build one AI application that serves three distinct markets through controlled deployment pipelines. Your SaaS runs on the main branch, white label partners get branded versions through config layers, and CodeCanyon releases ship from clean feature branches.

The result? One codebase, three revenue streams, and development time that stays linear instead of exponential.

Why GitHub Anchors Your Entire Operation

GitHub isn't just version control — it's the operational backbone that makes multi-market deployment possible. Every feature, fix, and release flows through GitHub's infrastructure, creating a single source of truth for your entire product ecosystem.

Safe Version History

Commit history protects your business from catastrophic rollbacks and mysterious bugs. Each commit captures the exact state of your project at that moment, creating the kind of traceability that supports AI-assisted code audit workflows. When a new feature breaks your billing system or corrupts user onboarding, you don't waste hours debugging in production.

You check the last stable commit, compare the diff, and roll back with confidence. Over months, this history becomes a decision log showing which refactors succeeded, which experiments failed, and which branches led to stable releases.

According to Calibrain, a shared codebase reduces development time by up to 60 percent compared to maintaining separate versions.

When a CodeCanyon buyer reports a bug in version 2.3, you locate the specific commit, patch it in a maintenance branch, and ship an update without touching your SaaS production code. The isolation keeps your main product stable while fixing marketplace issues.

Fast Recovery on New Devices

Your business should never depend on one laptop surviving. GitHub stores all source code, branches, and tags in infrastructure that outlasts hardware failures. If your development machine crashes, you log into a new device, clone the repository, install dependencies, and resume work from the exact same state.

This redundancy also supports multi-environment development. You might test builds on a clean virtual machine, keep your main environment optimized for daily coding, and run automated tests on a dedicated server. Every environment pulls the same repository, ensuring consistent behavior across your entire development stack.

Direct Integration with Vercel

GitHub and Vercel eliminate deployment friction through automated pipelines. You configure Vercel once, connect your repository, and assign branches to specific environments. Deployment becomes an extension of your commit process:

Push to main → Vercel updates your live SaaS platform
Push to dev → Vercel builds staging previews for testing
Push to agency-preview → Vercel creates temporary URLs for white label client reviews
Create release/codecanyon → Generate clean builds for marketplace packaging

You avoid manual file uploads, FTP sessions, and the deployment inconsistencies that create production bugs. Every release follows the same pipeline, supported by CI/CD automation checks, keeping your production environment predictable and your customers happy.

White Label Model A: Config-Driven Multi-Tenancy

The first white label approach treats your platform as a multi-tenant product with brand overlays. Agencies get their own version of your app, but the core logic stays centralized and shared.

Configuration Layer Architecture

A config-driven system separates branding and behavior from your core codebase. You define a configuration layer that loads different settings based on the requesting domain or selected tenant.

Each brand configuration includes:

Brand display name and tagline
Primary and secondary color schemes
Logo and favicon URLs
Custom domain and base URL
Contact email and support links
Feature flags controlling module visibility
Usage limits for projects, posts, or team seats

At runtime, your application reads the active config and renders the same React components with different styling, labels, and feature availability. When an agency signs a white label contract, you collect their brand assets, insert a new config entry, configure DNS, and point their domain to your existing infrastructure.

No code forks. No deployment duplication. No exponential maintenance overhead.

Pricing Structure for Agency Partners

Model A focuses on recurring revenue with clear service tiers that scale with agency needs:

Basic Agency: £49 per month for small agencies with limited team seats, basic reporting, and standard support response times.

Pro Agency: £149 per month for agencies managing multiple clients, with advanced features like team collaboration tools and custom template libraries.

Enterprise: £299 per month for high-volume agencies requiring priority support, custom integrations, and dedicated account management.

Setup Fee: £99 per new brand to cover initial configuration, DNS setup, and quality assurance testing.

This pricing model creates predictable monthly revenue while keeping setup costs low enough to attract smaller agencies. The recurring structure also ensures agencies stay engaged with your platform rather than treating it as a one-time purchase.

Basic Agency white label pricing starts at £49 per month, with Pro Agency at £149 per month and Enterprise at £299 per month.

As I detailed in my AI best practices guide, proper configuration management becomes critical when serving multiple clients through shared infrastructure. Your config layer needs to handle not just branding, but also feature flags, rate limits, and compliance requirements that vary by client.

Theme-Driven UI for Rapid Customization

Your user interface becomes the most visible differentiator between white label brands. A theme-driven approach lets you deliver completely different visual experiences while maintaining the same underlying functionality.

Component-Based Theming

Modern frameworks like React and Vue make component-based theming straightforward. You define base components that accept theme props, then create theme objects that override colors, typography, spacing, and layout patterns.

// Base theme structure
const baseTheme = {
  colors: {
    primary: '#3B82F6',
    secondary: '#10B981',
    accent: '#F59E0B'
  },
  typography: {
    headingFont: 'Inter',
    bodyFont: 'system-ui'
  },
  spacing: {
    unit: 8,
    containerPadding: 24
  }
}

// Agency-specific theme override
const agencyTheme = {
  ...baseTheme,
  colors: {
    ...baseTheme.colors,
    primary: '#7C3AED',
    secondary: '#EC4899'
  }
}

Your components consume these theme values through props or context, rendering different visual styles without requiring separate component files. When an agency requests a rebrand, you update their theme configuration and redeploy — no code changes required.

Asset Pipeline Management

Images, icons, and other static assets need systematic organization when serving multiple brands. You create an asset pipeline that maps theme identifiers to specific asset collections.

/assets
  /themes
    /agency-alpha
      /logo.svg
      /favicon.ico
      /hero-background.jpg
    /agency-beta
      /logo.svg
      /favicon.ico
      /hero-background.jpg
    /default
      /logo.svg
      /favicon.ico
      /hero-background.jpg

Your build process automatically includes the correct asset collection based on the target theme, ensuring each white label deployment ships with only the relevant branding files.

CodeCanyon Release Strategy

CodeCanyon represents a different market segment — developers who want to self-host your solution rather than subscribe to a hosted service. This requires a packaging approach that delivers complete functionality while protecting your competitive advantages.

Clean Branch Isolation

Your CodeCanyon releases should ship from dedicated branches that exclude sensitive infrastructure code and API integrations. You create a codecanyon-release branch that contains:

Core AI processing logic
User interface components
Database schema and migrations
Installation and configuration scripts
Documentation and setup guides

You exclude:

Production API keys and secrets
Proprietary algorithmic improvements
Advanced analytics and tracking
Integration with your own backend services
White label configuration systems

This approach gives CodeCanyon buyers a complete, functional product while preserving the competitive moats that make your SaaS and white label offerings more valuable.

Pricing for One-Time Licenses

CodeCanyon pricing follows a different model than recurring subscriptions:

Regular License: £39 to £69 for single-site usage, covering most individual developers and small projects.

Extended License: £199 to £349 for commercial usage, resale rights, or multi-site deployments.

The lower price point attracts budget-conscious developers while the extended license captures value from agencies and consultants who want to resell your solution. This creates a third revenue stream that complements rather than cannibalizes your SaaS and white label businesses.

Implementation Workflow

Your daily development process stays remarkably simple despite serving three distinct markets. You work on features in feature branches, test them in staging environments, and merge into your main branch when ready.

That same code then propagates to all three product surfaces:

SaaS deployment happens automatically through your Vercel integration
White label updates flow to agency partners through their configured domains
CodeCanyon releases get packaged from the latest stable branch when you're ready to update the marketplace

Bug fixes follow the same linear path. You identify issues, fix them once in your core codebase, and deploy the fix across all three channels through your established pipelines.

This unified approach eliminates the exponential complexity that kills most multi-market products. Instead of maintaining three separate codebases that drift apart over time, you maintain one codebase that serves three different customer segments through controlled deployment and configuration layers.

The result? Predictable development cycles, stable revenue growth, and a technical foundation that scales with your business rather than constraining it.

📦 Publishing Kit — Dev.to

Title Options (5)

Selected: One AI Codebase, Three Revenue Streams: GitHub Workflow for SaaS, White Label & CodeCanyon

Alternates:

How I Cut AI Development Time 60% Using GitHub to Power SaaS + White Label + Marketplace
Stop Building Three AI Products: GitHub Workflow for Multi-Market Deployment
From Three Codebases to One: GitHub Strategy for SaaS, Agency, and CodeCanyon Success
The GitHub Workflow That Turned One AI Project Into Three Revenue Streams

Slug

github-workflow-ai-saas-white-label-codecanyon

Tags

webdev, productivity, devops, saas