DEV Community: Julian Neagu

Why rel="noopener" Is Your Front Line Against Tab-Nabbing Phishing Attacks

Julian Neagu — Tue, 25 Aug 2026 17:55:37 +0000

TL;DR: The rel="noopener" attribute prevents tab-nabbing phishing attacks by severing the JavaScript window.opener reference between your page and any new tab opened with target="_blank". Modern browsers apply it automatically, but explicit declaration remains essential for security compliance, older browser coverage, and performance isolation.

When you add target="_blank" to a link, you're creating more than just a new tab. You're creating a JavaScript back-channel between two pages. The newly opened tab receives a reference to your original page through window.opener, and that reference can be weaponized. A malicious site can redirect your tab to a phishing page while the user is distracted. The user switches back, sees what looks like your login screen, and types in their credentials.

This isn't a theoretical vulnerability. It's a documented attack vector with a name: tab-nabbing. And the mitigation is trivially simple. Add six characters to every external link: rel="noopener".

How window.opener Creates a JavaScript Back-Channel

When you write this:

html <a href="https://external-docs.com" target="_blank"> Read the documentation </a>

The browser opens external-docs.com in a new tab. Nothing surprising there. But behind the scenes, the new tab gets a JavaScript object called window.opener that points back to your original page.

That reference is live. The external page can't read your DOM or access your cookies directly, because same-origin policy blocks cross-domain data access. But it can do one thing that breaks the entire security model: it can change where your tab points.

Without noopener, any page opened with target="_blank" can redirect your original tab to a phishing site using window.opener.location, even across completely different domains.

The external site runs this single line of JavaScript:

javascript window.opener.location = "https://fake-login-page.com";

Your original tab, sitting in the background, now displays a convincing replica of your login page. The user doesn't notice immediately because they're focused on the new tab. When they switch back, they see "session expired" and re-enter their credentials. The attacker captures everything.

The Attack Sequence

Here's how tab-nabbing plays out in practice:

User is logged into your app at yourapp.com
User clicks an external link that opens with target="_blank"
The new tab loads, runs JavaScript that executes window.opener.location = "https://fake-yourapp.com"
Your original tab (sitting in the background) is now showing a fake login page
User switches back, sees "session expired", types in credentials
Attacker logs the credentials and redirects to the real login page

The attack works because changing another window's location is one of the few cross-origin operations browsers historically allowed. You can't read the other window's content, but you can send it somewhere else. That's all a phishing attack needs.

The Performance Benefit: Process Isolation

Security is the main reason to use noopener, but there's a secondary benefit: performance isolation.

Without noopener, the new tab runs in the same browser process as your page. If the external site is resource-heavy (large JavaScript bundles, memory leaks, slow rendering), it can degrade your site's performance even though it's in a separate tab. Your UI might stutter. Your event loop might block.

Adding rel="noopener" allows browsers to spawn the new tab in a completely separate process, preventing external sites from degrading your original page's performance.

When you add rel="noopener", the browser can spawn the new tab in a completely separate process. That means a slow or broken external page won't jank your app's UI or consume your memory budget. This matters more on lower-end devices and mobile browsers where process isolation is less aggressive. It's not the primary reason to declare noopener, but it's a measurable side benefit.

What rel="noopener" Actually Does

The noopener attribute does exactly one thing: it sets window.opener to null in the newly opened tab. The tab still opens. The link still works. The user experience is identical. But the new tab has no reference back to your page, which means it can't touch it.

Here's the corrected version:

html <a href="https://external-docs.com" target="_blank" rel="noopener"> Read the documentation </a>

That's it. Six extra characters. The link behaves the same from the user's perspective, but the security hole is closed.

If the external site is one you don't fully trust (user-generated content, third-party integrations, ads), you can also add noreferrer:

html <a href="https://sketchy-site.com" target="_blank" rel="noopener noreferrer"> External link </a>

The noreferrer part prevents your URL from being sent to the external site as the Referer header. That's useful for privacy (the external site won't know where the traffic came from), but it's a separate concern. For most cases, noopener alone is sufficient.

Do Modern Browsers Apply noopener Automatically?

Yes, since 2021. Chrome version 88 made the change first. Firefox and Safari followed shortly after. Edge inherited the behavior when it switched to Chromium.

If you're targeting only modern browsers, you're already protected by default. The browser treats every target="_blank" as if it had rel="noopener", whether you wrote it or not.

So why still write it explicitly?

Three Reasons to Keep Writing rel="noopener"

Older browser support. Chrome 88 was released in January 2021. Firefox 79 (released July 2020) implemented the same default. Safari added it in version 12.2 (March 2019). But users on older versions of these browsers, or on outdated mobile browsers, are still vulnerable. Explicit declaration protects everyone.

Security scanners and compliance. Tools like static code analysis checkers flag missing noopener on target="_blank" links as vulnerabilities. SOC2 audits and penetration tests will call it out. If you're working in fintech, healthcare, or enterprise SaaS, auditors expect to see explicit noopener declarations. The fact that modern browsers handle it automatically doesn't satisfy the audit checklist.

Security scanners, linters, and compliance audits flag missing noopener on target="_blank" links as vulnerabilities, making explicit declaration necessary for SOC2 and pen test compliance.

Documentation and intent. Code is read more often than it's written. When another developer sees target="_blank" without noopener, they don't know if the omission was intentional or accidental. Explicit declaration signals that the security consideration was addressed.

Where Developers Forget to Add noopener

The most common mistake isn't forgetting to add noopener to handwritten links. It's forgetting to add it to dynamically generated links.

Links Generated from User Content

If your app renders Markdown or rich text from users, and that content includes links, you need to ensure the renderer adds noopener to every target="_blank" link. Most Markdown libraries don't do this by default.

For example, using a JavaScript Markdown library without configuring link safety:

javascript // Unsafe: user Markdown links open without noopener const html = marked.parse(userContent);

You need to configure the renderer:

`javascript const renderer = new marked.Renderer(); renderer.link = (href, title, text) => { const titleAttr = title ? title="${title}": ''; return${text}`;
};

const html = marked.parse(userContent, { renderer });
``

Links in Third-Party Embeds

If your app embeds content from third parties (documentation widgets, comment systems, social media feeds), you don't control the HTML those systems inject. You need to sanitize it on your end.

One approach: walk the DOM after the embed loads and add noopener to any target="_blank" link that's missing it.

javascript document.querySelectorAll('a[target="_blank"]:not([rel~="noopener"])').forEach(link => { const currentRel = link.getAttribute('rel') || ''; link.setAttribute('rel', `${currentRel} noopener`.trim()); });

This catches external content you didn't write but are responsible for serving.

Is the Risk Actually High?

The probability of a tab-nabbing attack is low. Most external sites you link to are not malicious. Most users don't click links to sketchy domains from within your app. And modern browsers already mitigate the issue automatically.

But the impact is high. A successful tab-nabbing attack can compromise user credentials, leading to account takeover, data exfiltration, and reputational damage. The cost of mitigation is negligible: six extra characters. The risk is low-probability, high-severity. That math always favors the fix.

From a strategic security perspective, noopener is a defense-in-depth measure. It's not your only layer of protection, but it's a layer that costs nothing to add and closes a documented attack vector.

How to Audit Your Codebase for Missing noopener

If you're working on an existing codebase and want to find every external link that's missing noopener, here's the audit process:

Step 1: Search for target="_blank"

Grep your codebase for every occurrence of target="_blank":

On macOS or Linux:

bash grep -r 'target="_blank"' ./src

On Windows PowerShell:

powershell Select-String -Path .\src\* -Pattern 'target="_blank"' -Recurse

This gives you a list of every file and line where a new-tab link exists.

Step 2: Check for Missing noopener

For each result, check whether the same line contains rel="noopener" or rel="noopener noreferrer". If it doesn't, add it.

You can automate this with a more specific grep:

On macOS or Linux:

bash grep -r 'target="_blank"' ./src | grep -v 'noopener'

On Windows PowerShell:

powershell Select-String -Path .\src\* -Pattern 'target="_blank"' -Recurse | Where-Object { $_.Line -notmatch 'noopener' }

Any result from this search is a link that opens a new tab without noopener.

Step 3: Add Linter Rules

To prevent future regressions, add an ESLint rule that flags missing noopener:

json { "rules": { "react/jsx-no-target-blank": ["error", { "enforceDynamicLinks": "always" }] } }

This rule enforces noopener on every target="_blank" link, including dynamically generated ones. It fails the build if the attribute is missing.

Step 4: Automate in CI

Add a script to your CI pipeline that runs the grep check and fails the build if any target="_blank" link is missing noopener. This makes the mitigation enforceable at the infrastructure level, not just at the code-review level.

The Bottom Line

The rel="noopener" attribute is a one-line fix for a documented security vulnerability. Modern browsers apply it automatically, but explicit declaration remains necessary for compliance, older browser support, and defense-in-depth strategy.

If you're writing new links, add noopener to every target="_blank" link. If you're auditing an existing codebase, search for target="_blank", check for missing noopener, and add linter rules to prevent regressions. If you're rendering user-generated content or third-party embeds, sanitize the output to ensure every external link includes noopener.

The cost is six characters. The risk is account takeover. The fix is trivial. Ship it.

📦 Publishing Kit — Dev.to

Title Options (5)

Selected: Why rel="noopener" Is Your Front Line Against Tab-Nabbing Phishing Attacks

Alternates:

The Security Vulnerability Hiding in Every target="_blank" Link
How window.opener Turns Your External Links Into Phishing Attack Vectors
rel="noopener" Explained: Stop Tab-Nabbing Attacks With Six Characters
The JavaScript Back-Channel You Created By Forgetting rel="noopener"

Slug

why-rel-noopener-prevents-tab-nabbing-phishing-attacks

Tags

webdev, security, javascript, performance

How to Embed a Free SaaS Audit Tool on Your Landing Page Without Breaking Production

Julian Neagu — Thu, 20 Aug 2026 08:16:01 +0000

TL;DR: Building a free audit tool for your landing page means fighting CSS collisions, SSR failures, and SSRF vulnerabilities. An iframe solves isolation. A protected API route blocks attackers. Redis enforces rate limits. The entire setup runs at $0 on Vercel's free tier and handles 5,000 requests per day.

Embedding a working lite version of your SaaS product on your landing page is one of the highest-converting additions you can make. Users get instant value before signup. You build trust by letting them test the real thing. The upgrade path becomes obvious because they've already experienced what the full product does.

Tools like Website Audit Dev demonstrate this pattern well. A free scan delivers real value. Then it surfaces the full product at exactly the right moment. The conversion rate speaks for itself. But the technical implementation is where most teams stumble. This guide shows you how to build and deploy one the right way, without breaking your site in production.

" width="799" height="600">

The Naive Approach Breaks in Production Every Time

The simplest idea is to take your standalone HTML tool and paste it directly into your site's custom code panel or a Next.js component. You preview it locally. It renders. You deploy. The build succeeds. You open the production URL and everything is broken.

This pattern repeats so predictably that you can set your watch by it. The reasons become obvious once you've hit them, but they're invisible until that moment.

Your site has global CSS classes. Classes like .layout, .content, .grid, .button, and .card are defined in your site's stylesheet. Your tool also has .layout, .content, .grid, .button, and .card. They collide. Your tool's button suddenly has your site's hover state. Your site's grid now has your tool's column sizing. The cascade is fighting itself and there's no clean winner.

Your site has a dark theme. Your tool expects light. The tool's white text on a white background becomes invisible. Or your site's dark background bleeds through the tool's transparent sections and makes the UI unreadable.

Your tool uses localStorage to cache results. This works in the browser. It throws during server-side rendering because window and localStorage don't exist on the server. Next.js tries to render the component at build time, hits localStorage.getItem(), and the build fails with a cryptic error about undefined properties.

Your tool loads Three.js via a runtime-injected script tag to render a WebGL globe. Your site has a Content Security Policy header that blocks inline scripts and third-party script sources. The browser silently refuses to execute the injected script. Your globe never appears. The console shows a CSP violation but only if you know to check the Network tab's failed requests.

And if any of this produces malformed HTML, like an unclosed tag or a mismatched quote, Vercel's build pipeline kills the deploy. You get next build exited with 1 and no useful error message.

The browser would have silently recovered from the same HTML. Vercel does not.

The Fix Is an Iframe

The fix is an <iframe>. It sounds old-fashioned, but it's exactly right here.

An iframe is a fully isolated browser context. Your site's CSS cannot reach inside it. Your site's JavaScript cannot touch its DOM. Your site's theme, CSP headers, and global state are invisible to the iframe's content. The tool renders exactly as it does standalone, pixel-perfect, every time.

The setup has three parts. Each one is simple. Together they solve the entire class of problems.

The Tool Itself

The tool is a single HTML file. It contains all its CSS in a <style> block and all its JavaScript in a <script> block. No external dependencies unless you need them. If you do need a library like Three.js or Chart.js, load it from a CDN with an integrity hash.

This file lives in your Next.js project's public/ folder. Vercel serves everything in public/ as static files at the root path. If you put audit-tool.html in public/, it's available at https://yoursite.com/audit-tool.html. No routing config needed.

Your landing page embeds it with one iframe tag:

html <iframe src="/audit-tool.html" width="100%" height="600" frameborder="0" sandbox="allow-scripts allow-same-origin" title="Free Audit Tool"> </iframe>

The sandbox attribute is key. allow-scripts lets the tool's JavaScript run. allow-same-origin lets it access localStorage and make fetch calls to your API route. Without allow-same-origin, the iframe is treated as cross-origin even though it's on the same domain, and fetch calls to /api/* fail with CORS errors.

The Backend Route

The backend route is a Next.js App Router serverless function at app/api/audit/route.js. Its only job is to fetch a URL server-side and return the HTML response plus the real HTTP response headers.

This matters because browsers hide cross-origin response headers from client-side JavaScript. If your tool fetches https://example.com from the browser, the response headers like Strict-Transport-Security, X-Frame-Options, and Content-Security-Policy are invisible. The browser returns an opaque response object and response.headers.get('x-frame-options') is always null.

Security checks that rely on HTTP headers can only be read when fetched from a server. That's the entire reason this route exists.

The route also enforces rate limiting, which we'll cover in the next section.

Why Vercel's Free Tier Doesn't Give You Rate Limiting

Vercel's free tier does not support built-in rate limiting through vercel.json configuration or dashboard settings. You can't add a rate limit rule that says "block IPs that hit this route more than 30 times in 60 seconds."

The second problem is that Vercel serverless functions are stateless. They reset on every cold start. If you try to implement in-memory rate limiting like this:

``javascript
// This DOES NOT WORK in production
const requestCounts = {};

export async function GET(req) {
const ip = req.headers.get('x-forwarded-for');
requestCounts[ip] = (requestCounts[ip] || 0) + 1;
if (requestCounts[ip] > 30) {
return new Response('Rate limit exceeded', { status: 429 });
}
// ... rest of logic
}
``

This works locally. It fails in production because requestCounts is wiped on every cold start. A cold start happens when Vercel spins up a new instance of your function. This happens unpredictably, often multiple times per day. The attacker sends 10 requests, triggers a cold start, and the counter resets to zero. They repeat this forever.

You need persistent storage. That storage needs to be fast enough to respond in under 50ms so it doesn't add noticeable latency to every request. Redis is the right answer here.

How Upstash Redis Solves This

Upstash offers a Redis service with a free tier that provides 10,000 commands per day. Each audit request consumes 2 Redis commands: one GET to read the current count for an IP, and one INCR to increment it. At 10,000 commands per day, the system can handle 5,000 audit requests per day before hitting the free tier limit.

The rate limit is set at 30 requests per IP address within 60 seconds before returning a 429 response. This is enough to block simple abuse while allowing legitimate users to run multiple tests.

Here's the full implementation:

``javascript
// app/api/audit/route.js
import { Redis } from '@upstash/redis';

const redis = new Redis({
url: process.env.UPSTASH_REDIS_REST_URL,
token: process.env.UPSTASH_REDIS_REST_TOKEN,
});

export async function GET(req) {
const ip = req.headers.get('x-forwarded-for') || 'unknown';
const rateKey = rate:${ip};

// Check current count
const current = await redis.get(rateKey);
if (current && current >= 30) {
return new Response('Rate limit exceeded. Try again in 60 seconds.', {
status: 429
});
}

// Increment and set expiry if first request
const count = await redis.incr(rateKey);
if (count === 1) {
await redis.expire(rateKey, 60);
}

// ... rest of audit logic
}
``

The expire call sets a 60-second TTL on the key. After 60 seconds, Redis automatically deletes the key. The next request from that IP starts fresh at count 1.

SSRF Protection: Why Your API Route Is a Security Hole Without It

A server-side fetch endpoint without SSRF protection is a security vulnerability. SSRF stands for Server-Side Request Forgery. It means an attacker can make your server fetch internal URLs that are not accessible from the public internet.

Here's what happens if you don't protect it:

``javascript
// VULNERABLE CODE - DO NOT USE
export async function GET(req) {
const { searchParams } = new URL(req.url);
const targetUrl = searchParams.get('url');

const response = await fetch(targetUrl);
const html = await response.text();
return new Response(html);
}
``

An attacker sends this request:

http GET /api/audit?url=http://169.254.169.254/latest/meta-data/iam/security-credentials/

That IP address is AWS's EC2 metadata endpoint. It's only accessible from inside an EC2 instance. If your Vercel function runs on AWS infrastructure (which it often does), this request succeeds. The attacker gets temporary AWS credentials. They use those credentials to access your S3 buckets, RDS databases, and Lambda functions.

The same attack works with internal services. If your company has an internal admin panel at http://admin.internal.company.com, an attacker can make your server fetch it and return the HTML. They now have access to internal tools that should never be public.

The Five Defensive Layers

The solution is to validate and sanitize every URL before fetching it. Here's the full defensive stack:

``javascript
// app/api/audit/route.js (continued)
const BLOCKED_IPS = [
'127.0.0.0/8', // localhost
'169.254.0.0/16', // AWS metadata
'10.0.0.0/8', // private network
'172.16.0.0/12', // private network
'192.168.0.0/16', // private network
];

function isBlockedIP(hostname) {
// Parse hostname to IP if it's an IP address
const ipRegex = /^(\d{1,3}.){3}\d{1,3}$/;
if (!ipRegex.test(hostname)) return false;

const parts = hostname.split('.').map(Number);

// Check against each blocked range
if (parts[0] === 127) return true; // localhost
if (parts[0] === 169 && parts[1] === 254) return true; // AWS metadata
if (parts[0] === 10) return true; // private
if (parts[0] === 172 && parts[1] >= 16 && parts[1] <= 31) return true;
if (parts[0] === 192 && parts[1] === 168) return true;

return false;
}

export async function GET(req) {
const { searchParams } = new URL(req.url);
const targetUrl = searchParams.get('url');

// Defense 1: Reject missing or empty URLs
if (!targetUrl) {
return new Response('Missing url parameter', { status: 400 });
}

// Defense 2: Cap URL length to prevent attack payloads
if (targetUrl.length > 2048) {
return new Response('URL too long', { status: 400 });
}

// Defense 3: Parse and validate the URL
let parsedUrl;
try {
parsedUrl = new URL(targetUrl);
} catch (e) {
return new Response('Invalid URL format', { status: 400 });
}

// Defense 4: Block non-HTTP(S) protocols
if (!['http:', 'https:'].includes(parsedUrl.protocol)) {
return new Response('Only HTTP(S) allowed', { status: 400 });
}

// Defense 5: Block private/internal IPs
if (isBlockedIP(parsedUrl.hostname)) {
return new Response('Access to internal resources blocked', { status: 403 });
}

// All checks passed, safe to fetch
const controller = new AbortController();
const timeout = setTimeout(() => controller.abort(), 10000);

try {
const response = await fetch(parsedUrl.href, { signal: controller.signal });
const html = await response.text();

return new Response(JSON.stringify({
  html,
  headers: Object.fromEntries(response.headers.entries()),
  status: response.status,
}), {
  headers: { 'Content-Type': 'application/json' },
});

} catch (e) {
if (e.name === 'AbortError') {
return new Response('Request timeout', { status: 504 });
}
return new Response('Fetch failed', { status: 500 });
} finally {
clearTimeout(timeout);
}
}
``

URL length is capped at 2,048 characters to prevent attack payloads. Fetch timeout is enforced at 10 seconds using AbortController. If the target site doesn't respond within 10 seconds, the request is aborted and the user gets a 504 error.

What Breaks and How to Debug It

Here's what goes wrong in production and how to fix it.

CSS Collisions

Symptom: Your tool's buttons have the wrong color. Your site's font leaks into the tool. The layout shifts unexpectedly.

Cause: Your site's global CSS is bleeding into the iframe.

Fix: Make sure the iframe has sandbox="allow-scripts allow-same-origin". If the problem persists, the tool's HTML might be loading your site's stylesheet. Check the <head> block in audit-tool.html. Remove any <link rel="stylesheet"> tags that point to your site's CSS.

CORS Errors

Symptom: The browser console shows Blocked by CORS policy. The tool can't fetch data from your API route.

Cause: The iframe doesn't have allow-same-origin in the sandbox attribute.

Fix: Add allow-same-origin to the iframe tag:

<iframe sandbox="allow-scripts allow-same-origin" ...></iframe>

For more detailed security patterns in user-facing tools, Security Audit Dev demonstrates a similar multi-layer validation approach before running any automated checks.

Rate Limiting Doesn't Work

Symptom: You've added the Redis rate limiting code but users can still send unlimited requests.

Cause: The environment variables aren't set in production.

Fix: Go to your Vercel dashboard. Click your project. Click "Settings" → "Environment Variables". Add UPSTASH_REDIS_REST_URL and UPSTASH_REDIS_REST_TOKEN. Redeploy.

Fetch Returns 403 for Valid URLs

Symptom: Your tool returns "Access to internal resources blocked" for legitimate public websites.

Cause: The isBlockedIP function is blocking valid hostnames.

Fix: The current implementation only blocks numeric IPs. If you're seeing false positives, check if the hostname resolves to a private IP. You might need to add a DNS lookup step and validate the resolved IP instead of the hostname.

The Full Deployment Checklist

Here's the step-by-step process to ship this:

Create the tool HTML file. Put it in public/audit-tool.html. Test it standalone by opening http://localhost:3000/audit-tool.html in your browser.
Add the iframe to your landing page. Use the full sandbox attribute shown earlier.
Create the API route. Put the full code from this guide in app/api/audit/route.js.
Sign up for Upstash. Create a new Redis database. Copy the REST URL and REST token.
Set environment variables. Add UPSTASH_REDIS_REST_URL and UPSTASH_REDIS_REST_TOKEN to your local .env.local and to Vercel's dashboard.
Test locally. Run npm run dev. Open your landing page. Try the tool. Check the Network tab for errors.
Deploy to Vercel. Run vercel --prod. Open the production URL. Test again.
Monitor rate limits. Check your Upstash dashboard to see how many commands you're using per day. If you hit the free tier limit, upgrade or adjust your rate limit values.

Building a free audit tool is one of the highest-leverage additions to your SaaS landing page. The iframe gives you perfect isolation. The API route blocks SSRF attacks. Redis enforces rate limits. The entire setup runs at $0 on Vercel's free tier and handles 5,000 requests per day.

For more on building automated workflows that tie into tools like this, check out this step-by-step guide on building your first automated workflow. And if you're looking for more ways to optimize your landing page strategy, the principles in landing page best practices apply directly here.

📦 Publishing Kit — Dev.to

Title Options (5)

Selected: How to Embed a Free SaaS Audit Tool on Your Landing Page Without Breaking Production

Alternates:

Building a Landing Page Audit Tool That Survives CSS Collisions, SSR, and Security Attacks
The Right Way to Embed a Free Tool Demo on Your SaaS Landing Page (Iframe + Redis + Vercel)
Stop Breaking Your Landing Page: How to Safely Embed a Free Audit Tool Using Iframes
From CSS Chaos to Production-Ready: Building a Free Landing Page Audit Tool for $0

Slug

embed-free-saas-audit-tool-landing-page-iframe-vercel

Tags

webdev, tutorial, performance, iframe

Stop Putting Everything in functions.php: The WordPress File Separation Guide

Julian Neagu — Sat, 15 Aug 2026 09:51:06 +0000

TL;DR: WordPress gives you three files with distinct jobs: functions.php for logic, style.css for visual rules, and theme.json for design tokens. Mix them up, and your theme becomes unmaintainable. Keep them separate, and you get a scalable design system that works across every block, page, and brand update.

Most developers dump everything into functions.php because it's the first file they learned to edit. Custom fonts go there. Visual styling gets scattered across PHP templates. Color palettes live in hardcoded inline styles. Then someone asks you to change the primary brand color, and you spend two hours hunting through fifteen files trying to find every instance.

This isn't just messy. It breaks the separation of concerns that makes modern WordPress themes maintainable. WordPress gives you three files that each do one job extremely well. When you understand what belongs in each file and why, building themes becomes faster, handoffs become cleaner, and design iterations don't break functionality.

Let's walk through exactly what each file does, what belongs where, and how they work together to create a cohesive design system.

functions.php Is Your Loader, Not Your Styler

This is the most misunderstood file in WordPress theme development. Developers treat it like a dumping ground for everything, but it has one clear purpose: load resources and enable features. It's a loader, not a styler.

functions.php tells WordPress what exists. It registers capabilities, enqueues assets, and adds hooks. But it should never apply visual presentation directly. No inline styles. No hardcoded CSS rules buried inside PHP functions.

What Actually Belongs in functions.php

Here's the complete list of what this file should handle:

Enqueue custom fonts from Google Fonts or local files
Enqueue custom CSS files (blog.css, blocks.css, components.css)
Enqueue JavaScript files for interactive features
Register theme supports (custom logo, post thumbnails, title tag)
Register custom image sizes for responsive layouts
Register custom Gutenberg block styles and variations
Register navigation menus and widget areas
Add filters for custom content (reading time, table of contents injection)
Add shortcodes for reusable content blocks
Register custom blocks with block.json manifests

Let's say you want to load the Inter typeface from Google Fonts. This is exactly what functions.php should do:

php function visionvix_enqueue_fonts() { wp_enqueue_style( 'visionvix-fonts', 'https://fonts.googleapis.com/css2?family=Inter:wght@300;400;600&display=swap', false ); } add_action('wp_enqueue_scripts', 'visionvix_enqueue_fonts');

This code loads the font into the page's <head> section. That's it. It doesn't apply the font to any element. It doesn't set font sizes. It doesn't touch typography rules. It just makes the resource available.

The same pattern applies when you want to load custom CSS conditionally. Maybe you have specific styling for blog articles that shouldn't load on product pages:

php function visionvix_enqueue_blog_styles() { if (is_singular('post')) { wp_enqueue_style( 'visionvix-blog-styles', get_template_directory_uri() . '/assets/css/blog.css', array(), '1.0.0' ); } } add_action('wp_enqueue_scripts', 'visionvix_enqueue_blog_styles');

This checks if you're viewing a single post, then loads blog.css only on that post type. The visual rules live in the CSS file, not in this PHP function.

If you're working with AI agents to generate or audit content, making your site structure clear and predictable helps those systems understand your intent. Following established patterns like proper JSON formatting best practices and ensuring your markup is AI agent-friendly makes your site more maintainable by both humans and automated tools.

functions.php enables features and registers resources. It never applies visual styling.

This separation is the foundation of maintainability. When you keep functions.php focused on logic and registration, you can swap out entire styling layers without touching functionality. When you mix them, every design change risks breaking a feature. You can't hand off styling to a designer if it's tangled with PHP logic. You can't test visual changes in isolation if they're scattered across procedural code.

What Doesn't Belong Here

Never put these in functions.php:

Inline CSS rules written inside PHP strings
Direct font-family or color declarations
Hardcoded spacing values (margin, padding)
Layout rules (flexbox, grid, positioning)
Typography scales or responsive breakpoints

If it changes how something looks on the page, it doesn't belong in functions.php. Period.

style.css Is Your Paintbrush

This is where every visual rule lives. If it affects how something appears in the browser, it goes in style.css or a CSS file you enqueue from functions.php.

Think of style.css as the styling layer. It applies visual presentation to everything that functions.php loads. The font you enqueued gets applied here. The colors you want to use get declared here. The spacing between headings gets defined here.

What Belongs in style.css

Here's what you should put in your main stylesheet:

Global font-family declarations for body, headings, and UI elements
Font sizes and weights for your complete typography scale
Color variables or direct color values for text, backgrounds, and accents
Spacing rules (margin, padding) for consistent vertical rhythm
Layout rules (flexbox, grid, positioning) for page structure
Blog article typography with reading-optimized line height and spacing
Code block styling with syntax highlighting and background colors
Highlight boxes, callouts, and blockquote styles
Table of contents styling with indentation and hover states
Reading time indicator positioning and color
Custom dividers and signature footer styling

Let's continue the Inter font example. In functions.php, you loaded the font. Now in style.css, you apply it to actual elements:

``css
body {
font-family: 'Inter', sans-serif;
color: #111;
line-height: 1.7;
font-size: 16px;
}

h1, h2, h3 {
font-weight: 600;
letter-spacing: -0.02em;
margin-top: 2rem;
margin-bottom: 1rem;
}

h1 {
font-size: 2.5rem;
}

h2 {
font-size: 2rem;
}

h3 {
font-size: 1.5rem;
}
``

This is where the visual transformation happens. Every pixel of spacing, every color choice, every typographic detail gets defined here. The user never sees your functions.php code. They see the output of your CSS.

Building a Premium Blog Aesthetic

If you're building a content-focused site, style.css is where you define the reading experience. Code blocks get their dark background and syntax highlighting here. Pull quotes get their border and italic treatment here. The table of contents gets its indentation, numbering, and hover states here.

Here's a concrete example for code blocks:

``css
pre {
background: #1e1e1e;
border-radius: 8px;
padding: 1.5rem;
overflow-x: auto;
margin: 2rem 0;
}

code {
font-family: 'Fira Code', monospace;
font-size: 14px;
line-height: 1.6;
color: #d4d4d4;
}

.hljs-keyword {
color: #569cd6;
}

.hljs-string {
color: #ce9178;
}

.hljs-function {
color: #dcdcaa;
}
``

Every visual detail lives in CSS. No scattered inline styles. No mystery styles buried in block patterns. When you need to change the brand color, you change it in one place. When you need to adjust spacing, you update the CSS variable and it propagates everywhere.

Centralizing your visual rules in style.css means every design change happens in one predictable location.

One pattern I see constantly: developers write inline styles in PHP templates or scatter CSS across multiple files with no naming convention. That's a maintainability nightmare. When you need to debug why a button looks wrong, you shouldn't have to grep through twenty files to find the rule. Centralize your visual rules in style.css or a small set of well-named files like blog.css, blocks.css, or components.css.

theme.json Is Your Design System

This is the modern WordPress approach to design tokens. If you're working with the block editor or building a block theme, theme.json is essential. It's not optional anymore.

Think of theme.json as your design system configuration file. It defines reusable tokens - colors, font sizes, spacing values - that WordPress uses everywhere. The editor uses these tokens. The frontend uses these tokens. Custom blocks inherit these tokens. Core blocks respect these tokens.

What Belongs in theme.json

Here's what this file controls:

Global typography scale with named sizes (small, base, large, extra-large)
Global color palette with semantic names (primary, secondary, accent, background, text)
Global spacing scale with consistent values (tight, base, loose, extra-loose)
Global layout rules (content width, wide width for constrained layouts)
Block editor defaults (which blocks are enabled, which settings appear in the sidebar)
Block editor restrictions (disable custom colors, lock down font sizes)
Custom block styles (alternative visual treatments for core blocks)
Link color rules (default link color, hover state)
Button appearance defaults (padding, border radius, background color)

Here's a simplified example showing the structure:

json { "version": 2, "settings": { "color": { "palette": [ { "slug": "primary", "color": "#0066cc", "name": "Primary" }, { "slug": "accent", "color": "#ff6b35", "name": "Accent" }, { "slug": "background", "color": "#ffffff", "name": "Background" }, { "slug": "text", "color": "#111111", "name": "Text" } ] }, "typography": { "fontSizes": [ { "slug": "small", "size": "14px", "name": "Small" }, { "slug": "base", "size": "16px", "name": "Base" }, { "slug": "large", "size": "24px", "name": "Large" }, { "slug": "extra-large", "size": "32px", "name": "Extra Large" } ], "fontFamilies": [ { "slug": "body", "fontFamily": "'Inter', sans-serif", "name": "Inter" } ] }, "spacing": { "units": ["px", "rem", "em"], "spacingScale": { "steps": 4, "increment": 0.5, "unit": "rem" } }, "layout": { "contentSize": "720px", "wideSize": "1200px" } } }

When you define colors in theme.json, WordPress automatically generates CSS custom properties for you. That primary color becomes --wp--preset--color--primary. You can use it anywhere:

css .custom-button { background: var(--wp--preset--color--primary); color: var(--wp--preset--color--background); }

The same applies to font sizes and spacing. Define them once in theme.json, use them everywhere through generated CSS variables. No manual coordination needed. The editor sidebar shows these options automatically. Users can pick from your design system without breaking the visual consistency.

theme.json turns your design decisions into reusable tokens that propagate automatically across WordPress.

This is how you scale a design system. When you need to change your primary color, you change one value in theme.json. Every block, every component, every template that uses --wp--preset--color--primary updates instantly. No grep, no find-and-replace, no missed instances.

Why This Matters for Consistency

The block editor gives users a lot of power. They can change colors, adjust font sizes, and add custom spacing. Without theme.json, they get access to every color in the browser color picker. With theme.json, they get a curated palette that matches your brand. They can still customize, but within guardrails.

This is especially important if you're handing off the site to a client. You don't want them choosing random shades of blue that clash with the brand. You want them picking from a defined palette. theme.json enforces that by limiting their options to the colors, font sizes, and spacing values you defined.

How They Work Together

Here's the mental model: functions.php loads the resources, theme.json defines the design system, and style.css applies the visual rules.

When WordPress renders a page, it follows this sequence:

functions.php runs first. It enqueues fonts, CSS files, JavaScript files, and registers theme supports. This sets up the environment.
theme.json generates CSS custom properties. WordPress reads your color palette, font sizes, and spacing scale, then outputs those as CSS variables in the page <head>.
style.css applies visual rules. Your stylesheet uses the fonts that functions.php loaded and the CSS variables that theme.json generated to style every element on the page.

This separation keeps each layer focused on one responsibility. When you need to add a new font, you touch functions.php. When you need to adjust spacing, you touch style.css. When you need to add a new color to the palette, you touch theme.json.

A Real-World Example

Let's say you want to add a custom accent color and use it in blog article headings. Here's how you'd implement it across all three files:

Step 1: Define the color in theme.json

json { "version": 2, "settings": { "color": { "palette": [ { "slug": "accent", "color": "#ff6b35", "name": "Accent" } ] } } }

WordPress now generates --wp--preset--color--accent automatically.

Step 2: Use the color in style.css

css h2 { color: var(--wp--preset--color--accent); font-weight: 600; margin-top: 2rem; }

Your h2 headings now use the accent color defined in theme.json.

Step 3: No changes needed in functions.php

You didn't need to load anything new or register anything additional. The color token already exists in the system.

This is the ideal workflow. Design changes happen in theme.json. Visual application happens in style.css. Functionality stays in functions.php.

What You Should Put Where

Here's a reference checklist you can use when building or refactoring a WordPress theme:

Put in functions.php:

wp_enqueue_style() calls for fonts and CSS files
wp_enqueue_script() calls for JavaScript files
add_theme_support() for features like custom logo, post thumbnails, title tag
register_nav_menus() for navigation menu locations
register_sidebar() for widget areas
add_action() and add_filter() for custom logic
register_block_type() for custom blocks
Custom shortcode definitions
Reading time calculators, table of contents generators, and similar utilities

Put in style.css:

Font-family declarations
Font sizes, weights, and line heights
Color declarations for text, backgrounds, borders, and accents
Margin and padding for spacing rhythm
Flexbox and grid layout rules
Code block background colors and syntax highlighting
Pull quote borders and italic styles
Table of contents indentation and numbering
Hover states for interactive elements
Media queries for responsive breakpoints

Put in theme.json:

Color palette definitions with semantic slugs
Typography scale with named font sizes
Spacing scale with consistent rem values
Font family declarations for body and headings
Layout constraints (content width, wide width)
Block editor feature toggles (disable custom colors, lock font sizes)
Custom block style variations
Default link colors and button styles

When you follow this structure, your theme becomes modular. You can hand off styling to a designer who only needs to touch style.css. You can onboard a developer who only needs to understand functions.php. You can update design tokens in theme.json without touching code.

This is how professional WordPress themes scale. Not by cramming everything into one file, but by respecting the purpose of each file and keeping concerns separated. When you structure your themes this way, maintenance becomes predictable, handoffs become smooth, and design iterations don't break functionality.

WordPress gives you three files for three jobs. Use them correctly, and your design system will scale with every new block, page, and brand update. Mix them up, and you'll spend more time debugging than building.

📦 Publishing Kit — Dev.to

Title Options (5)

Selected: Stop Putting Everything in functions.php: The WordPress File Separation Guide

Alternates:

functions.php vs style.css vs theme.json: What Actually Belongs Where
WordPress Theme Architecture: Separating Logic, Styles, and Design Tokens
The Three-File WordPress Theme System That Prevents Technical Debt
How to Structure WordPress Themes Using functions.php, style.css, and theme.json

Slug

stop-putting-everything-in-functions-php-wordpress-file-separation-guide

Tags

webdev, tutorial, beginners, wordpress

Stop Paying for Bot Traffic: A Two-Layer Defense for Next.js apps

Julian Neagu — Fri, 14 Aug 2026 05:17:02 +0000

TL;DR: Bot traffic isn't a fact of life - it's a leak in your infrastructure budget. A two-layer defense using robots.txt and Next.js middleware blocks 60-70% of resource-draining scrapers while keeping the bots you actually want. This approach runs at the edge, costs nothing to implement, and protects both your hosting bill and your analytics accuracy.

Most founders assume bot traffic is part of the cost of running a website. That assumption is expensive. Bots hammer your endpoints, consume metered server resources, pollute your analytics, and inflate your hosting bill. If you're on Vercel, AWS, or any pay-per-request platform, this traffic literally costs money every time it hits your server.

This is what a bot attack looks like - a sudden, unnatural spike that distorts your real user activity.

The standard advice is to add a robots.txt file and hope for the best. That only works if bots respect it. Most scrapers don't. They read your file, note what you've marked off-limits, and scrape it anyway. The file is a suggestion, not a firewall.

The real solution is a two-layer defense: a smart robots.txt configuration that filters well-behaved bots, and Next.js middleware running at the edge that enforces the rules before traffic reaches your application. Combined, they block 60-70% of unwanted bot traffic while keeping legitimate crawlers and AI assistants you actually want.

This isn't about locking out AI crawlers. It's about controlling who gets access and at what cost.

The Problem: Three Kinds of Visitors

Your website serves three categories of traffic. Real humans browsing your site. Legitimate bots that index your content for search engines or train helpful AI models. And scrapers that hammer your endpoints, steal your data, rotate IP addresses, and pretend to be browsers.

The third category is the problem. These bots ignore polite requests. They consume server resources you're paying for. They pollute your analytics by inflating visitor counts with fake traffic, making conversion rates appear worse than they are. They probe for vulnerabilities. If you're running on metered hosting, every request from a scraper costs you a fraction of a cent. When scrapers make up 30-40% of your traffic, that adds up fast.

60-70% of bot traffic can be filtered by a properly configured robots.txt, but only because most low-effort scrapers follow basic rules to avoid detection.

The remaining 30-40% requires enforcement. That's where middleware comes in. It runs before every request hits your application, inspects headers and user agents, and blocks anything suspicious. On Vercel, middleware runs at the edge, meaning blocked requests never consume your server resources. You're not paying to process traffic you don't want.

Layer 1: Smart robots.txt Configuration

Start with robots.txt. This file sits at the root of your domain and tells compliant bots what they can access. The key is being explicit about who you allow and who you block. Most sites either leave this file empty or use a generic "allow all" rule. That's leaving money on the table.

Here's a structure that works:

``txt

AI training & assistants - welcome

User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: anthropic-ai
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: CCBot
Allow: /

Search engines - welcome

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

Known scrapers - blocked

User-agent: SemrushBot
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: DotBot
Disallow: /

User-agent: PetalBot
Disallow: /

Default - be polite or leave

User-agent: *
Crawl-delay: 10
Disallow: /api/
``

This makes three categories clear. AI assistants and search engines get full access. Known scrapers get nothing. Everything else gets throttled with a 10-second crawl delay and blocked from API routes.

The crawl delay is critical. Aggressive bots hitting your site every few seconds will slow down or leave. Legitimate crawlers won't care about a 10-second pause between requests. This single line filters out bots that hammer your site hundreds of times per minute.

The Disallow: /api/ rule is equally important. API routes often trigger database queries, authentication checks, or external service calls. Bots crawling these endpoints waste expensive compute. Block them by default unless you're running a public API that needs discovery.

This configuration takes five minutes to set up. Copy the template, adjust the bot names if you have specific concerns, and deploy it to your site's root directory. For a deeper dive into robots.txt strategy and common mistakes, check out VisionVix's robots.txt best practices guide, which walks through edge cases and SEO implications.

Most low-effort scrapers follow basic robots.txt rules to avoid detection, making it effective against the majority of bad actors.

This handles the 60-70% of bot traffic that follows rules. It won't stop determined attackers, but it filters the lazy majority. Now we enforce the rest.

Layer 2: Middleware Enforcement at the Edge

robots.txt is polite. Middleware is the bouncer. It runs before every request hits your application, inspects the user agent and headers, and blocks anything suspicious. If you're using Next.js on Vercel, middleware runs at the edge. Requests get filtered before they consume your server resources. You're not paying to process blocked traffic.

Create a file called middleware.ts at the root of your Next.js project:

``typescript
import { NextResponse } from 'next/server'

const ALLOWED_BOTS = [
'googlebot',
'bingbot',
'gptbot',
'claudebot',
'perplexitybot',
'slackbot',
'twitterbot'
]

const BLOCKED_BOTS = [
'petalbot',
'blexbot',
'semrushbot',
'ahrefsbot',
'mj12bot',
'dotbot'
]

const SCRAPER_LIBS = [
'scrapy',
'python-requests',
'curl/',
'wget',
'java/',
'go-http-client'
]

export function middleware(req) {
const ua = (req.headers.get('user-agent') || '').toLowerCase()

// 1. Allow good bots
if (ALLOWED_BOTS.some(bot => ua.includes(bot))) {
return NextResponse.next()
}

// 2. Block empty user agents
if (!req.headers.get('user-agent')) {
return new NextResponse('Forbidden', { status: 403 })
}

// 3. Block aggressive crawlers
if (BLOCKED_BOTS.some(bot => ua.includes(bot))) {
return new NextResponse('Forbidden', { status: 403 })
}

// 4. Block scraper libraries
if (SCRAPER_LIBS.some(lib => ua.includes(lib))) {
return new NextResponse('Forbidden', { status: 403 })
}

// 5. Block missing Accept header
if (!req.headers.get('accept')) {
return new NextResponse('Forbidden', { status: 403 })
}

// Default: pass
return NextResponse.next()
}
``

Every request triggers this function. The middleware reads the user agent string and headers. If the user agent matches a known good bot, the request passes immediately. If it matches a blocked bot or scraper library, it gets a 403 response. If critical headers are missing, it's blocked.

The Accept header check is subtle but effective. Real browsers and legitimate bots send an Accept header telling the server what content types they understand. Scrapers often skip this. Blocking requests without an Accept header catches a lot of low-effort scraping attempts.

Middleware runs at the Vercel edge, meaning blocked requests never hit your application server or consume compute resources. This is what reduces the remaining 30-40% of bot traffic that ignored robots.txt.

What Gets Blocked and Why

Let's walk through what each check catches in practice.

Empty User Agents

typescript if (!req.headers.get('user-agent')) { return new NextResponse('Forbidden', { status: 403 }) }

Legitimate browsers always send a user agent. Tools like curl and wget do too, but scrapers built with low-effort scripts often don't. This single check blocks a surprising amount of junk traffic.

Scraper Libraries

typescript if (SCRAPER_LIBS.some(lib => ua.includes(lib))) { return new NextResponse('Forbidden', { status: 403 }) }

This targets common scraping tools. A user agent string like python-requests/2.28.0 or Scrapy/2.9.0 reveals exactly what's hitting your site. These are almost never legitimate traffic. Block them immediately.

Missing Accept Header

typescript if (!req.headers.get('accept')) { return new NextResponse('Forbidden', { status: 403 }) }

Real browsers send an Accept header like text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8. Bots that don't care about the response format skip this header. It's a strong signal that the request isn't coming from a browser or a well-behaved crawler.

The Conditional Middleware Pattern

Not every route needs the same level of protection. Public pages like your homepage or blog should allow all legitimate traffic. Admin panels, API routes, or expensive database queries need stricter rules.

You can add conditional logic to your middleware:

``typescript
export function middleware(req) {
const pathname = req.nextUrl.pathname

// Skip middleware for static assets
if (pathname.startsWith('/_next') || pathname.startsWith('/static')) {
return NextResponse.next()
}

// Apply strict rules to API routes
if (pathname.startsWith('/api')) {
return strictBotCheck(req)
}

// Apply normal rules to everything else
return normalBotCheck(req)
}
``

This pattern lets you enforce stricter checks on expensive routes while keeping public pages accessible. The strictBotCheck function could block all bots except explicitly allowed ones. The normalBotCheck function could allow most traffic but throttle suspicious patterns.

Honeypot Traps and Advanced Tactics

If you want to go further, add a honeypot trap. Create a page that's linked in your robots.txt under a Disallow rule. Good bots won't visit it. Bad bots will.

Add this to your robots.txt:

txt User-agent: * Disallow: /trap

Then create a /trap route in your Next.js app. Anyone who visits it gets their IP logged and blocked:

`typescript // pages/api/trap.ts export default function handler(req, res) { const ip = req.headers['x-forwarded-for'] || req.connection.remoteAddress console.log(Bot trap triggered by IP: ${ip}`)

// Add IP to blocklist
// (integrate with your edge config or database)

return res.status(403).json({ error: 'Forbidden' })
}
``

This catches bots that explicitly ignore your robots.txt rules. You can feed these IPs into your middleware blocklist or configure edge rules at the CDN level. For a broader look at proactive security measures like this, see the website security scanning guide, which covers automated vulnerability detection and bot behavior analysis.

What You're Actually Protecting

This defense isn't just about saving money. It protects three things founders underestimate:

Hosting costs. If bots make up 30-40% of your traffic, you're paying 30-40% more than you should. On metered platforms like Vercel or AWS Lambda, every function invocation costs money. Blocking bot traffic before it hits your application cuts those costs immediately.

Analytics accuracy. Bot traffic inflates your visitor counts and skews conversion rates. If 40% of your "users" are scrapers, your real conversion rate is higher than your dashboard shows. Clean traffic gives you accurate data to make decisions.

Attack surface. Bots probe for vulnerabilities. They test SQL injection patterns, brute-force login endpoints, and enumerate API routes. Blocking them reduces your exposure to automated attacks.

Blocking requests without Accept headers or empty user agents catches most low-effort scraping attempts.

Implementation Checklist

Here's what to do right now:

Deploy the robots.txt template to your site's root directory. Adjust the allowed and blocked bot lists based on your traffic logs.
Add the middleware file to your Next.js project. Copy the full code block above, save it as middleware.ts at the project root, and deploy.
Monitor your logs for a week. Check which user agents are getting blocked. If you see legitimate traffic being filtered, adjust your ALLOWED_BOTS list.
Review your hosting bill after 30 days. If bots were a significant portion of your traffic, you should see a measurable drop in compute costs.
Check your analytics for cleaner data. Visitor counts should drop, but conversion rates should improve because you're measuring real users instead of bots.

This setup takes less than an hour to implement. It costs nothing. It runs at the edge, so there's no performance penalty for legitimate users. And it blocks the majority of bot traffic that's wasting your budget and polluting your metrics.

If you've been treating bot traffic as inevitable, stop. It's not. Two layers of defense, five minutes of configuration, and you're done. The bots you want still get in. The ones you don't pay for anymore.

📦 Publishing Kit — Dev.to

Title Options (5)

Selected: Stop Paying for Bot Traffic: A Two-Layer Defense for Next.js Apps

Alternates:

How robots.txt + Next.js Middleware Cuts Bot Costs by 60-70%
Block Scrapers, Keep AI Crawlers: Smart Bot Management for Next.js
The $500/Month Bot Problem (And How to Fix It With Edge Middleware)
robots.txt Isn't Enough: Building a Real Bot Firewall in Next.js

Slug

stop-paying-for-bot-traffic-nextjs-middleware-defense

Tags

nextjs, webdev, devops, security

Blocking GPTBot Won't Remove You From ChatGPT Search Here's Why

Julian Neagu — Thu, 06 Aug 2026 10:44:19 +0000

TL;DR: Blocking GPTBot doesn't remove you from ChatGPT search results. OpenAI runs two separate bots: one that trains models (GPTBot) and one that fetches live citations (OAI-SearchBot). Every major AI company uses this dual-bot system. Block the wrong one and you vanish from AI search. Block the right one and you protect your content while staying visible.

Most people who block GPTBot expect their content to disappear from ChatGPT entirely. Then they search for their own article title in ChatGPT and find it sitting right there in the results with a working link and an accurate pull quote. The block didn't fail. You just blocked the wrong bot.

OpenAI doesn't run one bot. They run two. So does Anthropic. So does Google. Every major AI company operates a two-bot system because training and citation are fundamentally different jobs that need fundamentally different infrastructure.

The first bot harvests your content to train the model. It reads your articles, extracts patterns from your prose, and bakes those patterns into billions of model parameters. Your words become part of the AI's permanent knowledge. The model never cites you. It never links back to you. It never sends you traffic. Training bots give you nothing in return for what they take.

The second bot fetches your content in real time when someone asks a question your site might answer. The bot finds your page, reads the relevant section, and surfaces it in the AI's response with your name, a link, and a snippet. This is how you show up in ChatGPT search results. Citation bots drive referral traffic and keep you visible in AI-powered search.

Blocking the training bot does absolutely nothing to the citation bot. They don't share a blocklist. They don't check each other's permissions. They operate as if they work for separate companies.

The Two-Bot System Every AI Company Runs

Here's the full breakdown of how each major AI company splits training from citation:

OpenAI runs GPTBot for training. GPTBot crawls your site, reads your content, and uses it to improve future models. That's where your IP goes in and never comes back out. For live citations, OpenAI runs OAI-SearchBot and ChatGPT-User. These bots fetch content when a user asks a question. Block GPTBot and you stop training. Block OAI-SearchBot and you vanish from ChatGPT search results.

Anthropic operates ClaudeBot for training. ClaudeBot works exactly like GPTBot. It harvests content to build the model's knowledge base. For citations, Anthropic runs Claude-SearchBot and Claude-User. Block ClaudeBot to stop training. Leave Claude-SearchBot allowed if you want to appear in Claude's search answers.

Google uses Google-Extended for training. This is separate from Googlebot, which has been crawling the web for decades. Googlebot handles citations because Google already had a citation infrastructure before AI search existed. Block Google-Extended and you stop contributing to Gemini training. Block Googlebot and you disappear from Google Search entirely, which most sites can't afford.

Perplexity runs PerplexityBot and Perplexity-User. Both fetch content for live answers. Perplexity doesn't train foundation models, so they don't operate a separate training crawler. Every request from Perplexity is a citation request.

Meta operates meta-externalagent for training. They don't currently run a public citation bot for consumer search products, so blocking meta-externalagent has no downside for visibility.

Training crawlers need bulk access to millions of pages, running batch jobs that might revisit your content weeks or months later. Citation crawlers need low-latency access to fresh content, often fetching a single page in response to a specific user query.

This architecture exists because training and citation have fundamentally different requirements. Training bots run in batches. They might crawl your site once and never come back. They don't care about freshness. They care about volume. Citation bots run on demand. They fetch a single page in milliseconds when a user asks a question. They care about speed and freshness, not volume.

Building one bot that does both jobs well is harder than running two specialized systems. So every company runs two. As AI crawling vs traditional crawling explains, the infrastructure behind these systems diverges at nearly every layer, from request patterns to caching strategies to how they handle rate limits.

What You're Blocking Right Now (And What You're Not)

Most robots.txt files I see block training bots and stay silent on citation bots. That's not necessarily wrong. The robots.txt standard works by explicit permission. If you don't mention a bot, it's allowed by default. But I don't like relying on default behavior. Defaults change when companies push updates or rewrite their crawler logic.

Here's what a safe, explicit robots.txt configuration looks like if you want to block training but allow citations:

``txt
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /
``

The first block stops training. The second block guarantees you stay citable. Both blocks matter. Leaving citation bots unstated means trusting default behavior, and I've seen too many cases where defaults shifted after a company updated their crawler documentation.

You'll notice Googlebot isn't in either list. That's intentional. Blocking Googlebot means disappearing from Google Search entirely, which most sites can't survive. Google-Extended is the training-specific bot. Block that and leave Googlebot alone. You get the split you want: no training, yes citations.

One more detail. If you're running a Cloudflare-managed site, your robots.txt likely includes a comment that says # BEGIN Cloudflare Managed content. That marker means Cloudflare auto-generates part of your robots.txt based on dashboard settings. Don't edit robots.txt directly when that marker is present. Cloudflare will overwrite your changes on the next sync. Make changes through Cloudflare's AI Crawl Control panel instead.

How to Edit Robots.txt When Cloudflare Manages It

If you see the Cloudflare marker in your robots.txt, here's the correct path to making changes stick:

Log into your Cloudflare dashboard and select the domain you're managing.
Click Domains in the left sidebar.
Select your specific domain from the list.
Scroll down to AI Crawl Control or search for "AI" in the page search.
Toggle the bots you want to block or allow using the switches provided.

If you've been blocking GPTBot and wondering why your articles still show up in ChatGPT, now you know. The bot you blocked wasn't the bot you thought it was. And if you want to stay visible in AI search while protecting your content from training, you need to configure both. For a deeper look at how AI systems decide what to crawl and when, see this comparison of Claude vs GPT for coding tasks, which shows how different models prioritize different types of content even when using similar underlying architectures.

Cloudflare's AI Crawl Control panel handles the syntax for you. Changes take effect within a few minutes and won't be overwritten by automated syncs. If you need more granular control than Cloudflare's toggles provide, you'll need to disable Cloudflare management of robots.txt entirely and manage the file manually as we advance.

Checking Whether Your Block Actually Works

After updating your robots.txt, wait 24 hours and check your server logs or analytics dashboard. Look for requests from the bots you tried to block.

Training bots like GPTBot, ClaudeBot, and Google-Extended should show zero requests and zero bytes transferred after blocking. If you still see requests from GPTBot after a week, either your robots.txt syntax is wrong or the bot isn't respecting the directive. Cloudflare's WAF lets you write a firewall rule that blocks by user agent string if robots.txt alone doesn't work.

Citation bots like OAI-SearchBot and ChatGPT-User should show periodic requests when users ask questions your content might answer. These requests are low-volume and sporadic. You might see a handful per week for a niche topic or dozens per day if you rank well for common queries. This is normal and desirable. Each request represents a potential citation in an AI answer.

If you're not seeing any citation bot traffic after allowing them, check two things. First, make sure your Allow: / directive is present and correctly formatted. Second, verify that your content actually ranks for questions people ask AI tools. Not every page earns citations. The citation bots only fetch pages that match a user query.

Why This Matters More Than You Think

Training without attribution is economically asymmetric. Your content becomes part of a billion-dollar model's knowledge base. You get no traffic, no links, and no royalties. The model answers questions using patterns it learned from your prose, and you never see a visitor.

Citation with attribution reverses that asymmetry. Your content appears as a source. The AI links to your page. Users click through. You get referral traffic from people who were already looking for what you wrote about. This is the only version of AI-powered search that benefits content creators.

Blocking training bots while allowing citation bots is the rational default for most sites. You protect your IP from being absorbed into the model's weights. You stay visible in the AI-powered search results that are increasingly replacing traditional Google queries. You get the upside without the downside.

The mistake most people make is thinking GPTBot controls all of OpenAI's access to your site. It doesn't. GPTBot controls training. OAI-SearchBot controls citations. Block one, allow the other, and you get the outcome you actually want.

The two-bot system isn't going away. Every AI company has adopted it because the engineering requirements for training and citation are incompatible. One bot can't do both jobs well. So they split the work. And you need to split your robots.txt configuration to match.

📦 Publishing Kit — Dev.to

Title Options (5)

Selected: Blocking GPTBot Won't Remove You From ChatGPT Search — Here's Why

Alternates:

You're Blocking the Wrong AI Bot: The Training vs Citation Crawler Problem
Why Your Site Still Appears in ChatGPT After Blocking GPTBot
The Two-Bot System: How AI Companies Split Training From Citations
Block GPTBot, Still Show Up in ChatGPT: Understanding AI Crawler Architecture

Slug

blocking-gptbot-wont-remove-you-from-chatgpt-search

Tags

webdev, ai, opensource, seo

Auto-Submit Changed Pages to Search Engines with GitHub Actions and IndexNow

Julian Neagu — Wed, 29 Jul 2026 13:50:20 +0000

TL;DR: If you're pushing updates to a static site every week, manually notifying search engines is a waste of time. GitHub Actions can detect exactly which files changed in each commit, convert those paths to URLs, and submit them to IndexNow automatically. You build once, push to main, and search engines get notified within minutes - no batch uploads, no forgotten submissions, no resubmitting 388 URLs when only three changed.

If your static site gets a dozen updates a week, manually notifying search engines gets old fast. Most builders either skip the submissions entirely or batch them monthly, which means new pages sit invisible for weeks.

GitHub Actions combined with IndexNow solves this. Your repo already knows what changed, and search engines support instant notifications. You just need to wire them together.

Here's the real difference: instead of pushing code and then remembering to notify Bing or Yandex later, you build a workflow that detects changed files, waits for your deployment to finish, and submits only the modified URLs. No manual script runs. No forgotten submissions. No resubmitting all 388 URLs when only three changed.

This article walks through what GitHub Actions does, how IndexNow works, and how to set up submissions that fire on every push to main.

What GitHub Actions Actually Does

GitHub Actions is GitHub's built-in automation system. You write a workflow file, drop it in .github/workflows/, and GitHub runs it whenever a trigger fires. Common triggers include pushing code, opening a pull request, or hitting a cron schedule.

For static sites and micro-tools, this becomes a deployment and notification engine. You can:

Run a build step when you push to main
Detect which files changed in the commit
Convert those file paths into live URLs
Wait for your hosting platform to finish deploying
Submit only the changed URLs to search engines

The key advantage: GitHub already has the commit diff. It knows exactly which pages you modified. You don't need to scan your entire sitemap or keep a separate change log.

Run on Every Push

The most common trigger is push to a specific branch:

yaml on: push: branches: - main

Every time you merge a PR or push directly to main, the workflow fires. This matches how most teams deploy: main is production, and every merge should trigger a deployment notification.

For teams managing multiple repositories with similar automation needs, GitHub repository best practices covers how to structure .github/ folders, share workflows across repos, and maintain consistency when scaling to dozens of projects.

Detect Changed Files

GitHub Actions can extract the list of modified files from the commit diff. The official IndexNow Action supports multiple input modes:

urls-from: generate URLs from a shell command (like parsing the diff)
file: read URLs from a text file you generate in an earlier step
sitemap: submit URLs from an existing XML sitemap
urls: pass a direct list of URLs as a string

For changed-file workflows, urls-from is cleanest. You run a command that outputs one URL per line, and the Action reads that output.

Example shell logic on macOS or Linux:

This grabs modified files, filters for index.html entries (assuming each page is a folder with an index file), and converts paths into full URLs.

On Windows with PowerShell, you'd use native cmdlets instead:

powershell $changedFiles = git diff-tree --no-commit-id --name-only -r HEAD $urls = $changedFiles | Where-Object { $_ -like '*index.html' } | ForEach-Object { $_ -replace '^', 'https://yourdomain.com/' -replace '/index\.html$', '/' } $urls -join "`n"

Both produce the same output: a list of URLs, one per line, ready to feed into the IndexNow Action.

Submit URLs to IndexNow Automatically

The IndexNow Action handles the actual submission. It supports five search engines:

Bing
Yandex
Naver
Seznam
Yep

IndexNow submissions are shared across all supporting engines. You submit once, and Bing tells Yandex, Naver, Seznam, and Yep about the update.

The Action supports multiple feed formats:

XML sitemaps
RSS feeds
Atom feeds
Direct URL lists
Custom endpoints

For changed-file workflows, direct URL lists are simplest. You pass the output of your diff command and let the Action batch the submissions.

If you're building workflows for rapid deployment and want to see how to structure automation logic around changed files and deployment hooks, check out building your first automated workflow step for a deeper dive into event-driven automation patterns.

The Action also supports limits (submit only the first N URLs), time filtering (submit only URLs modified since X days ago), and custom key file locations.

Example workflow snippet:

``yaml

name: Submit changed URLs to IndexNow uses: bojieyang/indexnow-action@v2 with: sitemap-location: 'https://yourdomain.com/sitemap.xml' key: ${{ secrets.INDEXNOW_KEY }} key-location: 'https://yourdomain.com/indexnow-key.txt' ``

This submits URLs from your sitemap. For changed-file workflows, you'd replace sitemap-location with urls-from and pass your diff command.

Run on a Schedule

You can also trigger workflows on a cron schedule:

yaml on: schedule: - cron: '0 2 * * *' # 2 AM UTC daily

This is useful for submitting your full sitemap once a day, even if nothing changed. Some builders use this as a safety net: changed-file submissions run on every push, and a daily full-sitemap submission catches anything the diff logic missed.

Use Secrets for Keys

GitHub Actions supports encrypted secrets. You store your IndexNow key in repository settings under Secrets and variables, then reference it in your workflow as ${{ secrets.INDEXNOW_KEY }}.

This keeps the key out of your public repository. The workflow file itself is visible, but the key value is encrypted and only decrypted during workflow runs.

Generate Key Files Dynamically

The official IndexNow docs warn against committing your key file to a public repository. If the file is visible in your repo, anyone can see the key contents.

The recommended approach: generate the key file during deployment. Your hosting platform writes the key to disk as part of the build step, so it's available at https://yourdomain.com/indexnow-key.txt but never committed to version control.

Most static site hosts support build hooks or environment variable injection. You can write a simple script that creates the key file from a secret environment variable during the build process.

What You Get

A typical static site might have 388 URLs total but only 3 changed URLs per update.

With this workflow in place:

Every push to main triggers the workflow
GitHub extracts the diff and converts changed paths to URLs
The IndexNow Action submits only those URLs to all five supported search engines
Search engines crawl the updated pages within hours instead of weeks
You never manually submit a URL again

Traditional sitemap-based indexing can take days or weeks. IndexNow reduces this to hours. Sites that receive a dozen updates per week benefit the most, because each batch of changes gets notified immediately instead of sitting in a queue until someone remembers to submit them.

Common Mistakes to Avoid

Submitting the entire sitemap on every push. This works, but it wastes API quota and makes it harder for search engines to prioritize your fresh content. Only submit what changed.

Forgetting to wait for deployment. GitHub Actions can't detect deployment completion automatically. If your workflow submits URLs before your hosting platform finishes deploying, search engines will crawl stale versions. Add a delay step or poll your deployment status endpoint before submitting.

Committing your IndexNow key to a public repo. Use GitHub Secrets and generate the key file dynamically during deployment. Never hardcode the key in a workflow file or commit it to version control.

Assuming Google supports IndexNow. Google does not. You need a separate notification mechanism, either through the Search Console API or by updating your XML sitemap and pinging Google's submission endpoint.

Why This Matters for Static Sites

Static sites change frequently in small batches rather than all at once. You publish a new blog post, fix a typo on three pages, add a new product page. Each of these events is a small diff.

Most static site builds finish in under 60 seconds. That means your workflow can detect changes, wait for deployment, and submit URLs within two minutes of pushing to main.

This kind of speed doesn't matter for sites that update once a month. But if you're shipping updates every few days, the difference between manual batch submissions and automatic change detection is the difference between pages that index in hours and pages that sit invisible for weeks.

You build the workflow once. After that, every push to main handles its own search engine notifications. No manual steps. No forgotten submissions. No wasted API quota on unchanged URLs.

📦 Publishing Kit — Dev.to

Title Options (5)

Selected: Auto-Submit Changed Pages to Search Engines with GitHub Actions and IndexNow

Alternates:

Stop Manual Search Submissions: Automate IndexNow with GitHub Actions
GitHub Actions + IndexNow: Submit Only Changed URLs to Search Engines
How to Auto-Notify Search Engines When You Push Static Site Updates
Automate Search Engine Notifications for Static Sites with GitHub Actions

Slug

auto-submit-changed-pages-search-engines-github-actions-indexnow

Tags

webdev, devops, tutorial, indexnow

We Built 1,500 Keyword-Rich Domains Before Chasing Backlinks, Here's What Happened

Julian Neagu — Mon, 27 Jul 2026 14:20:50 +0000

TL;DR: We built 1,500+ keyword-rich domains hosting real AI tools before worrying about backlinks. Each domain is a production-grade product that passes enterprise security audits. The result: over 1 million AI citations in seven months, impressions within days of launch, and a compounding authority mesh that grows sideways rather than up.

Most founders obsess over backlinks. They build a product, launch it on a generic domain, then spend months begging for links from high-authority sites. We flipped that model.

We built the ranking infrastructure first. Then we filled it with products.

What started as a bootstrapped need for tools we couldn't afford became a 1,500-domain ecosystem. Every domain hosts a real AI tool. Every tool solves a real problem. Every site passes the strictest compliance audits from day one. This isn't a private blog network where hollow sites exist only to pass link juice. It's a distributed product mesh that compounds authority through authentic user engagement.

" width="800" height="465">

In seven months, this network-mesh approach drove more than one million AI citations to VisionVix.com. That number isn't a vanity metric. It's a massive indicator of how fast the ecosystem compounds when every node is real, useful, and built to rank.

Keyword-Rich Domains That Google Already Trusts

The foundation is a portfolio of 1,500+ keyword-rich domains spanning .dev, .app, .world, .ai, and .com extensions. These aren't vanity URLs. These aren't placeholder sites waiting for content. Each domain was chosen because it communicates intent instantly and carries inherent trust signals.

Google treats .dev and .app as secure-by-default TLDs because they enforce HTTPS at the registry level. Users click them more because the domain itself answers the question "what does this do?" When someone sees carvaluation.app, they know exactly what to expect. When they see genericstartup.com/tools/cars, they have to parse the path structure and guess.

The marketplace shows this strategy in action. Look at the names: koreanfood.app, [goldtrade.app](https://goldtrade.app/), carvaluation.app, pokezonline.app, and dozens more. Each name is a micro-product waiting to launch. When you own the exact-match domain for a tool category, you start with positioning advantage before writing a single line of code.

1,500+ keyword-rich domains across .dev, .app, .world, .link, and .com create instant topical relevance and trust signals that generic domains take months to build.

This is why new domains in the mesh begin receiving impressions within days, not months. The domain name itself is a relevance signal. No amount of on-page SEO on a generic domain can replicate the clarity of a purpose-built keyword domain. The difference between "try our car valuation tool at genericstartup.com/tools/cars" and "visit carvaluation.app" is the difference between explaining and being understood immediately.

Why TLD Choice Actually Matters

The choice of TLD matters more than most founders realize. We covered the strategic differences in depth when comparing .app versus .com domains, but the short version is this: modern TLDs carry semantic weight.

A .dev domain signals developer tools. A .app domain signals an application. A .world domain signals a platform or ecosystem. Google doesn't just see these as interchangeable alternatives to .com. It treats them as categorization hints.

For a deeper explanation of how this works at the registry level, understanding what a top-level domain actually is helps clarify why TLD choice affects both ranking speed and user perception.

Below you can see our Google Analytics data: in just about two weeks, roughly 580 of our .app domains went live, and the traffic skyrocketed almost immediately. The pattern is clear. These are keyword-rich domains that Google understands and rewards.

The engagement metrics prove this pattern. Analytics show 2.8k active users and 9.4k events tracked across the mesh. Traffic originates from the US, Hong Kong, Singapore, Germany, and the UK. Domains act as connectors, not isolated endpoints.

Some domains gain impressions within two weeks of launch. Some domains rank with zero backlinks because the on-page relevance and user engagement signals are strong enough to trigger indexing and ranking without external validation.

Our Domains: Purpose-Built, Keyword-Rich Assets for Digital Products

At VisionVix, we don’t collect domains—we build digital opportunities around them.

Every domain in our portfolio is selected with a clear objective: to become the foundation for a practical micro-tool, AI agent, software application, or SEO-driven growth asset. We focus on domains that combine strong keywords, user intent, and real commercial potential.

Our expertise lies in identifying premium .app and .dev domains that are clear, memorable, and aligned with future product opportunities. These domains are not simply names sitting unused—they are strategic digital assets designed for development, deployment, and long-term value creation.

Why Keyword-Rich Domains Create Value

A strong keyword domain gives a product an immediate advantage by connecting directly with what users are searching for. Keyword-rich domains can help:

Improve search visibility and relevance
Communicate product purpose instantly
Build credibility with users
Increase organic click potential
Support efficient customer acquisition
Strengthen SEO ecosystems and content strategies

When combined with a useful product and strong execution, a focused exact-match domain provides a powerful foundation for building online authority.

Our Domain Acquisition Strategy

We select domains based on a combination of market data, usability, and future development potential. Each domain is reviewed for:

Relevant search keywords
Commercial intent
Product alignment
Brand potential
Trademark availability
Historical quality, including Wayback and backlink checks
Integration potential within our 500+ tool development ecosystem

Our evaluation process considers:

Search volume
Keyword competitiveness
Traffic opportunity
CPC value
SERP landscape
Brandability
Compliance requirements

This approach ensures our portfolio consists of practical, build-ready assets rather than speculative domain holdings.

Examples of High-Intent .app Domains

Selected examples include:

koreanfood.app
goldtrade.app
ailoan.app
aieuct.app
carvaluation.app
pokezonline.app

Each domain is short, memorable, and instantly communicates its intended use case—making it suitable for AI tools, applications, marketplaces, and specialised digital services.

At VisionVix, we believe the right domain is more than an address—it is the starting point for building valuable products and scalable digital experiences.

Every Domain in the SEO Network Mesh Meets Product-Grade Compliance

Unlike a PBN where sites exist only to pass link juice, every domain in the mesh is a real product with real infrastructure. Each one ships with fully built landing pages, working AI tools or calculators, SSL certificates, strict security headers, privacy policies, terms of service, cookie consent, structured data, semantic HTML, agentic browsing readiness, high PageSpeed scores, and clean Next.js or static builds.

The audit results prove this. WebsiteAudit.dev scores 95/100. SecurityAudit.dev scores 100/100. VisionVix.world receives an A-grade security header report.

Agentic browsing audits pass with perfect scores. PageSpeed Insights consistently shows 100/100 performance, SEO, and best practices across the portfolio.

Production-Grade Infrastructure on Every Domain

These aren't placeholder sites thrown up to game search engines. They're production-grade applications built to survive compliance audits from enterprise customers, legal review, and accessibility standards. Every domain includes:

SSL certificates issued and auto-renewed through Let's Encrypt or Cloudflare
Security headers enforcing Content Security Policy, Strict Transport Security, and X-Frame-Options
Privacy and legal pages written by actual legal review, not copied templates
Structured data markup using JSON-LD for breadcrumbs, organization, and product schemas
Semantic HTML with proper heading hierarchy, ARIA labels, and landmark roles
Mobile responsiveness tested across viewport sizes and touch interactions
Cookie consent flows that comply with GDPR and CCPA requirements
Accessibility readiness including keyboard navigation, screen reader support, and contrast ratios

This level of quality creates real defensibility. When an enterprise evaluates a tool, they run security scans. When a journalist references a calculator, they check the about page. When an AI agent crawls a domain, it expects structured data.

The mesh survives these checks because every site was built to pass them. The infrastructure isn't optimized for SEO tricks. It's optimized for real use.

Why Compliance Matters for Rankings

Google's algorithm doesn't just measure backlinks and keywords. It measures user experience signals. Sites that load fast, work on mobile, respect privacy, and follow web standards send trust signals. Sites that break on mobile, load slowly, or lack legal pages send warning signals.

When you build 1,500 sites, you can't manually tune each one. You need infrastructure that makes compliance automatic. That's why every domain in the mesh inherits security headers, structured data, and legal pages from the same base template. One update cascades across the network.

This approach also protects against manual review. When Google's quality raters spot-check sites in the mesh, they find real tools with real users and real infrastructure. There's nothing to penalize.

For a detailed walkthrough of finding and fixing website accessibility issues, we covered the full process in an earlier post. The same principles apply across the mesh. Accessibility isn't a nice-to-have. It's a ranking signal.

The Keyword-Rich Domain Strategy Behind the Largest AI Agent Marketplace

The marketplace approach amplifies the mesh strategy. Instead of building one massive domain with hundreds of subdirectories, we built hundreds of standalone domains, each hosting a single tool or category. Then we connected them through the VEX Domain Marketplace.

Each domain in the marketplace is a live product. Users can visit goldtrade.app and get real-time precious metal price data. They can visit carvaluation.app and estimate vehicle worth. They can visit koreanfood.app and explore recipes with nutritional breakdowns.

Over 1 million AI citations to VisionVix.com in seven months proves the compounding effect of a distributed product mesh built on keyword-rich domains.

The marketplace itself becomes a hub. It lists every domain, categorizes them by function, and surfaces them to users searching for specific tools. This creates a natural discovery loop. A user finds one tool, sees the marketplace, discovers three more related tools.

Why Standalone Domains Beat Subdirectories

There's a persistent debate in SEO about whether subdomains or subdirectories rank better. We skipped the debate entirely and went straight to standalone domains. Here's why.

Each domain is an independent entity. It can rank for its own keywords without competing with sibling pages. It can earn its own backlinks. It can build its own brand. It can be sold, spun off, or repurposed without affecting the rest of the mesh.

Subdirectories force everything under one domain authority. If the parent domain gets penalized, every subdirectory suffers. If one subdirectory attracts spam, the entire domain risks contamination. Standalone domains isolate risk.

More importantly, standalone domains create more SERP real estate. When someone searches for "car valuation calculator," we can occupy multiple positions with carvaluation.app, carcalculator.dev, and vehiclevalue.world. Subdirectories under one domain would only let us occupy one position.

The mesh isn't just domains. It's functionality. The ecosystem includes mini tools and 160+ free calculators (but we're adding daily) distributed across the portfolio. These aren't demo apps or MVPs. They're production tools that thousands of people use daily to solve real problems.

The marketplace model also scales horizontally. We can launch 50 new domains in a weekend without touching existing infrastructure. Each new domain adds another node to the mesh. Each node compounds the authority of the network.

How We Choose Which Domains to Build

Not every keyword-rich domain is worth building. The decision framework is simple:

Does the domain name instantly communicate what the tool does? If someone sees goldtrade.app, do they immediately understand it's about precious metal trading? If yes, continue. If no, skip.
Is there enough search volume to justify the build? We don't need millions of searches per month. We need enough to generate a few hundred visits per month once ranked. Anything above 1,000 monthly searches is worth building.
Can we build a real tool, not just a landing page? If the domain is calculator.app, we need to build an actual calculator. If we can't deliver the function the domain promises, we don't launch it.
Does it fit into a category cluster? Isolated domains are harder to rank. Domains that connect to a category cluster inherit authority faster. If we already have five finance-related tools, adding goldtrade.app strengthens the entire finance cluster.
Is the domain short, memorable, and easy to type? Long domains don't convert. Domains with hyphens look spammy. Domains that require explanation lose users. We filter ruthlessly for clarity.

Using these filters, we've built a portfolio of 1,500+ domains. Each one passes the test. Each one adds value to the mesh.

Natural Cross-Linking Creates Sideways Authority Flow

Traditional SEO focuses on hub-and-spoke models. You build one authoritative hub, then create spokes that link back to it. The hub gets stronger, the spokes stay weak.

The mesh model works differently. Every domain links to related domains. Authority flows sideways, not just upward. When carvaluation.app links to carcalculator.dev and vehiclevalue.world, all three benefit. When goldtrade.app links to silverprices.world and platinumtrends.dev, the entire precious metals cluster gains strength.

This creates a compounding effect. Each new domain doesn't just add one node. It adds connections between existing nodes. The network becomes denser, not just larger.

Some domains gain traction and rank within two weeks of launch with zero backlinks due to strong on-page signals and user engagement.

Google doesn't penalize this because the links are contextually relevant and user-focused. A user on carvaluation.app who needs a more detailed breakdown genuinely benefits from a link to carcalculator.dev. The link serves the user, not just the algorithm.

The Difference Between a Mesh and a PBN

Private blog networks (PBNs) are built to manipulate rankings. Sites exist only to pass link juice. Content is thin. User engagement is nonexistent. Google actively hunts for PBNs and penalizes them.

The mesh is the opposite. Every site is a real product with real users. Links exist because they help users navigate between related tools. Content is dense, functional, and valuable. Engagement metrics are strong because people actually use the tools.

The difference shows up in the data. PBNs see zero engagement, high bounce rates, and no returning visitors. The mesh sees 2.8k active users, 9.4k tracked events, and traffic from five continents. Users don't land on a mesh site and immediately bounce. They use the tool, explore related tools, and return later.

This is why the mesh survives algorithm updates. Google's core updates target manipulation. They don't target real products with real users.

How We Structure Cross-Links

Cross-linking isn't random. We follow a structure:

Category clusters: All finance tools link to each other. All calculator tools link to each other. All AI tools link to each other. This creates topical authority within each cluster.
Related tools: A car valuation tool links to a car calculator, a car insurance estimator, and a vehicle history checker. These are natural next steps for the same user.
Hub pages: The marketplace acts as a central directory. Every tool links back to the marketplace. The marketplace links to every tool. This creates a strong hub without forcing all authority through a single domain.
User intent flows: We map user journeys and link accordingly. Someone using a gold price tracker might next want a silver price tracker, a precious metals calculator, or a market trend analyzer. We link those tools together.

The result is a network where authority flows naturally. No forced linking. No manipulation. Just a well-structured ecosystem where every link makes sense.

Engagement Data Confirms the Mesh Is Working

Traffic numbers tell part of the story. Engagement metrics tell the rest.

Across the mesh, we track 2.8k active users and 9.4k events. Traffic comes from the US, Hong Kong, Singapore, Germany, and the UK. Users aren't just landing on a page and leaving. They're clicking through tools, exploring calculators, and interacting with the products.

The Bing Webmaster Tools dashboard shows the compounding effect. Over 1 million AI citations to VisionVix.com in seven months. That's not a paid promotion number. That's organic discovery driven by the mesh.

Early Impressions on New Domains

Some domains start generating impressions within two weeks of launch. No backlinks. No outreach. Just strong on-page signals and keyword-rich domain names.

brandmonitoring.dev: 2.82k impressions
aiagentready.dev: 1.45k impressions
aieuact.dev: 435 impressions

These numbers aren't massive, but they prove the model works. A new domain with zero external authority can gain visibility purely through topical relevance, user engagement, and semantic clarity.

You can't manually configure 1,500 domains. You can't hand‑write privacy policies for 800 products. You can't individually optimize PageSpeed scores for hundreds of tools. The system works because the infrastructure is templated, automated, and tested. Our GitHub repository shows exactly how this happens:

When an AI agent like ChatGPT or Perplexity needs a calculator, it looks for structured data, semantic HTML, and clear function definitions. The mesh provides all three. That's why VisionVix.com receives over 1 million citations. The infrastructure is built for agentic discovery, not just human search.

PageSpeed and Core Web Vitals Across the Mesh

Google's ranking algorithm prioritizes user experience. That means fast load times, mobile responsiveness, and Core Web Vitals scores. Every domain in the mesh is built to pass these checks.

PageSpeed Insights consistently shows 100/100 performance, SEO, and best practices. This isn't an accident. The base template uses static generation where possible, lazy-loads images, inlines critical CSS, and preloads fonts. Every performance optimization is baked into the deployment pipeline.

The AI agent readiness score of 89/100 across the mesh confirms that these sites aren't just ranking for traditional search. They're optimized for the next wave of search: AI agents parsing structured data, answering user queries, and citing authoritative sources.

When you launch 1,500 sites, you can't manually optimize each one. The infrastructure has to make performance automatic. That's why we use Next.js with static export, host on Cloudflare's edge network, and compress all assets at build time.

The result: every site loads in under two seconds on mobile, scores 100/100 on PageSpeed, and passes Core Web Vitals. These signals compound with keyword relevance and user engagement to create fast-ranking domains.

Security Audits and Compliance Scores

Enterprise customers and AI agents both care about security. A site with weak security headers, missing SSL, or unpatched vulnerabilities gets flagged. A site with A-grade security, strict CSP, and HSTS enforcement gets trusted.

SecurityAudit.dev scores 100/100. VisionVix.world receives an A-grade security header report. Every domain in the mesh enforces HTTPS, sets secure cookies, blocks clickjacking, and prevents XSS attacks.

We didn't build this level of security because it's a nice-to-have. We built it because it's a ranking signal. Google's algorithm doesn't explicitly say "sites with CSP headers rank higher," but the correlation is clear. Sites that follow web standards, protect user data, and enforce security best practices consistently rank better than sites that don't.

For a detailed walkthrough of website security scanning for small business owners, we covered the full methodology in an earlier post. The same security infrastructure powers every domain in the mesh.

Scaling the Mesh: Deployment Pipeline and Automation

Building 1,500 domains manually would take years. We built them in seven months because the entire deployment pipeline is automated.

Every new domain starts from a base template. The template includes the full Next.js project structure, security headers, legal pages, structured data, and semantic HTML. When we decide to launch a new domain, we don't start from scratch. We copy the template, swap the branding, deploy the unique tool logic, and push to production.

The deployment script handles:

Domain registration through Cloudflare or Namecheap
DNS configuration pointing to Cloudflare's edge network
SSL certificate provisioning through Let's Encrypt
GitHub repository creation and initial commit
Vercel or Netlify deployment with environment variables
Sitemap generation and submission to Google Search Console

On macOS or Linux, the process looks like this:

./deploy-new-domain.sh carvaluation.app

On Windows, PowerShell handles the same flow:

$DomainName = "carvaluation.app"
.\deploy-new-domain.ps1 -Domain $DomainName

The script runs for about three minutes. At the end, the domain is live, SSL is active, and the site is submitted to search engines.

Template Once, Reuse Forever

The base template is the secret to scaling. Instead of rebuilding common infrastructure for every domain, we built it once and reused it 1,500 times.

The template includes:

Legal pages: Privacy policy, terms of service, cookie policy, GDPR compliance notices
Security headers: CSP, HSTS, X-Frame-Options, X-Content-Type-Options
Structured data: JSON-LD schemas for organization, breadcrumbs, and products
Semantic HTML: Proper heading hierarchy, ARIA labels, landmark roles
SEO metadata: Open Graph tags, Twitter cards, canonical URLs
Analytics integration: Google Analytics, Bing Webmaster Tools, Cloudflare Analytics
Cookie consent: GDPR-compliant consent banners with opt-in/opt-out flows

When we update the template, we can cascade the update across the mesh. One commit updates 1,500 domains. This is how we stay compliant as privacy laws evolve and web standards change.

" width="800" height="600">

WebsiteAudit.dev is the compliance gate for the entire mesh. It proves every domain is a real product. The audits cover analytics, consent, SEO, social tags, schema, security, performance, and accessibility.

AI Agents in the Deployment Loop

We don't manually configure every domain. AI agents handle repetitive tasks like generating legal pages, writing metadata, and creating structured data.

When we launch goldtrade.app, an AI agent reads the domain name, infers the tool category, and generates:

A privacy policy tailored to precious metals price tracking
SEO metadata describing the tool's function
Structured data schemas for financial calculators
A terms of service page covering data sources and disclaimers

This isn't lazy automation. It's precise automation. The agent follows templates, enforces legal compliance, and ensures consistency across the mesh. A human reviews the output, but the agent handles 90% of the work.

This is how we built 1,500 domains in seven months. We didn't hire a massive team. We built infrastructure that makes launching a new domain a three-minute task instead of a three-week project.

What This Means for Founders

Most founders think about SEO as a backlink arms race. They assume you need thousands of high-authority links to rank. The mesh proves otherwise.

You can rank with zero backlinks if your domain name is keyword-rich, your on-page signals are strong, and your user engagement is real. You can scale to 1,500 domains if your infrastructure is templated and your deployment is automated. You can generate over 1 million AI citations if every site is built for agentic discovery.

The mesh model works because it's built on real products, not manipulation. Every domain solves a real problem. Every tool serves real users. Every link helps someone find the next step in their journey.

If you're building in public, shipping fast, or scaling a portfolio of micro-products, this model is worth studying. The infrastructure compounds. The authority flows sideways. The rankings accelerate with every new node.

📦 Publishing Kit — Dev.to

Title Options (5)

Selected: We Built 1,500 Keyword-Rich Domains Before Chasing Backlinks — Here's What Happened

Alternates:

How 1,500 Keyword Domains Generated 1M+ AI Citations in 7 Months Without Traditional SEO
Building Authority Sideways: The Domain Mesh Strategy That Beat Traditional Link Building
From Zero to 1M AI Citations: Why We Built Products on 1,500 Domains First, SEO Second
Keyword-Rich Domains as Infrastructure: A Network-First Approach to Organic Authority

Slug

1500-keyword-domains-generated-1m-ai-citations-without-backlinks

Tags

seo, webdev, startup, domains

llms.txt vs mcp.json: Why Your Site Needs Both (And What They Actually Do)

Julian Neagu — Mon, 20 Jul 2026 11:11:51 +0000

TL;DR: llms.txt helps AI assistants read your site by mapping your best content. mcp.json lets AI agents call your site by exposing structured tools. They solve different problems, and your site needs both if you want to show up in agent-driven answers instead of getting skipped.

I updated our website generator last week to auto-deploy llms.txt files in every new site. Two days later, a founder asked me: "So what exactly does this do that my sitemap doesn't?" Fair question. I realized I'd been lumping llms.txt and mcp.json together as generic "AI stuff" when they're actually solving completely different problems.

Here's what I should have said from the start: we're building websites for two audiences now. The first audience is human visitors who click through your navigation and read your pages. The second audience is AI agents that crawl your site, extract information, and either cite your content in chat responses or call your site's APIs to take actions on behalf of users.

Those agents need different signals than humans do. Your homepage hero section might look great to a visitor, but an AI assistant reading your HTML sees twenty CSS classes, three analytics scripts, and a pile of nested divs before it finds your actual value proposition. That's the gap llms.txt and mcp.json are filling.

I check this on PageSpeed Insights for every site we ship now. A perfect 3/3 Agentic Browsing audit score requires:

They're not competing standards. They're complementary files that tell AI agents two fundamentally different things about your site. One is a map. The other is a menu of actions.

llms.txt: A Content Map for AI Reading

Think of llms.txt as a table of contents written specifically for AI assistants. It's a Markdown file that lists your most important pages with brief descriptions, so when ChatGPT or Perplexity encounters your site, it knows exactly where the valuable content lives.

Instead of scraping your homepage HTML and guessing which links matter, the AI reads your llms.txt and finds your pricing page, your feature comparison, your case studies, your best blog posts. No guessing. No hallucinated URLs from cached training data.

llms.txt gives AI assistants a clean, structured summary of your most important pages, so they cite canonical content instead of scraping cluttered HTML.

I built this into version 183.71 of our website generator. Every site that runs the sitemap agent now gets an auto-generated llms.txt that includes:

The homepage with your site's core value proposition
Key product or feature pages with descriptions
Your most important blog posts or content
Contact and about pages

What it doesn't include: privacy policies, terms of service, legal disclaimers. Those aren't "important content" per the emerging standard. AI assistants don't need to cite your cookie policy when someone asks what your product does.

The practical impact: when someone asks ChatGPT "What does [your product] do?" or "How does [your company] approach X?", the AI can pull from your llms.txt map instead of guessing which page to read. It's the difference between an assistant citing your actual pricing page versus hallucinating a price from a cached blog post written two years ago.

Writing llms.txt by Hand

The format is dead simple. Here's a real example from one of our sites:

``markdown

My Product Name

https://example. com/
The homepage. We help developers ship faster by automating repetitive setup tasks.

https://example. com/pricing
Pricing page. Three tiers: Free, Pro ($29/mo), Team ($99/mo).

https://example. com/blog/how-we-built-this
Case study: how we reduced deployment time from 4 hours to 12 minutes.

https://example. com/docs
Documentation hub. Getting started guides and API reference.

https://example. com/contact
Contact page. Email support@example. com or use the form.
``

That's it. One URL per line, followed by a one-sentence description. No XML. No schema markup. Just plain Markdown that both humans and AI can read. Save it as llms.txt at your site root.

Retroactive llms.txt for Existing Sites

If you're adding llms.txt to a site that's already live, you need to decide which pages deserve to be in the map. Here's the heuristic I use:

Start with your sitemap.xml and filter for pages with real content (not legal pages, not redirects)
Rank pages by how often you want them cited when someone asks about your product
Keep the top 10-15 pages, write a one-line description for each
Deploy the file and add it to your robots.txt

The robots.txt part matters. Add this line so AI crawlers know the file exists:

``txt
User-agent: *
Allow: /llms.txt

Sitemap: https://yoursite.com/sitemap.xml
``

" width="800" height="450">

Most AI agents don't actively look for llms.txt yet. But the ones that do (including some custom research agents and newer Perplexity iterations) respect it immediately. You're getting in early, which means your content gets cited while competitors' sites get skipped.

mcp.json: An Action API for AI Interaction

Now here's where mcp.json (also called WebMCP) does something entirely different. It's not about reading your content. It's about calling your site. It's a machine-readable manifest that defines specific tools an AI agent can invoke to get structured data back.

Think of it as an API specification, but designed for AI agents instead of developers. A typical mcp.json file might define tools like:

get_pricing → returns your current pricing tiers with exact dollar amounts
get_contact → returns your support email and contact methods
get_faq → returns your frequently asked questions with answers
check_availability → returns whether a product or service slot is available

When an AI agent reads your mcp.json, it sees these tools and can call them during a conversation. A user asks "How much does the pro plan cost?" and instead of reading your pricing page HTML, the agent calls your get_pricing tool and gets back clean JSON with the exact answer. No scraping. No guessing. No stale cache.

mcp.json defines callable tools that give AI agents exact, structured answers: your real prices, your real email, your real product availability.

Here's the key architectural difference: llms.txt is passive. It's a map. The AI reads it, then reads your pages. mcp.json is active. It's a set of endpoints. The AI calls it and gets immediate structured responses.

As we explain in our structured data tips for AI-ready websites, these agent-facing files complement traditional SEO signals like sitemaps and JSON-LD rather than replacing them.

Writing a Basic mcp.json Manifest

The WebMCP spec is still evolving, but here's a minimal working example that defines two tools:

json { "name": "My Product API", "version": "1.0.0", "description": "Tools for interacting with My Product", "tools": [ { "name": "get_pricing", "description": "Returns current pricing tiers and features", "inputSchema": { "type": "object", "properties": {} } }, { "name": "get_contact", "description": "Returns contact email and support options", "inputSchema": { "type": "object", "properties": {} } } ], "endpoints": { "get_pricing": "https://example.com/api/pricing", "get_contact": "https://example.com/api/contact" } }

Each tool has a name, a description, and an endpoint URL. When an AI agent calls get_pricing, it hits your /api/pricing endpoint and expects JSON back. You control what data gets returned. The agent gets structured facts instead of parsing HTML.

Deploying mcp.json to Three Locations

Here's the annoying part: the WebMCP standard is new, and different AI agents expect the manifest in different places. To maximize compatibility, deploy your mcp.json file to three locations:

/.well-known/webmcp.json (the official spec location)
/mcp.json (root fallback for agents that skip .well-known)
/agent.json (some tools look for this name)

Yes, it's redundant. Yes, it's a pain. But until the standard solidifies, this triple deployment gives you the best chance of being discovered by agent frameworks that are experimenting with different conventions.

On most hosting platforms, you can set up a single source file and three symlinks or redirects. Here's the setup for a static site:

``bash

Create the master file

cat > .well-known/webmcp.json << EOF
{
"name": "My Product API",
"version": "1.0.0",
...
}
EOF

Create fallback copies

cp .well-known/webmcp.json mcp.json
cp .well-known/webmcp.json agent.json
``

If you're on Windows and using PowerShell, the workflow is similar but with different commands:

``powershell

Create the directory if needed

New-Item -ItemType Directory -Force -Path .well-known

Create the master file (use your actual JSON content here)

@"
{
"name": "My Product API",
"version": "1.0.0",
...
}
"@ | Out-File -FilePath .well-known\webmcp.json -Encoding utf8

Create fallback copies

Copy-Item .well-known\webmcp.json mcp.json
Copy-Item .well-known\webmcp.json agent.json
``

Deploy all three files. When the standard settles, you can drop two of them. Until then, redundancy wins.

What Makes a Good WebMCP Tool

Not every API endpoint should be a WebMCP tool. The best tools return facts that change or actions that matter. Here's what works:

Good tools:

get_pricing → returns current prices (changes when you update pricing)
check_availability → returns real-time availability (changes as inventory moves)
get_latest_post → returns your most recent blog post (changes when you publish)
search_docs → searches your documentation (user-specific results)

Bad tools:

get_about_page → just returns static HTML (llms.txt already maps this)
get_logo → returns an image URL (not structured data)
subscribe_newsletter → writes data (WebMCP is read-only for now)

The pattern: if an AI agent calling your tool gets back structured data that's either dynamic or user-specific, it's a good tool. If it's just returning static content that could be read from a page, llms.txt is the better fit.

I built AI Agent Ready to solve exactly this gap. It generates both files automatically, deploys them to the right locations, and validates that your site passes the Agentic Browsing checks. If you're shipping multiple sites or managing a portfolio, automation is the only way to keep up.

Why Deploy Both (Even Though Adoption Is Still Low)

Neither llms.txt nor mcp.json is widely adopted yet. Most AI crawlers don't fully honor either one. So why am I building both into every site we ship?

Because the cost is near zero and the positioning advantage is real.

Here's my current ranking of what actually moves the needle for AI visibility today:

Sitemap + robots.txt that allow AI bots (most impactful right now, every major AI crawler respects this, non-negotiable)
JSON-LD structured data (actively used by Google today, immediate SEO benefit)
llms.txt (emerging standard with growing adoption for AI-search citation, low cost, future-facing)
mcp.json / WebMCP (most forward-looking, least adopted, but the agent-actions layer that could matter enormously as browsing agents mature)

The first two are table stakes. If your site doesn't have a sitemap or blocks AI bots in robots.txt, you're invisible. The second two are bets on where agent behavior is heading in the next 6-12 months.

Google added a fifth category to PageSpeed Insights in May 2026 called Agentic Browsing. It doesn't give a score out of 100; instead it shows a ratio like 3/3.

llms.txt present and properly formatted
mcp.json present with at least one callable tool
sitemap.xml with all key pages included
JSON-LD structured data on major pages
passing accessibility checks (agents rely on semantic HTML)

Most sites still score 0/3 or 1/3. Getting to 3/3 takes an afternoon of work. That gap is your positioning advantage right now.

The Four-Layer Agent-Ready Stack

Here's how I think about the full stack for agent-ready websites. You need all four layers:

1. Discoverable (Sitemap + Robots.txt)

AI crawlers need to know your site exists and which pages to index. This is the foundation. Without a sitemap, you're invisible.

2. Understandable (JSON-LD Structured Data)

Your pages need machine-readable context. JSON-LD tells AI agents what kind of page they're looking at (article, product, organization) and extracts key fields (author, price, date). This is how Google's Knowledge Graph works, and it's how AI search engines will cite you.

For implementation patterns, the JSON prompt guide for AI agents walks through specific schema markup for different page types.

3. Readable (llms.txt)

Once an AI agent lands on your site, llms.txt tells it which pages matter most. This is the content map layer. It answers: "If you can only read five pages, read these five."

4. Interactive (mcp.json)

Finally, mcp.json exposes callable tools that let AI agents get exact answers without scraping. This is the actions layer. It answers: "If you need real-time data or want to verify a fact, call these endpoints."

Each layer builds on the previous one. You can't skip straight to mcp.json if your sitemap blocks AI bots. But once you have layers 1 and 2 in place, adding layers 3 and 4 is straightforward.

What Happens If You Skip This

Here's the risk: your competitors deploy llms.txt and mcp.json, and their content starts getting cited in AI-driven search results while yours gets skipped. A user asks "What are the best tools for X?" and ChatGPT cites three competitors but not you, because their sites gave the AI a clean map and yours made it scrape HTML.

Or worse: a user asks "How much does [your product] cost?" and the AI hallucinates a price from a cached blog post written two years ago, because you didn't expose a get_pricing tool that returns your current pricing.

Agent-ready sites will win citations. Sites that ignore this layer will become invisible to the next generation of search behavior. That's not speculation. It's already happening in vertical AI search engines that prioritize structured data over keyword optimization.

Start With llms.txt, Add mcp.json Later

If you're resource-constrained, start with llms.txt. It's a single Markdown file. You can write it by hand in ten minutes. Deploy it to your site root, add it to robots.txt, and you're done.

Then, when you have time to build API endpoints, add mcp.json. Start with one or two simple tools like get_pricing or get_contact. Deploy the manifest to all three locations. Test it by checking the Agentic Browsing score on PageSpeed Insights.

That's the pragmatic path: llms.txt this week, mcp.json next month. Both files are small. Both are future-facing. Both give you positioning advantage while most sites are still ignoring this layer entirely.

But even if you're doing this manually for a single site, the work is worth it. We're not optimizing for today's AI agents. We're optimizing for the ones that launch six months from now, when agent-driven search is the default and sites without llms.txt and mcp.json are invisible.

📦 Publishing Kit — Dev.to

Title Options (5)

Selected: llms.txt vs mcp.json: Why Your Site Needs Both (And What They Actually Do)

Alternates:

AI Agents Need Two Things From Your Site: A Map and a Menu
llms.txt and mcp.json Aren't Competing—They're Solving Different Problems
How to Make Your Site Readable and Actionable for AI Assistants
The Real Difference Between llms.txt and mcp.json (With Implementation Examples)

Slug

llms-txt-vs-mcp-json-why-your-site-needs-both

Tags

webdev, ai, tutorial, llms

The Domain Extension Wars: Why .ai and .app Are Crushing .com

Julian Neagu — Mon, 29 Jun 2026 11:57:37 +0000

TL;DR: Tech companies are ditching .com for specialized TLDs like .ai, .app, and .dev. These extensions offer better SEO performance, clearer brand positioning, and signal innovation to investors. Anguilla earned $32 million from .ai domains alone in 2023 — that's 20% of their government revenue.

The domain extension wars are heating up, and .com is losing ground fast. When I see a startup pitch deck with a .ai domain, my immediate reaction is different than seeing the same company with a .com extension. One signals cutting-edge innovation; the other feels like they settled for what was available.

This shift isn't just cosmetic. New TLDs like .app, .dev, and .ai are becoming strategic business assets that communicate industry focus, improve search rankings, and attract investor attention in ways that generic .com domains simply can't match.

The Economics Behind the TLD Revolution

The numbers tell the story. In 2023 alone, the tiny Caribbean island of Anguilla generated approximately $32 million in revenue from .ai domain registrations — that's roughly 20% of their entire government budget. When a two-letter country code becomes a major revenue stream, you know something fundamental has shifted in how businesses think about domains.

According to the IMF, .ai domain revenue accounted for roughly 20% of Anguilla's total government revenues in 2023.

This explosion in .ai adoption reflects a broader trend: companies are choosing domains that immediately communicate their business focus rather than settling for generic .com extensions. The semantic clarity of these new TLDs creates instant brand alignment that traditional domains can't provide.

Understanding the TLD Landscape

A Top-Level Domain (TLD) is the suffix that follows the final dot in a web address — the ".com" in "google.com" or the ".ai" in "openai.ai". While this might seem like a technical detail, TLDs have evolved into powerful branding and positioning tools.

Traditional TLDs served their purpose in the early internet era, but they lack the contextual precision that modern businesses need. When every company from law firms to restaurants uses .com, the extension becomes meaningless as a differentiator.

For a deeper dive into how TLDs function and their technical implications, I covered the fundamentals in my comprehensive TLD guide.

How TLDs Shape User Perception

Users make split-second judgments based on domain extensions. A .dev domain immediately signals that you're looking at a developer-focused platform. A .ai extension suggests artificial intelligence capabilities, making clear brand and metadata signals even more important. These associations happen before users even read your content.

Consider these psychological triggers:

.ai domains create expectations of machine learning, automation, and cutting-edge technology
.app domains suggest mobile-first experiences and software applications
.dev domains signal developer tools, programming resources, and technical expertise
.com domains feel generic and established but lack industry specificity

Comparison Table

The .com vs Modern TLD Breakdown

The fundamental difference between .com and semantic TLDs comes down to signal clarity. A .com domain forces users to guess what your business does. A .ai domain tells them immediately.

Search Engine Optimization Advantages

Modern TLDs offer distinct SEO benefits that .com domains can't match. Search engines increasingly prioritize semantic matching between domain extensions and user intent. When someone searches for "AI writing tools," a website with a .ai domain has a natural advantage over the same content on a .com domain.

Key SEO factors favoring newer TLDs:

Keyword alignment: .ai domains naturally rank better for AI-related searches
Click-through rates: Users click on domains that match their search intent
Technical standards: Many new TLDs enforce HTTPS by default, boosting Google rankings
Core Web Vitals: Newer domain infrastructure often performs better on speed metrics

Investment and Market Appeal

Venture capitalists and angel investors are paying attention to domain choices. A well-chosen TLD signals that founders understand digital positioning and are thinking strategically about their brand presence.

Investors increasingly view .ai, .app, and .dev domains as indicators of future success and alignment with digital transformation trends.

This isn't superficial preference — it reflects deeper business acumen. Companies that choose strategic TLDs often demonstrate the same forward-thinking approach in product development and market positioning.

Why Businesses Are Abandoning .com

The migration away from .com stems from three core limitations: positioning weakness, SEO disadvantages, and availability constraints.

Positioning and Brand Clarity

When a business uses .app, .dev, or .ai, they immediately communicate their industry focus. This semantic precision helps companies:

Stand out in crowded markets
Build instant credibility with target audiences
Align their digital identity with their core services
Create memorable brand associations

Compare the positioning power: "mycompany.com" versus "mycompany.ai" for an artificial intelligence startup. The .ai extension does heavy lifting that .com simply cannot provide.

SEO Performance Gap

Search engines have evolved to understand semantic relationships between domain extensions and content. A .ai website discussing machine learning has inherent relevance advantages over identical content on a .com domain.

This semantic matching translates into:

Higher click-through rates from search results
Better keyword rankings for industry-specific terms
Improved user engagement metrics
Enhanced local and mobile search performance

The Availability Crisis

The .com namespace is effectively saturated. Finding a short, memorable .com domain that matches your business name requires either significant compromise or substantial financial investment, making strategic online positioning more important for new brands. Premium .com domains regularly sell for six or seven-figure amounts.

Meanwhile, newer TLDs offer:

Greater name availability for exact business matches
Shorter, more memorable domain options
Keyword-rich domains that support SEO strategies
Lower acquisition costs for premium names

Strategic TLD Selection by Business Type

Different business models benefit from different TLD strategies. The key is matching your domain extension to your primary value proposition and target audience expectations.

For a detailed comparison of .app versus .dev domains specifically, check out my analysis of these two popular extensions.

AI and Machine Learning Companies

Choose .ai for businesses focused on artificial intelligence, machine learning, automation, or data science. This extension immediately communicates technical sophistication and innovation.

Examples of effective .ai positioning:

SaaS platforms with AI-powered features
Machine learning consulting firms
Automated workflow tools
Predictive analytics platforms

Software Applications and Mobile Apps

Choose .app for companies building software applications, mobile apps, or digital tools. This extension suggests user-friendly interfaces and practical software solutions.

Ideal .app use cases:

Mobile app development companies
SaaS productivity tools
Consumer software platforms
Digital product marketplaces

Developer Tools and Technical Services

Choose .dev for businesses serving developers, offering programming tools, or providing technical infrastructure. This extension builds immediate credibility with technical audiences.

Perfect .dev applications:

API services and documentation
Developer tool companies
Code hosting and collaboration platforms
Technical consulting services

The Future of Domain Strategy

The trend toward semantic TLDs represents a fundamental shift in how businesses approach digital identity. Domains are evolving from simple web addresses into strategic brand assets that communicate value propositions before users even visit your site.

Companies that embrace this evolution gain competitive advantages in search rankings, user perception, and investor appeal. Those clinging to generic .com domains risk being perceived as outdated or unfocused.

The data supports this shift: when a small island nation can generate $32 million annually from a two-letter domain extension, it's clear that businesses see real value in semantic domain positioning.

As AI continues to reshape every industry, expect .ai domains to become even more valuable. Similarly, as software development becomes increasingly central to business operations, .dev and .app domains will likely appreciate in both practical utility and market value.

The question isn't whether semantic TLDs will replace .com — it's how quickly businesses will recognize that their domain extension is now a core component of their competitive positioning strategy.

📦 Publishing Kit — Dev.to

Title Options (5)

Selected: The Domain Extension Wars: Why .ai and .app Are Crushing .com

Alternates:

Why Smart Startups Are Ditching .com for .ai and .dev Domains
The $32M TLD Revolution: How New Domain Extensions Are Changing Tech
From .com to .ai: The Strategic Domain Shift Reshaping Startups
The Death of .com? Why Modern TLDs Are the New Business Standard

Slug

domain-extension-wars-ai-app-crushing-com

Tags

webdev, startup, branding, domains

AI Agent Architecture vs Workflow vs Modularity Know The Difference

Julian Neagu — Tue, 23 Jun 2026 11:45:09 +0000

TL;DR: Architecture defines system structure and roles, workflows control execution paths, and modularity enables component reuse. Focus on architecture for consistency issues, workflows for performance problems, and modularity for scaling pain. Start simple, then evolve.

Building AI agents that actually work in production requires understanding three distinct but interconnected concepts: architecture, workflow, and modularity. Most teams confuse these layers, leading to systems that demo well but crumble under real users.

Think of architecture as the wiring in your house, workflows as the electricity flowing through it, and modularity as the replaceable appliances you can swap without rewiring everything. Get these distinctions right, and you'll design cleaner systems that scale predictably.

The stakes are real. Teams that master these three layers ship faster, debug easier, and collaborate without stepping on each other's code. Those who don't end up rewriting everything when requirements change.

What Is AI Agent Architecture?

Architecture is your system's blueprint — the static map that defines who talks to whom, how data flows, and where decisions get made. It's not about the specific steps your agents take; it's about the foundation that makes those steps possible.

The core roles form a clear division of labor. The Planner breaks high-level goals into actionable tasks. The Router decides which agent or tool handles each task. The Executor does the actual work — calling APIs, processing data, or generating content. Memory preserves context and state across interactions. Guardrails enforce safety policies, access controls, and audit requirements.

// Example: Basic agent architecture
class AgentSystem {
  constructor() {
    this.planner = new TaskPlanner();
    this.router = new AgentRouter();
    this.memory = new ContextMemory();
    this.guardrails = new SafetyLayer();
  }

  async process(request) {
    const plan = await this.planner.createPlan(request);
    const route = this.router.selectAgent(plan.nextTask);
    return this.guardrails.validate(
      await route.execute(plan, this.memory.getContext())
    );
  }
}

Architecture also defines your data paths and decision points. How does information flow from initial request to final response? Where do you transform, validate, or enrich data? What happens when something fails?

Clean architecture prevents the most common AI system failures: inconsistent behavior, unclear hand-offs, and security gaps.

Good architecture centralizes policy enforcement. Instead of scattering safety checks across every agent, you build guardrails once and route everything through them. Role-based access control, PII handling, and audit logging become system-wide properties, supported by structured compliance audit processes, not per-agent responsibilities.

What Is an AI Workflow?

While architecture is your blueprint, workflow is what happens when you flip the switch. It's the step-by-step execution path that turns user intentions into actual results.

A typical workflow might look like: User query → Router analyzes intent → Research Agent gathers sources → Summary Agent condenses findings → Quality Reviewer validates output → System returns formatted response.

# Example: Debugging a slow workflow
curl -X POST http://localhost:3000/research \
  -d '{"query": "latest AI regulations", "depth": "comprehensive"}' \
  --trace-time

Workflows come in different patterns. Linear flows move predictably from step A to B to C — easy to test and debug. Branching flows add conditional logic based on content type, user permissions, or quality thresholds. Parallel flows speed things up by running independent tasks simultaneously. Event-driven flows react to triggers, webhooks, or queue messages.

The workflow layer is where performance becomes visible. You'll see bottlenecks, retry loops, and escalation paths play out in real time. A research workflow that takes 30 seconds usually has one slow step, not thirty slow ones.

Common workflow problems include: agents waiting unnecessarily for sequential tasks that could run in parallel, retry logic that creates infinite loops when external APIs fail, and handoff points where context gets lost between agents.

What Goes Wrong in Workflows

Here's a real error you might encounter:

Error: Research agent timeout after 45s
Stack: query="climate policy 2024" → router → research_agent → [TIMEOUT]
Cause: External API rate limit (429) → retry loop → eventual timeout

The fix usually involves adding proper retry backoff and parallel source gathering:

// Instead of sequential API calls
const sources = [];
for (const url of urls) {
  sources.push(await fetchWithRetry(url)); // Slow!
}

// Use Promise.allSettled for parallel fetching
const results = await Promise.allSettled(
  urls.map(url => fetchWithRetry(url, { backoff: 'exponential' }))
);

Start with simple linear workflows before adding branches or complexity. You can always optimize later once you understand where the bottlenecks actually occur.

What Is Modularity in AI Systems?

Modularity treats your AI stack like LEGO blocks. Each component has one clear job and connects to others through well-defined interfaces. When you need to upgrade your summarizer or swap language models, you replace one block without touching the rest.

The key principle: one responsibility per module. A retriever module only retrieves. A reranker only reranks. A formatter only formats. When each module has a single, clear purpose, the whole system becomes predictable.

Modular systems let teams ship upgrades in hours instead of months.

Real modularity benefits become obvious during maintenance. Imagine you built a research system six months ago using GPT-3.5. Today you want to upgrade to GPT-4. In a modular system, you swap the language model module and you're done. In a monolithic system, you're hunting through code to find every place that calls the old API.

# Modular approach: swap implementations easily
class SummarizerInterface:
    def summarize(self, text: str, max_length: int) -> str:
        pass

class GPT35Summarizer(SummarizerInterface):
    def summarize(self, text: str, max_length: int) -> str:
        # GPT-3.5 implementation
        pass

class GPT4Summarizer(SummarizerInterface):
    def summarize(self, text: str, max_length: int) -> str:
        # GPT-4 implementation
        pass

# Swap models without changing workflow code
summarizer = GPT4Summarizer()  # Was: GPT35Summarizer()
result = workflow.run(summarizer=summarizer)

Teams collaborate more effectively with clear module boundaries. The retrieval team owns the search module. The safety team owns the guardrails module. The UI team owns the formatting module. Each team can ship improvements without coordinating every change.

Consider building modules that solve these common needs across multiple workflows. I cover the security implications in detail in my AI agent security analysis, but the core insight is that reusable security modules prevent teams from rebuilding authentication and audit logging for every new agent.

When to Focus on Each Layer

System problems usually stem from one specific layer. Identifying the root cause helps you fix the right thing.

Focus on architecture when:

Agents behave inconsistently across similar requests
You're duplicating safety checks or access controls
Teams are stepping on each other's code
Adding new capabilities requires touching multiple files

Improve workflow when:

Requests take too long or stall unpredictably
Users abandon tasks before completion
You can't easily trace what went wrong
Similar tasks have wildly different performance

Invest in modularity when:

Upgrades require rewriting large portions of code
Testing one component breaks others
Teams can't work independently
You're rebuilding similar functionality across projects

How Architecture, Workflow, and Modularity Interconnect

These three layers form a virtuous cycle. Architecture provides the foundation and interfaces. Workflows run on that foundation and expose friction points. Modularity enables you to swap components and extend capabilities without rebuilding everything.

The cycle looks like this: solid architecture creates clear contracts → workflows reveal bottlenecks and edge cases → modular components let you optimize specific pieces → improved components make workflows faster → better workflows stress-test the architecture → architectural improvements unlock new modular possibilities.

Teams that master this cycle ship multi-agent systems that actually scale. They add new capabilities by plugging in modules, not by rewriting core logic. They optimize performance by swapping faster components, reducing technical debt through modular refactoring instead of hunting through monolithic code.

Practical Example: Building a Research System

Let's trace how these concepts work together in a real system. You want to build a tool that researches topics, synthesizes sources, and returns clean, cited answers.

Architecture: Your system needs a Planner to break research goals into searchable questions, a Data Fetcher to pull credible sources, a Summarizer to synthesize evidence, and a Reviewer to enforce citation standards and accuracy. Memory stores context between research steps. Guardrails validate source credibility and prevent harmful outputs.

Workflow: User submits research query → Planner generates search terms → Data Fetcher runs parallel searches → Results get filtered for credibility → Summarizer creates draft with citations → Reviewer checks accuracy and completeness → System returns formatted research brief.

Modularity: Each agent is swappable. Upgrade from basic web search to academic database access by replacing the Data Fetcher module. Improve summary quality by swapping the Summarizer. Add fact-checking by plugging in a new Reviewer module.

When something goes wrong, the layer tells you where to look. Inconsistent citation formats? That's architecture — centralize formatting rules. Slow research? That's workflow — parallelize the search phase. Hard to add new source types? That's modularity — abstract the fetcher interface.

Start simple with linear workflows and basic modularity. Add branching logic and sophisticated modules as you learn where optimization matters. The goal isn't perfect architecture on day one — it's a system that evolves intelligently as requirements change.

The teams shipping the most impressive AI systems aren't the ones with the most complex architectures. They're the ones who understand when to focus on structure, when to optimize flow, and when to increase modularity. Master those decisions, and your AI systems will scale from clever demos to dependable products.

📦 Publishing Kit — Dev.to

Title Options (5)

Selected: AI Agent Architecture vs Workflow vs Modularity: The Complete Guide to Building Production Systems

Alternates:

Understanding AI Agent Architecture, Workflows, and Modularity for Production-Ready Systems
The Three Pillars of AI Agent Development: Architecture, Workflow, and Modularity Explained
Building Scalable AI Agents: Mastering Architecture, Workflow, and Modularity Design
From Demo to Production: Understanding AI Agent Architecture, Workflows, and Modular Design

Slug

ai-agent-architecture-vs-workflow-vs-modularity-guide

Tags

ai, architecture, tutorial, agents

I Deployed 44 Live Websites in 10 Minutes With One PowerShell Script (Here's How)

Julian Neagu — Sat, 20 Jun 2026 14:27:25 +0000

TL;DR: I deployed 44 live websites in a single 10-minute session using a single PowerShell script. Zero manual clicks. Zero dashboard logins. This is how you operate 1,000+ domains as a solo founder without losing your mind.

Most solo founders hit a wall around product three or four. They talk about building. They don't talk about what happens when you're managing infrastructure for hundreds of live products at once.

I own 1,000+ premium .app domains, all listed through my domain portfolio & sales overview. Every single one needs to be a live landing page. Every one needs its own deployment, DNS configuration, security headers, sitemap, robots.txt file, and SSL certificate. Do that manually and you're staring at weeks of clicking through GoDaddy dashboards, copy-pasting DNS records one domain at a time, waiting for propagation, verifying each one individually.

That's not a workflow. That's a trap.

So I built a pipeline that turns all of it into one command.

The Three-Tool Stack That Makes This Possible

GitHub serves as the single source of truth. One monorepo. One subfolder per domain. Every domain folder contains exactly 11 files: the HTML landing page, hero image, Open Graph image, sitemap, robots.txt, manifest, favicons. Push to GitHub and everything downstream updates automatically.

Vercel CLI handles all deployments. Not the web dashboard. The command-line interface. Every subfolder becomes its own independent Vercel project. Each deployment takes about 6 seconds. The custom domain gets attached automatically in the same command.

GoDaddy API manages DNS records. Instead of logging into GoDaddy's dashboard and editing A records one by one, PowerShell calls the REST API directly and writes the record instantly. No forms. No waiting for pages to load.

One script coordinates all three. That's the entire deployment pipeline.

The Monorepo Structure

Here's what the repo looks like:

plaintext visionvix-domains/ ├── deploy.ps1 ← one script runs everything ├── domains.txt ← one domain per line │ ├── accessibilityaudit.app/ │ ├── index.html │ ├── og.jpg │ ├── hero.webp │ ├── sitemap.xml │ ├── robots.txt │ └── manifest.json │ ├── accountingassistant.app/ │ └── ... same 11 files │ └── ... one folder per domain

Simple. Consistent. Scalable. Every domain folder is identical in structure. The only thing that changes is the content inside index.html. Each landing page is customized for its specific AI tool or service, but the file structure never varies.

The entire repo is public on GitHub. That's intentional. It's a marketing asset as much as a technical one. Any developer who finds the repo can see the scale immediately: hundreds of domain folders, all consistently structured, all live. It shows what's possible as a solo founder operating with the right tools.

The Script That Does Everything

Here's the core deployment loop. Three API calls. One iteration. Zero manual steps.

On Windows, the deployment script looks like this:

`powershell$domains = Get-Content .\domains.txt

foreach ($domain in $domains) {
$slug = $domain -replace '.app$', ''

# Step 1: Deploy to Vercel (takes ~6 seconds per domain)
vercel deploy .\$domain --prod --yes --name $slug

# Step 2: Attach the custom domain and www subdomain
vercel domains add $domain $slug
vercel domains add www.$domain $slug

# Step 3: Write the A record to GoDaddy via REST API
Invoke-RestMethod -Method Put -Uri "https://api.godaddy.com/v1/domains/$domain/records/A/@"
-Headers $headers `
-Body '[{"data":"76.76.21.21","ttl":600}]'

# Step 4: Verify DNS was written correctly
$rec = Invoke-RestMethod -Method Get -Uri "https://api.godaddy.com/v1/domains/$domain/records/A/@"
-Headers $headers
Write-Host "✓ DNS configured: $($rec.data)"
}`

That's it. Every domain in the domains.txt file goes through all four steps automatically, one after another.

The $headers variable contains the GoDaddy API key and secret, stored in Windows environment variables. They're set once at the system level, available to every PowerShell session, never written to any file, never pushed to GitHub.

The Vercel A Record

The IP address 76.76.21.21 is Vercel's production A record. Every domain points there. Vercel's edge network handles routing to the correct project based on the domain name. The TTL is set to 600 seconds (10 minutes), which is low enough to make changes quickly if needed but high enough to avoid excessive DNS queries.

Why the CLI Instead of the Dashboard

The Vercel web dashboard works great for one or two projects. It completely breaks down at scale. Clicking through forms, waiting for pages to load, copying project URLs, manually adding domains. That's dozens of clicks per domain, which is exactly the kind of inefficiency solved by top workflow automation tools for small teams.

The CLI turns all of that into one command per domain. vercel deploy pushes the code, builds it, and returns a production URL. vercel domains add attaches the custom domain. Both commands are fully scriptable. That's what makes automation possible.

The Numbers After 44 Deployments

The first run was 5 domains. Then 38 more. Then the full 44 batch, all live, all verified, all returning HTTP 200.

44 domains deployed in a single session. 6 seconds average deploy time per domain. 0 manual clicks required. 100% success rate.

Metric	Result
Domains deployed in first session	44
Average deploy time per domain	~6 seconds
Manual clicks required	0
DNS propagation time	5-30 minutes
Total time from script to live	Under 10 minutes
HTTP 200 success rate	100%

When I ran the verification loop at the end, checking every domain for a 200 status code, every single one came back green. No failures. No retries. No manual fixes.

Security Headers Built Into Every Domain

Every domain automatically gets a full set of security headers via a vercel.json file that lives inside each domain folder. No extra deployment steps. No separate configuration, and can be validated with a website security audit tool. The headers deploy with the site.

Here's what every domain gets out of the box:

json { "headers": [ { "source": "/(.*)", "headers": [ { "key": "Content-Security-Policy", "value": "default-src 'self'; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline';" }, { "key": "Strict-Transport-Security", "value": "max-age=63072000; includeSubDomains; preload" }, { "key": "X-Frame-Options", "value": "DENY" }, { "key": "X-Content-Type-Options", "value": "nosniff" }, { "key": "Referrer-Policy", "value": "strict-origin-when-cross-origin" }, { "key": "Permissions-Policy", "value": "camera=(), microphone=(), geolocation=(), payment=()" } ] } ] }

Content-Security-Policy blocks malicious scripts and injection attacks. Strict-Transport-Security forces HTTPS for 2 years and covers all subdomains. X-Frame-Options: DENY prevents clickjacking. X-Content-Type-Options stops MIME type sniffing. Referrer-Policy controls how much referrer data leaks. Permissions-Policy disables camera, microphone, geolocation, and payment APIs.

Static assets like images and fonts are cached for a full year. The HTML is never cached, so users always get the latest version instantly.

Why These Architecture Decisions Matter

Why a Public GitHub Monorepo?

Security is handled without secrets ever touching the repo. GoDaddy API keys live in Windows environment variables only. Set once at the system level. Available to every PowerShell session. Never written to any file. Never pushed to GitHub.

The repo itself is public because it's a marketing asset. Developers who find it see the scale immediately. Hundreds of domain folders. All consistently structured. All live. It shows what's possible as a solo founder with the right automation in place.

Why Vercel Over the Alternatives?

The CLI is fully scriptable. That's the only reason that matters at this scale. Netlify and Cloudflare Pages both have solid products, but Vercel's CLI lets you deploy a subfolder, attach a domain, and get a production URL in a single command. That's what makes automation possible.

I can run the same script on 5 domains or 500 domains. The process doesn't change. The complexity doesn't increase.

Why GoDaddy API Instead of Manual DNS?

There is no "manual" at 600 domains. Logging into GoDaddy's dashboard and editing A records one by one would take days. The GoDaddy REST API accepts a PUT request with the new A record value and applies it instantly. PowerShell's Invoke-RestMethod makes the call in one line. The entire DNS step takes under a second per domain.

For readers who want to dive deeper into the strategy behind premium .app domains, I covered the positioning and branding implications in my earlier post on AI domain strategy.

What's Next: 1,000 Domains by Next Week

" width="800" height="482">

44 domains is the proof of concept. The pipeline works. Now it's just execution at scale.

The plan for the next 7 days:

Deploy the remaining 950+ domains using the same script
Index all pages with Google Search Console via sitemap submission
Track success rates for deployment, DNS propagation, and indexing
Monitor performance across the entire domain portfolio
Iterate on the landing page template based on traffic patterns

The script doesn't care if it's running on 50 domains or 500. The time per domain stays constant. The manual effort stays at zero.

The Bigger Point

This isn't really about domains. It's about treating infrastructure as code. One person can operate at enterprise scale when every manual process is replaced with an API call.

The traditional approach to scaling infrastructure is to hire people. You need someone to manage DNS. Someone to handle deployments. Someone to configure security. Someone to monitor uptime.

The alternative is to script everything once and run it a thousand times.

I'm not managing 1,000 domains. I'm managing one script that manages 1,000 domains. That's the difference between a job and a system.

📦 Publishing Kit — Dev.to

Title Options (5)

Selected: I Deployed 44 Live Websites in 10 Minutes With One PowerShell Script (Here's How)

Alternates:

How I Deploy 1,000+ Premium Domains as a Solo Founder Without Losing My Mind
Zero-Click Deployment Pipeline: Managing Hundreds of Live Domains With GitHub, Vercel & GoDaddy API
From Manual Hell to One-Command Deploy: Automating 1,000+ Domain Infrastructure
The Three-Tool Stack That Let Me Deploy 44 Websites in a Single 10-Minute Session

Slug

i-deployed-44-live-websites-in-10-minutes-with-one-powershell-script

Tags

devops, automation, productivity, domains

Building React Components: From JSX to Production-Ready Code

Julian Neagu — Fri, 19 Jun 2026 13:40:18 +0000

TL;DR: React components are reusable UI building blocks that combine structure, logic, and styling in isolated modules. Function components with Hooks are now the standard, replacing classes for cleaner, more maintainable code that scales across modern frameworks.

React components form the backbone of every React application. They break your user interface into small, logical pieces that handle their own structure, styling, and behavior independently. This modular approach transforms how developers build applications—making code more maintainable, bugs easier to isolate, and features faster to ship.

The numbers tell the story of React's dominance. According to JetBrains' 2024 Developer Ecosystem Report, 57% of JavaScript developers use React as their primary library. On NPM, React now pulls over 20 million weekly downloads, cementing its position as the most widely adopted UI framework globally.

React exceeds 20 million weekly downloads on NPM, confirming its position as the most adopted UI framework in the world.

What Makes a Component Work

A React component is fundamentally a JavaScript function that returns JSX—a blend of HTML-like syntax and JavaScript logic. Think of it as a custom HTML element that you define once and reuse anywhere.

function Welcome({ name }) {
  return <h1>Hello, {name}!</h1>;
}

This simple component takes a name prop and renders a personalized greeting. You can use it multiple times with different data:

<Welcome name="Sarah" />
<Welcome name="Mike" />
<Welcome name="Alex" />

Each instance behaves identically but displays unique content based on the props you pass in. This pattern—write once, use everywhere—drives React's efficiency.

JSX and TSX: The Language of Components

JSX Explained

JSX (JavaScript XML) lets you write HTML-like code directly in JavaScript. Behind the scenes, React transforms this syntax into function calls that create virtual DOM elements.

const element = <div className="header">Welcome to React</div>;

This compiles to:

const element = React.createElement('div', { className: 'header' }, 'Welcome to React');

JSX makes component code readable and intuitive, especially for developers familiar with HTML.

TSX and Type Safety

TSX extends JSX with TypeScript's type system. It catches errors before your code runs and provides better autocomplete in editors.

interface ButtonProps {
  label: string;
  onClick: () => void;
  disabled?: boolean;
}

const Button = ({ label, onClick, disabled = false }: ButtonProps) => (
  <button onClick={onClick} disabled={disabled}>
    {label}
  </button>
);

90.6% of frontend developers now use TypeScript, making TSX the standard format for professional React development.

Component Architecture Breakdown

Every React component has four essential parts that work together to create interactive UI elements.

Props: External Input

Props are data passed from parent to child components. They define what information a component receives and how it should behave.

<ProductCard 
  name="Digital Camera" 
  price={299} 
  inStock={true} 
/>

Props make components flexible. The same ProductCard component can display any product by changing the props you pass to it.

State: Internal Data

State holds data that can change during a component's lifetime. When state updates, React automatically re-renders the component with the new data.

const [count, setCount] = useState(0);

const increment = () => setCount(count + 1);

State enables interactivity—user actions trigger state changes, which update the UI immediately.

Hooks: Lifecycle Management

Hooks like useState, useEffect, and useRef give function components access to React's lifecycle features without writing class components.

useEffect(() => {
  document.title = `Count: ${count}`;
}, [count]);

Return: UI Output

The return statement defines what the component renders to the screen. It must return a single JSX element (or fragment).

Function vs Class Components: The Modern Standard

React originally used class components, but the introduction of Hooks in 2019 shifted the entire ecosystem toward function components.

Class Component (Legacy):

class Welcome extends React.Component {
  constructor(props) {
    super(props);
    this.state = { message: 'Hello' };
  }

  render() {
    return <h1>{this.state.message}, {this.props.name}</h1>;
  }
}

Function Component (Modern):

function Welcome({ name }) {
  const [message, setMessage] = useState('Hello');
  return <h1>{message}, {name}</h1>;
}

Function components require less boilerplate, avoid this binding issues, and integrate naturally with modern tooling. The React team now recommends function components for all new development, especially as teams move toward cleaner code maintainability practices.

The Power of Reusability

React's component model promotes code reuse through composition. Instead of copying similar code, you create one component and customize it with props.

<Button variant="primary" size="large">Save Changes</Button>
<Button variant="secondary" size="small">Cancel</Button>
<Button variant="danger" size="medium">Delete Account</Button>

The same Button component handles three different use cases by changing props. This approach keeps your codebase smaller and more consistent.

A Stack Overflow 2024 Developer Survey found that over 70% of React developers cite "reusability" as the primary reason for adopting React in their projects. When teams can build once and reuse everywhere, development velocity increases dramatically.

Over 70% of React developers cite "reusability" as the primary reason for adopting React in their projects.

Universal Building Blocks Across Frameworks

React components aren't limited to basic React apps. They power the entire modern JavaScript ecosystem:

Next.js uses components to define pages (page.tsx) and layouts (layout.tsx)
Gatsby generates static sites from React components
Remix renders components server-side for faster performance
Expo builds mobile apps using React Native components

In Next.js 14's App Router, every file in the app/ directory exports a React component. This structure automatically converts your file system into routes—no configuration required.

Under the Hood: JSX Compilation

React doesn't understand JSX natively. Build tools like Babel or SWC compile JSX into JavaScript before it runs in browsers.

Before React 17:

React.createElement('div', { className: 'container' }, 'Hello World');

React 17+ (New JSX Transform):

import { jsx as _jsx } from 'react/jsx-runtime';
_jsx('div', { className: 'container', children: 'Hello World' });

This modern transform eliminates the need to import React in every file and makes builds faster. Most developers never see this compiled output, but understanding it helps debug complex JSX issues.

Streamlined Development with Modern Tooling

For teams working on complex React projects, automation tools can significantly speed up development workflows. When migrating from basic React apps to production-grade Next.js structures, manual conversion becomes time-consuming and error-prone.

Modern development practices emphasize consistency—from file organization to code formatting. Teams often spend hours manually restructuring components, organizing folders, and ensuring proper TypeScript types when scaling projects.

I covered similar automation patterns in my cursor best practices guide, where consistent tooling and formatting become critical as codebases grow beyond simple prototypes.

Component Design Best Practices

Keep Components Focused

Each component should handle one responsibility. A UserProfile component manages user data display, while a separate EditUserForm handles user data updates.

Use Descriptive Names

Component names should immediately convey purpose:

ProductCard instead of Card
SignupForm instead of Form
PaymentButton instead of Button

Embrace Composition

Build complex interfaces by combining simple components rather than creating monolithic components with many features.

<Card>
  <CardHeader title="Product Details" />
  <CardBody>
    <ProductImage src={product.image} />
    <ProductInfo name={product.name} price={product.price} />
  </CardBody>
  <CardFooter>
    <AddToCartButton productId={product.id} />
  </CardFooter>
</Card>

Type Everything

Add TypeScript interfaces for all props to catch errors early and improve developer experience:

interface ProductCardProps {
  product: {
    id: string;
    name: string;
    price: number;
    image: string;
  };
  onAddToCart: (productId: string) => void;
}

Common Component Mistakes to Avoid

Mixing Concerns: Don't handle data fetching, business logic, and UI rendering in the same component. Separate these responsibilities into different layers.

Prop Drilling: Avoid passing props through multiple component layers. Use React Context or state management libraries for deeply nested data.

Ignoring Keys: Always provide unique key props when rendering lists to help React optimize re-renders.

Mutating Props: Never modify props directly. Treat them as read-only data.

Overusing useEffect: Many side effects can be handled in event handlers instead of effects, leading to simpler code.

The Future of Component Development

React continues evolving with features like Server Components and Concurrent Rendering. However, the core principles of component-based architecture remain constant: modularity, reusability, and clear separation of concerns.

Components represent more than just a React pattern, they're a fundamental shift in how we think about building user interfaces. By breaking complex UIs into simple, composable pieces, React components support more maintainable AI-powered development workflows and make frontend development more predictable.

The next time you build a React application, remember that every component is an opportunity to create something reusable. Start small, think modular, and let composition guide your architecture decisions.

📦 Publishing Kit — Dev.to

Title Options (5)

Selected: Building React Components: From JSX to Production-Ready Code

Alternates:

React Components Explained: The Complete Guide to Modern UI Development
Mastering React Components: JSX, Props, and Component Architecture
React Component Fundamentals: Building Scalable UI with Modern Patterns
The Developer's Guide to React Components: From Basics to Best Practices

Slug

building-react-components-jsx-production-ready-code

Tags

react, webdev, javascript, components