How to Block AI Web Scrapers on Your WordPress Site with robots.txt (2025 Update)

Mouaz — Thu, 10 Jul 2025 12:27:27 +0000

AI web scrapers are increasingly crawling WordPress sites to gather content for training large language models and powering AI search results. As a site owner, you can use your robots.txt file to help protect your original content from being scraped and used without your permission.

This guide explains:

What the robots.txt file does
Why blocking AI bots matters
How to add up-to-date blocking rules

A current table of the main AI bots and the exact syntax you need

What Is robots.txt?

The robots.txt file is a simple text file in your website’s root directory (e.g., https://yourdomain.com/robots.txt). It tells web crawlers which parts of your site they can or cannot access. Most reputable AI and search engine bots will respect these rules.

Why Block AI Bots?

Protect your original content from being used to train AI models without your consent
Maintain control over your website’s data
Limit how your content appears in AI-powered search and chatbots

How to Edit robots.txt in WordPress

Access your website’s root directory (via FTP, file manager, or a WordPress SEO plugin).
Open or create the robots.txt file.
Add the blocking rules from the table below.
Save and upload the file.

Table: AI Bots and robots.txt Syntax (2025)

AI Bot / Model	Syntax to Block (add to robots.txt)

GPTBot (OpenAI)	User-agent: GPTBot Disallow: /

Google-Extended	User-agent: Google-Extended Disallow: /

ClaudeBot (Anthropic)	User-agent: ClaudeBot Disallow: /

PerplexityBot	User-agent: PerplexityBot Disallow: /

CCBot (Common Crawl)	User-agent: CCBot Disallow: /

Bytespider (ByteDance)	User-agent: Bytespider Disallow: /

Amazonbot	User-agent: Amazonbot Disallow: /

Applebot	User-agent: Applebot Disallow: /

Copy and paste the lines for each bot you want to block.

Example robots.txt

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

Important Notes

robots.txt is not foolproof. Only bots that respect the protocol will comply. Some scrapers may ignore it.
Update regularly. New bots appear all the time—keep your file current.
Don’t block search engines (like Googlebot or Bingbot) unless you want your site removed from search results.

Next.js Routing in 2025

Mouaz — Tue, 25 Mar 2025 20:20:23 +0000

Overview of Next.js Routing

Next.js provides two primary routing systems: the Pages Router and the App Router. Understanding these systems is crucial for building efficient and scalable applications.

Pages Router

Introduction: The Pages Router was the original routing system in Next.js, available before version 13. It uses a file-system-based approach where each file in the pages directory corresponds to a route.
Features:
- Server-Side Rendering (SSR): Supports SSR, which can enhance SEO and initial page load performance.
- Static Site Generation (SSG): Allows for pre-rendering pages at build time, improving performance.
- Simple Setup: Easy to set up and understand, especially for smaller applications.
Use Cases: Ideal for applications where SEO and initial performance are critical, or for maintaining legacy projects.

App Router

Introduction: Introduced in Next.js 13, the App Router is designed to leverage modern React features like Server Components and Streaming.
Features:
- Server Components: Enables server-side rendering by default, with client-side rendering for interactive components.
- Dynamic, Nested, and Parallel Routes: Supports complex routing scenarios, making it suitable for dynamic and interactive applications.
- Performance Optimization: Offers advanced caching and streaming capabilities to improve user experience.
- Flexibility: Allows for more flexible rendering and handling of UI scenarios.
Use Cases: Recommended for single-page applications or projects requiring advanced routing features and performance optimizations.

Key Differences

Feature	Pages Router	App Router
Routing Approach	File-system based	More flexible, supports dynamic and parallel routes
Server-Side Rendering	Optional, via `getServerSideProps`	Default for most components
SEO & Performance	Better SEO and initial performance due to SSR/SSG	Advanced caching and streaming for improved performance
Complexity	Simpler setup	More complex, but offers advanced features
Recommended Use	Legacy projects, SEO-focused applications	New projects, single-page applications, dynamic routing

To improve your skills in Next.js, focus on the following:

Familiarize yourself with the App Router
Practice with Real Projects: Start building projects that utilize the App Router's advanced features like Server Components and parallel routes.
Stay Updated with Documentation: Regularly check the official Next.js documentation for updates and best practices.
Experiment with Different Routing Scenarios: Try implementing dynamic, nested, and parallel routes to understand how they can enhance your application's user experience.

Happy Coding! 🤓

DEV Community: Mouaz