Protecting Your Website with Cloudflare: Security, Performance, and Reliability [Part 2]

  

Cloudflare is an integral service for the modern web, delivering performance, integrity, and security on the edge of the network. In Part 1 of this series, the basic infrastructure deployment domain set-up, onboarding of Cloudflare, domain routing, and origin server preparation were covered.

If you haven’t read Part 1 yet, it’s recommended to start there to understand the baseline architecture and configuration before moving forward:

👉 Read Part 1: Cloudflare Fundamentals & Setup

As this guide has grown to cover multiple layers of infrastructure, it makes sense to continue with a focused follow-up.

In Part 2, we’ll dive into edge-level security and traffic control, including:

• AI crawl control and bot behavior management
• Rate limiting for APIs and sensitive endpoints
• Turnstile for user-friendly request validation

This part is all about hardening your application at the edge while keeping performance fast and user experience smooth.

AI Crawl Control

Why controlling AI crawlers matters

AI crawlers are no longer limited to traditional search engines. Many modern bots are designed to collect content for AI training, assistants, or large-scale analysis, often without providing direct value back to your site.

Uncontrolled AI crawling can lead to:

• Increased bandwidth and infrastructure costs
• Higher origin load and degraded performance
• Unwanted content harvesting or dataset creation
• Reduced visibility into who is accessing your content and why

At the same time, not all crawlers are bad. Search engine bots are critical for discoverability and SEO. The challenge is allowing the right bots while blocking or limiting the rest.

This is exactly where Cloudflare’s AI Crawl Control becomes essential.

Navigating to AI Crawl Control in Cloudflare

To access AI Crawl Control:

1. Log in to your Cloudflare dashboard
2. Select the domain you want to manage
3. From the left-hand navigation menu, click AI Crawl Control
4. Open the Crawlers tab

  

This section provides a centralized view of all detected crawlers accessing your site, along with:

• Crawler name and organization
• Category (Search Engine Crawler, AI Crawler, AI Assistant, Archiver, etc.)
• Request volume and recent activity
• Allow / Block controls per crawler

Blocking specific AI crawlers

Within the Crawlers tab, Cloudflare lists individual bots such as AI trainers, assistant crawlers, and large-scale data collectors.

To block a specific crawler:

1. Locate the crawler in the list
2. Review its category and request behavior
3. Click Block in the Action column

  

Once blocked, Cloudflare enforces this rule at the edge, preventing requests from reaching your origin server. This helps reduce unnecessary load and protects your content from being harvested for unintended use.

You can selectively block:

• AI training crawlers
• AI assistant fetchers
• Data aggregation bots

Keep search engine crawlers allowed (critical for SEO)

Search engine crawlers should almost always remain allowed, as they play a fundamental role in how your site is discovered and ranked. Bots such as Googlebot, Bingbot, and other verified search engine crawlers are responsible for indexing your content, updating search rankings, and driving organic traffic to your site. Blocking these crawlers can have serious consequences, including reduced visibility in search results, ranking drops, and even complete removal of pages from search engine indexes. For most websites, restricting search engine crawlers should only be done with a very specific and well-understood reason.

Rate Limiting

Rate limiting is a critical defense mechanism for protecting websites and APIs from abuse, brute-force attempts, scraping, and excessive automated traffic. By limiting how frequently a client can make requests within a defined time window, you can prevent malicious or misconfigured clients from overwhelming your application—while still allowing legitimate users to operate normally.

In Cloudflare, rate limiting is enforced at the edge, meaning abusive traffic is stopped before it reaches your origin, reducing load, latency, and infrastructure costs.

Navigating to Security Rules

To create a Rate Limiting rule:

1. Log in to the Cloudflare dashboard
2. Select your domain
3. From the left-hand menu, go to Security
4. Click Security rules

  

This section is where Cloudflare allows you to define how incoming requests are evaluated and mitigated using custom rules, managed rules, and Rate Limiting rules.

Creating a Rate Limiting rule

Inside Security rules:

1. Click Create rule
2. From the dropdown, select Rate limiting rules

  

This opens the rate limiting rule builder, where you define what traffic to monitor, how often it’s allowed, and what action Cloudflare should take when limits are exceeded.

Configuring the rate limiting rule

Rule name

Start by giving your rule a clear, descriptive name. For example:

api-rate-limit

A meaningful name makes future maintenance and troubleshooting much easier.

When incoming requests match…

This section defines which requests the rule applies to.

• Field: URI Path

• Operator:

  • Use Wildcard if you want to match a group of URLs
  • Use Equals if you want to target a single, exact path

• Value:
  
  

  /api/*

Using a wildcard is ideal for APIs or grouped endpoints, as it applies the rule to all matching paths under /api/.

When rate exceeds…

This section defines how much traffic is allowed before Cloudflare takes action.

• Requests: 10
• Period: 10 seconds

This means a single client (based on IP, by default) can make up to 10 requests within 10 seconds. Any additional requests beyond this threshold will trigger the configured action.

Then take action…

Choose what Cloudflare should do once the rate limit is exceeded.

• Action: Block

Blocking immediately stops matching requests and prevents Cloudflare from evaluating additional rules for that request, making it an effective mitigation for abusive behavior.

• For duration…

This setting defines how long the block remains active.

• Duration: 10 seconds

After the duration expires, the client is allowed to send requests again—unless the rate limit is exceeded once more.

Saving and deploying the rule

Once all fields are configured:

1. Review the rule settings
2. Click Deploy

The rate limiting rule becomes active immediately and is enforced at Cloudflare’s edge across all incoming traffic that matches your criteria.

  

To validate the rate-limiting rule, we can intentionally send multiple rapid requests to the same endpoint to simulate abusive behavior. Once the defined threshold is exceeded (for example, more than 10 requests within 10 seconds), Cloudflare immediately blocks further requests from the same client. At this point, the client receives Error 1015 – You are being rate limited, indicating that Cloudflare has temporarily restricted access. This confirms that the rule is working as expected and that excessive or abusive traffic is being stopped at the edge before reaching the origin server.

  

Setting Up Cloudflare Turnstile: Step-by-Step Guide

Cloudflare Turnstile provides bot protection without traditional CAPTCHAs, making it both user-friendly and secure. In this section, we’ll walk through the complete setup process, from creating a Turnstile widget in the Cloudflare dashboard to applying security rules and integrating it into your application.

Navigate to Turnstile in Cloudflare Dashboard

1. Log in to your Cloudflare dashboard.
2. Select your account or domain.
3. From the left navigation panel, go to: Application Security → Turnstile
4. This section is where all Turnstile widgets are managed.

  

Create a New Turnstile Widget

1. Click the “Add widget” button under Turnstile widgets.
2. Provide a Widget Name (e.g., turnstile-test) to identify it later.
3. Under Hostname Management, click Add Hostnames.
4. Select an existing hostname or add a custom hostname where Turnstile should be active.
5. Click Add to confirm the hostname selection.
6. This ensures Turnstile challenges are only valid for the specified domain(s).

Widget Mode

Choose how Turnstile behaves for users:

• Managed (Recommended) Cloudflare automatically decides whether interaction is needed based on request risk.
• Non-interactive Shows a loading-style challenge without user input.
• Invisible Runs entirely in the background with no UI.

For most applications, Managed mode offers the best balance between security and user experience.

Once configured, click Create.

  

Retrieve Site Key and Secret Key

After successful creation, Cloudflare displays:

• Site Key – used on the client side (frontend)
• Secret Key – used on the server side for verification

Important: Store the Secret Key securely. It should never be exposed in frontend code.

  

Client-Side Integration

Add the Turnstile script and widget container to your HTML page:

<script src="https://challenges.cloudflare.com/turnstile/v0/api.js" async defer></script>

<div class="cf-turnstile"
     data-sitekey="YOUR_SITE_KEY">
</div>

What Happens Here

• The script loads asynchronously to avoid blocking page load.
• Turnstile automatically runs risk analysis in the background.
• Legitimate users usually see no visible challenge.
• Bots or suspicious traffic are blocked or challenged.

Apply Turnstile Using Security Rules

To enforce Turnstile on specific routes or actions:

1. Navigate to Security → Security rules.
2. Click Create rule → Custom rules.
3. Define matching conditions (e.g., URI Path equals / or /api/*).
4. Set the Action to Managed Challenge.
5. Choose rule order (usually First for critical paths).
6. Click Deploy.

  

This ensures Turnstile is triggered only where it’s needed, such as login pages or sensitive APIs.

Verify and Monitor

Once active:

• Use View analytics on the widget to track human vs bot traffic.
• Monitor challenge success rates and request patterns.
• Adjust rules or widget mode if necessary.
• Cloudflare’s analytics help fine-tune protection without degrading user experience.

  

Cloudflare provides far more than a single layer of protection—it offers a comprehensive security ecosystem designed to defend modern applications against a wide range of threats. In this article, we explored how Turnstile delivers frictionless bot protection without degrading user experience, and how it can be combined with other Cloudflare features such as custom security rules, managed challenges, and rate limiting to build a stronger defensive posture.

When used together, these controls allow you to protect sensitive endpoints, reduce abusive traffic, and handle sudden spikes or malicious behavior with confidence. Instead of relying on one mechanism, Cloudflare enables a layered security strategy—balancing usability, performance, and protection. By thoughtfully configuring these features, teams can stay ahead of automated abuse while keeping applications fast, accessible, and resilient at scale.