Cloudflare Cache's Blind Spot: The Cost of Bypass Rules

Last month, I started noticing unexplained spikes in server load on my blog, mustafaerbay.com, especially when publishing new articles or when popular content received traffic. I manage over 13 Docker containers on my own VPS, and when one started misbehaving, others would swap, kcompactd would eat 90% of the CPU, and even SSH connections became impossible to establish. This was surprising for a site sitting behind Cloudflare. I even recall a moment when the Astro build process consumed 2.5 GB of RAM, straining the system's 7.6 GB. Surely Cloudflare's cache should have sorted everything out, right?

The problem wasn't Cloudflare itself, but rather my reliance on cache bypass rules and the blind spot created by the Cache-Control headers coming from my origin server. It took me a while to realize how Astro's default max-age=0 setting was completely disrupting the caching logic I had established with Cloudflare. In this post, I'll explain how I diagnosed this cache blind spot, its causes, and how I gained control over cache behavior using Cloudflare and Nginx.

The Problem: Unexpected Side Effects of Cache Bypass Rules

On my own blog, I had defined some cache bypass rules on Cloudflare for specific dynamic-looking sections like mustafaerbay.com/feed.xml or backend services like /api/*. My goal was to ensure these particular URLs always fetched the latest data from the origin. It seemed logical. However, over time, I noticed that these rules were causing unintended side effects.

Even static content in the /blog/* path of my blog posts occasionally started hitting my origin server directly. Although I could see cf-cache-status: HIT for some requests in the Cloudflare panel, my origin server's CPU and I/O load indicated that caching wasn't working as it should. This situation led to my server becoming overloaded, especially during peak traffic, and at one point, it escalated into a Docker disk fire. 33 GB of build cache and 23 GB of unused images filled the disk to 100%, causing the VPS to completely freeze. This wasn't just a caching issue; it became a chain reaction that turned into an operational nightmare.

Symptoms and Initial Observations

• High Origin Load: My server's CPU and RAM usage were significantly higher than expected, despite Cloudflare being in front. I was seeing node processes peaking in htop.
• Inconsistent Page Load Times: Some users experienced lightning-fast page loads, while others faced noticeable delays. This was a clear sign of inconsistent caching.
• cf-cache-status: BYPASS: In browser developer tools or during curl tests, I started seeing the cf-cache-status: BYPASS header for some static content. This was my first red flag.
• Disk Fullness: Increased traffic and unexpected origin requests caused log files to grow rapidly and temporary files for Docker containers to bloat. This led to the 100% disk usage mentioned earlier.

ℹ️ Sharing Experience

Performance issues like these rarely stem from a single point. While a cache bypass rule might be a trigger, as I experienced, underlying issues like poor disk management or application resource consumption can exacerbate the situation. I always try to see the whole picture.

Diagnosis: Where Did I Start?

To understand the problem, I proceeded step-by-step. Instead of panicking, I used the data and tools at hand to figure out what was happening.

Cloudflare Logs and Analytics

First, I checked the Analytics section in the Cloudflare panel. While the edge cache hit ratios generally looked good, I noticed a high Cache Bypass rate for certain URL patterns. This provided the initial clue as to which requests were reaching the origin. I specifically examined the Security and Firewall logs to see which rules were being triggered and how they were affecting cache behavior.

Origin Server Logs

I dove into the Nginx and Node.js (where my Astro application was running) logs. In Nginx's access.log file, I started seeing continuous 200 OK responses for static blog post URLs that should have been served by Cloudflare. I confirmed whether a request came from Cloudflare or directly by looking at the X-Forwarded-For header in these logs. Using grep and awk commands, I quickly compared request counts for specific URL patterns.

# Count requests containing "/blog/" in Nginx access log
grep "/blog/" /var/log/nginx/access.log | wc -l

# Filter requests from a specific IP (Cloudflare IP range)
grep "172.68.XXX.XXX" /var/log/nginx/access.log | grep "/blog/" | wc -l

Tests with curl and httpie

I obtained the most concrete data through curl commands. I tested cache behavior by sending different headers (especially Cookie or User-Agent). Cloudflare's cf-cache-status header clearly indicates whether a request came from the cache (HIT), went to the origin (BYPASS), or was cached for the first time (MISS).

# Check cache status for a blog post
curl -svo /dev/null https://mustafaerbay.com/blog/cloudflare-cachein-kor-noktasi-bypass-kuralinin-bedeli 2>&1 | grep -i "cf-cache-status"

# Test request with a Cookie (Cloudflare usually bypasses if Cookie is present)
curl -svo /dev/null -H "Cookie: my_session_id=123" https://mustafaerbay.com/blog/cloudflare-cachein-kor-noktasi-bypass-kuralinin-bedeli 2>&1 | grep -i "cf-cache-status"

💡 Quick Check

The command curl -svo /dev/null https://mysite.com/path 2>&1 | grep -i "cf-cache-status" is an indispensable tool for instantly checking Cloudflare cache status.

Browser Developer Tools

Finally, I opened my browser's developer tools (F12), went to the Network tab, and inspected the headers of the requests made when I refreshed the page. The cf-cache-status and cache-control headers were clearly visible here as well. Seeing the combination of cf-cache-status: BYPASS and Cache-Control: max-age=0 made me understand the root of the problem much better.

Root Cause Analysis: Why Was It Bypassing?

All these diagnostic steps led me to two primary issues: the broad scope of Cloudflare's cache bypass rules and the misinterpretation of Cache-Control headers from my origin server.

Cloudflare Page Rules vs. Cache Rules

Cloudflare has two main rule sets: Page Rules and Cache Rules. Page Rules are older and more general-purpose, while Cache Rules are designed for more specific control over cache behavior. In my scenario, a rule defined in Page Rules had Cache Level: Bypass set for specific paths. While this rule targeted dynamic endpoints like *mustafaerbay.com/api/*, it unintentionally affected static content like /blog/* due to its broad scope.

Cloudflare's default caching behavior was also important here. It generally caches GET requests, but it bypasses requests containing Cookie headers, Query Strings, or directives like Cache-Control: no-cache by default. When my bypass rule combined with this default behavior, even static content was being routed to the origin.

The Role of the Cache-Control Header

The biggest blind spot was the Cache-Control header that my Astro application was returning by default. Although Astro is a static site generator, when run as a Node.js-based SSR (Server-Side Rendering) or API endpoint, it can often return headers like Cache-Control: max-age=0, must-revalidate by default.

This header instructed Cloudflare to "check the validity of this content immediately" or "don't cache, always go to the origin." Even without my Cloudflare bypass rule, this max-age=0 directive from the origin was preventing Cloudflare from caching the content. Instead of seeing Cache-Control: public, max-age=X, seeing max-age=0 meant Cloudflare wouldn't cache the content. I can say that Astro's max-age=0 gave me quite a bit of trouble on my own blog. This situation rendered even Cloudflare's powerful caching mechanism ineffective.

⚠️ Important Information

Cache-Control headers from your origin server can even override your Cloudflare Edge Cache TTL settings. If the origin sends max-age=0 or no-cache, Cloudflare will generally respect this directive and not cache the content.

The Solution: Taking Control of Cache Behavior

After understanding the root cause of the problem, I took steps to gain full control over cache behavior using Cloudflare and Nginx.

More Precise Control with Cloudflare Cache Rules

The first step was to review the old Page Rules in Cloudflare and define more specific Cache Rules instead.

1. Review and Narrow Down Existing Bypass Rules: I refined the Cache Level: Bypass rules in the existing Page Rules to be specific only to paths that truly needed to be dynamic, like /api/* or /login/*. I excluded static content like /blog/* from these rules.
2. Define Cache Rules for Static Content: I created a new Cache Rule for all content under the /blog/* path. This rule sets the Edge Cache TTL to a specific duration (e.g., 1 hour or 1 day) and sets the Cache Level to Cache Everything. I also kept options like Bypass Cache on Cookie active only where truly necessary (like /account/*).

An example of a Cloudflare Cache Rule (when defining via API or Terraform, it would be similar to this format):

{
  "rules": [
    {
      "id": "mustafa_blog_cache_rule",
      "action": {
        "id": "cache_settings",
        "value": {
          "edge_cache_ttl": 3600, // 1 hour
          "cache_level": "cache_everything"
        }
      },
      "expression": "(http.request.uri.path contains \"/blog/\") and (not http.cookie)"
    },
    {
      "id": "mustafa_api_bypass_rule",
      "action": {
        "id": "cache_settings",
        "value": {
          "edge_cache_ttl": 0, // Bypass
          "cache_level": "bypass_cache"
        }
      },
      "expression": "http.request.uri.path contains \"/api/\""
    }
  ]
}

This example caches requests under the /blog/ path that do not contain cookies for 1 hour, while bypassing all requests under the /api/ path. Creating these rules through the Cloudflare UI is much easier and more visual.

Nginx Override on the Origin Server

Cloudflare Cache Rules are great for managing cache on Cloudflare's side. However, the Cache-Control: max-age=0 header from my origin server was still an issue. Cloudflare tended to respect this directive. Therefore, I decided to override this header using Nginx.

Using Nginx's proxy_hide_header and add_header directives, I hid the Cache-Control and Pragma headers coming from the origin and added my own desired Cache-Control header instead. This prevented Cloudflare from seeing the max-age=0 directive and allowed it to cache the content with my specified cache duration.

server {
    listen 80;
    server_name mustafaerbay.com;

    # Redirect HTTP requests from Cloudflare to HTTPS (optional, but generally good practice)
    # if ($http_x_forwarded_proto != 'https') {
    #     return 301 https://$host$request_uri;
    # }

    location /blog/ {
        # Hide Cache-Control and Pragma from Astro
        proxy_hide_header Cache-Control;
        proxy_hide_header Pragma;

        # Add our own Cache-Control: public cache for 1 hour
        add_header Cache-Control "public, max-age=3600";

        proxy_pass http://localhost:3000; # Port where Astro app is running
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    # Other location blocks (e.g., /api/ or root path)
    location / {
        proxy_pass http://localhost:3000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    # ... other Nginx settings
}

With this Nginx configuration, for all requests under the /blog/ path, the Cache-Control header from Astro is hidden by Nginx, and Cache-Control: public, max-age=3600 is added instead. This allows Cloudflare to cache this content for 1 hour.

⚠️ Caution: Nginx Override

You must be very careful when overriding cache headers from the origin. If you accidentally cache dynamic content as static, users might see old or incorrect data. Therefore, I only applied the override for paths that truly needed to be static.

Implementation Steps and Verification

After applying these changes, I performed extensive tests to ensure everything was working as expected.

Step 1: Review and Clean Up Existing Rules

I reviewed all Page Rules and Cache Rules in the Cloudflare panel. I either deleted rules that conflicted or seemed unnecessary, or I narrowed their scope. I specifically disabled all bypass rules affecting the /blog/* path.

Step 2: Check Origin Headers

Before changing the Nginx configuration, I checked the headers coming directly from the Astro application with commands like curl -I http://localhost:3000/blog/post-name. I made sure I was seeing Cache-Control: max-age=0.

Step 3: Define Cloudflare Cache Rule

I defined the mustafa_blog_cache_rule mentioned above through the Cloudflare UI. I set the Edge Cache TTL to 1 hour and Cache Level to Cache Everything. I disabled options like cookie bypass for this rule, ensuring it would cache under all conditions (provided the Cache-Control header from Nginx was considered).

Step 4: Nginx Configuration Changes

I SSH'd into my VPS, edited the /etc/nginx/sites-available/mustafaerbay.com file, and applied the Nginx configuration above.

sudo nano /etc/nginx/sites-available/mustafaerbay.com

After saving the changes, I tested the Nginx configuration and restarted it:

sudo nginx -t
sudo systemctl reload nginx

Step 5: Testing and Verification

Finally, I re-tested using curl and browser developer tools:

curl -svo /dev/null https://mustafaerbay.com/blog/cloudflare-cachein-kor-noktasi-bypass-kuralinin-bedeli 2>&1 | grep -i "cf-cache-status\|cache-control"

I was now seeing cf-cache-status: HIT and Cache-Control: public, max-age=3600 headers. This meant Cloudflare was successfully caching the content and not putting unnecessary load on the origin. My server's CPU and RAM usage also dropped significantly. Even when traffic to my blog increased, I started seeing 304 Not Modified or HIT instead of BYPASS in the Nginx logs.

Lessons Learned and Best Practices

This incident taught me several important lessons:

• Headers Are Critical: HTTP headers, especially Cache-Control, can be the smallest yet most impactful details in a system architecture. When using a CDN like Cloudflare, it's crucial to understand how headers from the origin server are interpreted.
• Thorough Testing is Essential: Just because everything looks fine in the Cloudflare panel doesn't mean everything is truly okay. Testing different scenarios with curl is essential to see the actual behavior.
• Beware of Bypass Rules: Cache bypass rules are powerful, but they should be kept as specific as possible. Broadly scoped rules can lead to unexpected outcomes, as I experienced. Always ask, "What will this rule affect?"
• Monitoring is Indispensable: Continuously monitoring performance metrics is critical for early detection of issues like abnormal origin load or disk fullness. This is even more important for someone like me managing their own server. This situation reminded me once again of the "preflight resource guard" principle in pipeline reliability. It's always better to check a resource before loading it than to put out fires later.
• Understand Trade-offs: Caching isn't always good. Bypassing might be necessary for dynamic content. However, if we don't want static content to behave like dynamic content, managing this trade-off with an intermediate layer like Nginx is necessary.

Conclusion

Cloudflare's caching mechanism is a powerful tool, but managing bypass rules and Cache-Control headers from the origin server correctly is vital for performance and operational stability. My experience demonstrated that even static content can overload the origin due to misconfigurations. By combining Cloudflare Cache Rules with Nginx's proxy_hide_header and add_header directives, I overcame this blind spot and ensured my blog ran much more stably.

The lessons learned from this process carry important insights for those who, like me, manage their own servers and pay attention to every millisecond of performance. Do you have a similarly frustrating caching story? Or have you used different approaches to solve these problems? I'd love to hear about it in the comments.