A few days ago I saw people discussing Cloudflare defaulting to blocking Googlebot for sites with "Block AI Training" enabled. That thread nagged at me all evening, because it reminded me of something dumber I did to myself a few weeks earlier — and I only fully understood how dumb it was last night.
I run a few Cloudflare-fronted SaaS products, including a video processing tool. That one matters for this story, because part of what it does is offload heavy work — frame extraction for content-compliance checks, video format conversion — to AWS Lambda, which then talks back to my Cloudflare-fronted domains to sync data and files.
The CPU spike
Early this month I opened the Cloudflare dashboard for something unrelated and noticed Workers CPU time was roughly double the previous month's level. I run several domains through Cloudflare, so I started going through the logs one by one, starting with a domain that gets almost no real traffic anymore.
Sure enough — something was batch-scraping the content on that site. I figured I'd found my CPU culprit. Since Cloudflare is supposed to be good at this kind of thing, I asked Gemini what to do about it. It recommended turning on Bot Fight Mode. That felt like a clean, official answer, so I enabled it — on every domain in my account, not just the one I was investigating.
The slow-motion breakage
A few days after flipping the switch, I did my usual log check and something felt off. Signups were still trickling in — but actual usage had basically flatlined. Nobody was really using the product, even though people were still showing up.
It took me an embarrassingly long time to connect this to Bot Fight Mode. When I finally went digging through Cloudflare's security event log, I found it: repeated blocked requests from the IP of my own AWS Lambda service — the one doing frame extraction and format conversion — trying to call back into my main domain. I could reproduce the failure on demand just by triggering that callback.
Once I had the cause, I spent the next few weeks doing the wrong thing about it: instead of turning Bot Fight Mode off to isolate the variable, I treated it as a fixed constraint and rewrote parts of my app to work around it. The workaround I landed on was routing the Lambda callbacks through S3 URLs and a workers.dev subdomain instead of my protected main domain — different enough hostnames that Bot Fight Mode left them alone, while still letting data and files sync where they needed to. Both Claude Code and Codex suggested more than once, in passing, that I could just disable Bot Fight Mode temporarily to confirm the cause. I didn't want to — I'd turned it on for a reason, so turning it off felt like giving up on a "real" fix.
Somewhere in that stretch, the product got a 3-star review. I'm fairly sure it landed during the exact window when the product silently didn't work. Sorry to whoever that was — you had every right to be unimpressed.
By the end of the month, after enough workarounds, everything was technically functioning again. I told myself the Bot Fight Mode problem was "solved." Not elegantly, but solved.
The actual realization
Then came the Googlebot-blocking discussion, and it sat in the back of my head until right before bed, when it clicked: I locked myself out of my own product for a month, and I never actually checked whether Bot Fight Mode was appropriate for these sites in the first place.
So I asked a different, better question this time. Not "how do I fix high CPU usage" but: what kind of site should turn Bot Fight Mode on, and what kind shouldn't?
The short version of the answer: it's reasonable for content sites that just want to deter scraping. It's a bad idea for SaaS products with API traffic, webhooks, or server-to-server calls, because it can't reliably tell your legitimate automated traffic from a bot attack — and it fails silently, with no clear signal pointing back to itself.
I'd had the tool right, applied it to the wrong category of site, and then spent a month debugging the symptom instead of the setting.
I turned it off the next morning.
The bonus mistake
The site where this all started — the low-traffic one actually being scraped — genuinely was a good candidate for Bot Fight Mode. But digging back into it, I also realized it never should have been running on Workers at all. It's a static-ish content site; it belongs on Cloudflare Pages. Running it on Workers is exactly how a scraper turns into a CPU bill instead of a non-issue.
What I'd tell past-me
A global, account-wide toggle is a bad default when your domains have fundamentally different traffic profiles (content vs. SaaS/API). Apply security settings per-zone, deliberately, not as a blanket fix.
Ask the specific question, not the generic one. "What feature fixes X" gets you a tool. "Is this tool right for my situation" gets you a decision. I only got a useful answer once I asked the second kind of question.
When something changes and behavior gets weird shortly after, disable the change first to confirm causation, then build the real fix. Don't treat your own recent setting change as a fixed constraint to engineer around.
Silent failures are the worst kind. Cloudflare's security event log did have a record of the blocked requests the whole time — I just wasn't looking there, because nothing pointed me to it. No alert, no error surfaced in my own app, nothing. I only noticed because usage numbers looked wrong, and only found the log entry weeks later when I went looking on purpose.
If you run multiple Cloudflare-fronted domains with different traffic types, it's worth checking right now whether Bot Fight Mode is on somewhere it shouldn't be.
I build a small video processing tool on the side, which is where this particular fire started.
Top comments (0)