DEV Community

Cover image for Cloudflare and PSI Automation: Challenges, Allowlists & Monitoring
Apogee Watcher
Apogee Watcher

Posted on Originally published at apogeewatcher.com

Cloudflare and PSI Automation: Challenges, Allowlists & Monitoring

#webdev #webperf #seo

The same client URL opened in PageSpeed Insights from a laptop often completes. Put that URL on a schedule, call the PageSpeed Insights API from a monitoring stack, or re-run Lighthouse through a headless pipeline, and the document request returns a Cloudflare challenge, a 403, or an empty error shell. The score that ends up in the ticket is not a Core Web Vitals regression. It is a security edge deciding that automation is not a visitor worth serving HTML.

That contrast is the whole problem for agencies. Manual checks feel fine, so the account team trusts the green screenshot. Overnight automation red-flags the homepage, clients ask what broke, and engineering digs into LCP subparts on a page that never reached the real document. Cloudflare is doing its job. Monitoring is measuring the wrong response. What follows is how to tell those cases apart, which Cloudflare controls usually cause them, and how to allow legitimate PageSpeed Insights traffic without opening the site to every scraper on the internet.

For the wider network path (DNS, TLS, CDN cache), see Network Performance for Web Teams: DNS, TLS, HTTP, CDN, and Cache Rules. That guide covers where edge routing and cache hits change Time to First Byte before paint work begins. For why one-off PageSpeed Insights runs are a weak substitute for a schedule, see PageSpeed Insights vs Automated Monitoring: When Manual Checks Aren't Enough.

Why PageSpeed Insights works in a browser but fails under Cloudflare automation

PageSpeed Insights and Lighthouse need a full HTML document, then a rendered page, to compute lab metrics. An interactive browser session often looks enough like a human visitor for Cloudflare to pass the first request, especially if cookies from a prior challenge exist, JavaScript challenges can run, or the operator is on a residential network that is not already associated with bulk scraping. Scheduled and API-driven runs typically do not share that context. They arrive from data-centre ranges, use predictable automation User-Agents (including Chrome-Lighthouse strings), lack challenge cookies, and cannot complete managed challenges the way a person with a browser can.

Path Typical Cloudflare outcome What Lighthouse / PageSpeed Insights measures
Interactive browser PageSpeed Insights / DevTools Often allowed after challenge or soft pass Real HTML and assets (when allowed)
PageSpeed Insights API / scheduled headless Challenge interstitial, 403, connection error Challenge page, error document, or failed fetch
Monitoring tool from fixed egress IPs Blocked until IP Access or Skip rule exists Same failure modes until allowlisted

When the fetch fails, Google’s tooling often surfaces messages such as ERRORED_DOCUMENT_REQUEST or a generic connection failure. In Cloudflare Security Events you may see Bot Fight Mode, Super Bot Fight Mode, a WAF custom rule, rate limiting, or Under Attack Mode labelled against the same URL. Treating that as an LCP incident wastes a sprint; treat it as automation that never received the origin HTML.

Agencies also hit a quieter failure: the run “succeeds” but the lab screenshot shows a Cloudflare interstitial or a stripped challenge document. Scores look absurdly bad or oddly empty. Field data in PageSpeed Insights can still look normal when enough real users reach the site, which is another reason not to collapse lab and field into one narrative. For missing URL-level field metrics, see When PageSpeed Insights Shows No CLS or INP for Your URL. For when synthetic and real-user signals should disagree on purpose, see When to Use Synthetic vs Real User Monitoring for Performance.

Which Cloudflare controls block PageSpeed Insights and Lighthouse

Several Cloudflare products can interrupt PageSpeed Insights automation. They are not interchangeable, and some cannot be bypassed with the allowlist patterns people copy from blog posts. Matching the failed run to the right product saves a week of rewriting Largest Contentful Paint budgets on a document that never left the edge.

Bot Fight Mode (free)

Cloudflare documents that Bot Fight Mode challenges traffic matching known bot patterns and that you cannot bypass or skip it with WAF custom rules or Page Rules, because it does not run on the Ruleset Engine. If portfolio monitoring is blocked and Bot Fight Mode is on, Skip rules aimed at Chrome-Lighthouse will not save you. Cloudflare’s own guidance points teams toward Super Bot Fight Mode when they need exceptions, or toward IP Access rules that match the connecting IP before Bot Fight Mode evaluates the request.

Super Bot Fight Mode (Pro and above)

This product adds configurable actions and supports Skip actions in WAF custom rules. It is the practical path when you need to allow verified bots or monitoring User-Agents while still challenging other automated traffic. Misconfigured “Definitely automated” or “Likely automated” actions still catch Lighthouse and PageSpeed Insights traffic that Cloudflare does not classify as a verified bot.

Under Attack Mode

This mode raises friction for almost everyone. Community threads regularly report PageSpeed Insights failing with ERRORED_DOCUMENT_REQUEST while Under Attack Mode is enabled, even when a User-Agent allowlist appears to match in the logs. Temporary attack posture and continuous performance monitoring do not coexist cleanly unless verified bots (Known Bots) are explicitly skipped from remaining security checks. Under Attack Mode fits active incidents; it is a poor permanent monitoring configuration.

WAF custom rules and managed rules

Country blocks, ASN blocks, rate limits, and managed rule sets can 403 Google’s lab infrastructure or your monitoring egress. A rule that challenges “automated” traffic without an earlier Skip for known good bots will catch PageSpeed Insights. Order matters: exceptions for verified bots and monitoring IPs need to run before the block or challenge.

Rate limiting and Bot Management scores

Burst schedules across many URLs on one hostname can trip rate limits even when a single manual test works. Enterprise Bot Management customers can use score fields for finer thresholds; most agency clients on Free or Pro still need coarser Skip / IP Access patterns. When you investigate a Cloudflare block on PageSpeed Insights, open Security Events for the exact timestamp of the failed run, note the Service label, and match it to the product above before you rewrite Largest Contentful Paint budgets. Guessing “CDN is slow” from a challenged document wastes time that belongs in the security configuration, so screenshot the event for the client ticket so the zone admin sees the same evidence you used.

Origin versus edge: what a blocked lab run is actually measuring

A healthy synthetic run should measure the HTML and assets your visitors receive after Cloudflare’s edge has applied caching and routing, not a challenge interstitial and not a bypass that skips the edge entirely unless that bypass is intentional. Mixing those paths without labels is what makes weekly scores look volatile when nothing in the application changed. Three measurement modes show up in agency work:

  1. Edge as users see it. Requests go through Cloudflare with the same security posture as public traffic, except for narrow exceptions for known good lab traffic. This is the default you want for scheduled PageSpeed Insights once allowlists are correct.
  2. Challenge / block response. Automation measures Cloudflare’s security HTML or a failed document fetch. Lab metrics are invalid for Core Web Vitals decisions. Report the failure as a monitoring access issue, not as a performance regression.
  3. Origin-direct bypass. Some teams point monitors at the origin hostname or an IP Access “grey cloud” path so tests always pass. That can prove origin PHP or application latency, but it is not the same as visitor experience through the CDN. Use it as a diagnostic split, labelled clearly, not as the only portfolio score.

In our experience, the worst client reports skip those labels. One week’s “green” run used a temporary Under Attack exception; the next week’s “red” run hit Bot Fight Mode again. The narrative looks like volatility. The cause is inconsistent access to the document.

Origin and edge stay honest the same way Cache-Control stays honest: name which hostname, which Cloudflare mode, and which allowlist were in effect for the run you are quoting. Pair that with fixed performance budgets so a challenge-induced score does not auto-open a “fix LCP” ticket. Budget templates live in Performance Budget Thresholds Template.

Allowlists that work: verified bots, IP Access, and Skip rules

There is no single checkbox labelled “allow PageSpeed Insights forever.” You assemble a small set of exceptions that match how Cloudflare classifies the traffic, then you verify with a scheduled re-run. The patterns below are the ones that usually hold up under Security Events review.

Prefer verified / known bots over User-Agent string matching alone

Cloudflare documents custom-rule patterns that allow search-engine and other verified bots using fields such as cf.client.bot (and Bot Management fields where available). User-Agent contains "Chrome-Lighthouse" rules are fragile: they are easy to spoof, easy to miss when Google changes strings, and insufficient under Under Attack Mode when the wrong Skip scope is selected. Known Bots / verified bot Skip rules for remaining WAF modules and Super Bot Fight Mode (where the plan supports them) are a stronger starting point, with monitoring-specific exceptions added only if Security Events still show blocks.

Use IP Access allow rules for known monitoring egress

If your PageSpeed Insights automation or third-party monitor exits from stable IP ranges, an IP Access allow that matches the connecting IP can prevent Bot Fight Mode from firing on those requests (Cloudflare notes Bot Fight Mode will not trigger when an IP Access rule matches first). Document the ranges with the vendor and rotate them when the vendor publishes changes. Allowlisting entire cloud regions is usually too broad for the risk you are trying to reduce.

Move from Bot Fight Mode to Super Bot Fight Mode when you need Skip

Free Bot Fight Mode cannot honour WAF Skip the way Super Bot Fight Mode can. Agencies stuck on Free with constant false positives on monitoring have a product constraint, not a missing regex. The realistic options are to disable Bot Fight Mode after the attack window, upgrade so Skip rules work, or accept that scheduled PageSpeed Insights on that zone will stay unreliable.

Scope Skip narrowly

A Skip that disables all remaining security for any bot-shaped request is a self-inflicted outage waiting to happen. Narrowing by hostname path if needed (public marketing URLs versus authenticated app), by verified-bot field, and by known monitoring IPs keeps the exception honest. Review Security Events after the change to confirm PageSpeed Insights is skipped for the right reason and that random scrapers are still challenged.

Confirm with the same automation that failed

A manual browser pass after a rule change proves little. Re-run the scheduled job or API call that produced the original error, and only then mark the incident closed. Scheduling discipline for portfolios is covered in PageSpeed Monitoring: How to Schedule Test Frequency and Priority Across Your Portfolio.

User-Agent myths, warm-up, retries, and what not to do

Spoofing Googlebot to “pass” Cloudflare from a monitoring host violates Google’s crawler guidelines, teaches the wrong security lesson, and still fails when Cloudflare verifies bots beyond the User-Agent string. If you need Googlebot behaviour, you are not debugging PageSpeed Insights automation; you are debugging crawl access. Turning security off globally to get a green Lighthouse score is the other common shortcut, and it trades one KPI for a real incident risk. Time-boxed Under Attack Mode, narrow Skips, and IP Access for monitors are safer than disabling Bot Fight Mode or WAF managed rules site-wide for a performance project.

User-Agent allowlists help on some plans and some products, but they are not complete. They do not override Bot Fight Mode. They do not always survive Under Attack Mode. They do nothing for rate limits keyed on IP.

Warm-up and retries have limits. A single warm-up request that solves a challenge in a full browser does not automatically mint cookies for a separate PageSpeed Insights API worker. Retrying a challenged URL every minute usually deepens rate-limit pain without producing a valid document, so retries fit transient network errors after allowlists are confirmed, not as a substitute for a Skip rule.

Runbooks stay clearer when “tool blocked” and “site slow” stay separate. Security Events and HTTP status come first; LCP subparts come after. If the document request never returned 200 HTML from the application, the performance playbook stops and a Cloudflare configuration ticket with the client’s zone admin is the right next step.

How agencies should report Cloudflare blocks in client monitoring

Clients rarely want a lecture on Ruleset Engine phases. They want to know whether the site is slow for users or whether the monitor was refused. A short script in the report usually covers it:

  1. State the failure mode in plain language: “Scheduled PageSpeed Insights received a Cloudflare challenge / 403, so lab metrics are invalid for this run.”
  2. Attach evidence: Security Events screenshot or rule name, HTTP status, and whether field data in PageSpeed Insights still looks normal.
  3. Name the owner: client Cloudflare admin versus agency monitoring egress. Most allowlist changes require the zone owner.
  4. Record the fix: verified-bot Skip, IP Access allow, Super Bot Fight Mode change, or temporary Under Attack Mode turned off after the incident.
  5. Re-run on schedule and only then restore budget alerting for that URL.

Keep synthetic budgets armed for real regressions once access is restored. Muting an entire client because one zone had Bot Fight Mode misconfigured hides unrelated regressions; mute or annotate the affected URLs until the next clean run instead. That keeps Apogee Watcher-style portfolio schedules useful: continuous checks, early detection, clear reports, without pretending every red score is an LCP bug. If you run Apogee Watcher across many Cloudflare-fronted clients, recurring challenge failures belong on the onboarding checklist: confirm Bot Fight Mode posture, document whether Super Bot Fight Mode Skip is available, and store monitoring egress expectations with the client before you promise daily lab history.

FAQ

Why does PageSpeed Insights work manually but fail on a schedule behind Cloudflare?

Manual runs often complete challenges or arrive from networks Cloudflare treats more gently. Scheduled API and headless runs look like automation, lack challenge state, and hit Bot Fight Mode, WAF rules, or Under Attack Mode. Fix allowlists and product mode rather than only re-testing from a laptop.

Can a WAF custom rule Skip Bot Fight Mode for Chrome-Lighthouse?

No. Cloudflare documents that Bot Fight Mode cannot be bypassed with WAF custom rules or Page Rules. Super Bot Fight Mode with Skip, IP Access allows that match first, or disabling Bot Fight Mode when the attack window is over are the documented paths.

Should we allowlist by User-Agent or by verified bots?

Prefer verified / Known Bots fields and stable monitoring IPs. User-Agent matching is a weak secondary signal and is easy to get wrong under Under Attack Mode.

Is a green score through an origin bypass good enough for clients?

Only as a labelled diagnostic. Visitors still hit the Cloudflare edge. Report edge-path PageSpeed Insights once security exceptions are correct.

What if field data looks fine but lab keeps failing?

That pattern often means real users pass while automation is blocked, or that URL-level field metrics are missing for other reasons. Investigate Cloudflare events first, then reconcile lab versus field with your usual synthetic-versus-RUM rules.

When Cloudflare blocks PageSpeed Insights automation, the monitoring problem is access, not paint timing. Diagnose the product that challenged the request, apply the narrowest Skip or IP Access that still protects the zone, and re-run the same scheduled path before you rewrite performance budgets. If you want that loop across a multi-tenant portfolio without babysitting every PageSpeed Insights tab, start a free Apogee Watcher trial or run a quick check on a priority URL once Cloudflare allows the document through.

References

Top comments (0)