DEV Community

Oleksandr Gamanyuk for Hipa.ai

Posted on Edited on Fully Autonomous

~600K bot requests a day hit my origin. A 24-hour Cloudflare cache with purge-on-change fixed it (on the Free plan)

In mid-October I looked at Cloudflare analytics for hipa.ai, the clinical-trials search site I build alone. On a typical day there were about 455,000 requests marked DYNAMIC and another 141,000 marked MISS. Every one of them went through to my Next.js origin on Azure App Service, which rendered the page from scratch against MongoDB and Elasticsearch.

Very few of those requests were people. When I later reviewed our own client-side analytics beacon, about 95% of the events came from bots too. Real people were somewhere around 250–370 visitors a week.

This post covers who was sending the traffic, the cache rule that took it off the origin, and the four ways that rule could have served wrong pages for a full day.

Who was hitting the origin

There were three groups:

  1. A headless residential-proxy botnet. User agent X11; Linux x86_64 … Chrome/154, about 8,500 distinct US IPs, ran JavaScript, ignored robots.txt. It looped over /, /platform, /for-* and /auth/* all day. Blocking by IP doesn't work when every IP is someone's home router.
  2. OAI-SearchBot, which fetched each /condition/<slug> page about 1,000 times a day. Same page, same content, a thousand fresh renders.
  3. ClaudeBot, working through trial listings and /study/* pages.

The pages they hit render the same for every visitor. But they either matched no Cache Rule or sent Cache-Control: private, no-store, so Cloudflare passed every request through to origin.

The prequel: when per-render cost takes down a deploy

I already knew what this traffic could do, because on June 1 it had taken down a deploy.

A helper that wrapped drug names in links on /study/[nctId] and every trial listing ran this on every render:

const drugs = await db.collection("drug_articles").find({}).toArray(); // 398 docs, per page view
Enter fullscreen mode Exit fullscreen mode

On localhost it was fine. After a slot swap, though, the caches were cold, Cloudflare's deploy purge sent a wave of misses, and the crawlers kept going. The Mongo client saturated, App Service went past its 60-second health-check timeout, and the swap left a broken build live in production. Recovery was an emergency swap back, then a refactor that served the catalog from a static manifest. Afterwards a warm /study render took about 250 ms.

That became a house rule: never pull a whole collection on the request path. The October traffic was the same problem at a different layer. Even a cheap render is expensive at 600K a day.

The fix: cache everything for 24 hours, and purge when it changes

A short TTL (say 5 minutes) would still give every crawler a fresh origin render roughly every 5 minutes per URL, multiplied by hundreds of thousands of URLs. I wanted the opposite: cache for a day, and purge the moment something actually changes.

This works because of one property. These routes render the same for every visitor (no cookies() or headers() reads; geo ads and attribution run client-side), and most of them are dynamic at origin. So origin never holds a stale copy. A purge followed by one origin render is always fresh.

The rule is a single Cloudflare Cache Rule, [BOT-SHIELD], built from a small TypeScript module that also drives every purge:

export const BOT_SHIELD_EDGE_TTL_SECONDS = 86_400;

export const BOT_SHIELD_WILDCARDS = [
  "/condition/*", "/study/*", "/research/*", "/news/*",
  "/drug/*", "/companies/*", "/clinical-trials/*", "/aba-therapy-in-*",
] as const;

export const BOT_SHIELD_EXCLUDE_WILDCARDS = [
  "/news/*/providers*",   // gated downloads: per-user
  "/research/*/studies*",
  "*.xml",                // sitemaps keep their own s-maxage
] as const;

export const BOT_SHIELD_BYPASS_COOKIES = ["logged_in", "refresh_token", "fpa_session"] as const;
Enter fullscreen mode Exit fullscreen mode

The rule's action looks like this:

action: "set_cache_settings",
action_parameters: {
  cache: true,
  edge_ttl: {
    mode: "override_origin",
    default: 86400,
    status_code_ttl: [
      { status_code_range: { from: 500, to: 599 }, value: -1 }, // never store 5xx
      { status_code: 404, value: 300 },
      { status_code: 410, value: 300 },
    ],
  },
  browser_ttl: { mode: "respect_origin" },
},
Enter fullscreen mode Exit fullscreen mode

A unit test fails the build if any cached path has no matching purge path, if a gated download is cacheable, or if a new route.ts shows up under a cached family without anyone deciding whether it should be cached.

Purging on the Free plan

I assumed prefix purge needed a paid plan. On this zone it works on Free. One call like {"prefixes": ["hipa.ai/condition"]} evicts every query-string and RSC variant under that path, and one call takes up to 30 prefixes. Purging all the HTML went from about 800 per-URL calls to about 2 seconds.

Each thing that changes content then purges exactly what it touched:

Trigger Purges
Deploy (after slot swap) every HTML family + /
Weekday trials import (after ES alias swap) trial pages + 4 sitemap XMLs
News / research pipelines /news, /research
Drug/condition update posts that /drug/<slug> + linked /study/<NCT>
Facility directory rebuild /facility/, /network/
Portal "hide this trial" toggle every trial-rendering family

Two rules I hold to: purge after the write (and after any Next revalidateTag call), and offline scripts skip the purge when they wrote to a non-production database. Otherwise a test run against a scratch DB happily purges the live zone.

The result, from the Azure bill

I can't show you the Cloudflare graph here, but the origin's egress meter in Azure Cost Management makes the point well enough:

  • Oct 10–15, before the rule: 8.8–9.7 GB/day out of the web app ($0.76–0.85/day)
  • Oct 16, launch day: 12.7 GB (deploy, purges, refill)
  • Oct 17–31, after: 1.1–2.5 GB/day ($0.09–0.22/day)

That's roughly an 80% drop in origin egress overnight, from a free-plan cache rule and some purge plumbing. The bigger win doesn't show up as a dollar line: those Mongo and Elasticsearch round-trips stopped happening at all, and they were the thing that had already taken a deploy down once.

Four traps that would have served wrong pages for 24 hours

override_origin caches whatever origin returns. That's the point, and it's also the danger.

1. Error pages got cached. Reviewing the rollout on launch day, I found that a /study 404 was a HIT on the second request. A 404 or 500 caused by a passing Elasticsearch hiccup would have stuck for a day. That's why the rule has status_code_ttl: 5xx is never stored, and 404/410 are held for 5 minutes, which is just long enough to absorb a crawler looping over dead URLs.

2. Friendly "temporarily unavailable" pages returned 200. /companies caught Elasticsearch failures and rendered a polite notice with status 200. The edge would have kept it for a day. Now the page rethrows, and a segment error.tsx renders the same notice with a 500, which is never stored. One more trap here: the first version of that error.tsx imported a server-only page shell, and the production build broke on node:fs. tsc and the tests passed. Only next build caught it.

3. Per-user responses under cached paths. Gated downloads under /news/* and /research/* return a file for signed-in users and a 302 to /register for everyone else. If either version got cached, the gate would be broken, so both paths are excluded. On top of that, any request carrying a session cookie always goes to origin, so a real session never fills the cache. Crawlers don't send cookies, so they still get HITs.

4. The purge token had been dead for two weeks. I'd rotated the Cloudflare API token on September 30. The production app setting still had the old one. Every in-app purge since then had failed, and the only sign was a warning in the logs. Before BOT-SHIELD, few pages were cached for long, so nobody noticed. With a 24-hour TTL it would have meant day-old pages after every portal change. Now the token is a Key Vault reference, and I test new tokens with a harmless prefix purge on a path that doesn't exist.

What caching didn't fix

The botnet still calls /api/* and /auth/*, which can't be cached. Pages that are only reachable through facet combinations still get crawled. Next on the list are a rate-limit rule for that one user agent and a robots Disallow on facet parameters. And because the trials import deliberately doesn't expire origin-side aggregation caches all at once (that's exactly the cold-cache storm from June), trial lists can lag an import by up to a day.

If you run a content site behind Cloudflare

  • Look at your DYNAMIC and MISS counts before you look at your servers. You may be paying to render the same page for crawlers a thousand times a day.
  • A long TTL plus precise purges beats a short TTL, as long as your pages render the same for everyone.
  • Write down every cached path in one module and make the tests enforce that each one has a purge.
  • Assume the edge will freeze your worst response. Design status codes so it can't.

Everything above runs hipa.ai, where you can search recruiting clinical trials by condition and city.

Oleksandr Gamaniuk

Top comments (0)