Four alarms fired overnight. Two database contention pages, one read-replica collapse, and a batch of metrics that came back empty.
Four alarms, one cause, and the cause was not anything we shipped.
The wrong first instinct
The reflex when four alarms land at once is to open the deploy log and start reading diffs. I did that for a few minutes. It is almost always wasted time, and there is a cheaper question that settles it.
Did request volume change?
workersInvocationsAdaptive, grouped by script, per hour
prod-api 3.6K/hr -> 18K/hr at 01:00Z
prod-web 3.6K/hr -> 17K/hr at 01:00Z
Both workers stepped up five times at the same minute and stayed there. A deploy does not do that. Traffic does. One query, and the entire code-review lane is closed.
The database side matched:
d1AnalyticsAdaptiveGroups, by databaseRole
primary reads 15-28K/hr -> sustained 37-46K/hr at 01:00Z
D1's primary is single-threaded. Roughly ten queries a second is about 36K an hour, so we were sitting above the ceiling. Everything downstream, the refusals, the replica page, the empty metrics, is one queue backing up.
The 50 seconds that named it
wrangler tail aiapplyd-prod-web --format json
Fifty seconds of live requests. 187 of them. What was in there:
- 155 of 187 from one hosting provider's VPS range, across eight or more addresses, geolocated to a single country.
- Plain desktop Chrome user agents, rotating across five major versions plus a macOS build. No bot user agent anywhere. Nothing in
robots.txtwould have stopped it, because it never asked. - 152 requests to
/jobs/{id}and 29 to/companies/{slug}. Nothing else.
That last line is the whole diagnosis.
Not one CSS file. Not one JS chunk. No font. No favicon.
A browser cannot render a page without them. A human cannot browse without a browser. So whatever this was, it was reading HTML and discarding it, at a rate of roughly four pages a second, walking two URL families in ID order.
The rotating user agent is what makes this worth writing down. It is a deliberate signal that the client wants to look human. The asset requests are the signal it forgot to fake, and they are free to check because they are already in your logs.
Why caching was not the answer
The first three suggestions in the room were all cache-shaped: longer TTL, more edge caching, a stale-while-revalidate window.
None of them help. The two families being walked hold hundreds of thousands of distinct IDs. Every request is a unique URL, on a first visit, at a rate the origin cannot absorb. A cache that never gets a second request is an extra hop. You cannot cache your way out of an enumeration.
What does work, in order of how quickly it can be deployed:
- Rate limit by ASN or by IP prefix, not by user agent. The user agent is attacker-controlled. The network path is not.
- Serve the enumerable families cheaply. If they are not indexable anyway, they do not need a database read per request.
- Verified-bot checks are not free. On our plan, the verified-bot field is enterprise-only. We reached for it, we could not have it, and knowing that in advance would have saved twenty minutes.
The two instrumentation lessons
Tail the worker that is actually being hit. Our API worker sees requests from our own frontend worker, which forwards without the original user agent. Tailing the API would have shown a wall of identical internal requests and nothing else. The client's fingerprint only exists at the true front door. Name the process that receives the traffic, then tail that one.
And check the alarm's window before you believe it. One of these alarms re-fired the next morning at full severity for an incident that had been fixed the previous afternoon, because it computed its rate over a rolling 24 hours. The burst was inside the window. The verdict was stale.
A rate wants a wide window. A verdict wants a recency gate. If you only have one window, you get paged for your own fixes, and the day you stop trusting the page is the day it is real.
I build AI Applyd. We scrape job listings ourselves, on a strict budget and only from sources that allow it, which is exactly why I recognise the shape of a client that does not read your CSS.
Top comments (8)
Hi Ava,😊
Your article is absolutely brilliant! The "asset-request ratio" concept is pure gold, and it actually helped me solve a mystery that had been worrying me for days.
I run purely static HTML/JS websites, nothing complex. Recently, I panicked when my daily visitors dropped from 400 to 140. I couldn't figure out why—the site was still running perfectly, no changes, no errors.
Then I read your post and decided to check my DDoS mitigation logs. And there it was:
┌─────────────────────────────────────┐
│ DDoS Mitigation Stats │
├─────────────────────────────────────┤
│ Allowed: 458 🟢 │
│ Blocked: 2.0k 🔴 │
│ Challenged: 3.6k ⚠️ │
│ Rate Limited: - 📊 │
└─────────────────────────────────────┘
Over 3,800 blocked requests from AWS IPs blindly hunting for
.env,phpinfo, and.gitfiles—treating my static site like a high-value target!It made me laugh out loud: it's like a thief bringing heavy-duty explosives to crack a bank vault, only to realize they are standing in front of a wooden shack! 😂
Your article helped me completely reframe this situation—not as "lost traffic," but as a "successful filtration of digital noise." I wasn't losing visitors; I was finally seeing the bot traffic that had always been there.
Quick question for you: Have you noticed similar blind scanning patterns from AWS IPs, or do you mostly encounter scraping bots specifically targeting dynamic content?
Thank you for providing the mental framework to understand bot behavior. It gave me total peace of mind and a really good laugh!
🧊🌊🗻
This made my morning, and the wooden shack line is going to live in my head for a while.
To your question: both, and they look completely different once you know what to compare.
The blind vulnerability scan is what you caught. It does not care what your site is. It asks for .env, .git/config, phpinfo, wp-login, a handful of admin paths, gets a 404 on every one, and leaves. It never requests a stylesheet or an image, because it is not rendering anything. That is the asset-request ratio doing its job.
Targeted content scraping looks almost the opposite, and it is the one that actually cost us. Ours walked a specific URL family in order, in the small hours, at a steady rate, requesting only the HTML and nothing else. No 404s at all, because every URL it asked for was real. It read as ordinary traffic on every dashboard we had, and we only found it because the database load had a shape that human traffic does not have: perfectly flat, all night, on pages nobody clicks.
So the tell for the first one is a high 404 rate with zero asset requests. The tell for the second is a low 404 rate with zero asset requests. The zero is the common part, and it is the one signal neither of them can fake without paying for a real browser.
And your reframe is the right one. You did not lose 260 visitors. You found out they were never visitors.
Thank you for this detailed clarification!
The distinction between blind vulnerability scanners and targeted content scrapers is incredibly useful. I hadn't thought about looking at the database load shape as a signal—that's a brilliant observation.
Your article gave me both insight and motivation.
This entire conversation has been an enjoyable learning experience. Thank you for sharing your creativity. I'll definitely be applying some of these lessons to my own projects.
Wishing you continued success and more groundbreaking achievements!
Best regards,
Glad it was useful, and the wooden shack is still the best description of that traffic I have read.
On your question: nearly all of what we see is the blind kind, the same as yours. One path per request, .env, .git/config, wp-login, phpinfo, arriving in bursts from cloud ranges, and none of it cares what the site actually is. A targeted scraper looks completely different. It requests real pages in a sensible order and it holds a session, which is why the asset ratio catches it and does not catch the blind ones.
The practical split is what it costs you. A blind scanner costs log noise and nothing else, so filtering it is the whole fix. A targeted scraper costs real database work, so if you ever need to tell them apart in a hurry, look at load shape rather than request count. The blind ones spike your request graph and leave your database flat.
Prefix-wide limits need a criterion or you'll blackhole a whole cloud. Two things worked for us: only trust a proxy/VPN verdict on datacenter IPs when two independent reputation sources agree, since a single source flags entire AWS and Hetzner ranges as proxy. Then grade the /24 by share of already-flagged neighbours. Under 10%, limit the IP, not the block.
That's a fair hit on what I wrote. I said rate limit by ASN or prefix and gave no criterion, and you're right that's how you end up blackholing Hetzner for everyone who happens to be on it.
Requiring two independent reputation sources to agree is the part I hadn't thought about. We were about to trust a single verdict, and single-source datacenter flags are exactly as noisy as you describe.
The /24 neighbour share is nice because it degrades gracefully instead of flipping. Under 10% you're limiting one tenant, over it you're probably looking at a range genuinely being used for this.
One thing from our side that composes with yours and costs nothing: the asset-request ratio per IP over a short window. A real browser pulls CSS, JS and fonts alongside the HTML. A client that has taken 200 HTML responses and zero assets doesn't need a reputation lookup at all. It works well as a pre-filter, so you only spend the paid reputation calls on the ambiguous middle.
Did you find the neighbour grading needed recomputing often, or is it stable enough to run daily?
Daily holds. Bands are coarse (10%, 30%), so small drift rarely flips a tier.
Daily plus coarse bands is the answer I was hoping for, because it means the whole thing is a cron job and a lookup rather than anything live on the request path.
Thanks for coming back with the actual numbers. Between the two of us this thread now has a complete recipe: asset-request ratio as the free pre-filter, two independent reputation sources before trusting a datacenter verdict, then a daily-recomputed /24 neighbour share with 10 and 30 percent bands deciding whether you limit the address or the block.
That's better than what either of us walked in with.