I've been "the computer guy" my whole life. The one you call when the Wi-Fi is haunted. But I never wrote code until this year, when AI assistants closed the gap and I built a movie-discovery site (flickomatic.com).
Current user count: me and my wife. So imagine my delight when the weekly hosting bill hit $82. And climbing. Here's what I found when I went digging. Four culprits, none of which I'd have guessed.
An impossible number
The biggest line item: "ISR Writes: 8.69M ($34.75)." ISR is the scheme where a page renders once and gets served from cache for a week. My site has about 2,000 pages. There's no arithmetic where that becomes 8.69 million.
Except there is, one sentence deep in the pricing docs: writes are billed in 8 KB chunks. That "8.69M writes" was really 70 GB of payload shoveled into the cache. New question: shoveled by whom?
1. Bots were inventing infinite versions of my pages
My URLs look like /movie/603/the-matrix, but it turns out the title part was decorative. /movie/603/absolutely-anything rendered the same page, and each variant got cached as a brand-new entry at ~13 billed writes a pop. No human would ever notice. Crawlers, which hoard URLs in every variant they've ever seen, noticed constantly.
I now know the word "canonicalization": wrong URL gets a permanent redirect to the one true URL. One cached page per movie. Novel concept.
2. I was archiving movie posters by accident
My link-preview cards fetched each movie poster with a "cached fetch," which sounds responsible. What it actually does is save the entire JPEG into the billed cache. About 19 writes per poster, thousands of movies, read back approximately never. A very expensive museum nobody visits. One changed line killed a third of the problem.
3. My site's biggest customer was my site
The traffic dashboard: homepage 58.7K requests/day, internal API 58.1K. Suspiciously identical. Top visitor identity: something called "node." At, you guessed it, 58.1K.
"node" is my own server. The homepage loaded its movie list by making a full HTTP request to its own API, once per visit, cached under a key that included every filter combination. My site was DDoSing itself. Politely. On a schedule. And I was paying for both directions.
If your function invocations outnumber your page views, go find yourself in your own logs before blaming outsiders.
4. My domain has a past life
Heavy bot traffic kept hitting /blog/1, /blog/2, pages I never built. Bing's dashboard explained it: my domain was first "discovered" in January 2010. I bought it this year. Someone ran a blog here fifteen years ago and crawlers are still knocking on that dead door. They now get HTTP 410, "Gone," a status code I didn't know existed and now love.
The bot zoo
"Bot traffic" turned out to be four different animals:
- Search and AI-assistant crawlers (Googlebot, Bingbot, the crawlers behind Claude and ChatGPT search): how anyone might ever FIND my site. Welcome guests. The goal was making their visits cheap, never blocking them.
- AI training scrapers: bulk-download everything, send nothing back. Asked to leave via robots.txt.
- SEO-tool crawlers: AhrefsBot alone was a THIRD of my traffic. It indexes your site to sell reports to marketers. Asked to leave; complied within hours. Most courteous freeloader on the internet.
- A scraper farm on Alibaba Cloud, 51K requests/day, claiming to be Chrome on Windows and also Chrome on a Mac. Pick a lane. Every request shared one identical TLS fingerprint (a "JA4"). Real devices differ at that layer; 51K identical handshakes is one program in costumes. The firewall can challenge by fingerprint: humans pass, scripts fail. Gone in 60 seconds. Most satisfying minute of the month.
Results, two days later
- Cache writes: $2.30/day down to $0.75, still falling
- Data transfer: 65 GB/day down to 8
- Compute: down 75%
- Every crawler I actually want: untouched
Back to hobby money. Site still wide open to the bots that matter.
What the computer guy learned
- Read what the billing UNITS mean. One sentence explained an impossible number.
- Every URL your site will answer is a tiny liability. I was offering the internet infinite URLs. Free for them, anyway.
- Your own server is a visitor too. It wears a name tag that says "node."
- Domains have exes. Check what they left behind.
- Before changing anything, write down what number should move, and by how much, if your theory is right. I borrowed that habit from people far more experienced than me. It's the difference between fixing something and poking it.
Full disclosure: I didn't untangle this alone. I build with an AI coding assistant. I bring the dashboards and make the calls; it brings the patience to explain ISR billing to me twice. Twenty-five years of being everyone's tech guy, and it turns out what I was missing wasn't aptitude. It was a collaborator who never gets tired of my questions.
The site all this drama was protecting: https://flickomatic.com. Feedback welcome. I'm told that's how you get better at this.
Top comments (2)
I really appreciated how you broke down the unexpected costs of your hosting bill into specific issues, especially the part about bots inventing infinite versions of your pages. I've had similar issues with crawlers and SEO tools hitting my site, and I've found that implementing a robust canonicalization strategy and using tools like Google Search Console can help mitigate these problems. One thing I'm curious about is how you handled the process of identifying and blocking the scraper farm on Alibaba Cloud - did you use a specific tool or service to detect the identical TLS fingerprints, or was it a manual process?
OP here. This is my first project. It was something fun that I was looking for... a better movie search engine, not something I think will solve the worlds problems.