DEV Community

Christian Anderson
Christian Anderson

Posted on

Why I built warden: fail2ban can't see attackers behind a Cloudflare tunnel

I already wrote up what warden does and the bugs I hit building it in Homelab security on 512 MB. This one is about why it exists at all. I didn't set out to write a security tool. I kept finding gaps in the setup I already had, and warden is what was left after I filled them.

The address you want to ban isn't on the wire

My public services sit behind a Cloudflare tunnel. That's good for lots of reasons, but it breaks the usual homelab reflex of "see a bad IP, drop it in iptables".

Behind the tunnel, the source address at the network layer is cloudflared or the Cloudflare edge. The attacker's real address only exists in the HTTP layer, in a header the proxy passes along. So a local firewall ban keyed on the attacker never matches a single packet. And if you ban what does show up at L3, you've banned Cloudflare and cut off your own estate.

I wrote that down before writing any code, and it's why warden has no iptables code in it. Any ban has to happen at the Cloudflare edge, before traffic reaches the tunnel. Cloudflare's own edge ranges sit on warden's allowlist from the start, next to my own addresses.

Request path from attacker through the Cloudflare edge, tunnel and reverse proxy to the app. The real client IP only exists in an HTTP header, so a local iptables ban never matches; the ban has to happen at the edge

CrowdSec was running. It just couldn't see everything.

I'd had CrowdSec in place since June, sending its local bans to Cloudflare. On paper that was covered. When I looked properly in August, it wasn't.

The web tier was basically unwatched. My reverse proxy doesn't write the standard nginx log format, so when I mounted its log as nginx, every line landed in "unparsed". That's coverage that looks real and detects nothing. I wrote a custom parser for it, but the bigger hole was the tunnel itself. Some hostnames go straight from the tunnel to a service and skip the proxy completely, and nothing was reading cloudflared's log. That's where I found someone probing one of those direct routes for a Vite file-read bug, against a service that wasn't even running any more.

So warden started on 30 August as a small, dependency-free second opinion to CrowdSec. It reads the logs CrowdSec can't see or doesn't parse (the proxy log, my SSO's log, and the tunnel log) and scores what it finds. CrowdSec keeps running alongside it.

Why it asks before it acts

CrowdSec bans automatically. My sync pushed its local bans to the edge every 60 seconds. That's fine until you look at who is getting flagged.

The only local CrowdSec alert in one September week was my own WAN IP: the Immich app on my phone asking for 11 thumbnails I'd deleted, all within four seconds. Warden agreed with it: 27 of the 29 app-layer events that week came from that same address. On 7 October, 281 of the 295 edge events I pulled were me as well. On 10 October it was 1,526 of 1,577 events in 24 hours, until I filtered my own rows out of the counts.

The loudest "attacker" on my network is me. A detector you don't fully trust, pointed at traffic that's mostly yours, shouldn't be holding the ban button.

I'd also seen what automation does with bad readings. My security daemon once restarted CrowdSec every ten minutes for a day. CrowdSec was never down. An SSH timeout came back empty and got read as "service dead". Warden's own first ingest put weeks of July history inside the live ban window, which in enforcing mode would have banned people for July (the details are in the other post).

So warden ships detect-only. It records what it would ban and changes nothing. When I want something blocked at the edge, it proposes it in Discord and I tap ✅ or ❌. Approved bans go into one WAF rule and lapse after 30 days. A ❌ is remembered for 90 days so it doesn't keep asking. It refuses to propose my own addresses, private ranges or Cloudflare's. Patching works the same way: it shows me the upgrade plan first, takes a snapshot, and never reboots a Proxmox node on its own.

One place, on a small box, with nothing installed

The other reason was sprawl. Logs in one tool, CVEs nowhere, patches done by hand, and no real list of what was on my network. When I needed a SIEM layer in September I chose to extend warden instead of deploying Loki or Wazuh.

Everything it does is agentless. Vulnerability scanning copies the package list off each box and checks it against OSV.dev, the open database that Debian, Ubuntu, Alpine, GitHub and the Go team publish their advisories into. For Docker it reads what's actually in each running container, not just the image it came from. Integrity checks are one read-only shell collector per box. Nothing gets installed on the things being watched. I measured the core at 1 vCPU, 512 MB of RAM and 300 MB of disk.

It used to run Trivy for the vulnerability side, and that was the weak point: a 161 MB binary, a server process, a 1.4 GB database, and a project whose own release got backdoored this year (CVE-2026-33634). So I swapped it out on 10 October. On my Proxmox node the new scanner found 680 of the 691 CVEs Trivy found, both agreed nothing was fixable, and the odd ones out were the two databases updating at different times. It also binned two "actively exploited" Trivy alerts for a package one container doesn't even run any more.

None of the pieces are new. Wazuh, CrowdSec, Vuls, Security Onion and others all exist, and warden leans on OSV.dev's data, Lynis and Suricata for the hard parts. What I couldn't find was that combination: agentless, small, and asking in chat before it does anything. It's AGPL-3.0 since 8 October.

Agents read it, they don't decide

I run Hermes agents at home, and the obvious temptation is to let one of them do the security work. I don't. Warden has a read-only MCP server, and my sec agent connects to it as a reader: summaries, detections, devices, CVEs. My IT admin agent includes warden's facts in its daily report, and its rules say it reports and never approves. Scripts gather the facts and I make the call. The agent is there to explain things, not to decide them.

The newest bit: the house itself

The newest gap was the house. Warden knew about servers, but not the bulbs, plugs and TVs. On 10 October it gained a device inventory that merges ARP sweeps with the router's DHCP leases: 107 real devices from 133 addresses, with 5 left as honest unknowns rather than guesses.

Then I gave each device a DNS baseline from the router's query log. Replaying a week of history gave 1.17 million lookups across 85 devices. Some devices are very predictable. One WiZ bulb has talked only to wiz.world for nine days. A smart lock bridge talks only to its maker.

That made scored decisions possible instead of "count over threshold". When a fixed-function device that's past its 7-day learning period hits a domain it has never used, warden starts at 40 points and adjusts. Plus 25 if no other device uses that domain, minus 25 if two or more do, minus 30 if the same kind of device all changed together, minus 20 if the maker's name is in the domain, plus 15 for 00:00 to 05:00, plus 15 for a risky TLD. Under 25 it just learns it, 25 to 54 gets logged, and 55 or more asks me.

Replaying the real week produced zero reviews. A simulated WiZ bulb calling update-check.xyz at 3am scored 95 and showed up for review.

A simulated WiZ bulb looking up update-check.xyz at 3am: base 40, plus 25 because nothing else uses it, plus 15 for the hour, plus 15 for the TLD, 95 in total, well into the review band

It's still limited. Device scoring only sees DNS, my IoT VLAN has no devices on it yet, and the edge ban path is brand new: three bans approved so far, all on 10 October. But each part exists because something I already ran was missing it, not because a feature list said so.

If you run services behind a Cloudflare tunnel, how do you deal with bans when the address you'd block only exists in a header?


🤖 Drafted with AI assistance from my own homelab notes, logs and repos, then reviewed and edited before publishing.

Top comments (0)