A bot enumerated every payment URL on our site and our logs never saw it

LinkBook — Tue, 15 Sep 2026 08:22:36 +0000

We migrated a customer panel's nginx container last week for an unrelated SEO fix (a trailing-slash redirect bug), rebuilt it, and redeployed. Routine stuff.

A few days later I was pulling GA4 numbers to check on organic traffic and noticed something odd: roughly a quarter of our "Direct" channel sessions were landing on /app/customer/login?returnTo=/payment/<plan>/<period> for a huge spread of plan slugs - not two or three, but nearly every pricing tier across every product we sell. Same browser fingerprint every time: (not set), desktop, Germany. Zero engagement on all of them.

That's not a customer. That's a script walking through every checkout URL it could find, probably scraped off our own pricing page.

I went to pull the nginx access log to find the source IP and got nothing. The container had been rebuilt days earlier for that unrelated SEO fix, and access_log was pointed at a file inside the container (/var/log/nginx/access.log), not stdout. The rebuild wiped it. docker logs showed nothing because there was nothing on stdout to show. Our Promtail/Loki pipeline, which only scrapes the Docker logging driver, had never seen a single line of it either.

The bot's IP is gone. There's no way to get it back now.

The actual fix was two lines:

access_log /dev/stdout main;
error_log /dev/stderr warn;

nginx treats /dev/stdout and /dev/stderr as regular files it can write to, so this isn't some special integration, it just means the same log lines that used to sit in a file now go through the container's stdout/stderr, which docker logs already reads and which our existing Promtail config was already scraping. Nothing else changed.

The lesson wasn't really about the bot. Low-rate GET enumeration of public pricing URLs isn't a very interesting threat on its own. It was that we had a monitoring pipeline that looked complete (Promtail, Loki, Grafana, the works) with one container quietly opting out of it, because its base nginx config still defaulted to writing log files the way it would on a bare-metal box from ten years ago. If this container gets rebuilt again next month, whatever it's doing then, we'll actually be able to see it.

If you're running nginx in Docker and haven't checked, it's worth a look at your own access_log/error_log directives. A monitoring stack is only as complete as the containers that actually write to it.

The Docker Compose Restart Policy Gotcha That Cost Us Two Weeks of Silent Downtime

LinkBook — Sun, 30 Aug 2026 10:26:50 +0000

We run a dozen+ services in plain docker compose on a single box — no Swarm, no Kubernetes, just docker-compose.prod.yml and restart: always on everything. Or so I thought.

What actually happened

A host reboot (unrelated maintenance) brought everything back up except three containers. Nobody noticed for two weeks, because the services that stayed down were dev-only instances and one internal monitoring stack — nothing customer-facing paged anyone.

I found it by accident, running docker ps -a for something unrelated:

docker ps -a --format '{{.Names}}\t{{.Status}}' | grep -iE "exited"

Three containers, all Exited (0) from two weeks ago.

The root cause

docker compose down (run once, on purpose, weeks earlier, for an unrelated fix) stops and removes containers. When they got recreated on the next up -d, a few of them had drifted from the rest of the fleet: no explicit restart policy in their compose block, which means Docker defaults to restart: "no".

restart: "no" means exactly what it says — if the container exits for any reason (a clean down, a host reboot, an OOM kill), Docker will never bring it back. Not on daemon restart, not ever, until someone runs up -d again by hand.

The other ~90% of our services had restart: always explicitly set, so a reboot was invisible for them. These three were added later by someone (me) who forgot the line.

The one-liner that would've caught it immediately

for c in $(docker ps -aq); do
  policy=$(docker inspect --format '{{.HostConfig.RestartPolicy.Name}}' $c)
  [ "$policy" = "no" ] && echo "$(docker inspect --format '{{.Name}}' $c): $policy"
done

Now it's a step in our monthly infra audit, not something we find by accident.

The second half of this problem: zero-downtime redeploys

This same fleet has another sharp edge worth knowing about if you're running plain compose in production: docker compose up -d <service> for a redeploy briefly stops the old container before the new one is healthy — a few seconds of 502s behind Traefik/nginx.

We ended up writing a small blue-green script instead of reaching for Swarm/K8s just for this one problem:

start a second container alongside the live one (docker compose run -d --no-deps --name <container>_green <service> — has to be run, not up, since up always reconciles to one container per service)
poll its healthcheck until healthy
remove the old container, rename the new one into its place, restore restart: always
on healthcheck timeout, remove the new container and leave the old one untouched — a safe no-op, not a rollback of something that already happened

Script's public if useful: https://github.com/Fanpino/deploy-bluegreen