DEV Community: ogurexxx

Five outages in three days: finding where my server's connection breaks

ogurexxx — Wed, 11 Nov 2026 16:48:45 +0000

In three days the site dropped off the network five times, the longest for five hours. The server itself ran without a single error the whole time: no reboots, no crashes, stats kept coming in. Only the road from it to the internet kept disappearing. Here is how I tracked that down and what I changed so that next time the answer is ready on its own.

The setup

A small server that runs my portfolio site and an SSH honeypot for bots. The site lives in Docker behind a reverse proxy; next to it a watchdog made of bash and systemd timers checks every ten minutes that everything works and sends a push notification if something breaks. An external monitor checks the site from outside every five minutes.

"0 problems" while the site is down

I was not the first to learn about the outages: an external monitor was. It emails me when the site stops answering. At the same time the server's own check was writing "0 problems" to its log every ten minutes.

It was not wrong. It checks what is inside: whether the data is fresh, containers and services are running, there is disk space left. Inside, everything was fine. That is the first lesson of this story: a check from inside sees the server, not the path to it. A visitor does not care that the processes are alive if nobody can reach them.

What the logs showed

First I ruled out the obvious: the server had not rebooted in six days, no container had restarted, the kernel log had no out-of-memory or disk errors. Docker, however, kept logging the same error at the right minutes: it could not ask DNS for the address of anything, from the certificate authority to my own domains. Timeout, timeout, timeout.

To see where the outages begin and end, it is enough to bucket these errors into ten-minute slots:

journalctl -u docker --since today -o short-iso --no-pager | grep 'failed to query external DNS' | cut -c1-15 | uniq -c

The picture came out neat: silence, a half-hour burst, silence again, then an hour and a half of solid errors. The bursts matched the monitor's emails to the minute. The server could neither reach outside nor accept connections from outside, so the whole link was gone.

Teaching the watchdog to notice a drop

The first fix is obvious: also check the outside connection. Now the watchdog asks DNS and tries to connect to a couple of public addresses every ten minutes.

The tricky part is elsewhere: while the link is down there is no way to report it, a push simply will not leave the server. So the watchdog remembers when the first check failed, and when the link comes back it sends one message, "offline from … to …", and prepares a draft write-up by itself. The day turns red on the uptime bar right away, without waiting for me to write the text.

A false alarm

Two days later the watchdog caught three drops. I was about to write them up, but first decided to check whether visitors had actually noticed. The best witness here is the site's own access log: if external requests kept arriving, the site was reachable, whatever the watchdog thought.

python3 -c '
import sys, json, datetime, ipaddress, collections
c = collections.Counter()
for line in sys.stdin:
    e = json.loads(line)
    if ipaddress.ip_address(e["request"]["remote_ip"]).is_global:
        c[datetime.datetime.fromtimestamp(e["ts"]).strftime("%H:%M")[:4] + "0"] += 1
print(sorted(c.items()))' < access.log

The script only counts requests from public addresses: internal ones, such as checks from the local network, do not count.

Two drops were confirmed: not a single request came in from outside during those hours. The first one, ten minutes long, turned out to be false. Only DNS was failing, and visitors were getting in just fine. The watchdog saw that the server could not reach outside and concluded that the site was down.

That is the second lesson: "the server cannot reach outside" and "the site is down" are different events, and both need checking. Now, before writing a draft, the watchdog looks at the access log itself. If requests came in during at least half of the gap, it only sends a push saying the site was reachable all along, and the uptime bar stays green.

Where exactly it breaks

What was left was to find out whose side the drop is on. That takes two addresses: the gateway the server uses to reach the network, and the first provider hop behind it. The gateway comes from the routing table, and the first provider hop can be found with a simple trick: send a ping with a time-to-live of 2.

ping -n -c1 -t 2 1.1.1.1

The packet dies at the second hop, and that hop answers "Time to live exceeded", giving away its address. While the link is up, the watchdog remembers both addresses once a day. During an outage it pings them every ten minutes and writes a verdict into the draft:

the gateway did not answer: the fault is on our side;
the gateway answered but the provider hop did not: the line to the provider failed;
both answered: the problem is further out in the provider's network, or DNS only.

With a verdict like that you can go to support not with "it sometimes doesn't work" but with exact times and a location.

An outside view that works during an outage

The final touch: a status page that opens even when the site is down. It lives on GitHub Pages, checks the site every five minutes from GitHub Actions and sits on the subdomain status.ogurexxx.ru.

There was a trap here too. The site has HSTS enabled for all subdomains: the browser opens them only over HTTPS with a valid certificate and does not let you click through a warning. Until the certificate for the new subdomain is issued, the page simply does not open, even if DNS is set up correctly. And some monitoring services only issue a certificate for your own domain on a paid plan.

How much downtime can I afford

After this story the incidents page got a reliability table: number of outages, total downtime, mean time to recover (MTTR) and mean time between failures (MTBF). And an error budget, SRE-style. The target is 99.5% availability per month, which is about 3 hours 40 minutes of downtime.

November burned through that budget in its first week: the two outages on 5 November alone took more than five hours. It is an unpleasant number, but a useful one. While budget is left, I can experiment freely. Once it is gone, there is one priority, finding the cause of the drops, and new features wait.

What I took away

A check from inside and a check from outside answer different questions. You need both.
During an outage the server cannot report it by itself. So the report has to arrive after recovery, with the facts: how long it lasted and where it broke.
Before writing a write-up, ask a witness the outage did not touch. The access log honestly showed which drop was real.
Five drops in three days is no longer chance but a fault. Now I have the data to find it.

Write-ups of each outage are on the Incidents page, the current state is at status.ogurexxx.ru.

This post was first published in Russian on Habr.

An SSH honeypot on my own server: 24,000 sessions in five days and one very stubborn bot

ogurexxx — Wed, 11 Nov 2026 16:39:56 +0000

I set up an SSH honeypot on my server: it pretends to be a defenceless Linux box, accepts almost any password and records everything done inside. There is no real system behind it. In the first five days it received 24,218 sessions from 683 addresses, and almost nine in ten came from a single bot.

I wanted to see attacks on my own server rather than in reports and textbooks: who shows up, what they type, and how one attack differs from another. Other companies' statistics answer "how many"; a honeypot shows "how".

The setup

The honeypot lives next to my portfolio site. It is Cowrie in an unprivileged, read-only container with every outbound connection blocked: whatever an attacker types, nothing leaves the box. Next to it run a tarpit, a port that pretends to be SSH and never finishes its greeting, and Redis, MySQL and Elasticsearch decoys. Incoming web traffic is filtered by CrowdSec.

Every five minutes the logs are turned into summaries: commands are cleaned (links, addresses and random file names become placeholders such as <url> and <file>), and attacker addresses are never published. Everything below is visible live on the site.

Found quickly, and the obvious gets tried first

A new session typically arrives about five seconds after the last one. The server was never advertised: it is found by scanners that constantly sweep the whole internet.

The most common passwords are predictable: 123456, 123, 1234, 12345678. Besides root, the usernames include ubuntu, admin, deploy, postgres and also claude: bots are already looking for accounts of developers who use AI tools. The conclusion is simple: password login on a public server does not last long.

One bot made almost 90% of the noise

The most interesting part was the distribution. 21,217 sessions, or 88%, came from a single family using 18 addresses. It logs in and runs one command:

uname -s -v -n -r -m

Then it leaves. This does not look like a breach so much as an inventory: the bot checks whether the password worked and what system is inside. I cannot prove it, but the behaviour fits collecting a list of accessible servers to return to later or hand to someone else.

That gave the first practical lesson: counting sessions is useless. One noisy bot drowns out everything else, so I rebuilt the experiments to count addresses instead of sessions.

The most widespread family leaves a key

The family with the most addresses, 291, does something different: it removes the protection from the .ssh folder, deletes it, recreates it and writes its own key. The whole thing takes seconds:

cd ~ && rm -rf .ssh && mkdir .ssh && echo <key> >> .ssh/authorized_keys

Deleting the old folder matters: on a real server it would lock out both the owner and other bots. And changing the password afterwards does not help, because the key stays. So when you suspect a breach, check not only passwords but also the authorized_keys file, cron jobs and the user list.

What bots left in the honeypot

The honeypot may not make outbound connections, so it cannot download anything. But bots upload files themselves: 22 files have piled up so far, 17 of them Linux programs for several architectures, including x86-64, ARM and AArch64. Several carry signs of XMRig, a cryptocurrency miner. The files are only examined as data and never run. The goal of most attacks is boring: someone else's CPU.

The tarpit: how much time you can steal from bots

The tarpit follows RFC 4253: before the banner a server may send any lines it likes, and the client has to wait. It sends one random line every ten seconds and holds the connection for up to a day. All that time the bot attacks nobody.

A week after launch the counter showed over 2,000 hours stolen from attackers and over 3,400 connections caught. The record is 24 hours, because that is as long as the tarpit holds anyone. If each of those bots had tried one password per second, that would be about 7.5 million password guesses that never happened. At any moment a couple of dozen bots are stuck in it.

How half my defences stayed silent for two days

The most useful part turned out not to be about bots at all. For two days the tarpit and decoy counters sat at zero. The service was running, the firewall rule was in place, and its counter was even growing. A local check passed.

tcpdump helped: requests were arriving, but no replies went out. In the end a second firewall turned up. Permission in one does not override a block in the other: a packet has to pass both. The first firewall's counter said all was well, and that was only half true.

The conclusion: a check from inside does not prove a service is reachable from outside. A firewall counter does not prove it either: it only speaks for its own part of the path. The most reliable answer came from a simple experiment: a real connection from outside while watching the packets.

What's next

The next question is whether bots pay attention to what they are dealing with. Every day the honeypot changes its disguise: the server name and the system version. In a few weeks it will be clear whether bots react or do not care. If they do not care, that is a result too.

The honeypot and the site also carry bait keys: a fake .env with an API key unique to everyone who downloads it. If someone tries the key, I will see how many minutes passed since it was stolen and whether it came from the same address. That gets its own post once there is data.

Live honeypot data is on the live page, every botnet family with its lineage is in the bestiary, and you can check your own auth.log against these bots right in the browser: the log never leaves your machine.

This post was first published in Russian on Habr.