DEV Community: Daanial Farrukh

Postfix in a Chroot Cannot See Your MySQL Socket — Why Mail Stopped Both Ways

Daanial Farrukh — Sun, 08 Nov 2026 17:21:12 +0000

A field note about a self-hosted mail server where nothing moved in or out for weeks — the one master.cf column almost nobody reads — and the four checks I now run before touching a single password.

The first sign was not an error. It was silence.

No bounce. No angry reply. No “message could not be delivered”. Just a mailbox that had stopped receiving anything, and outgoing mail that never arrived anywhere.

The server was a self-hosted grommunio box on Ubuntu 24.04. Postfix in front, MySQL behind it holding the mailboxes and aliases. A normal, modern stack.

I opened the queue.

postqueue -p

It was full. The oldest message was about six weeks old.

And the part that changed the whole investigation: some of the stuck messages were from one mailbox on the domain to another mailbox on the same domain. Same server. No internet in the path. No remote MX, no blacklist, no DNS.

If internal mail is stuck, the problem is not out there. It is inside the box.

That’s the trap. A broken mail server usually shouts. This one only deferred — politely, forever.

What the Log Actually Said

The mail log had the same pair of lines over and over:

tail -n 50 /var/log/mail.log

postfix/cleanup: warning: mysql:/etc/postfix/grommunio-virtual-mailbox-alias-maps.cf lookup error
... virtual_alias_maps map lookup problem ... message not accepted, try again later

Read that second line slowly. “Try again later” is a temporary failure. Postfix did not reject the mail — it asked every sender to come back. Remote servers obeyed and retried. Local submissions sat in the queue and retried. Nobody got a bounce, because a temporary failure is not a bounce until the queue lifetime runs out.

That is why it stayed quiet for weeks.

The first guess almost everyone makes from this log is the database: wrong password in the .cf file, MySQL down, the grommunio user lost its grant. So I tested exactly that — the same map, the same lookup, by hand:

postmap -q user@example.com mysql:/etc/postfix/grommunio-virtual-mailbox-alias-maps.cf

It answered. Instantly. Correct result.

Here’s the part that trips people up. The lookup works from the shell and fails inside Postfix. Same file, same credentials, same database. When the same lookup works for you and fails for the daemon, the problem is not the lookup — it is where the daemon is standing.

What a Postfix Chroot Actually Is

A chroot is a jail: the process is told that one directory is the whole filesystem, and it cannot see anything outside it.

For Postfix, that directory is the queue directory, usually /var/spool/postfix. When a service in master.cf is chrooted, a path like /run/mysqld/mysqld.sock is really read as /var/spool/postfix/run/mysqld/mysqld.sock. That file does not exist. So the MySQL client cannot connect, and every mysql: map lookup fails.

The flow, inside the jail:

cleanup → mysql: map → host = localhost → Unix socket /run/mysqld/mysqld.sock → not found inside /var/spool/postfix → lookup error → "try again later"

And the same flow from your root shell:

postmap -q → mysql: map → host = localhost → /run/mysqld/mysqld.sock → found → answer

The cleanup daemon matters more than people think. It runs on every message, inbound, outbound and local, and it is where virtual_alias_maps is expanded. Break it and you break mail in every direction at once — which is exactly what the queue showed.

Where this setup is common:

Debian and Ubuntu style master.cf files, where many services are marked chrooted.
Mail stacks that keep users and aliases in MySQL or MariaDB: grommunio, Postfix admin panels, hand-built virtual mailbox servers.
Servers rebuilt or upgraded where master.cf came from a package and the MySQL maps were added later.

Why Nobody Sees It in Testing

Three reasons, and they stack.

First, every manual test runs outside the jail. postmap -q, the mysql client, a PHP admin page — all of them see /run/mysqld/mysqld.sock and all of them pass. The test you trust is the one that cannot fail.

Second, the failure is a deferral. No bounce reaches a user, no alert fires on a queue that is “only” growing, and senders’ servers keep retrying quietly.

Third, the fix that looks obvious makes it worse. From the log, the natural move is to edit the .cf map and change hosts = localhost to hosts = 127.0.0.1 — “use TCP instead of the socket”. On this server that is a new failure, not a fix. The MySQL client treats localhost as “use the Unix socket” and 127.0.0.1 as a TCP connection, and MySQL matches accounts by host. The grommunio database user was granted on localhost. Change the host and the login itself stops matching. You have swapped a jail problem for an access-denied problem.

How to Actually Fix This

Everything up to Step 4 is read-only.

Step 1: Confirm mail is deferred, not rejected.

postqueue -p | tail -n 1

The last line gives the count of queued requests and their size. Then look at the oldest entries at the top of the full list. If messages are days or weeks old and include local-to-local mail, stop looking at DNS and remote servers.

Step 2: Read the exact error.

grep -E "lookup error|map lookup problem" /var/log/mail.log | tail -n 5

You are looking for mysql: named in the warning and map lookup problem with try again later. The map file named in the warning is the one to test next.

Step 3: Run the same lookup as root, outside Postfix.

postmap -q user@example.com mysql:/etc/postfix/grommunio-virtual-mailbox-alias-maps.cf

Use an address you know exists. If it returns a result, the credentials, the grant and the database are fine. This is the step that turns “database problem” into “Postfix environment problem”.

If it fails here too, you do have a database or credentials problem, and the rest of this article is not your fix.

Step 4: Read the chroot column in master.cf.

grep -E '^(cleanup|smtp\s|smtpd|pickup)' /etc/postfix/master.cf

The columns are: service, type, private, unpriv, chroot, wakeup, maxproc, command. The fifth column is the one that matters. A y there means that service runs inside /var/spool/postfix.

Postfix can also print the value for every service in one go:

postconf -F '*/*/chroot'

On the broken server, cleanup, smtpd and smtp were all y. That plus a working postmap -q is the proof.

Step 5 (changes state): Turn chroot off for all services.

postconf -F '*/*/chroot=n'

This edits master.cf for you — no hand editing, no missed line. Then restart Postfix and flush the queue:

systemctl restart postfix

postqueue -f

postqueue -f asks Postfix to retry everything in the queue now instead of waiting for the next retry interval.

Step 6: Verify.

This step is where most tutorials stop and where mine starts.

postconf -F '*/*/chroot'

Every line must now end in = n.

tail -f /var/log/mail.log

You should see the old messages leave: status=sent lines for remote delivery, local delivery lines for internal mail, and no new lookup error warnings. Then watch the queue drain:

postqueue -p | tail -n 1

What success looks like: the count drops, and finally Postfix reports Mail queue is empty. What a failure looks like: the count stays flat and the lookup error lines keep appearing — in that case the restart did not happen, or a service still carries y.

Last, send one fresh test message from one local mailbox to another and one to an outside address. The old queue draining proves the fix; a new message proves the server is healthy now.

On this server, mail flowed again in both directions right after the restart.

Why Not Keep the Chroot?

A fair question. A chroot is a security feature, and turning it off is a trade-off, not a free win.

You can keep it, but then the socket has to exist inside the jail — for example by making MySQL’s socket available under /var/spool/postfix/run/mysqld/. That works until a reboot, a package update or a systemd change removes it, and then you are back to a silent queue. And moving the maps to TCP needs matching database grants for the TCP host, which, as above, is exactly what this server did not have.

For a mail server whose users and aliases live in a database, I prefer the boring setup that cannot quietly break: no chroot on the Postfix services, the database socket where the client expects it, and the grants left alone. Postfix still drops privileges and runs its daemons as an unprivileged user; the chroot is one extra layer, not the only one.

The Checklist I Run Now

When a mail server goes quiet, in this order:

Is anything queued, and how old is the oldest message? postqueue -p
Is local-to-local mail stuck too? If yes, the cause is inside the box. Read the sender and recipient in the queue list.
Which map is failing? grep "lookup error" /var/log/mail.log
Does the same lookup work outside Postfix? postmap -q <address> <map>
Which services are chrooted? postconf -F '//chroot'
After any fix: does the queue drain to empty? postqueue -p | tail -n 1

Plain takeaway: a lookup that works as root and fails in the daemon is an environment problem, not a credentials problem.

What I Tell Every Customer Now

Whenever someone runs their own mail on Postfix with a database behind it, I ask one question in the review:

“Who gets told when the oldest message in the queue is more than an hour old?”

If the answer is a named person and a working alert, they’re in good shape. If the answer is “users would complain”, we’re not done yet — because in this failure, nobody complains. The mail just waits.

I write about the questions I ask in real customer rooms — the ones that change architectures and surface the problems no one else noticed. If you found this useful, drop your email below and I’ll send the next one when it’s ready.

FAQs

Why does postmap -q work but Postfix says “map lookup problem”?

postmap runs in your shell, outside any chroot, so it can reach the MySQL socket at /run/mysqld/mysqld.sock. A chrooted Postfix daemon such as cleanup looks for the same path inside /var/spool/postfix, where it does not exist. Same file and credentials, different view of the filesystem.

What does “virtual_alias_maps map lookup problem … try again later” mean?

Postfix could not complete a lookup in the virtual alias map, so it refused the message with a temporary error instead of guessing. Senders retry and local mail stays queued, so nothing bounces for days. The fix is to make the lookup work again, then flush the queue.

How do I check if Postfix services run in a chroot?

Run postconf -F ‘//chroot’ to print the chroot setting for every service, or read the fifth column of each line in /etc/postfix/master.cf. A y means that service is chrooted to the Postfix queue directory.

Should I change hosts = localhost to 127.0.0.1 in the MySQL map files?

Not as a fix for this problem. With the MySQL client, localhost means the Unix socket and 127.0.0.1 means TCP, and MySQL accounts are matched by host. If the mail user is granted only on localhost, the change makes the login fail.

Is it safe to turn off chroot for Postfix?

It removes one layer of isolation, so it is a trade-off. Postfix still runs its daemons as an unprivileged user. For a database-backed mail server, a setup that cannot silently lose access to the database socket is usually the safer choice in practice.

How do I get the stuck mail delivered after the fix?

Restart Postfix so the new master.cf is read, then run postqueue -f to retry every queued message now. Watch /var/log/mail.log for delivery lines and run postqueue -p until it reports that the mail queue is empty.

Download the PDF

Originally published at quabin.com.

Dozens of VPN Credentials in a Leak — What to Do in the First 24 Hours

Daanial Farrukh — Sat, 07 Nov 2026 17:14:22 +0000

A field note about a threat report that said "no evidence of valid credentials" — the one sentence almost nobody challenges — and the order of work I now put on the table the moment a VPN domain shows up in a leak.

I was reading a threat-intelligence appraisal.

It was a careful document. Sources, dates, a clean summary. It covered one of the large credential leaks of this year and it ended with a calm conclusion: "no evidence of valid or active credentials."

That sentence is the one people forward to their managers.

So I asked one question. Has anybody searched for the domain?

I searched. The lookup took less than a minute.

The organisation's own domain was there — dozens of SSL-VPN accounts, listed against its own edge firewalls. Not a staff member reusing a password on a shopping site. Real VPN portals, on the organisation's own public ranges, several on non-standard ports and a few on plain HTTP.

"No evidence" had only meant nobody had looked.

That's the trap. A leak doesn't show up as an alert. Your firewall is healthy, your patch level is fine, your monitoring is green — and someone else is holding the password.

What a Credential Leak Actually Is

A credential leak is not a vulnerability. It is a list. Someone already has the username and the password, and the attacker's next step is an ordinary login.

The public example this year was FortiBleed. Researchers reported a dataset covering about 74,000 FortiGate firewall URLs across 194 countries, and public analysis confirmed that sampled logins were real. Fortinet's position is that this was not a new vulnerability — the data came from earlier incidents and from brute-force attacks. That detail matters more than the headline number.

It means the path looks like this:

Old theft → dataset → password spray or direct login → a normal-looking VPN session

Nothing in that flow is an exploit. Your IPS has nothing to match. The login succeeds, and the log line looks exactly like Monday morning.

This pattern is common in three places:

SSL-VPN portals with local accounts and no MFA
Firewall management pages that were left open to the internet
Directory bind accounts stored in the firewall configuration, which rarely change

Why the Usual Response Misses It

The usual response is one word: patch.

Patching is necessary. It is not sufficient. Here's why.

A patch closes a door. It does not change a key that was copied last year. If the password was taken before the patch, it still works after the patch. Public guidance on this leak says the same thing: if your domain or IP appears, assume compromise, and treat the upgrade as one step among several.

There is a second reason. Fortinet documented an attacker technique where a symbolic link was planted in the folder that serves SSL-VPN language files, and that link could survive a firmware upgrade. So an upgrade can leave you with a patched box that is still not clean.

And there is a third reason, the one that surprised me most. The date of the leak is not the date of the theft. In the case I was reading, the most likely source was not the firewall at all. It was a Windows server on the same network — logged in daily as the built-in Administrator, no antivirus, passwords saved in the browser. An infostealer had taken about a hundred saved logins, a handful of them corporate, years before the leak made the news.

That is still a theory until the cleartext list is compared, and I'd rather say so. But it changes the question. "Did we change our passwords after the news?" is the wrong test. The right test is: was this password changed after the date it was stolen?

How to Actually Fix This — The First 24 Hours

This is the order I recommend, and the order matters. The commands are FortiOS because that was the case in front of me. The sequence works on any VPN gateway.

Commands marked as read-only change nothing. Run those first.

Step 1: Hour 0 — confirm, then freeze the evidence. (Read-only.)

Search your domains and public IPs in a reputable lookup, such as the public checker Hudson Rock published for this dataset. Then take a configuration backup and export the logs before you change anything. In the next steps you will be removing the very things you want to study.

Record the firmware on every listed device:

get system status

Plain takeaway: a listing is a reason to start, not proof of a breach. It is enough to act on.

Step 2: Hour 1 — see who is connected, then cut them off.

List the active SSL-VPN users. (Read-only.)

get vpn ssl monitor

Read every line. Look for a source IP you can't explain, an account of someone on leave, a service account with an interactive session. Then remove the tunnels you do not trust. (This changes state.)

execute vpn sslvpn del-tunnel <index>

Cut sessions first and rotate second. A live session does not always care that the password changed underneath it.

Step 3: Hours 1–6 — rotate everything the firewall trusts. (This changes state.)

Most teams change the VPN user passwords and stop. That is about a third of the job. The real list is:

Local administrator accounts on every firewall
Local SSL-VPN users
LDAP and RADIUS bind accounts and shared secrets
API tokens and SNMP strings
Any password that was reused anywhere else

Item 3 is the one that hurts. A directory bind account stored in a firewall configuration is often years old, rarely reviewed, and very useful to an attacker. Treat the whole configuration as disclosed.

Check the bind account's age on the directory side. (Read-only.)

Get-ADUser <vpn-bind-account> -Properties PasswordLastSet,LastLogonDate

If PasswordLastSet is older than the suspected theft, that password is still in play — no matter what the news cycle says.

One more detail from the public advice: have each administrator log in after the rotation so the firewall can re-hash the password with PBKDF2. Older hashes are the ones that crack.

Step 4: Hours 2–12 — hunt for what the login left behind. (Read-only.)

Rotation locks out the password. It does not remove an account the attacker created with it.

List every administrator:

show system admin

List every local VPN user:

show user local

List the local-in policies and the automation stitches:

show firewall local-in-policy

show system automation-stitch

Fortinet's published incident write-ups name accounts like audit, backup, itadmin, secadmin and support as ones attackers have created. A plausible name is the whole point. Do not ask whether the name looks suspicious. Ask who created it and on what date.

Then read the login history. Filter the event log for logins:

execute log filter category 1

execute log filter field action login

execute log display

You are looking for a source that has never appeared before, a success after a burst of failures, and a dormant account that suddenly works. Then search the same window for configuration downloads and for changes to trusted hosts, MFA, logging or DNS.

Here's the part that trips people up. Log memory on a small firewall is short. If logs are not forwarded off the box, "nothing in the logs" may only mean the logs have rolled over. That is the appraisal's mistake again.

If you find a rogue account or an unexplained change, stop cleaning. Restore a known-clean configuration or rebuild the device, and escalate it as an incident.

Step 5: Hours 6–18 — close the door the password opened. (This changes state.)

First, patch to a current release. Then remove the exposure:

Take firewall management off the internet. Researcher Kevin Beaumont noted in his analysis of this leak that management was exposed on most of the affected devices.
Restrict each administrator to known sources:

config system admin
edit <admin-name>
set trusthost1 203.0.113.10 255.255.255.255
end

Enforce MFA on every admin and VPN account, with no exceptions for service desks or "just this one director".
Force HTTPS. A VPN portal on plain HTTP sends its login in the clear.

This is the same pattern as any admin lockdown: the port should answer only the people who need it.

Step 6: Hours 12–24 — find the machine the passwords came from.

This is the step everyone skips, because the firewall team has gone home feeling finished.

If the source was an infostealer, the firewall is only where the stolen password was used. The infected host has other secrets. Ask four questions:

Which machine had these logins saved in a browser?
Is it still in service? If so, reimage it. Do not clean it.
Is it used daily as Administrator? Stop that.
Does it run endpoint protection that someone actually watches?

Then widen the rotation to every login on that machine — mail, control panels, personal accounts that share a pattern. A password scheme with one changing word is one password.

Step 7: Verify — the step that proves any of this happened.

This step is where most tutorials stop and where mine starts.

Check three things. First, list the bind account again:

Get-ADUser <vpn-bind-account> -Properties PasswordLastSet

The date must be today's. Second, run show system admin and match every line to a named person. Third, from outside your network, try to reach the management page. It must time out.

curl -k -m 10 https://203.0.113.20:8443/

Expected result: a timeout. If you get a login page, step 5 didn't take.

Then do the check nobody enjoys. Have one user try to connect with a password but without the second factor. It must fail.

A refused login here is not a fault. It is the verification.

The Bigger Lesson

Three things from this case have stayed with me.

"No evidence" is not a finding. It is a statement about how hard you looked. Before you accept it, ask what was searched, and for which domains.

A patch is a date, not a state. It tells you when the door closed. It says nothing about who already has a key.

The firewall is rarely where the story begins. One unprotected server used for daily work can feed a VPN, a mail system and a control panel at the same time.

The Checklist I Run Now

Domains and public IPs searched in a leak lookup — by a person, with a date
Active sessions listed and untrusted ones cut: get vpn ssl monitor
Admin, VPN, bind, RADIUS, API and SNMP secrets rotated, not just user passwords
PasswordLastSet newer than the suspected theft date
Every administrator matched to a named person: show system admin
Management closed from outside, proven with curl from an outside network
MFA proven by a failed login without the second factor
Logs forwarded off the box, so the next review has a history to read

Plain takeaway: rotate first, hunt second, and do not close the ticket until you have found the source.

What I Tell Every Customer Now

When a report lands saying the organisation is not affected, I ask one question:

"Who searched for our domain in the dataset, on what date, and which of our passwords are older than the theft?"

If the answer is a name, a date and a list, they're in good shape. If the answer is "the report says we're fine", we're not done yet.

FAQs

What should I do first if my VPN credentials appear in a leak?

Confirm the listing, back up the configuration and export the logs, then list active VPN sessions and cut any you cannot explain. Do this before rotating passwords. Assume the listed accounts are compromised until the passwords have been changed.

Does patching the firewall fix a credential leak?

No. A patch closes a software flaw, but a stolen password still works after the upgrade. You must also rotate credentials, check for rogue accounts and enforce MFA. Fortinet has also documented an attacker technique that could survive a firmware upgrade, so a patched box is not automatically clean.

Which passwords need to change after a firewall credential leak?

All of them that the firewall holds or trusts: local administrators, local VPN users, LDAP and RADIUS bind accounts, shared secrets, API tokens and SNMP strings. Also change any password reused elsewhere. Treat the whole configuration as disclosed, because a backup of it may have been taken.

How do I know if a leaked VPN password was actually used?

Read the login history for new source addresses, successes after a burst of failures and dormant accounts that suddenly log in. Then check for new administrators, changed trusted hosts and configuration downloads. If logs were not forwarded off the firewall, an empty result proves little.

Why would old passwords appear in a leak from this year?

The date a dataset is published is not the date the passwords were stolen. An infostealer on one machine can capture saved browser logins years before they appear in a public dataset. That is why the useful test is whether a password was changed after the theft, not after the news.

Is it enough to enable MFA after a VPN credential leak?

It is one of the strongest steps, but it does not replace rotation. Cut existing sessions, rotate the passwords and look for rogue accounts as well, because an attacker who already got in may have created a way back. Then prove MFA works by trying a login without the second factor.

Download the PDF: https://quabin.com/wp-content/uploads/2026/11/quabin-vpn-credentials-leak-first-24-hours.pdf

Originally published at quabin.com.

WHM Is IP-Locked, So Why Can't I Reach It? — Cloudflare Proxied Hostnames vs Origin IPs

Daanial Farrukh — Fri, 06 Nov 2026 17:17:31 +0000

A field note about the day my own firewall rule locked me out of WHM — a rule that was correct, on a hostname that was orange — and the read-only checks I now run before I touch a single admin allowlist.

This one has no customer quote at the start, because the person who got it wrong was me.

A cPanel server was sitting behind a distributed firewall, the kind that is enforced outside the VM. The job was simple and sensible: stop the world from reaching the admin ports. SSH on 22 and WHM on 2086 and 2087 would answer one source only — my admin IP.

So I wrote the rules the way you are supposed to write them. Allow from the admin IP. Then an explicit drop from any, placed above the catch-all allow at the bottom of the policy. The firewall reads top-down and the first match wins, so the order was doing real work.

SSH still worked. WHM did not.

My IP had not changed. The rule was enabled. The order was right. I had just locked myself out with a rule I could read line by line and find nothing wrong with.

There was nothing wrong with it.

That's the trap. An IP allowlist does not check who you are. It checks who connected last. And when your hostname is orange in Cloudflare, the last one to connect was not you.

What Actually Happens to a Request on Port 2087

Most people think of Cloudflare as a web thing. Port 80, port 443, done. It isn't.

Cloudflare's proxy also listens on a published list of other ports, and that list includes the whole cPanel family:

2082 and 2083 — cPanel
2086 and 2087 — WHM
2095 and 2096 — Webmail

So a proxied hostname does not fail on 2087. It works, and that is the problem. Your browser reaches Cloudflare, and Cloudflare opens a brand-new TCP connection to your origin:

Admin laptop → Cloudflare edge → Firewall → WHM

There are two connections here, not one. The first one is yours. The second one belongs to Cloudflare. The firewall in front of the origin only ever sees the second one. Its source address is a Cloudflare address, so the walk down the policy goes like this:

Source = Cloudflare IP → Allow admin IP? No → Drop from any? Yes → Dropped

The rule did exactly what I told it to do. It dropped a connection from a stranger. The stranger just happened to be carrying my request.

Here is the part that trips people up: Cloudflare does pass your real address along, in the CF-Connecting-IP header. That is no help at this layer. A network firewall decides on the TCP source address before any HTTP header exists. The header is inside a connection the firewall never lets start.

Why SSH Tells You Nothing — and Why That Helps

Port 22 is not on Cloudflare's normal proxy list. So SSH to the origin IP goes straight to the server, and the firewall sees your real address. (Point SSH at an orange hostname and it simply fails, because Cloudflare's normal proxy does not carry it. That is a different symptom.)

That difference misleads you first. "SSH works, so my IP is allowed, so the firewall is fine." You go looking in WHM, in cPHulk, in the service status.

Then it helps you. Same machine, same firewall, same allowlist, same laptop. One port works and the other does not. The rule cannot be the difference, because both ports use the same rule logic. The difference is the path. When SSH works and WHM does not, stop reading the firewall and start asking which IP WHM's traffic is really coming from.

Why You Never See This in Testing

Three things hide it.

First, the rule is usually tested with the wrong URL. Many admins test against the raw IP while building the rule, it passes, and then their bookmark goes to the friendly hostname from then on.

Second, the hostname may have turned orange long after the firewall was built. Somebody importing a zone into Cloudflare, or proxying everything to be safe, can flip it months later. The firewall did not change. The DNS did.

Third, the error points away from the cause. A dropped connection gets no answer, so a browser going through Cloudflare should show Cloudflare's own 522 connection-timed-out page, not anything from your firewall or from cPanel. If the firewall rejected instead of dropped, the same mistake would normally show up as a 521. Either way, the page tells you the origin is unwell. The origin is fine.

How to Actually Fix This

Everything up to Step 3 is read-only.

Step 1: Find out what your hostname really resolves to.

dig +short server.example.com

If you get your origin IP back, the hostname is grey and this article is not your problem. If you get addresses that are not yours, usually starting 104. or 172.67., you are proxied. Compare against the list Cloudflare publishes:

curl -s https://www.cloudflare.com/ips-v4

Step 2: Ask the port who answered.

curl -skI https://server.example.com:2087/

Look at the server: line. If it says cloudflare and there is a cf-ray: header, then whatever you are talking to on 2087, it is not your WHM. A status of 522 here is the whole diagnosis in one line.

Step 3: Bypass Cloudflare for one request and compare.

This forces curl to the origin IP while keeping the correct hostname for TLS. It changes nothing anywhere.

curl -skI --resolve server.example.com:2087:203.0.113.10 https://server.example.com:2087/

Run it from the admin IP. Now the server line should name cpsrvd, which is cPanel's own service. Same hostname, same port, same laptop — and it answers. That is your proof: the rule allows you, and only the proxied path is blocked.

If this one fails too, the firewall or the service really is the problem, and you should go back to the rule. That result is useful as well.

Step 4: Make the admin hostname grey. (This changes state.)

In Cloudflare, go to DNS → Records, find the server hostname, and set Proxy status to DNS only. Leave the public websites orange. They need the proxy and they are not the ones being locked.

This matters even if you plan to browse to the IP. When I open the origin IP on port 2087, WHM redirects me to the server hostname on 2087. If that hostname is orange, the redirect walks me straight back into Cloudflare and into the drop rule. The IP gets you to the door. The grey hostname keeps you there.

After the change, wait for the old TTL to expire, then repeat Step 1. You should see the origin IP. Then bookmark the direct path:

https://203.0.113.10:2087

Step 5: Verify from both sides.

This step is where most tutorials stop and where mine starts. Working for you proves half of it. The other half is failing for everyone else.

From the admin IP:

curl -skI https://server.example.com:2087/

Expect the cpsrvd server line and no cf-ray header.

Then, from anywhere else — a phone hotspot is fine:

curl -skI --connect-timeout 10 https://203.0.113.10:2087/

Expect a timeout and nothing else. curl exits with code 28.

This error is not a bug. It is the verification. If that second command returns a login page, your lock is not locking.

On the server itself, one more read-only line shows which address WHM thinks you are:

tail -n 5 /usr/local/cpanel/logs/access_log

The first field of each line is the client. It should be your admin IP.

One warning from the way this firewall works: when the drop happens outside the VM, a capture inside the VM shows nothing at all. An empty tcpdump on port 2087 does not mean no traffic was sent. It means the traffic died before it arrived. Use the hit counters on the firewall rule for that view.

The Fix You Must Never Make

Sooner or later somebody says it: "Just add Cloudflare's IP ranges to the WHM allow rule."

It will work. WHM will load. Please don't.

Those ranges are not your ranges. They are shared by every Cloudflare customer on the planet. Anyone can add a domain to a Cloudflare account, create a proxied record pointing at your origin IP, and send their browser through it on 2087. The connection then arrives from an address you allowed. You did not narrow your admin port. You opened it to the world with one extra step.

The same goes for SSH. Whichever port you lock, the allow rule names your addresses, and no one else's.

If your admin IP changes often, the answer is a VPN or a jump host with a fixed address. If you truly want WHM behind Cloudflare, that is a different design: Cloudflare Tunnel with Access in front of it, and nothing listening to the public internet. It is a good design. It is not a one-line allow rule.

The Honest Trade-Off

A grey hostname publishes your origin IP in public DNS. Some people will object to that, and they are not wrong to care.

On a cPanel server the mail hostnames usually publish that IP anyway, because mail cannot go through Cloudflare's normal proxy. Hiding the origin was never going to be the control here. The control is the drop rule. A published IP with a closed port is safe. A hidden IP with an open port is only safe until someone finds it.

The Checklist I Run Now

Is the admin hostname grey? dig +short server.example.com
Who answers on the admin port? curl -skI https://server.example.com:2087/
Does the bypass work from the admin IP? curl -skI --resolve server.example.com:2087:203.0.113.10 https://server.example.com:2087/
Does the origin refuse a stranger? curl -skI --connect-timeout 10 https://203.0.113.10:2087/ from another network.
Is the drop rule above the catch-all allow? Read the policy top-down, and check the hit counter is rising.
Is 443 a side door? cPanel's proxy subdomains can carry whm.example.com over ordinary 80 and 443, which a rule on 2087 does not touch. grep proxysubdomains /var/cpanel/cpanel.config
Is anything else on the same list? Check 2082, 2083, 2086, 2095 and 2096 the same way, on every hostname that points at the box.

Plain takeaway: lock the port, then prove it from outside, every time.

What I Tell Every Customer Now

When someone tells me their admin panel is locked to one IP, I don't ask to see the rule. I ask this:

"From a network that is not on the list, what does port 2087 on your origin IP do — and is your admin hostname orange or grey?"

If the answer is "it times out, and it's grey," they're in good shape. If the answer is "I think so, the rule is in there," we're not done yet.

FAQs

Why can't I reach WHM after locking it to my IP in the firewall?

If your WHM hostname is proxied through Cloudflare, the origin firewall sees a Cloudflare address, not yours. The allow rule misses and the drop rule fires. Check with dig +short on the hostname: if it does not return your origin IP, set the record to DNS only in Cloudflare or reach WHM through the origin IP.

Does Cloudflare proxy port 2087?

Yes. Cloudflare's proxy supports a published list of HTTP and HTTPS ports, and it includes the cPanel and WHM ports 2082, 2083, 2086, 2087, 2095 and 2096. A request to an orange-cloud hostname on 2087 reaches Cloudflare first, and Cloudflare opens its own connection to your server.

Should I allow Cloudflare IP ranges to reach WHM?

No. Those ranges are shared by every Cloudflare customer, so anyone can create a proxied record pointing at your origin and reach your WHM login from an allowed address. Keep WHM limited to your own admin IP, and use a VPN, a jump host or Cloudflare Tunnel with Access if your address changes.

Why does SSH work when WHM does not?

SSH on port 22 is not carried by Cloudflare's normal proxy, so a session to the origin IP arrives directly and the firewall sees your real address. That makes the mismatch a useful clue: the same allowlist passes one port and blocks the other, so the rule is not the cause. The path taken by WHM traffic is.

Why do I get a Cloudflare 522 error on port 2087?

A 522 means Cloudflare could not complete a TCP connection to your origin. On an IP-locked server, that usually means your firewall is silently dropping Cloudflare's connection, which is what the rule was written to do. Your server is healthy; only the proxied path is blocked.

Does a DNS-only hostname expose my server's IP?

Yes, the origin IP becomes visible in public DNS. On a cPanel server, the mail hostnames usually show it anyway. Protection should come from the firewall rule that closes the admin ports, not from keeping the address secret.

Download the PDF: https://quabin.com/wp-content/uploads/2026/11/quabin-whm-ip-locked-cloudflare-proxied-hostname.pdf

Originally published at quabin.com.

FortiGate Upgrade Dropped LAN-to-LAN Traffic — allow-traffic-redirect and the Implicit Deny Nobody Planned For

Daanial Farrukh — Fri, 06 Nov 2026 09:25:19 +0000

A war story about a firewall upgrade that finished cleanly and cut two LANs apart — traffic that had never had a policy, never written a log line, and never been on anyone’s test list — and the checks I now run before and after any firmware change.

The firmware upgrade was done at night. It finished. The firewall came back. The team sent the usual note asking the customer to verify their services.

The customer found the problem the next morning. Virtual machines on one of their LANs could not reach virtual machines on the other. Packet loss between them was 100 percent, and the system they called their key system was down.

The fix itself was small — one explicit policy, in place by early afternoon. The conversation was harder.

“Whose lapse is this?” their lead asked in the group. “And what measures will follow?”

Two explanations came back from the provider’s side. Monitoring could not validate LAN-to-LAN traffic. And the customer should have flagged the problem after the verification note.

Neither one answered the question. The customer rejected both, and they were right to.

I accepted it as our lapse, and I apologised. Then I went looking for the question we should have asked before the change window opened:

“What on this firewall is working without a rule?”

The Trap, Named

That’s the trap. A firewall review reads the policy table, and a change plan tests what the policy table says. But a flow that passes with no policy does not appear in that table at all. It has no rule to read, no owner to ask, and — as you will see — often no log line to find.

The most dangerous traffic on your firewall is the traffic nobody ever had to allow.

What Actually Happened to the Traffic

The customer had two LANs behind the same firewall interface. There was no LAN-to-LAN policy, because nobody had ever needed one. The old firmware passed the traffic. The new firmware dropped it under the implicit deny.

The path between the two LANs looks like this:

VM on LAN A → firewall interface → route lookup says “out the same interface” → VM on LAN B

Traffic that arrives on an interface and leaves on the same interface is called hairpin traffic. On a FortiGate, what happens next is decided by one global setting, allow-traffic-redirect.

enable — same-interface traffic can pass on the routing decision alone, with no matching policy.
disable — it needs a firewall policy whose source and destination interface are the same. Without one, implicit deny drops it.

Here’s the part that trips people up. That default changed. Fortinet’s FortiOS 7.4.10 release notes list it under “Changes in default behavior”: the default moved from enable to disable, for IPv4 and for IPv6. Fortinet’s technical tip gives the same change for 7.6.5 and 8.0.0 onward. And the release note says the upgrade sets it to disable even if it was enabled before.

So a box can be fully healthy, with every rule intact, and lose a whole class of traffic in the reboot.

I want to be exact about what is proven. The symptom, the old behaviour, the new behaviour and the fix are proven in this case. The specific firmware versions are not in my notes, so I cannot prove this setting was the mechanism on that box. It is the leading explanation, because it matches on every point. The first check in this article is the one that confirms it on yours.

This shape is common. You will find it wherever:

Two subnets share one interface, usually through a secondary IP address.
A router or layer 3 switch sits on the same LAN as the firewall, and the firewall is the default gateway that hands traffic back.
A site grew a second LAN years ago and someone added an IP address instead of an interface.

Why Nobody Sees It in Testing

Three things hide this flow, and they stack.

It has no rule. A policy review cannot find what is not there.

It often has no log. Fortinet’s tip notes that traffic logs are generated only when traffic passes a firewall policy. Hairpin traffic passing without a policy leaves no trail in the traffic log, so a “top talkers” report will not show the dependency either.

It is not on the monitor. Most monitoring checks the firewall itself, its uplinks and its published services. All of those were green. The traffic between two groups of a customer’s own VMs was never a monitored item, and “firewall is up” says nothing about it.

So the post-upgrade check passed, honestly, on every item it contained. The list was the failure, not the box.

How to Actually Fix This

Steps 1 to 3 are read-only. Step 4 changes state, so do it in a window. I have used documentation addresses here: 192.0.2.0/24 for LAN A, 198.51.100.0/24 for LAN B, and port2 for the shared interface.

Step 1: Read the setting and the version.

get system status

get system global | grep allow-traffic

If you are on 7.4.10, 7.6.5, 8.0.0 or later, expect allow-traffic-redirect: disable. On a box still below those versions, it may say enable — and that box is the one to worry about, because it may depend on this today.

Step 2: Prove the drop with a flow trace. Start a ping from a VM on LAN A to a VM on LAN B, then run this on the firewall. Keep the count small; this is a production box.

diagnose debug flow filter addr 192.0.2.10

diagnose debug flow show function-name enable

diagnose debug enable

diagnose debug flow trace start 20

When you are done, switch it off:

diagnose debug disable

The line you are looking for is this:

msg="Denied by forward policy check (policy 0)"

Policy 0 is the implicit deny. This is not a bug in the firmware. It is the firewall doing exactly what the new default tells it to do.

Step 3: Find the dependency before you upgrade. On a box that has not been upgraded yet, the log will not help you, so look at the wire. Start a fresh ping between the two LANs and run:

diagnose sniffer packet port2 'net 192.0.2.0/24 and net 198.51.100.0/24' 4 20

The same packet going in on port2 and then out on port2 is hairpin traffic. Now read the policy table:

show firewall policy

Look for a policy whose srcintf and dstintf are the same. If the sniffer shows the traffic and the table has no such policy, that flow is alive on the old default alone. It will die in the upgrade.

Step 4: Write the policy (this changes state). Create the two address objects, then the policy. Set these up in a window, and take a config backup first.

config firewall address
    edit "LAN-A"
        set subnet 192.0.2.0 255.255.255.0
    next
    edit "LAN-B"
        set subnet 198.51.100.0 255.255.255.0
    next
end

config firewall policy
    edit 0
        set name "LAN-A-to-LAN-B"
        set srcintf "port2"
        set dstintf "port2"
        set srcaddr "LAN-A"
        set dstaddr "LAN-B"
        set action accept
        set schedule "always"
        set service "ALL"
        set logtraffic all
    next
end

Repeat it with source and destination swapped if sessions start from both sides. Then narrow service once the logs show you what really flows.

There is a one-line alternative: set allow-traffic-redirect enable under config system global. It restores the old behaviour and it will stop the outage in a second. I use it only as an emergency measure. Fortinet’s own article warns that it bypasses the policy check for hairpin flows, which leaves you exactly where you started — a production dependency with no rule, no log and no owner.

Plain takeaway: the setting hides the dependency; the policy documents it.

Step 5: Verify. This step is where most tutorials stop and where mine starts. Run the same ping, then the same trace. You should now see the policy id in the trace instead of policy 0. Then confirm the flow has its own trail: open Log & Report › Forward Traffic and filter on the policy name. You should see entries for LAN A to LAN B.

Now test real traffic, not just ping. If ping works but TCP sessions hang, Fortinet’s tip points at anti-replay on the same-interface policy: set anti-replay disable under that policy. Ping alone would have given you a green tick and a broken application.

The Checklist I Run Around Any Firewall Upgrade

Read “Changes in default behavior” in the release notes for every version on the path, not just the last one. This section exists so that you do not meet it in production.
Save the setting before and after. get system global | grep allow-traffic goes in the change record, both times.
Sniff for hairpin traffic on every interface that carries more than one subnet. Any flow without a same-interface policy gets a policy first, before the upgrade.
Ask the customer for east-west flows in writing. “Which of your servers talk to each other through this firewall?” is a different question from “which services do you publish?”
Give notice, and get a named person on their side. Ask them to test during the window, not the next morning.
Test it yourself after the reboot. A provider’s “please verify” note is not a verification. Keep one host in each LAN that can ping the other, and watch that path during the window.
Add east-west probes to monitoring. If the path matters, a monitor should measure the path, not the firewall that carries it.
Check the rest of the fleet. If one box has this shape, others built by the same people probably do. I have no proof of that for any specific firewall — that is exactly why you run the check.

Absence of errors is not health. A clean upgrade log proves the firmware installed. It does not prove the customer’s traffic still flows.

What I Tell Every Customer Now

Before any firewall change, I ask one question:

“Name the traffic between your own servers that would stop the business if it failed — and tell me who tests it, and when, after our change.”

If the answer is a list of flows and a name on each side, they’re in good shape. If the answer is “the firewall doesn’t touch that”, we’re not done yet.

FAQs

Why did LAN-to-LAN traffic stop after a FortiGate upgrade?

A leading reason is the changed default of allow-traffic-redirect. From FortiOS 7.4.10, 7.6.5 and 8.0.0, it is disable, and the upgrade sets it to disable even if it was enabled before. Two subnets on one interface then need an explicit policy with the same source and destination interface, or implicit deny drops the traffic.

What does “Denied by forward policy check (policy 0)” mean?

It means the traffic matched no firewall policy and was dropped by the implicit deny at the end of the policy table. You see it in a debug flow trace. For same-interface traffic after an upgrade, the fix is a policy with the same interface as source and destination.

Should I set allow-traffic-redirect back to enable?

Only as an emergency measure. It brings the traffic back quickly, but that traffic again passes without a policy check. An explicit same-interface policy is better, because it can be logged, inspected and owned by someone.

Do two VLANs on a FortiGate need this policy too?

Two VLAN interfaces are two different interfaces, so that traffic is not hairpin traffic and has always needed a normal policy between them. The setting matters when ingress and egress are the same interface, such as two subnets on one interface with a secondary IP address.

How can I find hairpin traffic before an upgrade if it is not logged?

Use the sniffer on the shared interface with a filter on the two subnets, and start a fresh ping or session. Packets going in and back out on the same interface are hairpin traffic. Then check whether a policy with matching source and destination interfaces exists.

Is the customer or the provider responsible for testing after a firewall upgrade?

Whoever makes the change owns the proof that it worked. The customer knows which flows matter, so collect that list before the window and agree a named tester. A note asking the customer to verify, sent after the work, moves the risk without removing it.

Download the PDF

Originally published at quabin.com.

Three Data Centres and No High Availability — Why a Second Site Without a Load Balancer Is Not Redundancy

Daanial Farrukh — Wed, 04 Nov 2026 17:28:56 +0000

A presales war story about an infrastructure review where three data centres looked like the safest design in the room — until I asked which box the traffic passes through first — and the four-layer check I now run before I let anyone say the word "redundant".

The slide had three buildings on it.

"We run three data centres. If one goes down, the others carry the load. We are fully redundant."

The audience nodded. It is a good slide. Three sites in different parts of the city, their own power, their own fibre, their own racks. An auditor had been happy with it. A customer's procurement team had been happy with it. Three is more than two, and two is already more than most.

I asked one question.

"When a visitor opens the portal, which device receives the packet first?"

"The load balancer."

"Where is the load balancer?"

Silence.

"Site A."

"And if Site A is the one that goes dark?"

Longer silence. Then the honest answer that every good engineer eventually gives.

"Then nothing reaches Site B or Site C. The load balancer is the only way in."

Three data centres. One front door. One key.

The Trap, Named

That's the trap. A single point of failure does not show up as a single point of failure when it sits in front of a lot of redundancy. It hides in plain sight, because every diagram draws the load balancer as a small box at the top and the three big data centres underneath, and the eye goes to the big boxes.

Redundancy is counted at the narrowest point of the path, not the widest. Three sites behind one load balancer give you exactly the availability of that one load balancer. Not more.

What High Availability Actually Requires

High availability is a property of a path, not of a building. For a request to survive the loss of a site, every hop it touches must have a living twin somewhere else, and the switch between the twins must happen without a human.

Draw the path and the problem is obvious:

Visitor → DNS → Load balancer → Web / app tier → Database

Each of those four layers has to span sites on its own terms:

DNS has to be able to point visitors somewhere else, or to point them at an address that is itself not tied to one site.
The load balancer has to exist in more than one place, or be a service that lives outside all your sites.
The web and app tier has to be running, warm and current in the second site, not "we can restore it from backup in four hours".
The database has to be replicated to the second site with a defined way to promote the replica, or the app tier there has nothing to talk to.

Lose any one layer and the other three do not matter. In the review I described, the provider had done the hardest part — the buildings — and skipped the cheapest part, the front door.

This pattern is common in three places:

Providers and enterprises that grew from one site to two or three, and kept the original load balancer because "it works".
Hardware load balancer deployments where a second appliance was quoted, judged expensive, and quietly dropped from the purchase order.
Virtual load balancers that were deployed as one VM in one cluster and never moved to an HA pair because nobody owned that line of the design.

Why Nobody Sees It in Testing

Failover tests are almost always run from inside. Someone powers off a web server, watches the load balancer mark it down, and sees the site stay up. That proves the load balancer works. It proves nothing about what happens when the load balancer is the thing that is gone.

The honest test — pull the uplink of the site that hosts the load balancer — is one almost nobody schedules, because it takes the whole platform down for the duration if the design is wrong. So the design is never proven wrong. It is simply believed.

Absence of a failed test is not evidence of a passing one.

There is a second reason the gap stays hidden. Monitoring usually runs from the same site as the load balancer. When that site dies, the monitor dies with it, and the dashboard that would have shown red is itself unreachable. The first alert comes from a customer.

How to Actually Check This

Before anyone promises redundancy, I run four read-only checks. None of them change anything. All of them fit in ten minutes.

Step 1: Find the single address everything resolves to.

dig +short www.example.com

If the answer is one IP address, and that address is a physical or virtual device in one building, you have found the front door. Write down the address and the site it lives in. If the answer is a set of Cloudflare or other anycast addresses, the front door is already outside your sites, and the question moves to how the origins behind it are configured.

Step 2: Ask where that address is announced from.

traceroute -n <that IP>

Read the last three hops. If they are routers in Site A every time, the address is reachable only through Site A. Some teams answer "we can re-announce it from Site B" — ask how long that takes and who does it at three in the morning. A manual re-announcement is a recovery procedure, not high availability.

Step 3: Count the load balancers.

On a Linux load balancer pair, this one command tells you if a virtual IP is in play at all:

ip addr show | grep -A1 "inet "

If the public-facing address appears as a secondary address on an interface and a keepalived or haproxy process is running, you are looking at one member of a pair. If the address is the only one on the box and there is no VRRP process, you are looking at a single device. Then check its twin:

systemctl status keepalived

No such unit, no twin.

Step 4: Ask the database the same question.

This is the layer teams forget. A second site with a load balancer and an app tier but a database that lives only in Site A is still a single site. Confirm replication is real and current:

mysql -e "SHOW REPLICA STATUS\G" | grep -E "Replica_IO_Running|Replica_SQL_Running|Seconds_Behind_Source"

Both Running values must be Yes, and the lag has to be a number you can live with. On older MySQL and MariaDB the command is SHOW SLAVE STATUS\G and the fields are Slave_IO_Running and Seconds_Behind_Master. If this returns nothing, there is no replica, and no amount of load-balancer work will give you a second site.

This step is where most tutorials stop and where mine starts. The checks above only tell you the shape of the problem. The fix is below.

How to Actually Fix the Front Door

There are two honest answers. Which one is right depends on whether you already use Cloudflare.

Option A — Move the load balancer outside your buildings (Cloudflare Load Balancing)

If the site is already proxied through Cloudflare, this is the fastest path, because it adds no hardware and no new network design. The provider in my story was already on Cloudflare for WAF and SSL, which is exactly why I recommended it.

The idea is simple. Cloudflare's edge becomes the load balancer. Each of your sites becomes an origin inside a pool. A health monitor probes every origin from the outside, and when Site A stops answering, traffic is steered to Site B and Site C with no device of yours in the path.

Visitor → Cloudflare edge (LB + health checks) → Origin in Site A | Site B | Site C

Step 1: Give each site its own public origin address. Each data centre needs an address that is reachable without passing through another data centre. If your three sites today share one address that lives in Site A, this step is the real work: you need a web or app entry point per site. It can be a small HAProxy or Nginx VM in each site, or a direct address on the app tier.

Step 2: Create a health monitor. Cloudflare → Traffic → Load Balancing → Manage Monitors → Create. Use an HTTPS probe to a path that touches the application, not just the web server — a /health endpoint that opens a database connection is far more honest than /. Set the expected code to 200.

Step 3: Create a pool per site, attach the monitor. Cloudflare → Traffic → Load Balancing → Manage Pools → Create. One pool per data centre, with that site's origin inside it. Attach the monitor and set a notification email, so a site going red tells a human even though failover is automatic.

Step 4: Create the load balancer on the hostname. Cloudflare → Traffic → Load Balancing → Create Load Balancer. Hostname www.example.com, pools in failover order, proxied on. Choose a steering policy — "Off" gives strict failover in pool order, which is the right starting point for an active–standby design. Save.

Step 5: Verify from outside — twice.

curl -sI https://www.example.com | head -5

A 200 and a cf-ray header prove you are reaching the application through Cloudflare. Now the test that nobody ran before: block the health-check path on Site A's origin for two minutes, or simply stop its web service, and run the same curl again. The response must still be 200, and Cloudflare → Traffic → Load Balancing must show Site A's pool as Unhealthy and the others Healthy. Bring Site A back and confirm the pool returns to Healthy.

This error is not a bug. It is the verification. If the curl fails while Site A is down, the second site is not yet an independent origin, and you have found the real work.

Option B — Keep it on-premises (keepalived + virtual IP)

If the traffic cannot leave your network — a private portal, a regulatory constraint, an internal API — the on-premises answer is a pair of load balancers sharing a virtual IP with VRRP, implemented by keepalived. Two HAProxy or Nginx nodes hold the public address between them. One is MASTER and answers; the other is BACKUP and takes the address within a few seconds when the master disappears.

Visitor → Virtual IP (VRRP) → HAProxy-1 (MASTER) | HAProxy-2 (BACKUP) → App tier

Step 1: Install on both nodes.

apt install keepalived haproxy

Step 2: Configure VRRP on the first node. This is the entire file for a minimal pair. Mark this as a change — it adds an IP address to the box.

vrrp_script chk_haproxy {
    script "/usr/bin/pgrep haproxy"
    interval 2
    weight -20
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 101
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass <SHARED_SECRET>
    }
    virtual_ipaddress {
        203.0.113.10/24
    }
    track_script {
        chk_haproxy
    }
}

On the second node, the same file with state BACKUP and priority 100. The track_script block is the part people leave out: without it, keepalived only fails over when the whole node dies, not when HAProxy itself has crashed while the node is still up.

Step 3: Start and verify the address is held.

systemctl enable --now keepalived

ip addr show eth0 | grep 203.0.113.10

The virtual IP must appear on the MASTER and not on the BACKUP.

Step 4: Fail it and watch. Stop HAProxy on the master and run the same ip addr show on the backup. The address must move within about three seconds. Start HAProxy again and it moves back. If it does not move, the track script is wrong or the two nodes cannot see each other's VRRP multicast — check that the switch and any firewall between them allow protocol 112.

A word of honesty about Option B: a VRRP pair protects you against the loss of one load balancer. It does not by itself protect you against the loss of a whole site, because both nodes normally sit on the same subnet in the same building. To span sites with a virtual IP you need a stretched layer 2 network or a routed design with BGP, and at that point Option A is usually cheaper and safer.

The Bigger Lesson — the Four-Layer Redundancy Checklist

Before the word "redundant" goes on a slide, each of these must have an answer and a command that proves it.

DNS. Can visitors be steered away from a dead site without a human editing a record? Check: dig +short www.example.com — is it a single site-bound address?
Load balancer. Is there a second one, somewhere else, that takes over on its own? Check: systemctl status keepalived on-prem, or Cloudflare → Traffic → Load Balancing → pools listed as Healthy.
App tier. Is the second site running the current release right now? Check: curl -sI https://<site-B-origin>/health — it must be 200 today, not "after a restore".
Database. Is the replica current and is promotion documented? Check: SHOW REPLICA STATUS\G — both threads Yes, lag acceptable.
Monitoring. Does at least one probe run from outside all your sites? Check: where does the monitor that would page you actually live?
The real test. Has the site that hosts the primary load balancer ever been taken offline on purpose, with customers watching? If not, the design is a belief.

Plain takeaway: three sites behind one door is one site with two very expensive spare rooms.

What I Tell Every Customer Now

In the next infrastructure review, ask one question:

"If the building with the load balancer loses power, what is the first thing a visitor sees — and who had to do anything to make that happen?"

If the answer is "the same page as always, and nobody", they're in good shape. If the answer is "we re-announce the address from the other site" or "we have never tried", we're not done yet, and the fix is a health check and a pool away.

FAQs

Is a second data centre enough for high availability?

No. A second data centre is a place where redundancy can live, not redundancy itself. The request path has four layers — DNS, load balancer, application tier and database — and each one has to exist and fail over in the second site on its own. If any one of them lives only in the first site, the second site cannot serve a single request when the first is down.

Why is a single load balancer a single point of failure even with three data centres?

Because every request has to pass through it before it reaches any of the three sites. Availability is set by the narrowest hop in the path, and a lone load balancer in one building is exactly as available as that building. Three sites behind it inherit its failure, not the other way round.

How does Cloudflare Load Balancing provide high availability across data centres?

Cloudflare runs the load balancer at its own edge, outside all of your sites. You define each data centre as an origin in a pool, attach a health monitor that probes it from the internet, and set a steering policy. When a site stops passing the health check, Cloudflare routes new requests to the remaining healthy pools automatically, with no device of yours in the path and no DNS change by a human.

What is keepalived and when should I use it instead of a cloud load balancer?

keepalived implements VRRP on Linux so two load balancer nodes share one virtual IP address. The backup node takes over the address within seconds when the master or its HAProxy process fails. Use it when traffic must stay inside your own network. Remember that a standard VRRP pair sits on one subnet in one site, so it protects against a dead load balancer, not a dead building.

How do I test whether my setup really fails over?

Take the primary load balancer, or the whole site hosting it, offline on purpose and run an external check at the same time. A plain curl -sI https://www.example.com from outside your network must still return 200 while the primary is down. If it does not, you have not tested failover before; you have only tested a healthy system.

Does the database need to be in both data centres too?

Yes. An application tier in the second site with no database to talk to is a web page that returns errors. The database must be replicated to the second site, the replica must be current, and there must be a documented, preferably automated, way to promote it. Check with SHOW REPLICA STATUS\G that both replication threads are running and the lag is acceptable.

Download the PDF

Originally published at quabin.com.

RPO Zero Is a Network Question, Not a Backup Question — What 'No Data Loss' Really Costs Between Two Sites

Daanial Farrukh — Wed, 04 Nov 2026 17:19:41 +0000

A presales war story about a revision request that read "RPO ≈ 0 and the lowest possible RTO" — the sentence everyone files under backup and almost nobody files under latency — and the four numbers I now collect before I promise any recovery point.

The first version of the proposal had been accepted in principle. Primary site in one city, a warm disaster-recovery site in another, the database replicated asynchronously, a recovery time of a few hours. Normal. Defensible. Then the revision request arrived.

"We now need RPO of approximately zero and the lowest RTO you can commit to. Please update the technical and financial proposals."

I joined the call with their database team already on it.

"Zero means zero," said their lead DBA. "Our team will check the design. Do not write 'near zero' and hope."

"Understood. Which component do you expect to deliver it?"

"Your backup. You proposed a backup service. Make it continuous."

"A backup cannot give you zero. It can give you a copy from a point in time. How far apart are your two sites, in milliseconds?"

Silence.

"And what is the peak redo rate of the database, in megabytes per second?"

"Why does that matter for a backup?"

That question — asked honestly, by a competent team — is the whole article.

The Trap, Named

That's the trap. RPO is a phrase from the backup world, so people hand it to the backup product. But a recovery point of zero cannot be produced by anything that copies data on a schedule, however short the schedule. It can only be produced by a system that refuses to confirm a write until a second site has it. That system is synchronous replication, and whether it is possible at all is decided by one network link — its round-trip time and its bandwidth against the database's peak write rate.

RPO zero is not a feature you buy. It is a latency budget you either have or do not have.

What RPO Actually Means for Each Kind of Copy

Here's the part that trips people up. Three different things get called "a copy of the database", and each one carries a different recovery point.

A backup writes a restore point at intervals. Nightly full, hourly incremental, whatever the policy says. Your RPO is the gap between restore points — hours, sometimes a day. In the design I was defending, the backup tier was a hardened repository with a 7-day immutability window. Its job is to survive ransomware and human error. Its job is not to be current.

Asynchronous replication streams every change to the standby as fast as the link allows, but the primary commits first and ships later. The recovery point is the transport lag — normally seconds, sometimes a minute when the link is busy. The primary never waits.

Synchronous replication sends the change to the standby and waits for the acknowledgement before the primary tells the application "committed". The recovery point is zero. The cost is that every single commit now includes a round trip to the other city.

In arrow form, the synchronous path for one transaction looks like this:

Application commit → primary writes redo → redo crosses the link → standby writes it → acknowledgement crosses the link back → primary returns "committed" to the application

Two trips across the link, inside every commit, for as long as the database runs. If the round-trip time between the sites is 20 milliseconds, every commit grows by roughly 20 milliseconds. A busy order-entry screen that used to commit in 2 milliseconds now commits in 22. Nobody changed the application. The requirement did it.

Where this confusion is common:

Tender requirement sheets, where "RPO" and "backup" sit in the same table row and get read as one thing.
Marketing for "continuous data protection", which sounds like zero and usually means seconds.
Database teams and infrastructure teams who each assume the other side owns the replication link.

Why Nobody Sees It in Testing

A disaster-recovery drill is the quietest moment in the database's year. It runs at night, on a weekend, with the application mostly idle. The redo rate is a trickle. The standby is "fully in sync" within a second of any change because there is almost nothing to send. Commit latency looks fine because almost nobody is committing.

Then the design goes live, month-end arrives, the redo rate climbs to its real peak, and one of two things happens. In asynchronous mode the transport lag stretches — your "seconds" become "a minute or two" exactly when the data matters most. In synchronous mode the application slows down across the board, and if the link flaps, the primary stalls or drops back to asynchronous on its own. Either way, the drill never showed it, because the drill never carried load.

Absence of lag at 2 a.m. is not proof of RPO. It is proof that nothing was happening.

Why Replication Must Never Be Sold as Backup

This is the second thing the revision request nearly flattened, and the customer's own tender had a section demanding the opposite — a clear separation between backup and disaster recovery. Their DBA team would have caught any blurring, and rightly so.

Replication copies every change, including the bad ones. A dropped table arrives on the standby in seconds. An application bug that corrupts rows corrupts them in both cities. Ransomware that encrypts data through the application layer is replicated faithfully. The standby is a mirror, and a mirror reflects the damage.

The backup is the thing that does not follow. It holds restore points from before the mistake, in a repository the production credentials cannot modify, for a fixed retention window. In the design that stayed, the database was backed up natively by the database's own tooling to that repository, and the replication product was listed under "disaster recovery" with its own column. Two products, two columns, two jobs.

Plain takeaway: if a proposal has one line that claims to be both your backup and your zero-RPO replica, it is neither.

How to Actually Decide: The Four Numbers

Before anyone writes "RPO 0" into a commitment, collect these four numbers. None of them come from the backup product.

Step 1: Measure the round-trip time between the two sites.

This is read-only and takes a minute. From a server at the primary site:

ping -c 100 203.0.113.20

Read the last line. The average round-trip time is the cost you will add to every synchronous commit. The maximum tells you what a bad moment looks like. If you want the path as well as the number:

mtr -r -c 100 203.0.113.20

Expected output is a stable average with a small spread. A failure looks like an average that is fine and a maximum ten times higher — that spike is what a synchronous commit will wait for.

Step 2: Measure the peak redo rate, not the average.

The database already knows how much change it generates. For Oracle, the archived redo gives you a per-hour history over the last days:

SELECT TRUNC(completion_time,'HH24') hour, ROUND(SUM(blocks*block_size)/1024/1024) mb FROM v$archived_log WHERE completion_time > SYSDATE-7 GROUP BY TRUNC(completion_time,'HH24') ORDER BY 1;

Find the busiest hour, divide by 3,600, and you have peak redo in megabytes per second. That peak — not the daily average — is what the link must carry without the lag growing.

Step 3: Size the link against the peak, with headroom.

Oracle's own Data Guard guidance sizes the transport link by dividing the redo rate by 0.7 to leave room for protocol overhead, then converting to bits:

Required Mbps = ( peak_redo_bytes_per_second / 0.7 ) x 8 / 1,000,000

If the number comes out above the link you have, asynchronous lag will grow at peak and synchronous commits will queue. In the case that started this article, the peak redo rate was one of the three facts we had to request from the customer before the revision could be honest, together with the measured latency and the current link bandwidth. Nobody had them to hand. That is normal, and it is exactly why the question matters.

Step 4: Choose the protection mode and name who presses the button.

With the three numbers in front of you, the mode chooses itself. If the round-trip time is a few milliseconds and the link has headroom at peak, synchronous is affordable. If the sites are a city apart and the link is shared or thin, asynchronous with a committed lag ceiling is the honest answer. Read the current state from the database, not from the diagram:

SELECT name, value FROM v$dataguard_stats WHERE name IN ('transport lag','apply lag');

Transport lag is your real RPO right now. Apply lag is how far behind the standby is in applying what it already has. Run it during the busiest hour you found in Step 2, not on a Sunday.

Then write down who performs the failover. Automatic failover needs an observer process that watches both databases from a third location and makes the switch. If the customer's database team owns that observer — which is what we agreed, with the observer VM provided in a third data centre — then the recovery time splits into two owned parts: the provider's part (the standby platform is running and traffic is redirected) and the customer's part (the database role switch). Commit to the part you control. In our case that was 15 minutes for the platform and the traffic switch, and the commitment said so in plain words.

Step 5: Verify under load. This step is where most tutorials stop and where mine starts.

Run the drill with the application busy, or replay a representative load. Measure commit latency on the primary before and after switching the mode. Watch transport lag during the run:

SELECT name, value, time_computed FROM v$dataguard_stats WHERE name = 'transport lag';

If the lag climbs steadily during the test and only recovers when load stops, the link is undersized for that peak. If commit latency doubled when you enabled synchronous transport, that is not a bug. It is the verification — the round trip you measured in Step 1, showing up exactly where the arithmetic said it would.

What We Committed Instead of "Zero"

The honest version was shorter than the ambitious one, and the customer's database team accepted it.

Replication stayed asynchronous over one dedicated link between the two sites. No extra relay hop, no second link.
RPO committed as under one minute — typically seconds, explicitly "not zero" — because that is what the measured path could promise without slowing production.
RTO 15 minutes, scoped to the provider's part: the disaster-recovery platform kept hot at full production size, and the traffic switch at the edge. The database failover belonged to the customer's observer.
The observer ran on a VM in a third data centre, owned and configured by the customer's team, with the provider guiding.
Backup stayed a separate line — native database backups into a hardened, immutable repository — and was never described as part of the recovery-point story.
One yearly drill, under load, with the lag query above as the exit criterion.

Nothing in that list is exotic. All of it is measurable. That is the point.

The Bigger Lesson: Six Checks Before You Promise a Recovery Point

You know the round-trip time between the sites. ping -c 100 <standby-ip> — the average is your synchronous commit tax.
You know the peak redo rate, by hour, for the last week. The v$archived_log query above — size the link to the peak.
The link has headroom at that peak. Redo bytes per second ÷ 0.7 × 8 — if the result is above the link, lag will grow when it matters.
The protection mode is stated in words a DBA can verify. v$dataguard_stats during peak hour — transport lag is the real RPO.
Failover has a named owner and a named observer location. If the answer is "the platform handles it", ask which process, on which host, in which site.
Backup and replication are separate lines with separate jobs. One holds history behind immutability; the other mirrors the present. Never the same box.

What I Tell Every Customer Now

When a requirement says "RPO zero", I ask one question before I touch the proposal:

"What is the round-trip time between your two sites, and what is your peak redo rate?"

If they answer with two numbers, they are in good shape — the design is arithmetic from there. If the answer is "our backup vendor handles that", we're not done yet. The backup vendor was never asked that question either.

FAQs

Can a backup product deliver RPO zero?

No. A backup writes restore points at intervals, so the recovery point is always the gap between the last restore point and the failure. Even "continuous" backup products capture changes with a small delay. Zero data loss needs synchronous replication, where the primary database waits for the standby to acknowledge each change before confirming the commit.

What does synchronous replication cost in practice?

Every commit waits for a round trip to the standby site. If the two sites are 20 milliseconds apart, every commit grows by roughly 20 milliseconds, all day, for every transaction. Measure the round-trip time with ping or mtr first; that number is the price of zero.

What is the difference between transport lag and apply lag in Oracle Data Guard?

Transport lag is how much redo the primary has generated that has not yet reached the standby — that is your real recovery point at this moment. Apply lag is how far the standby is behind in applying redo it already holds, which affects recovery time rather than data loss. Both come from the view v$dataguard_stats and should be read during peak hours.

Is Oracle Data Guard a backup?

No. Data Guard mirrors changes continuously, so a dropped table, a corrupting bug or ransomware acting through the application reaches the standby within seconds. A backup holds restore points from before the mistake, ideally in an immutable repository. Keep them as separate products with separate lines in any proposal or contract.

How do I size the network link for database replication?

Find the peak redo rate — the busiest hour from v$archived_log divided by 3,600 — not the daily average. Oracle's guidance divides that rate by 0.7 for overhead and converts to bits per second. If the result exceeds the link, asynchronous lag grows at peak and synchronous commits queue.

Who should own the failover in a provider-hosted DR design?

Name it explicitly. Automatic database failover needs an observer process in a third location; whoever configures and runs that observer owns the database part of the recovery time. The provider commits to its own part — the standby platform running and traffic redirected — and the agreement should state both parts and both owners.

Download the PDF

Originally published at quabin.com.

Locked Out of a Windows VM After a Cloud-Admin Restart — Cloudbase-Init and the Password Nobody Set

Daanial Farrukh — Fri, 30 Oct 2026 17:18:54 +0000

A presales war story about an Administrator password that changed with "SYSTEM" as the subject, the OpenStack service almost nobody remembers is still installed, and the one log line I now read before anyone says "we were hacked".

The call started with two groups of people certain they had done nothing.

"We did not touch the password. We did not touch the network card. Your team did."

That was the customer. Their application server, a Windows VM on a private OpenStack cloud, had been shut down the previous evening by their own administrator, from the console. The cloud team started it again a few minutes later. When the customer tried to log in the next morning, the Administrator password was wrong. The machine name had changed. The network adapter was disabled.

"We only pressed Start. Nothing else," said the cloud team.

I asked for one thing before anyone said the word "breach".

"Can you export the Security log from the VM?"

"It says SYSTEM did it. So it was not a person. So it was the platform."

"SYSTEM is not a person. SYSTEM is a service. Which service?"

Silence.

Forty minutes later I had the answer, and it was neither group. It was a service that had been sitting on that VM since the day the image was built, doing nothing for months, waiting for a reason to run again.

The Trap, Named

That's the trap. When a Windows event says the subject was SYSTEM, people read it as "no human was involved" and stop there. Some read it as proof of a hack. Some read it as proof the hypervisor did something. Almost nobody asks the next question: which process, running as SYSTEM, made that change at that second.

A SYSTEM-subject password reset is not the end of the investigation. It is the first clue about which service to go and read.

What Cloudbase-Init Actually Is

Here's the part people miss. Every Windows image built for OpenStack carries a small service called Cloudbase-Init. It is the Windows equivalent of cloud-init on Linux. On the first boot of a new instance it reads metadata from the cloud and does the things that make a fresh VM usable:

Cloud metadata → Cloudbase-Init service → set hostname → create user → set password → extend disk → run user-data → reboot

Each of those is a plugin. The ones that matter in this story:

SetHostNamePlugin renames Windows to the instance name the cloud knows, then schedules a reboot.
CreateUserPlugin makes sure the admin account exists.
SetUserPasswordPlugin sets that account's password. If the cloud passed one in metadata, it uses it. If not, it generates a random one.

The design is correct for a brand-new instance. The random password is meant to be encrypted with the SSH public key the VM was launched with and posted back to the metadata service, so the owner can retrieve it from the cloud. If there is no key, there is nothing to encrypt with, and Cloudbase-Init says so in its log and moves on.

Cloudbase-Init remembers which plugins have already run. It keeps a per-instance status in the Windows registry, under HKLM\SOFTWARE\Cloudbase Solutions\Cloudbase-Init\<instance-id>\Plugins. On every later boot the service starts, checks the status, and for each plugin logs "execution already done, skipping". That is the normal life of the service: start, skip everything, stop. For months.

Until one boot where it does not skip.

Why Nobody Sees It

Three reasons.

First, the service is invisible by design. It runs for a few seconds at boot, writes to a log under C:\Program Files\Cloudbase Solutions\, and exits. Nobody opens that folder on a server that has been in production for a year.

Second, the symptom looks exactly like an intruder. A password reset you did not do. A hostname you did not choose. A network card you did not disable. If you are the customer, that is a compromise. If you are the cloud team, that is the customer forgetting what they did. Both readings are emotionally satisfying and both are wrong.

Third, the Security log is honest but incomplete. Event 4724, "An attempt was made to reset an account's password", records the subject as SYSTEM. It does not record the process. The one log that names the process is the one nobody reads.

Absence of a human in the event log is not absence of a cause. It is a pointer to a service.

How to Actually Prove It

Everything here is read-only. Run it from an elevated PowerShell on the affected VM, or against an exported log on another machine.

Step 1: Find the password resets and the exact second they happened.

Get-WinEvent -FilterHashtable @{LogName='Security'; Id=4724} -MaxEvents 20 | Format-List TimeCreated, Message

In this case there were two 4724 events ten seconds apart, both with Subject Account Name SYSTEM, both targeting Administrator. Note the timestamps to the second. You will match them in the next step.

Also pull 4723, "An attempt was made to change an account's password". That is the event you get when a logged-in user changes a password themselves. If you see 4723 with a named user, a person was at the keyboard. In this case there was one 4723, the next evening, from an interactive Administrator session, and that one was the customer's own recovery attempt. Keep the two event IDs apart in your head: 4724 is a reset done to an account; 4723 is a change done by the account.

Step 2: Find out who rebooted the machine and why.

Get-WinEvent -FilterHashtable @{LogName='System'; Id=1074} -MaxEvents 10 | Format-List TimeCreated, Message

Event 1074 names the process and the reason for every shutdown and restart. Here it showed two entries in the window. The first was the customer's own shutdown from the console the previous evening. The second, a minute after the cloud team pressed Start, read "Cloudbase-Init reboot" in the reason field. That one line is the whole case. A service called Cloudbase-Init rebooted the server, and it only reboots after it has renamed the host.

Step 3: Read the Cloudbase-Init log for the same seconds.

Select-String -Path "C:\Program Files\Cloudbase Solutions\Cloudbase-Init\log\cloudbase-init.log" -Pattern "Generating a random user password|Password succesfully updated|No SSH public key|already done, skipping" | Select-Object -Last 40

Yes, "succesfully" is spelled with one s in the log. Search for the misspelling or you will not find it.

What you want to see is the contrast. In every earlier boot, line after line of:

[-] Plugin 'cloudbaseinit.plugins.common.setuserpassword.SetUserPasswordPlugin' execution already done, skipping

And then, in the boot after the restart, at the exact second of the first 4724 event:

setuserpassword [-] Generating a random user password
setuserpassword [-] No SSH public key available for password encryption

Read it in words. The service decided to run its first-boot plugins again. It created or confirmed the admin user, generated a random password, found no key to encrypt it with, and set it anyway. The password now exists in exactly one place: the account itself. Nobody can retrieve it. Nobody did anything wrong at a keyboard. The machine is locked.

When the timestamp of "Generating a random user password" matches the timestamp of the 4724 event with subject SYSTEM, the cause is proven. Not a theory. Proven.

Step 4: Confirm the rename came from the same service.

Select-String -Path "C:\Program Files\Cloudbase Solutions\Cloudbase-Init\log\cloudbase-init.log" -Pattern "sethostname|hostname" | Select-Object -Last 10

The hostname plugin logs the new name it applied and the reboot it requested. Match it to the 1074 "Cloudbase-Init reboot" event. Now you have one author for two of the three symptoms.

The Part I Could Not Prove

I want to be honest here, because this is where it is tempting to make the story tidier than it was.

I proved the password reset. I proved the rename and the reboot. I did not prove why the plugins ran again after months of skipping, and I did not prove that the disabled network adapter came from the same service.

The documented mechanism is the per-instance plugin status in the registry. The status lives under a key named after the instance ID the service reads from metadata. If that key is not found, every plugin is new and runs. That can happen when the instance ID the service sees changes, or when the status key is gone. On this VM I could see that the plugins re-ran; I could not get the metadata history to say which of those it was. So in the report I wrote exactly that: cause of lockout proven, trigger of the re-run not proven, network card not attributed.

A report that says "this part is proven and this part is not" is worth more than one that sounds complete. The customer trusted the proven half because I did not stretch it.

How to Actually Fix It

Two jobs. Get back in, then make sure a restart can never do this again.

Step 1: Get back in. This changes state.

If the VM was launched with an SSH key pair and the cloud still has it, the random password is retrievable from the cloud side. On the OpenStack CLI:

nova get-password <server> <private-key-file>

If, like here, there was no key, the cloud team resets the password from the platform instead. On OpenStack that is a password change through the compute service or the console. Any reset done this way is a new 4724 event. In our log, the following morning's resets were exactly that, and they did not appear in the Cloudbase-Init log at all, which is how you tell an admin's reset from the service's reset.

Step 2: Decide what Cloudbase-Init is allowed to do on a server that is already in production. This changes state.

Open the config file:

notepad "C:\Program Files\Cloudbase Solutions\Cloudbase-Init\conf\cloudbase-init.conf"

Find the plugins= line. It is a comma-separated list of every plugin the service may run. On a VM that has been handed over to a customer, there is no reason for the service to ever rename the host or set the admin password again. Remove SetHostNamePlugin, CreateUserPlugin and SetUserPasswordPlugin from the list and save. A restart is not required; the file is read at the next service start.

Also check two settings that control the password behaviour:

Select-String -Path "C:\Program Files\Cloudbase Solutions\Cloudbase-Init\conf\cloudbase-init.conf" -Pattern "first_logon_behaviour|inject_user_password|plugins"

inject_user_password is what posts the encrypted password back to the cloud; it only helps if the VM has a key. first_logon_behaviour decides whether the user is forced to change the password at first logon. Neither of them stops the plugin from running. Only the plugins= list does that.

If the customer never wants cloud-side changes on this VM again, the stronger option is to disable the service entirely:

Set-Service cloudbase-init -StartupType Disabled

I prefer trimming the plugin list over disabling the service, because the service also handles useful things like extending the disk after a resize. But both are legitimate. The point is to make a decision, not to leave the default.

Step 3: Keep a break-glass account. This changes state.

Create a second local administrator with a password stored in the customer's vault, not on the VM. Cloudbase-Init only manages the account named in its username= setting. A second account is untouched by anything the service does. On this VM that one account would have turned a two-day outage into a two-minute login.

Step 4: Verify. This step is where most tutorials stop and where mine starts.

Restart the VM once, on purpose, in a maintenance window. Then re-run the Step 1 and Step 3 checks from the proof section:

Get-WinEvent -FilterHashtable @{LogName='Security'; Id=4724} -MaxEvents 5 | Format-List TimeCreated, Message

There must be no new 4724 event at boot time. Then:

Select-String -Path "C:\Program Files\Cloudbase Solutions\Cloudbase-Init\log\cloudbase-init.log" -Pattern "Generating a random user password" | Select-Object -Last 3

The last match must be the old one from the incident, not a fresh one from your test boot. If a new one appears, the plugin is still in the list, or the service is reading a different config file than the one you edited, and you go back to Step 2.

Plain takeaway: a first-boot service you have not told to stop is a first-boot service that is waiting.

The Bigger Lesson — Four Questions Before Anyone Says "Breach"

Whenever a Windows VM in a cloud comes back from a restart with something changed that "nobody did", run these in order.

What did the 1074 events say? The reason field often names the process. "Cloudbase-Init reboot" ends the argument in one line.
Who is the subject on the 4724 events? A named user means a person. SYSTEM means a service, and your next job is to find which one.
Does the Cloudbase-Init log have activity at the same second? If yes, the service is the author. If no, the reset came from the hypervisor side or from a real intruder, and you escalate properly.
Is the first-boot service still allowed to run its first-boot plugins? Check plugins= in the config. On any VM older than its first week, the answer should be no for hostname and password.

If the answers point at Cloudbase-Init, the fix is a config edit and a break-glass account. If they do not, you have a genuine incident and you have reached it in twenty minutes instead of two days.

What I Tell Every Customer Now

In the next review, ask the team one question:

"On your Windows cloud VMs, which Cloudbase-Init plugins are still enabled, and when did you last read its log?"

If the answer is "we trimmed the list at handover and the log shows nothing but skips", they are in good shape. If the answer is "what is Cloudbase-Init?", we are not done yet.

FAQs

Why did my Windows VM's Administrator password change after a restart on OpenStack?

Cloudbase-Init, the first-boot service installed in every OpenStack Windows image, re-ran its SetUserPasswordPlugin and generated a random password. The Security log shows this as event 4724 with subject SYSTEM. Confirm it by finding "Generating a random user password" in cloudbase-init.log at the same timestamp.

What does "No SSH public key available for password encryption" mean in cloudbase-init.log?

Cloudbase-Init wanted to encrypt the new random password with the key pair the VM was launched with, so the owner could retrieve it from the cloud. The VM had no key pair, so the password was set but never stored anywhere retrievable. That line is why the account is locked rather than just changed.

What is the difference between Windows event 4724 and 4723?

Event 4724 is a password reset performed on an account by someone or something else, such as an administrator or a service. Event 4723 is a password change made by the account itself from a logged-in session. A 4724 with subject SYSTEM points at a service; a 4723 with a user name points at a person at the keyboard.

How do I stop Cloudbase-Init from changing the hostname or password again?

Edit cloudbase-init.conf under C:\Program Files\Cloudbase Solutions\Cloudbase-Init\conf and remove SetHostNamePlugin, CreateUserPlugin and SetUserPasswordPlugin from the plugins= list. Alternatively disable the cloudbase-init service. Then restart once in a maintenance window and confirm no new 4724 event appears at boot.

Can I recover the random password Cloudbase-Init generated?

Only if the VM was launched with an SSH key pair and inject_user_password was enabled; then the encrypted password is in the metadata service and nova get-password decrypts it with the private key. Without a key pair the password cannot be recovered, and the cloud team must reset it from the platform.

Is a SYSTEM-subject password reset proof that the server was hacked?

No. SYSTEM means a process running as the local system account made the change, which includes legitimate services like Cloudbase-Init. Match the event's timestamp against service logs before treating it as an intrusion. If no service accounts for it, then escalate as a security incident.

Download the PDF: https://quabin.com/wp-content/uploads/2026/10/quabin-windows-vm-locked-out-cloudbase-init-random-password.pdf

Originally published at quabin.com.

Webmail Returns 502 Only When People Are Actually Using It — The One Proxy Flag cPanel Forgot

Daanial Farrukh — Thu, 29 Oct 2026 17:17:48 +0000

A presales war story about a webmail login page that worked for one tester and failed for forty staff, the Apache error almost nobody reads, and the single ProxyPass flag I now check on every cPanel server.

The ticket had been open for two weeks.

"Webmail is unstable. Users get Bad Gateway. It comes and goes."

The server was a customer's cPanel box, a few hundred mailboxes, staff logging in through webmail.example.com in a browser. The hosting team had already done the usual rounds.

"We added RAM. We restarted Apache twice. We even moved the site off Cloudflare for a day. Nothing changes."

I asked one question.

"When does it fail?"

"Mornings. Nine, half past nine. By eleven it's fine."

"And when you test it yourself?"

"It always works when I test it."

That was the whole case, in two answers. One person testing: fine. Forty people logging in within the same ten minutes: 502. Nobody had tested forty logins. Nobody had looked at what sits between Apache and the webmail service on a cPanel server.

I asked for one line from one log, and the answer was there before the call ended.

The Trap, Named

That's the trap. A 502 Bad Gateway from webmail does not look like a cPanel bug. It looks like a load problem. So the team feeds the server more memory, blames the CDN, blames the firewall, and tests with one browser tab, where it never fails.

The failure is not in the server's resources. It is in how Apache talks to cPanel's own webmail service on the loopback.

What cPanel Webmail Actually Is

Here's the part people miss. On a cPanel server, Roundcube is not served by Apache the way a normal PHP site is. It is served by cpsrvd, cPanel's own daemon, on ports 2095 and 2096.

When a user types webmail.example.com, they are hitting Apache on port 443. Apache then proxies the request over the loopback to cpsrvd:

Browser → Apache :443 (webmail.example.com) → ProxyPass → cpsrvd 127.0.0.1:2095 → Roundcube

That ProxyPass is generated by cPanel from a template. In the stock configuration it looks like this:

ProxyPass / http://127.0.0.1:2095/ max=1 retry=0

Two settings matter. max=1 means Apache keeps at most one pooled connection to cpsrvd per worker. And there is no disablereuse=on, which means Apache is allowed to reuse that one connection for the next request, from whoever it comes.

That is exactly the design that breaks under load.

The same proxy pattern is used on cPanel servers for all the service subdomains:

webmail.example.com → 127.0.0.1:2095
cpanel.example.com → 127.0.0.1:2082
whm.example.com and webdisk.example.com → their own local ports

Webmail is where it shows first, because webmail is where many different people log in at the same minute.

Why the Shared Connection Breaks

cpsrvd is strict about sessions. Each connection it accepts is tied to the authentication it saw on that connection's first request. The function doing this is named plainly in its own log: ensure_authn_has_not_changed_from_initial_request.

Now put the two facts together.

Apache has one pooled connection to cpsrvd and is allowed to reuse it.
User A's request goes through it. Then user B's request is sent down the same connection.
cpsrvd sees a session that does not match the first request on that connection, and it resets the connection.
Apache was mid-request. It never gets a status line back. It logs an error and sends the browser a 502.

With one tester, user B never exists. The connection carries one session, cpsrvd is happy, and the page loads every time. With forty staff at nine in the morning, user B exists every few seconds.

Absence of errors in testing is not health. It is a test that never created the condition.

Why Nobody Sees It

Three reasons, and I have watched all three happen on the same ticket.

First, the obvious dashboards are green. Memory is fine. Load average is fine. The 502 is not a resource problem, so no resource graph will show it.

Second, the error is in a log almost nobody opens for a webmail complaint. It is in Apache's error log, as a proxy error, with a code that looks like noise: AH01102. And the matching line on the other side is in cpsrvd's log under /usr/local/cpanel/logs/, not under /var/log.

Third, the direct URL works. If you tell a user to go to https://example.com:2096 instead of webmail.example.com, the problem disappears. The team reads that as "the server is fine, the user's network is weird". It is actually the single most useful clue in the case: port 2096 bypasses the Apache proxy entirely. If the proxy path fails and the direct path works, the problem is the proxy.

How to Actually Prove It

Step 1: Read the Apache error log for the proxy error.

This is a read-only check.

grep AH01102 /etc/apache2/logs/error_log | tail

If this is your problem, you will see lines like:

[proxy_http:error] [pid 12345] (104)Connection reset by peer: [client 203.0.113.10:51234] AH01102: error reading status line from remote server 127.0.0.1:2095

Read it in words: Apache sent a request to cpsrvd on the loopback, and the connection was reset before a status line came back. The client IP is the user who saw the 502. The remote server is cpsrvd on 2095. Nothing else in the chain is involved.

Step 2: Read cpsrvd's side of the same moment.

grep "Session does not match" /usr/local/cpanel/logs/error_log | tail

The matching line is:

reset_connection("Session does not match first request")

That is cpsrvd saying, in its own words, that it received a request for a different session on a connection that already belonged to someone else, and it closed the connection on purpose. When you see this line and the AH01102 line at the same timestamps, the cause is proven. Not a theory. Proven.

Step 3: Confirm the generated ProxyPass is missing the flag.

grep -n "2095" /etc/apache2/conf/httpd.conf

Look at the webmail ProxyPass lines. If they say max=1 retry=0 and nothing about disablereuse, you have the stock configuration that produces the fault.

Step 4: Confirm the direct port works while the proxy fails.

Ask one affected user to log in at https://example.com:2096 during the failure window. If that works and webmail.example.com fails, you have the behavioural proof as well as the log proof.

How to Actually Fix It

The fix is one flag: disablereuse=on. It tells Apache to close the backend connection after each request instead of pooling it. cpsrvd then sees a fresh connection for every request, and the session check never fails.

There is a quick way and a permanent way. Do the quick way first if users are failing right now, then make it permanent.

Step 1: Quick fix, applied directly to the running config. This changes state.

sed -i 's#ProxyPass / http://127.0.0.1:2095/ max=1 retry=0#ProxyPass / http://127.0.0.1:2095/ max=1 retry=0 disablereuse=on#' /etc/apache2/conf/httpd.conf

Then restart Apache the cPanel way:

/scripts/restartsrv_httpd

The 502s stop immediately. But this edit lives in a generated file. The next time cPanel rebuilds httpd.conf, which happens on EasyApache changes, account changes and some updates, your flag is gone and the 502s return. That is why the next step matters.

Step 2: Permanent fix, through the template cPanel actually uses.

The webmail ProxyPass is generated from the main template, ea4_main, not from the SSL vhost template many people edit first. cPanel's supported way to change a template is to copy .default to .local and edit the copy. This changes state.

cp /var/cpanel/templates/apache2_4/ea4_main.default /var/cpanel/templates/apache2_4/ea4_main.local

Open ea4_main.local and find the webmail ProxyPass lines. There are two, one for the HTTP vhost and one for the HTTPS vhost, roughly 180 lines apart. Add disablereuse=on to both:

grep -n "2095" /var/cpanel/templates/apache2_4/ea4_main.local

After editing, rebuild the config from the template and restart:

/scripts/rebuildhttpdconf && /scripts/restartsrv_httpd

Step 3: Verify. This step is where most tutorials stop and where mine starts.

First, prove the flag survived the rebuild:

grep -c "disablereuse=on" /etc/apache2/conf/httpd.conf

You want a count of at least two, one per webmail vhost. Zero means the rebuild overwrote your change, which means you edited the wrong template.

Second, prove the fault is gone. Wait for the next busy window, or simulate it: have several people, or several browser sessions logged in as different mailboxes, hit webmail.example.com within the same minute. Then:

grep -c AH01102 /etc/apache2/logs/error_log

Note the number, wait through a busy period, run it again. If the count stops growing, the fix is in. If it keeps climbing with timestamps after your restart, the flag is not in effect on the vhost that is serving the traffic, and you go back to Step 2.

Plain takeaway: a fix you cannot see in the log after a rebuild is not a fix.

The Part That Is Not Your Fault

I want to be fair to cPanel here, because this is not a misconfiguration by the hosting team.

The stock template ships this way. I first saw the fault on a server running cPanel version 134, and it was reported widely after servers upgraded to 136.0.20. I checked the stock ea4_main.default on 136.0.23 and it still generates max=1 retry=0 without disablereuse=on. This is a regression in the default template, not something anyone on the customer side typed.

That has two consequences.

First, you will meet it on servers that have never been touched. I have now confirmed the identical fault on two unrelated cPanel servers at two unrelated customers. Same version range, same AH01102, same cpsrvd reset line, same fix.

Second, freezing ea4_main.local has a cost. ea4_main is a core template. Once you have a .local copy, cPanel stops applying its own future changes to that template on your server. You will not get fixes or new directives that land in ea4_main.default later. For a small server that is acceptable; for a fleet, it is a maintenance debt you have to track.

The better long-term options are either a post-rebuild hook that re-applies the flag to the generated config, or a ticket with cPanel asking them to add disablereuse=on to the default. I would open the ticket regardless. The more servers that report it, the sooner the default changes and the .local copy can be deleted.

The Bigger Lesson — Four Checks Before You Add RAM

Whenever a web service "fails under load but works in testing", run these before you touch resources.

Does the direct port work while the proxied name fails? Try https://example.com:2096. If yes, it is the proxy layer, not the application.
What does the proxy log say, in its own words?

grep -i "proxy" /etc/apache2/logs/error_log | tail -20

What does the backend log say at the same timestamps? For cpsrvd that is /usr/local/cpanel/logs/error_log. Match the seconds.
Is a connection being shared between users? Any ProxyPass with a connection pool and no disablereuse=on, in front of a backend that ties sessions to connections, will produce exactly this pattern.

If all four point at the proxy, more memory will not help, and a CDN change will not help, because the failing hop is a loopback connection inside the same machine.

What I Tell Every Customer Now

In the next review, ask the hosting team one question:

"When webmail returns a 502, what does the Apache error log say about 127.0.0.1:2095?"

If the answer is "nothing, we've never seen a proxy error there", they are in good shape. If the answer is "AH01102, error reading status line", we are not done yet, and the fix is one flag away.

FAQs

Why does cPanel webmail give 502 Bad Gateway only sometimes?

Because the fault needs two different users on the same Apache-to-cpsrvd connection within a short time. One tester never triggers it. A busy login window does. The intermittent pattern is the signature of a shared backend connection being reset, not of a server running out of resources.

What does Apache error AH01102 mean on a cPanel server?

AH01102 means Apache's proxy module sent a request to a backend and the connection was closed before a status line came back. On cPanel, when the remote server in the message is 127.0.0.1:2095 or 2096, the backend is cpsrvd serving webmail, and the reset is cpsrvd refusing a connection whose session changed.

What does disablereuse=on do in Apache ProxyPass?

It stops Apache from keeping the backend connection open for the next request. Each proxied request opens its own connection and closes it afterwards. That removes the chance of two users' requests sharing one connection, which is what cpsrvd's session check rejects.

Will my fix to httpd.conf survive a cPanel update?

Not if you edited httpd.conf directly. cPanel regenerates that file from templates. To keep the change, copy ea4_main.default to ea4_main.local in /var/cpanel/templates/apache2_4, add the flag there, and run rebuildhttpdconf. Then confirm the flag is present in the rebuilt httpd.conf.

Is this a cPanel bug or a configuration mistake?

It is the stock template. Servers on cPanel 136.0.20 and later generate the webmail ProxyPass without disablereuse=on, and the default was still unchanged at 136.0.23. A hosting team that never touched the config will still see it. A support ticket to cPanel is worth opening so the default gets fixed.

Does the same problem affect cpanel.example.com and whm.example.com?

Yes, the same ProxyPass pattern, max=1 and retry=0 without disablereuse=on, is used for the cpanel, whm and webdisk service subdomains. Webmail shows it first because many users log in at the same time there. If those other subdomains also return 502 under load, apply the same flag to their ProxyPass lines.

Download the PDF: https://quabin.com/wp-content/uploads/2026/10/quabin-cpanel-webmail-502-bad-gateway-disablereuse.pdf

Originally published at quabin.com.

"We've Been Hacked" — No, That Is Cloudflare: Reading Edge Traffic Before You Blame the VM

Daanial Farrukh — Wed, 28 Oct 2026 17:13:40 +0000

A presales war story about a firewall full of "abnormal sessions", a team ready to block their own origin IP, and the three questions I now ask in Cloudflare before anyone calls a server compromised.

The call came on a weekend.

"The VM is compromised. The firewall is showing abnormal sessions all over it."

The server was a customer's cPanel box — WordPress sites and email on one VM, with Cloudflare in front of the websites on the free plan.

The plan was already moving.

"We're running an antivirus scan now. And we want to block the VM's IP on the firewall until it's clean."

I asked one question.

"Where are the sessions coming from?"

Someone read a few source addresses off the firewall view. They all started with 172.64, 172.68, 162.158.

"Those are Cloudflare."

Silence.

"So… Cloudflare is attacking us?"

The Trap, Named

That's the trap. Inbound traffic from Cloudflare to your origin is not an attack on your VM. It is your website working. A busy, angry firewall view looks exactly like a breach to someone who does not know which way the arrows point.

And blocking the origin IP does not stop anything. It just takes every site on the server offline — the attacker's job, done for free.

What You Are Actually Looking At

When a site sits behind Cloudflare's proxy, nobody on the internet talks to your server directly. The path is:

Visitor → Cloudflare edge → Origin VM

Every request your origin sees on port 443 arrives from a Cloudflare edge IP. Cloudflare publishes those ranges — 173.245.48.0/20, 103.21.244.0/22, 104.16.0.0/13, 172.64.0.0/13, 162.158.0.0/15 and a handful more. So when a firewall lists thousands of inbound :443 sessions from 172.64.0.0/13 and 162.158.0.0/15, it is showing you reverse-proxy traffic, nothing else.

A compromised VM is the source of traffic, not the destination. The signs point outward:

the VM opening connections to addresses nobody configured — a command-and-control server, a mining pool
spam or scans leaving the server from processes that should not send anything
a login that succeeded from an address or a tool that should never have one

Inbound sessions from Cloudflare are the opposite direction. They tell you about load, not about a breach.

This mix-up is common in three places:

teams where the network side and the web side are run by different people
monitoring that alerts on session count without knowing what sits behind the proxy
any shop that has added Cloudflare recently and never updated its "normal" baseline

So What Was It?

The sessions were real, and there were a lot of them. I opened Cloudflare's Security Analytics for the zone and read the last 24 hours.

About 22.7 million requests in 24 hours, for sites that normally see a tiny fraction of that.
About 22.7 million GET and about 7,200 POST. So this was not a password attack on wp-login.php or xmlrpc.php — those are POSTs.
Two user-agent strings carried almost all of it, roughly 50/50: one Firefox on Windows, one Firefox on iOS. Real Chrome — the browser most real visitors use — was about 0.25 million.
Cloudflare mitigated about 1.18 million (~5%) and passed about 20 million (~88%) to the origin as dynamic, uncached requests.

That is a distributed layer-7 GET flood from a botnet pretending to be two kinds of Firefox. Every one of those 20 million requests reached the VM through Cloudflare. That is what lit up the firewall.

Here's the part that trips people up. Nothing on the VM was wrong. The antivirus scan would find nothing because there was nothing to find. The VM was doing its job under a flood.

Why the Usual Check Misses It

The first place most people look in Cloudflare is Security → Events. On this zone it was nearly empty. A few blocks, a few challenges — nothing that matched 22 million requests.

That empty log is the second trap. On the free plan, a flood that looks like real browser traffic does not show up as ddos_block or managed-challenge events. Cloudflare did not decide it was an attack, so it did not log an action. An empty events log means Cloudflare did not act. It does not mean nothing happened. Ask a dashboard "was there a DDoS?" and it will look at the events log and say no.

You judge a flood by volume and shape, not by the events log:

request count against the normal baseline
the GET/POST split
the user-agent mix
the top source ASNs

All four live in Security → Analytics, not in Events.

How to Actually Tell — and Fix It

Step 1: Confirm the sources are Cloudflare

Before anything else, check the source addresses the firewall is worried about against Cloudflare's published list:

curl -s https://www.cloudflare.com/ips-v4

If the "abnormal" sources sit inside those ranges, you are looking at proxied traffic. Stop the VM-blocking plan here.

Step 2: Read the shape of the traffic in Cloudflare

Cloudflare → your zone → Security → Analytics. Set the window to the last 24 hours and read four panels in this order:

Total requests — compare with a normal day.
HTTP methods — mostly GET means a page flood; a POST spike on login paths means a credential attack.
User agents / browsers — real traffic is spread across many browsers. One or two identical strings carrying most of it is a botnet template.
Mitigated vs served by origin — how much got through to you.

Plain takeaway: if requests are huge, mostly GET, and two user-agents carry the load, it is a flood. It is not a breach.

Step 3: Turn on the free plan's strongest lever

Cloudflare → Security → Settings → Security Level → I'm Under Attack.

This puts a JavaScript challenge in front of every visitor. A real browser passes it in a few seconds without the user doing anything. A raw GET-flood script cannot run JavaScript, so it never reaches the origin. On the free plan, this single switch does more than anything else.

Optionally, add a custom rule with Managed Challenge for the exact user-agent strings the botnet is using, so you can dial the global level back down later without letting the template through.

Step 4: Lock the origin to Cloudflare

A flood that cannot pass Cloudflare will try to go around it. Allow port 443 on the origin only from Cloudflare's ranges, at the firewall in front of the VM or on the VM itself. Then the botnet has no direct path. This is the one change on the network side that actually helps — the opposite of blocking the origin IP.

Step 5: Verify — this step is where most tutorials stop and where mine starts

Back in Security → Analytics, the "served by origin" line should fall to a small fraction of the total, and the challenged count should rise to match the flood. On the origin, the number of established sessions on :443 should drop:

ss -Htn state established '( sport = :443 )' | wc -l

If origin traffic does not drop, something reaches the server without passing Cloudflare — a DNS-only record that points at the same IP, or port 443 still open to the world. That is not a failed fix. It is the verification telling you where the gap is.

One more thing that confused the team: requests to wp-cron.php coming from the origin's own IP. That is WordPress calling its own scheduler through the public URL. It is the site talking to itself, not an attacker.

What a Real Compromise Looked Like — on the Same Server

I want to be honest here: this same server did have a real compromise, found a few days later. It is the best proof that the two look nothing alike.

First, there was probing. A scanner with an empty user-agent asked a backup subdomain for a list of common web-shell file names. Probing is not a breach. The question is whether any of those requests got a real answer. One grep on the access log decides it:

grep -Ei "sh3ll|ah25|xwpg" access_log | grep " 200 "

Nothing means the scanner knocked and nobody answered. A 200 means a file with that name existed and ran — then you investigate that file.

The real entry, when it came, was quiet. One successful login to the cPanel account, from a script user-agent (python-requests), followed by a password change on that account. No flood, no firewall alarm. It was found in the cPanel login and access logs, not in Cloudflare and not on the firewall.

And during the cleanup, two process names in top looked like malware. Each one was checked before anyone called it:

ls -l /proc/<PID>/exe

One pointed to a mail server's own helper binary — legitimate, just stuck at full CPU from brute-force load on the mailboxes, cleared by restarting the mail service. The other was an admin tool running from root's home directory. Neither was malware.

A breach is a quiet login and something leaving the server. A flood is loud traffic arriving through the proxy. Mixing them up costs you both ways — you block your own sites for a flood, and you miss the one login that mattered.

The Bigger Lesson — Five Checks Before You Say "Compromised"

Which direction is the traffic? Inbound from Cloudflare ranges is your site. Outbound to unknown addresses is the alarm. ss -tnp state established shows both, with the process.
Is there a successful login you cannot explain? For cPanel, read /usr/local/cpanel/logs/login_log. For SSH, grep Accepted /var/log/auth.log.
Did any probe get a 200? Grep the access log for the probed paths and the status code.
Is the scary process really what its name says? ls -l /proc/<PID>/exe before you kill or report it.
Is the edge absorbing the load? Security → Analytics: mitigated vs served by origin.

If all five are clean, you have a load problem, not a security problem. Fix it at the edge.

What I Tell Every Customer Now

When a team says "the VM is hacked", I ask one question before anything is blocked or scanned:

"Show me one connection that started from the server."

If the answer is "there isn't one, they're all coming in from Cloudflare", we are dealing with traffic, and the fix is in Cloudflare. If the answer is "yes, it's talking to an address we don't know", we're not done yet — and now we look at the VM.

FAQs

Why does my firewall show so many connections from Cloudflare IPs?

Because your site is behind Cloudflare's proxy. Every visitor's request reaches your server from a Cloudflare edge address, not from the visitor. Thousands of inbound sessions from ranges like 172.64.0.0/13 or 162.158.0.0/15 are normal reverse-proxy traffic, and a spike means more requests, not a breach.

How do I know if my server is compromised or just under a DDoS?

Look at direction. A compromised server starts connections outward — to a command server, a mining pool, or other mail servers with spam — or shows a login you cannot explain. A layer-7 flood is huge inbound request volume through the proxy. Check outbound connections and login logs on the server, and request volume in Cloudflare Security Analytics.

Why is the Cloudflare Security Events log empty during an attack?

Events only lists requests Cloudflare took an action on. On the free plan, a flood that imitates real browsers is often not classified as an attack, so it is passed to the origin without an event. Judge the attack by total requests, method split and user-agent mix in Security Analytics instead.

Does Cloudflare's I'm Under Attack mode stop a layer-7 GET flood?

It stops most simple ones. It shows a JavaScript challenge that real browsers pass automatically and plain request scripts cannot. Combine it with a firewall rule that allows port 443 on the origin only from Cloudflare's ranges, so the flood cannot go around the proxy.

Should I block my origin IP during an attack?

No. Blocking the origin IP takes every site on the server offline, which is what the attacker wanted. Block or challenge at Cloudflare, and restrict the origin to Cloudflare's ranges so nothing reaches it directly.

Why do I see wp-cron.php requests from my own server's IP?

That is WordPress triggering its own scheduled tasks by requesting wp-cron.php through the public URL. The source is your origin because the server is calling itself. It is expected and not an attack.

Download the PDF: https://quabin.com/wp-content/uploads/2026/10/quabin-cloudflare-traffic-not-vm-compromise.pdf

Originally published at quabin.com.

The Root Password Changed at 10:17 and Nobody Logged In — The VM Console Path Your Firewall Never Sees

Daanial Farrukh — Tue, 27 Oct 2026 17:18:33 +0000

A presales war story about a Linux VM whose root password changed with no login in any log — the swapped guest-tools agent behind it, the password stealer that came with it, and the checks I now run before anyone types a password on a suspect server.

The message came late one morning.

"The root password doesn't work anymore. Nobody changed it."

The server was a customer's Ubuntu 24.04 VM on a hosted private cloud. A web server with a handful of sites, SSH on a non-standard port, a firewall in front of it that had been reviewed twice that year.

Their first theory came fast.

"Someone brute-forced SSH. Check the firewall logs."

I asked one question.

"When was the last successful SSH login as root?"

They looked. "Two days ago. Ours."

"And the password changed this morning?"

"10:17."

A pause. "So how did they get in?"

That is the question this whole article is about. Because nobody got in the way anyone in that room expected. Nobody got in through the network at all.

The Trap, Named

That's the trap. When root changes under you, everyone stares at the network door — SSH, the firewall, the web application firewall, the panel.

A VM has a second door that is not on the network at all: the hypervisor console. It hides in plain sight because it is the door the cloud provider uses. Nobody treats it as an attack path, so nobody watches it.

What Actually Happened — In Order

It took two nights of log work to put the chain together. Here it is in one line:

Cloud portal login → VM console opened → old build-time user logs in → sudo → three backdoors installed → guest agent replaced → "reset password" called from inside the VM → root changed at 10:17

Now each link, with where the evidence was.

The portal login was not the customer's. The cloud portal's audit showed a login to the customer's tenant from an address that was not the customer's office or their admin's home. Minutes later the web console to the VM was opened.

Here is the part that trips people up. A console session leaves no trace in auth.log by itself. But it does leave one in the kernel log. When the hypervisor attaches a console, the guest sees a virtual USB tablet — the console's mouse — plug in. That one kernel line dated the entry to the second.

Root failed on the console. A forgotten user did not. On tty1, two failed root logins. Then a login as a user created when the VM was first built, months earlier, whose password had never been changed. It worked. sudo su. Root, in under three minutes, with a password that was probably in some old build document.

Three persistence mechanisms in twenty minutes. Once root, the attacker did not do one thing. They did three.

A fake service: a static binary at /usr/sbin/kernel-agent with a systemd unit called kernel-agent.service, running as root. No package owned it.
A password stealer in /etc/ld.so.preload: a shared object with a glibc-sounding name, loaded into every process on the system, hooking the functions that read passwords from a terminal.
A PAM backdoor: one line added to common-auth, sshd, su and sudo, plus a tty variant in login:

auth optional pam_exec.so quiet expose_authtok /usr/libexec/glibc-langpack

expose_authtok hands the plaintext password to that script on every authentication. The attacker left a marker comment in each file and saved the original files in a hidden folder under /var/lib/private/ — their own backup, so they could restore the files if they ever needed to look clean.

The guest-tools agent was swapped. Every cloud platform ships a small agent inside the VM that the portal talks to over localhost: reset the password, run a script, report the IP. For about an hour the attacker probed that agent from inside the VM — the signature checks on its API kept failing. Then they replaced the binary with a modified copy and left a .bak next to it. The running file's hash no longer matched the copy the vendor's package ships.

Then they cleaned up. /root/.bash_history emptied. Old records cut out of /var/log/wtmp, so last started two days late. The relevant lines deleted from auth.log. They missed one place: journald still had every line.

Hours later, 10:17. With no login of any kind — no SSH, no console, no panel — the journal shows calls to the agent's reset-password and run-script endpoints from localhost, and at 10:17:12 chpasswd changed root. The attacker did not need to be logged in anymore. The swapped agent did it for them, on a schedule.

Why the Usual Checks Missed It

The firewall was never in the path. The console goes hypervisor → VM, not network → VM. The distributed firewall, the port knocking, the custom SSH port — all irrelevant to that door.

The logs people trust had been edited. auth.log was missing lines. wtmp had been truncated. last and who showed a quiet server, not because the server was quiet but because the file that would have said otherwise had been cut.

The file timestamps lied. The attacker set the modification time of every file they touched back to an old date. ls -l showed nothing recent. find -mtime found nothing. That is why the PAM backdoor was found a full day after the first cleanup — it was caught by a search on change time, which a normal user cannot fake without changing the system clock.

And the first-page checks were all green. No extra UID 0 user. Root's crontab untouched. No new SSH keys. Kernel not tainted. Everything a standard "am I hacked" checklist starts with was clean.

Absence of evidence in a log the attacker could edit is not evidence of absence.

How to Actually Find This

Everything below is read-only. Run it from your own terminal, and — this matters — do not type a password on the suspect server until Step 3 comes back clean. If the stealer is loaded, every password typed is captured.

Step 1: Ask the journal, not the file.

journalctl --since "-3 days" _COMM=chpasswd

On this server it printed password changed for root with a time nobody could explain. Then compare what the journal knows about logins with what the file knows:

journalctl --since "-3 days" | grep -c "session opened for user"

grep -c "session opened for user" /var/log/auth.log

If the journal has more lines than the file for the same period, the file was edited. On this server it had.

Step 2: Find console sessions in the kernel log.

journalctl -k --since "-3 days" | grep -i "usb"

A virtual tablet or mouse attaching, with no physical change to the VM, is a console being opened from the hypervisor. This is how we knew the exact minute the attacker arrived — and that they arrived through the portal, not the network.

Step 3: Check the two places that capture passwords.

cat /etc/ld.so.preload

The file should not exist, or should be empty. Any library listed there is loaded into every process on the box. On this server it named one .so in /usr/libexec/ that no package owned.

grep -r "pam_exec" /etc/pam.d/

A pam_exec.so line with expose_authtok that you did not add is a password stealer. Full stop.

This error is not a bug. It is the verification. If both of those come back clean, passwords typed on the server are safe. If either is dirty, nothing typed on this server from the moment of compromise can be trusted.

Step 4: Search by change time, not modification time.

find / -xdev -type f -ctime -3 -printf '%CY-%Cm-%Cd %CH:%CM %p\n' | sort

Modification time is one touch away from any date. Change time is set by the kernel when the inode changes and cannot be set from userspace. Sorted by ctime, the attacker's whole session appeared in order: the PAM files, the .so, the agent, the hidden backup folder — all within one twenty-minute window, no matter what ls -l said.

Step 5: Find services that no package owns.

systemctl list-units --type=service --state=running --no-pager

For any name you do not recognise:

dpkg -S /usr/sbin/kernel-agent

dpkg-query: no path found matching pattern means no package put that file there. A root-running service with a generic name and an unowned binary is yours to explain.

Step 6: Hash the guest agent against its own package copy.

Vendor tools usually keep a pristine copy of the agent in an architecture folder next to the installed one. Hash both:

sha256sum /usr/local/<vendor-tools>/agent /usr/local/<vendor-tools>/x64/agent

They must match. On this server they did not, and an agent.bak sat beside them with the original hash. Also check the vendor's own update log: no update had been applied, so the only way the running hash could differ was a manual swap.

Step 7: Find the user who should not exist.

awk -F: '$3>=1000 && $7!~/nologin|false/' /etc/passwd

Then for each one:

chage -l <user>

"Last password change" from the week the VM was built is the door. That user also sat in sudo, adm and lxd — three groups, any one of which is root.

How to Actually Clean It — In This Order

The order matters because of the stealer. Clear the password capture first, then rotate passwords, never the other way round. Rotating first hands the attacker the new password.

Snapshot the VM from the portal. That is your evidence. Everything after this changes state.
Copy evidence out. The unowned binaries, /etc/pam.d/, the .so, the agent and its .bak, the journal export. Tar it, pull it off the server over your own key, hash the tarball.
(changes state) Remove the preload. rm /etc/ld.so.preload, move the .so to a quarantine folder. Nothing typed after this is captured by that component.
(changes state) Restore PAM. Diff every file in /etc/pam.d/ against the attacker's own backup copies or against the package versions, and put the originals back. The attacker's backup folder made this fast; dpkg -V libpam-runtime confirms it.
(changes state) Remove the fake service. systemctl disable --now kernel-agent, then remove the unit and the binary to quarantine.
(changes state) Restore the guest agent. Reinstall the vendor tools from the package, confirm the hash matches the package copy, restart the agent.
(changes state) Lock the entry user. usermod -L -e 1 -s /usr/sbin/nologin <user> and deluser <user> sudo — and the same for adm and lxd.
Now rotate every password. Root, every sudo user, the panel, the database. On this server the portal's own "reset password" failed, so root was set from the installer ISO: boot the Ubuntu installer, Help › Enter shell, mount the LVM root at /mnt, chroot /mnt passwd root.
Re-run Steps 1 to 7. On this server a full second pass — unowned binaries, UID 0, cron, timers, kernel modules, webshells, authorized keys — came back clean.
Plan the rebuild. A cleaned server is a server you hope is clean. A rebuilt server is one you know is clean. Treat the cleanup as the bridge to a rebuild, not as the end.

Two things I can say with certainty from the evidence, and one I cannot.

The stealer stored what it captured in a hidden file under /var/log, encrypted with a key kept beside it — it did not send anything out over the network by itself. It had two records, both from passwords that were changed anyway.

The attacker's session proved outbound internet worked: a Python package install ran from the server during their window, with no login at that time. A compromised server with open egress is a staging post. Egress filtering is not a nice-to-have on a web server.

What I cannot tell you is how the portal credential got into someone else's hands. The provider's own platform logs and the firewall records for that address were still being pulled when I wrote this. I will not guess at it here, because a guess presented as a cause is how the next incident gets missed.

The Second Server

Within a day, a second VM at another customer on the same platform was reported with the same symptoms. Different OS family, same pattern: portal console, build-time user, the same three backdoors.

We wrote the checks above into a read-only script and ran that first, before anyone touched the server. Then the snapshot, then the evidence, then the quarantine, then the agent, then the lock, and only then the passwords.

Plain takeaway: when the first server tells you the attacker's method, the second server is a checklist, not an investigation.

The Bigger Lesson — Six Checks for Any Hosted VM

The console is a login path. Treat portal accounts like root. MFA on the cloud portal, named users only, and an alert on every console open. journalctl -k | grep -i tablet tells you when one happened.
No build-time users survive go-live. awk -F: '$3>=1000' /etc/passwd and chage -l for each. Expire or remove what nobody owns.
/etc/ld.so.preload must not exist. test -s /etc/ld.so.preload && echo ALERT in a daily job that mails you.
PAM under change control. sha256sum /etc/pam.d/* > /root/pam.baseline on a clean day, then sha256sum -c /root/pam.baseline on a schedule.
Hash the guest agent. sha256sum of the running binary against the package copy, on the same schedule.
Filter egress. A web server needs to reach its update mirror and its DNS resolvers. It does not need to reach the whole internet. Prove it with curl -m 5 https://pypi.org from the server — it should fail.

Each one is a minute. Together they close the door this attacker walked through, and the two they would have used next.

What I Tell Every Customer Now

When a customer shows me a VM on any hosted platform, I ask one question:

"Who can open a console to this VM from the portal, and would you know within the hour if they did?"

If the answer is "these three people, with MFA, and the console event alerts us", they're in good shape. If the answer is "the provider handles the portal", we're not done yet.

FAQs

Why did the root password change when nobody logged in?

Because the change was made by a program, not a person. A tampered guest-tools agent inside the VM accepted a "reset password" call from localhost and ran chpasswd. Look for journalctl _COMM=chpasswd to see the exact time, and then check whether the agent binary matches the vendor package.

How can someone access a VM without going through the firewall?

Through the hypervisor console in the cloud portal. The console is attached by the hypervisor directly to the VM, like a keyboard and screen plugged into a physical server. It never crosses the VM's network interface, so no firewall, security group or SSH setting applies to it.

What is ld.so.preload and why is it dangerous?

It is a file that lists shared libraries the dynamic loader must inject into every program on the system. Legitimate systems almost never use it. An attacker can list a library there that hooks password-reading functions and captures every password typed, in every program, without any process looking unusual.

What does pam_exec expose_authtok do?

The pam_exec.so module runs an external program during authentication. With expose_authtok, PAM passes the user's plaintext password to that program on standard input. Added to common-auth, sshd, su and sudo, it captures every password used on the server. Any such line you did not add yourself is a backdoor.

Why search files by ctime instead of mtime?

Modification time can be set to any value with touch -d, and attackers do this to hide. Change time is updated by the kernel whenever the inode changes and cannot be set from userspace without changing the system clock. find / -xdev -ctime -3 shows what really changed, in order.

Should I clean a compromised server or rebuild it?

Clean it first so you can collect evidence, stop password capture and get control back, then rebuild. A cleanup proves what you found. A rebuild from a known-good image is the only thing that proves nothing else is there.

Download the PDF: https://quabin.com/wp-content/uploads/2026/10/quabin-root-password-changed-no-login-vm-console-backdoor.pdf

Originally published at quabin.com.

Mail Stopped Both Ways — "451 Temporary Local Problem" and the One DNS Resolver Nobody Checked

Daanial Farrukh — Mon, 26 Oct 2026 17:15:18 +0000

A presales war story about a cPanel server that could neither send nor receive mail — no blacklist, no full disk, no broken config — and the single DNS line I now check before I open any mail log.

The call was short.

"Email is down. Completely. We can't send, and nothing is coming in."

The server was a customer's cPanel/WHM box on Rocky Linux 8. It relayed outbound mail through their provider's smart host. It had been quiet and boring for a long time, which is exactly what you want from a mail server.

Users saw this in Roundcube every time they pressed Send:

SMTP Error (451): Failed to add recipient - Temporary local problem

The customer's IT lead had a theory ready.

"It must be the provider's relay. Or we got blacklisted again."

I asked one question.

"If it's the relay, why can't anyone send to you either?"

A pause. "...Both directions at the same time?"

Both directions at the same time. A relay problem breaks outbound. A blacklist breaks one side for some senders. Something that breaks everything, in and out, at once, usually sits underneath both.

The Trap, Named

That's the trap. "Temporary local problem" sounds like the mail server is sick. So people look at the mail server — the queue, the disk, the Exim config, the last cPanel update.

The mail server was healthy. It just could not ask a single DNS question.

And the reason was one line in a file that almost nobody reads during a mail outage.

What "Temporary Local Problem" Actually Means Here

Exim needs DNS for nearly every step of a mail transaction. Not only to find the recipient's MX. It needs DNS to find its own smart host, and it needs DNS to check that a sender is real before it accepts a message.

Outbound looks like this:

User → Roundcube → Exim → DNS lookup of the smart host → smart host → recipient

Inbound with sender verification looks like this:

Sender → Exim → DNS lookup of the sender's domain → callout to the sender's MX → accept or reject

If DNS does not answer, Exim cannot finish either flow. It does not bounce the mail permanently, because a DNS timeout might fix itself. It answers with a temporary 451 and tries again later. Roundcube shows that as "Temporary local problem". The sending server on the other side sees a deferral and keeps retrying.

A 451 with "local problem" often means the local server could not finish its own lookups — not that the message or the recipient is wrong.

The Exim log said it plainly once we looked:

temporarily rejected RCPT <user@example.com>: host lookup for smarthost.example.net did not complete (DNS timeout?)

And WHM › Email › Track Delivery was full of the same line:

Could not complete sender verify

Both are the same failure seen from two sides. The smart host lookup breaks outbound. The sender verify lookup breaks inbound.

This setup is common in three places:

Servers built from a template where someone typed 8.8.8.8 as the only resolver because "it always works".
Networks where outbound DNS to the internet is filtered or routed differently from web traffic.
Servers with one resolver line and no second one, so there is no fallback when that one goes away.

Why Nobody Saw It

From the outside, a server in this state looks fine. WHM loads. SSH works. Websites keep serving, because answering a web request does not need the server to resolve anything.

The usual checks stay green too: disk, the Exim service, the last update. The queue grows, but a growing queue is a symptom, not a cause.

Here is the part that trips people up. Resolver problems hide behind every other check. A web request comes in, the server answers it, done — no outbound DNS needed. Mail is different. Mail is the one service on a cPanel box that makes outbound DNS lookups for every message.

The file that mattered was this one:

cat /etc/resolv.conf

It had exactly one line:

nameserver 8.8.8.8

One resolver. No second line. And from that network, 8.8.8.8 did not answer.

One dead resolver with no fallback stops all mail, in both directions, while everything else on the server keeps working.

Why 8.8.8.8 was unreachable from that network is a question for the network team, and I won't guess at it here. What mattered on the day was proving that the resolver, and only the resolver, was the problem.

How to Actually Fix This

Everything below is read-only until Step 3, which I mark clearly.

Step 1: Read the Exim log for DNS timeouts.

grep "did not complete" /var/log/exim_mainlog | tail -20

If you see host lookup for ... did not complete (DNS timeout?), stop reading other logs. You are looking at a DNS problem, not a mail problem.

Step 2: Prove it is the resolver, not DNS as a whole.

First, ask the resolver the server is configured to use:

host google.com 8.8.8.8

On this server it hung and then failed with a timeout. Then ask a different public resolver the same question:

host google.com 1.1.1.1

That one answered at once.

Same server, same network, same name. One resolver answers, one does not. This error is not a bug. It is the verification. It tells you the server's network and DNS in general are fine, and only the configured resolver is out of reach.

Step 3 (changes state): Point the server at resolvers it can reach.

Back up the file first:

cp /etc/resolv.conf /etc/resolv.conf.bak

Then set two resolvers instead of one. On this server we used:

nameserver 1.1.1.1
nameserver 1.0.0.1

Two lines, not one. A second resolver is the fallback the server never had.

Step 4 (changes state): Flush the queue.

The mail that piled up during the outage is still sitting in the queue. Check how much:

exim -bpc

Then force a delivery run of everything in it:

exim -qff

Step 5: Verify — in both directions.

This step is where most tutorials stop and where mine starts.

Send from Roundcube to an outside mailbox. The 451 must be gone. Then send from an outside mailbox into the server, and follow the log:

tail -f /var/log/exim_mainlog

You want to see the smart host resolve and the outbound message delivered, and the inbound message accepted without a sender verify failure. Run exim -bpc again a few minutes later — the queue should be falling. In our case mail flowed in both directions as soon as the resolver changed.

Step 6 (changes state): Make it survive a reboot.

This is the step almost everyone skips, and it is the one that brings the outage back after the next reboot.

On Rocky Linux 8, /etc/resolv.conf is written by NetworkManager. Edit it by hand and it works — until the next reboot or network restart, when NetworkManager writes 8.8.8.8 straight back.

Find the connection name:

nmcli con show

Then set the resolvers on the connection itself:

nmcli con mod "<connection-name>" ipv4.dns "1.1.1.1 1.0.0.1"

Check that it is stored:

nmcli -g ipv4.dns con show "<connection-name>"

It should print 1.1.1.1,1.0.0.1. Now the fix lives in the place that generates the file, not only in the file.

Plain takeaway: if a tool generates a config file, fix it in the tool, not in the file.

One Thing the DNS Fix Did Not Fix

One more failure was in the logs, and it had its own cause. Sender verify for the server's own no-reply address kept failing with "Mailbox is full / Blocks limit exceeded".

That mailbox was over its quota. A verify check against a full mailbox fails no matter how healthy DNS is.

I mention it because this is how outages hide a second problem. Once the big failure is fixed, look at what is still failing, and check whether it has its own cause. A fix that clears the big failure has not cleared the small ones — it has made them easier to see.

The Bigger Lesson — And a Caveat From My Last Article

A mail server is a DNS client that also happens to deliver mail. Treat its resolver as part of the mail design, not as a network detail.

There is one caveat I owe readers. In my last article on Spamhaus and the open-resolver error, the lesson was that a mail server should not send its blacklist lookups through big public resolvers, because Spamhaus refuses them. Switching to 1.1.1.1 was the right move on this day, because it brought mail back within minutes. But if the server checks Spamhaus blocklists, a public resolver will break those lookups. The long-term design for that case is a local recursive resolver on the server, with the network allowing it to reach the internet on port 53.

The two cases teach the same thing from opposite ends: know exactly which resolver your mail server asks, and know that it will answer.

Here is the short checklist I now run on any mail server I inherit:

Count the resolvers. cat /etc/resolv.conf must show at least two nameserver lines.
Test each one directly. host google.com <resolver-ip> for every line in the file. Each must answer.
Search for DNS timeouts. grep -c "did not complete" /var/log/exim_mainlog should be zero or close to it.
Check who writes the file. nmcli -g ipv4.dns con show "<connection-name>" must match what is in resolv.conf.
Watch the queue. exim -bpc on a normal day tells you what "normal" is, so a growing queue stands out.
Check system mailbox quotas. WHM › Account Information › List Accounts, then the account's Email Accounts page, for any no-reply or system address near its limit.

Each item takes seconds. Together they turn "email is down" into a two-minute diagnosis.

What I Tell Every Customer Now

When a customer shows me their mail server design, I ask one question:

"If the DNS resolver in resolv.conf stopped answering right now, what would happen to your mail?"

If the answer is "it would fall back to the second resolver, and we test both", they are in good shape. If the answer is "there is only one, and it's Google, so it never goes down", we're not done yet.

FAQs

What does "SMTP Error (451) Failed to add recipient - Temporary local problem" mean?

It means the mail server could not finish its own work for that recipient and deferred the message instead of rejecting it. On cPanel servers a common cause is a DNS lookup that timed out, for example the lookup of the smart host or of the sender's domain. Check the Exim log for "did not complete (DNS timeout?)".

Why does a DNS problem stop both sending and receiving?

Exim needs DNS to find the smart host or recipient MX for outbound mail. It also needs DNS for sender verification on inbound mail. If the only resolver does not answer, both flows fail at the same time, while websites on the same server keep working.

How do I test if my server's DNS resolver works?

Run host google.com <resolver-ip> for each nameserver listed in /etc/resolv.conf. Each one should answer at once. If one times out while another public resolver answers, the configured resolver is unreachable from that network.

Why did my resolv.conf change back after a reboot?

On Rocky Linux, AlmaLinux and RHEL 8, NetworkManager usually writes /etc/resolv.conf. A manual edit is overwritten on the next reboot or network restart. Set the DNS servers on the connection with nmcli con mod "<connection-name>" ipv4.dns "..." so the change persists.

Should a mail server use 8.8.8.8 or 1.1.1.1 as its resolver?

A public resolver can work for basic mail flow, but always list at least two and test them. If the server checks Spamhaus blocklists, public resolvers will break those lookups, so a local recursive resolver is the better long-term design.

What does "Could not complete sender verify" mean in Track Delivery?

The server tried to confirm the sender's address before accepting the message and could not finish the check. A DNS timeout causes it for every sender at once. A full or missing mailbox causes it for one address only.

Download the PDF: https://quabin.com/wp-content/uploads/2026/10/quabin-exim-451-temporary-local-problem-dns-resolver.pdf

Originally published at quabin.com.

Every Email Rejected at Once — The Open-Resolver Mistake That Makes Spamhaus Answer "No"

Daanial Farrukh — Mon, 26 Oct 2026 05:52:44 +0000

A presales war story about a mail server that suddenly rejected every sender on the internet — Gmail, Outlook, the bank — and the two DNS queries I now run on every mail server before anyone blames the blacklist.

The message came in early in the morning.

"Nobody can email us. Gmail bounces. Outlook bounces. Even the bank statements are not arriving."

The server was a customer's cPanel/WHM box on Ubuntu. It had been running mail for years without much drama. Nothing had been deployed. Nobody had touched the mail settings.

The bounce text was the same for every sender:

is in an RBL: Error: open resolver

The first reaction on the call was the natural one.

"So Gmail is on a blacklist?"

I asked one question back.

"All of Gmail, all of Outlook and your bank — on the same blacklist, on the same morning?"

Silence. Then: "That doesn't sound right, does it?"

It didn't. When every sender on the internet is "listed" at once, the senders are not the problem. The thing doing the checking is.

The Trap, Named

That's the trap. An RBL rejection reads like a verdict on the sender. So people look at the sender. They open the blacklist websites, paste in Google's IP addresses, and find nothing.

The error was not about the senders at all. It was about how this server asked the question.

And the answer to that was hiding in one file that nobody had looked at in years.

What "Open Resolver" Actually Means Here

A blacklist like Spamhaus ZEN is queried over DNS. When mail arrives from an address, the server reverses the IP and asks Spamhaus a DNS question about it. The flow looks like this:

Sender connects → Exim takes the sender's IP → local resolver → Spamhaus DNS → answer → accept or reject

If the answer is empty, the address is clean. If the answer is an address in 127.0.0.x, the address is listed and Exim rejects the mail.

Here is the part that trips people up. Spamhaus does not answer queries that come through big public resolvers such as 1.1.1.1 or 8.8.8.8. Those resolvers carry traffic for millions of users, so Spamhaus blocks them. Instead of a real answer, it returns a special code: 127.255.255.254. That code means "you asked through a public resolver, I will not tell you".

Now look at that code again. It starts with 127.. A mail server that treats any 127.x answer as "listed" now sees every single sender as listed. Gmail, Outlook, the bank, the customer's own staff at home — all rejected.

That is what "open resolver" in the bounce means. Not "the sender is an open resolver". It means "the server asked me through a public resolver, so I refused to answer".

This setup is common in three places:

Servers where someone "fixed slow DNS" once by forwarding to public resolvers.
Images or templates that ship a resolver config pointing at 1.1.1.1 or 8.8.8.8.
Networks where outbound DNS was once blocked, and forwarding was the quick workaround.

Why Nobody Saw It

The obvious check is /etc/resolv.conf. On this server it looked perfect:

nameserver 127.0.0.1

That line says "use the local resolver". Most people stop there. Local resolver, so it must do its own recursion. Green tick.

It didn't. The local resolver was Unbound, and Unbound has its own config. Inside /etc/unbound/unbound.conf there was a block like this:

forward-zone:
    name: "."
    forward-addr: 1.1.1.1
    forward-addr: 8.8.8.8

name: "." means "for every domain on the internet". So Unbound was not resolving anything itself. It took every question, including the Spamhaus ones, and passed it to Cloudflare and Google DNS. From Spamhaus's point of view, the question came from a public resolver.

resolv.conf told the truth about the first hop. It said nothing about the second.

Why did it suddenly start that morning, and not years ago? I never proved the exact moment the block started to bite, so I won't pretend I know. What I can say is that the config was wrong all along, and the safe design does not depend on a third party's tolerance.

How to Actually Fix This

Everything below is read-only until Step 3, which I mark clearly.

Step 1: Prove the RBL lookups are broken — not the senders.

Spamhaus keeps two test entries for exactly this. 127.0.0.2 is always listed. 127.0.0.1 is never listed. Ask the server's own resolver about both:

dig +short 2.0.0.127.zen.spamhaus.org

dig +short 1.0.0.127.zen.spamhaus.org

A healthy server returns one or more 127.0.0.x addresses for the first query and nothing at all for the second.

A broken server gives the same answer to both. If that answer is 127.255.255.254, you have your cause: Spamhaus sees your queries coming from a public resolver.

This error is not a bug. It is the verification. The test entries exist so you can tell "my resolver is wrong" apart from "the sender is listed" in ten seconds.

Step 2: Find where the forwarding lives.

First confirm the first hop:

cat /etc/resolv.conf

If it points to 127.0.0.1, the local resolver is in charge. Now look inside it:

grep -rn "forward" /etc/unbound/

Any forward-zone: with name: "." and public addresses is the problem. If the server uses a different local resolver, look for the same idea there: a forwarder that sends everything to 1.1.1.1, 8.8.8.8 or your provider's shared resolvers.

Step 3 (changes state): Remove the forwarder.

Back up the file, then comment out the whole forward-zone: block — all four lines, including name: ".":

cp /etc/unbound/unbound.conf /etc/unbound/unbound.conf.bak

#forward-zone:
#    name: "."
#    forward-addr: 1.1.1.1
#    forward-addr: 8.8.8.8

With no forwarder, Unbound does full recursion itself, starting from the root servers. The queries to Spamhaus now come from the server's own IP, which is exactly what Spamhaus wants.

Check the config before you restart, so a typo does not take DNS down for the whole box:

unbound-checkconf

It must say no errors. Then:

systemctl restart unbound

Step 4: Verify — again, with the same two queries.

This step is where most tutorials stop and where mine starts.

dig +short 2.0.0.127.zen.spamhaus.org

dig +short 1.0.0.127.zen.spamhaus.org

First query: one or more 127.0.0.x answers. Second query: empty. Two different answers means the lookups are real again.

Then watch real mail come in. Send a test from an outside mailbox and follow the log:

tail -f /var/log/exim_mainlog

You want to see the message accepted, not rejected with in an RBL. In our case inbound mail from the big providers started flowing as soon as Unbound restarted.

Step 5: Leave the whitelist alone if it is there on purpose.

This server also had a WHM RBL whitelist for two Google mail ranges, 209.85.0.0/16 and 74.125.0.0/16. It was tempting to clean it up during the fix. We kept it. Another list, SpamCop, does sometimes list a real Gmail relay for a short time, and that whitelist keeps business mail flowing when it happens. It had nothing to do with this outage, so it stayed.

Plain takeaway: fix the cause you proved, and don't tidy up things you have not proven wrong in the same change.

The Bigger Lesson — A Mail Server Must Do Its Own Recursion

A web server can live happily behind a public resolver. A mail server cannot. Its spam defence depends on DNS questions that the blacklist operators only answer for resolvers they can trust to be one server, not millions of users.

Here is the short checklist I now run on any mail server I inherit:

Test the RBL path with the test entries. dig +short 2.0.0.127.zen.spamhaus.org must answer, dig +short 1.0.0.127.zen.spamhaus.org must not.
Check the first hop. cat /etc/resolv.conf should point to a local resolver.
Check the second hop. grep -rn "forward" /etc/unbound/ should find no name: "." forwarder to public DNS.
Validate before restart. unbound-checkconf must say no errors.
Search the logs for the symptom. grep -c "open resolver" /var/log/exim_mainlog should be zero.
Know why every whitelist entry exists. Look at WHM › Exim Configuration Manager and write down the reason next to each RBL whitelist range.

Each item takes seconds. Together they turn "the whole internet is blacklisted" into a one-line fix.

What I Tell Every Customer Now

When a customer shows me their mail server design, I ask one question:

"When this server checks a blacklist, whose DNS resolver asks the question?"

If the answer is "the server's own resolver, doing its own recursion", they are in good shape. If the answer is "it uses Google DNS, it's faster", we're not done yet.

FAQs

What does "is in an RBL: Error: open resolver" mean?

It means your mail server asked a blacklist like Spamhaus through a public DNS resolver, and Spamhaus refused to answer. The refusal code starts with 127, so the mail server reads it as "listed" and rejects the sender. The sender is usually not listed at all.

Why does Spamhaus block 1.1.1.1 and 8.8.8.8?

Public resolvers carry queries for millions of users, so Spamhaus cannot tell who is asking or how much. It refuses those queries and returns 127.255.255.254 instead of a real answer. Mail servers should query from their own resolver and their own IP.

How do I test if my RBL lookups work?

Run dig +short 2.0.0.127.zen.spamhaus.org and dig +short 1.0.0.127.zen.spamhaus.org on the mail server. The first must return 127.0.0.x and the second must return nothing. If both give the same answer, your lookups are broken.

My resolv.conf already says 127.0.0.1. Can I still have this problem?

Yes. 127.0.0.1 only means a local resolver such as Unbound answers first. If that resolver has a forward-zone for "." pointing at public DNS, every query still leaves through the public resolver. Check the local resolver's own config.

Is it safe to remove the forward-zone from Unbound?

Yes, as long as the server can reach the internet on port 53. Unbound then resolves from the root servers itself. Back up the file, run unbound-checkconf, and restart the service, then repeat the two test queries.

Should I whitelist Google and Microsoft ranges to stop the rejections?

No, not for this problem. A whitelist only hides the symptom for the ranges you add, and every other sender stays blocked. Fix the resolver. Keep a whitelist only where you have a proven reason for it.

Download the PDF: https://quabin.com/wp-content/uploads/2026/10/quabin-spamhaus-open-resolver-all-mail-rejected.pdf

Originally published at quabin.com.