DEV Community: LinkBook

PostgreSQL 14 End of Life on 12 November 2026: How to Upgrade Postgres in Docker

LinkBook — Tue, 03 Nov 2026 05:27:42 +0000

Liquid syntax error: Unknown tag 'endraw'

Roundcube 1.6.20 and 1.7.5: 20 Fixes, and an Older SQL Injection Now Being Exploited

LinkBook — Mon, 02 Nov 2026 04:46:54 +0000

On 19 October 2026 the Roundcube project released versions 1.6.20 and 1.7.5, with fixes for 20 reported vulnerabilities, and wrote: "We strongly recommend to update all productive installations of Roundcube 1.6.x and 1.7.x". Two weeks earlier, on 3 October, Singapore's Cyber Security Agency warned that an older Roundcube flaw, a SQL injection that works before login, "is reportedly being actively exploited".

Those are two separate problems with different fixed versions. If your webmail runs Roundcube, here is how to tell which one applies to you and what to do this week.

The exploited one: CVE-2026-48842

According to the NVD entry, Roundcube 1.6.x before 1.6.16 and 1.7.x before 1.7.1 has a pre-authentication SQL injection in the virtuser_query plugin, caused by a backslash escape that slips past preg_replace(). The fix shipped in May, in the 1.6.16 and 1.7.1 releases of 24 May 2026. CSA rates it 8.1 under CVSS v3.1 and says it lets "an unauthenticated attacker" inject SQL into the underlying database.

Two details decide whether you are exposed. First, the bug is in a plugin. virtuser_query maps logins to mailbox users through a database query, and like every Roundcube plugin it only runs if it is listed in your configuration. Second, the fix is five months old, so a server that has taken any Roundcube update since late May is already past it. The servers at risk are the ones nobody has touched since spring, which is exactly the kind of box attackers scan for.

The new batch: 1.6.20 and 1.7.5

The October release fixes 20 issues. The project does not say any of them are being exploited. The NVD published several CVE records for them on 22 October, including:

CVE-2026-114727, scored 8.3: potentially unsafe attachment content is passed to ImageMagick when images are resized.
CVE-2026-114730, scored 7.2: a bypass of the server-side request forgery filter through the NAT64 range 64:ff9b::/96.
CVE-2026-114732, scored 6.8: on multi-host installations, an alias login was not tied to its IMAP host.

The rest of the list, from the release post, is worth reading if you run Roundcube with extra plugins or drivers: IMAP command injection through CRLF in the _uid and _flag parameters, several header injection paths, a CSRF bypass in the mark-as-junk action, an XSS in password driver error handling, and a bug that let the Modoboa password driver change another account's password. Several of these need a logged-in user, so the risk depends on how many accounts can sign in.

Step 1: find your real version

The version is in program/include/iniset.php as RCMAIL_VERSION, and also shown under About in the settings menu if you have not hidden it. Compare against these:

1.6 branch: you need 1.6.20. Anything from 1.6.0 to 1.6.15 is also open to the exploited CVE-2026-48842.
1.7 branch: you need 1.7.5. 1.7.0 is also open to CVE-2026-48842.
1.5 or older: these releases do not cover it. Plan a move to 1.6 LTS.

If Roundcube came bundled with something else, such as a control panel, a mail server suite or a container image, the bundle's own version number tells you nothing. Check the Roundcube version inside it, and check whether the vendor has shipped 1.6.20 or 1.7.5 yet.

Step 2: update, then check the plugin list

Back up the Roundcube database and the config/ directory.
Install 1.6.20 or 1.7.5 from the official GitHub releases or your distribution's package, and run the update script so the database schema matches.
Open config/config.inc.php and look at the plugins array. Remove anything you do not use. If virtuser_query is there and you were below 1.6.16 or 1.7.1, treat the server as possibly compromised and go to step 3.
Confirm that PHP itself is current. Roundcube is a PHP application, so the PHP2Shell flaws disclosed on 21 October apply to it too, and updating Roundcube does not update PHP.

Step 3: if you ran an exposed version with virtuser_query

A SQL injection against the Roundcube database can read what that database holds: user preferences, address books, identities and, depending on configuration, the session table. Practical checks:

Search the web server logs for unusual characters, especially backslashes and SQL keywords, in login requests from before you patched.
Look at the database user Roundcube connects with. If it has rights on other databases on the same server, assume those were reachable too, and reduce it to the Roundcube database only.
Expire all sessions by clearing the session table after the update, and rotate the des_key in the config.
If the same database server holds mail-server user tables, rotate those credentials as well.

The pattern behind both

Webmail sits on the public internet and talks to IMAP, SMTP, a database, an image library and PHP, so a flaw in any of them is a flaw in the mail service. The Roundcube fixes this month are the third reminder in a few weeks, after the Zimbra SNMP command injection and PHP2Shell, that optional add-ons and the runtime underneath matter as much as the mail server itself. Enable only the plugins you use, keep the database account narrow, and track the runtime version, not just the application's. If you are rethinking the stack, our Mailcow, Stalwart and Mailu comparison covers how each one handles webmail.

Frequently asked questions

Which Roundcube version fixes the actively exploited SQL injection?

CVE-2026-48842 is fixed in Roundcube 1.6.16 and 1.7.1, released on 24 May 2026. The latest releases, 1.6.20 and 1.7.5 from 19 October 2026, include that fix plus 20 more security fixes.

Is CVE-2026-48842 exploitable on every Roundcube server?

No. It is in the virtuser_query plugin, so a server is exposed only if that plugin is enabled and Roundcube is older than 1.6.16 on the 1.6 branch or 1.7.1 on the 1.7 branch.

Are the October 2026 Roundcube vulnerabilities being exploited?

The Roundcube project did not report exploitation of the 20 issues fixed in 1.6.20 and 1.7.5. The exploitation warning from Singapore's CSA on 3 October concerns the older CVE-2026-48842.

Does updating Roundcube also protect against PHP2Shell?

No. PHP2Shell is a flaw in the PHP runtime. Roundcube needs both its own update and a fixed PHP version, such as 8.2.35, 8.3.36, 8.4.27 or 8.5.12, or a distribution build with the backported fix.

Sources: Roundcube, 19 October 2026, Roundcube, 24 May 2026, CSA Singapore alert AL-2026-131 and the NVD entries linked above.

Originally published on fanpino.com.

Zimbra CVE-2026-73570: Patched in July? Check Whether Someone Got In First

LinkBook — Sun, 01 Nov 2026 22:27:14 +0000

Zimbra fixed CVE-2026-73570 in version 10.1.20 on 20 July 2026. The flaw lets anyone who can send mail to a Zimbra server run shell commands on it, with no login and no user action, as long as the optional zimbra-snmp package is installed and SNMP notifications are switched on. On 30 September Microsoft Threat Intelligence published what attackers did with it: web shells, root access, stolen authentication keys and mailbox archives, across organisations in more than one region and industry.

If you upgraded in July and never had SNMP notifications on, you can stop here. Everyone else has two jobs. The first is to patch. The second, which most advice skips, is to check whether someone got in before you did, because the upgrade closes the hole but does not remove a web shell, a sudoers entry or a stolen key.

Why patch dates matter more than usual here

According to Microsoft, two separate scanning tools probed the vulnerable path between 28 July and 7 August. That was after the fix shipped on 20 July but before the public disclosure on 13 August. In other words, attackers were testing servers while most admins had no reason yet to rush. A server that took the update in late August was exposed during a window when the bug was already being worked.

How the attack works, in one paragraph

Zimbra's monitoring daemon, swatchdog, watches the mail log for "service status change" lines and, when SNMP notifications are enabled, sends an snmptrap through a shell. Elastic Security Labs reproduced the chain: an attacker sends an SMTP command with a malformed address, Postfix writes it to /var/log/zimbra.log, and if that text looks like a status line, the service name part reaches /bin/sh. The result is code execution as the zimbra user. The CVE entry carries a CNA score of 8.9.

Step 1: patch, or remove the trigger

Upgrade every Zimbra node to 10.1.20 or later. Microsoft and Elastic both call this the only complete fix.
If you cannot upgrade today, Microsoft's interim advice is to uninstall the zimbra-snmp package, disable SNMP notifications, and restrict SNMP and SMTP access to trusted hosts.

Do not trust a quick exploit test as proof you are safe. Elastic found that several public proof-of-concept scripts write a payload that swatchdog never matches, so a failed test can make a vulnerable server look patched. Check the installed version instead.

Step 2: look for what attackers left behind

These checks come straight from the behaviour Microsoft describes. Run them on every mailbox and MTA node, not only the one facing the internet, because the attackers moved between nodes using Zimbra's own SSH key.

Web shells. Look for JSP files that do not belong to your Zimbra version in the Jetty and mailboxd web application directories, including servlet work directories. Microsoft saw several copies per server, plus copies on peer nodes.
Sudo access for the zimbra account. Run sudo -l -U zimbra and search /etc/sudoers and /etc/sudoers.d/ for NOPASSWD: ALL. The attackers created exactly that entry by briefly taking ownership of /etc/pam.d/sudo, then restored the PAM file but kept the sudoers line.
A fake systemd service. Check for /etc/systemd/system/zimlog.service. The name sounds like part of Zimbra, but Microsoft found it installed by hand in the system-wide systemd directory, outside Zimbra's own folders, and the attackers changed its timestamps to match older system services, so sorting by date will not reveal it.
Cron jobs and odd processes. Review crontabs for the zimbra user and root, and look for a named pipe at /tmp/s or an openssl s_client process wired to a shell, which Microsoft observed as an encrypted reverse shell.
Log evidence. Search /var/log/zimbra.log for "Illegal address syntax" lines that contain $(, backticks or the text "Service status change", and web logs for the ZB73570 User-Agent used by the early probes.

Step 3: if you find anything, rotate the keys, not just passwords

Microsoft says the attackers went after Zimbra's central secrets rather than individual passwords. They ran zmlocalconfig -s, which prints the LDAP, MySQL, Postfix, Amavis and replication credentials, then used LDAP to pull zimbraPreAuthKey, zimbraAuthTokenKey and zimbraTwoFactorAuthSecret. With the pre-auth or auth-token key, an attacker can sign in as any user without knowing their password, so resetting user passwords alone does nothing. On a server with confirmed compromise, plan to:

Rotate the service credentials shown by zmlocalconfig -s.
Regenerate the pre-auth keys for each domain and the auth token key, so tokens signed with the stolen keys stop working.
Re-enrol two-factor authentication, since the shared secrets may have been copied.
Replace the zimbra SSH key in /opt/zimbra/.ssh/ on every node.
Treat mailbox contents as read. Microsoft observed archives being created and transferred out, which may trigger breach notification duties depending on where you operate.

In many cases rebuilding the affected nodes from a clean install and restoring data is quicker and more certain than cleaning them, given that the attackers had root and edited timestamps.

The wider lesson for self-hosted mail

This bug sat in an optional monitoring add-on, not in the mail path most admins think about. It is the same pattern as the NetScaler zero-days: the patch closes the door, but secrets taken before the patch stay valid until you rotate them. Two habits help on any self-hosted mail server. Install only the packages you use, and keep a written list of every key the server holds, so that rotating them after an incident is a checklist rather than a research project. If you are comparing self-hosted platforms, our Mailcow, Stalwart and Mailu comparison covers how each one is put together.

Frequently asked questions

Is my Zimbra server affected by CVE-2026-73570?

It is affected if it runs a version before 10.1.20, has the optional zimbra-snmp package installed and has SNMP notifications enabled. Servers without SNMP notifications are not exposed to this path, but should still be upgraded.

Is CVE-2026-73570 being exploited?

Yes. Microsoft Threat Intelligence reported confirmed compromises in several regions and industries, and probing of the vulnerable path from 28 July 2026, before public disclosure on 13 August.

Does upgrading to 10.1.20 remove an existing compromise?

No. The upgrade fixes the flaw, but web shells, sudoers entries, systemd units and stolen keys remain. Check every node and rotate Zimbra's authentication keys if you find signs of access.

Can I test my server with a public exploit?

It is not a reliable test. Elastic Security Labs found that several public proofs of concept never trigger the vulnerable code, so a failed test can hide a vulnerable server. Check the installed version and SNMP settings instead.

Sources: Microsoft Threat Intelligence (30 September 2026), Elastic Security Labs (23 October 2026) and NVD CVE-2026-73570.

Originally published on fanpino.com.

PHP2Shell: Updating WordPress Won't Patch It. Here's How to Patch the PHP Underneath

LinkBook — Wed, 28 Oct 2026 04:41:53 +0000

Liquid syntax error: Unknown tag 'endraw'

MCP Server Audit Logging: What to Log So You Can Answer "Were We Affected?"

LinkBook — Tue, 27 Oct 2026 04:42:47 +0000

When the Deadbugz campaign was published in August 2026, the first question most teams asked was "were we affected?" Most could not answer it. The malicious MCP server behaved like a text formatter for its first three tool calls and only then changed the instructions it sent to the agent. A team that had logged nothing but "server connected" had no record of what the server said, what the agent did next, or which files it touched. This guide is the logging setup we wish we had had before our own MCP audit: what to record, where, and how long to keep it.

What should an MCP audit log contain?

An MCP audit log should record every server launch with its exact command and version, every tool definition the server sends and any change to it, every tool call with its arguments and result size, the identity and token scopes behind each call, and outbound network requests. Together these let you rebuild what an agent did and who it acted for.

Why ordinary application logs miss MCP attacks

An MCP server has two channels into your systems. It runs code, and it sends the agent text, such as tool names, descriptions and prompt templates, that the model treats as instructions. Most logging only covers the first channel, if that. The Deadbugz payload arrived through the second: new instructions in later tools/list and prompts/get responses, after a counter reached three calls. Nothing at install time looked wrong. Only a log that stores what the server said, and when it changed, would have shown it.

The protocol's own security best practices point the same way in several places. Proxy services that spawn local servers should "log all stdio transport usage for security monitoring". Servers should log scope elevation events "with correlation IDs". Clients should "log suspicious authorization URLs". And the document warns that token passthrough breaks the audit trail, because the server can no longer tell clients apart and downstream logs show the wrong identity.

The events worth logging

Event	Fields to record	What it catches
Server launch	Full command and arguments, package version or commit hash, config file it came from, user it runs as	Unpinned launches, servers added by a pull request, a changed command
Tool definitions	Hash and full text of each tools/list and prompts/list response, per session, with a timestamp	Descriptions that change after approval or after a few calls
Tool call	Tool name, server, arguments (secrets redacted), result size, duration, error	A formatter that suddenly reads files, unusual argument patterns
Identity and scopes	Human user, agent session ID, token audience and scopes, correlation ID	Over-broad tokens, token passthrough, calls nobody can attribute
Scope elevation	Scope requested, subset granted, which tool triggered it	Privilege creeping up during a session
Outbound requests	Destination host, port, bytes sent, which server or tool made it	Quiet exfiltration, calls to metadata addresses like 169.254.169.254
File access outside the project	Path, read or write, process	Reads of ~/.ssh, cloud credential files, shell history

Log tool definitions, not just tool calls

This is the one most teams skip. Save the full output of tools/list and prompts/list each time a session starts, and again after several calls, then compare hashes. A changed description on a server you already approved should raise an alert and need a person's approval again; that is the recommendation Pillar Security gave MCP client builders after Deadbugz. Storing the full text, not only the hash, matters for the investigation afterwards, because the changed description is the evidence.

Where the logs should live

Not on the same machine as the agent, and not anywhere the agent can write. If an agent can be steered into deleting files, it can be steered into deleting its own log. Ship the events off the host as they happen. A practical pattern: have the MCP client or a small local proxy write one JSON line per event to stdout, and let your existing log shipper collect it, the same way container logs should go to stdout, not files. Network events come more reliably from an egress proxy than from the agent itself, since a compromised process will not report its own exfiltration.

Redact before you store

Tool arguments and results carry exactly the things an attacker wants: tokens, file contents, customer records. Redact known secret patterns before writing, store result sizes instead of full results by default, and keep full payloads only for tools you have marked sensitive, in a store with tighter access. An audit log that leaks credentials becomes the next incident.

How long to keep MCP logs

Keep them at least as long as your other security logs, and long enough to cover the gap between an attack and its public disclosure. Deadbugz pull requests were opened on 10 August and the write-up came out on 12 August; other campaigns surface months later. If your incident response policy already sets a retention period for authentication and endpoint logs, use the same one rather than inventing a shorter one for AI tools.

A minimal setup for a small team

Inventory every MCP server and pin its version, as in our audit checklist.
Wrap launches so the exact command, version and user are logged once per session.
Snapshot tools/list and prompts/list at session start and after the third call, and alert on any hash change.
Log every tool call with redacted arguments and result size, tagged with the human user and session ID.
Route agent traffic through an allow-listed egress proxy and keep its logs off the host.

The same principle shapes how we build internal assistants: an assistant should not hold more power than the person it works for, and every answer should be traceable to a source. We covered the access side in scoping an internal AI assistant to the person asking. In FanMind, our document assistant, outbound calls go through one allow-listed tunnel and confidential documents are excluded from AI indexing by default, which keeps both the attack surface and the log volume small.

Frequently asked questions

What should I log for MCP servers?

Log each server launch with its exact command, version and user, the full text and hash of every tools/list and prompts/list response, every tool call with redacted arguments and result size, the human user, session ID and token scopes behind each call, scope elevation events, and outbound network requests made by the server or agent.

Why log MCP tool definitions?

Because tool names, descriptions and prompt templates are instructions the model follows, and they can change after you approve a server. The Deadbugz server only changed what it sent after three tool calls. Comparing snapshots of tools/list and prompts/list over a session is the only way to see that change and prove what the agent was told.

Does the MCP specification require audit logging?

It does not define one audit log format, but its security best practices recommend logging in several places: proxies should log all stdio transport usage for security monitoring, servers should log scope elevation events with correlation IDs, and clients should log suspicious authorization URLs. It also warns that token passthrough damages the audit trail.

Where should MCP audit logs be stored?

Off the host where the agent runs and out of the agent's write access, so a steered agent cannot delete them. Emit events as JSON lines to stdout and ship them with your existing log pipeline, and take network events from an egress proxy, because a compromised process will not reliably report its own outbound traffic.

Sources

Exchange's Last ESU Patch Is Out, and a Windows Update Is Knocking Servers Over

LinkBook — Mon, 26 Oct 2026 18:34:41 +0000

On 13 October 2026 Microsoft shipped what it calls the last security update that Exchange Server 2016 and 2019 will ever get. Three days later the Exchange team published a second post: some Exchange servers were becoming unstable, and the cause was not the Exchange update at all but that month's Windows Server patches. If you run Exchange on your own hardware, both posts land on your desk in the same week.

We wrote about the end-of-support dates and the three ways out in our Exchange 2016/2019 exit plan. This is the shorter, more urgent follow-up: what the October release fixes, how to handle the Windows problem, and what changes from November.

What the October 2026 release covers

According to the Exchange team's release post, the updates are available for Exchange Server Subscription Edition (SE) RTM, Exchange 2019 CU14 and CU15, and Exchange 2016 CU23. The 2016 and 2019 builds are only available to organisations enrolled in the Period 2 Extended Security Update (ESU) program. Exchange Online customers are already protected, but the post is clear that hybrid setups still need the update on every on-premises Exchange server and on any workstation running the Exchange Management Tools.

Microsoft's October security release lists seven Exchange Server CVEs. The one to read first is CVE-2026-70571, a stack-based buffer overflow that Microsoft rates Critical, with a CVSS score of 8.1, allowing an unauthorised attacker to run code over the network. Next is CVE-2026-70279, a cross-site scripting flaw scored 9.3 that an unauthenticated attacker can use for spoofing. Microsoft marked both, along with the elevation-of-privilege bug CVE-2026-96940, as "Exploitation More Likely". None of the seven was listed as publicly disclosed or exploited at release. That is the good news, and it usually does not last long for Exchange.

SUs are cumulative. If your server is on a supported CU, you install only the latest SU, not every one you missed.

The Windows update that is taking Exchange down

The 16 October post describes servers that run fine for a while after patching and then degrade. Reported symptoms include OWA failing with MapiExceptionMdbOffline and event ID 1006 in the Application log, mail queues growing through the day until delivery almost stops, and DAG members failing to reach quorum. A reboot helps, then the problem comes back.

Microsoft names four Windows Server updates as the trigger:

KB5126646 on Windows Server 2025
KB5126655 on Windows Server 2022
KB5126651 on Windows Server 2019
KB5126868 on Windows Server 2016

Any Exchange version on those Windows builds can be affected, SE included. Busy servers show it sooner. Some may never show it.

Microsoft's advice is not to uninstall the Windows updates. Instead, from an elevated command prompt on the Exchange server, run:

reg add "HKLM\SOFTWARE\Microsoft\Rpc" /v EpmapSavedContextCap /t REG_DWORD /d 65536 /f

Then reboot and watch. If the problem returns, Microsoft says to raise the limit. In practice that means you should schedule the registry change in the same maintenance window as the October patches, rather than waiting for the help desk to report that Outlook cannot connect.

A patch-week checklist

Run Microsoft's Exchange Server Health Checker on every server, as the release post recommends, to see which CU and SU each one is on.
Bring each server to a CU the October SU supports before installing it.
Install the October Windows update and the October Exchange SU in the same window, then apply the RPC registry value.
After the reboot, confirm every Exchange service has started. Disabled services usually mean an interrupted setup.
For the next few days, watch queue length and OWA errors closely, not just "server up".
Patch the management-tools workstations too. They are easy to forget.

From November, 2016 and 2019 go quiet

The release post spells out what happens next. After October 2026, updates will be released for Exchange SE only, and Microsoft's announcements and CVE entries will stop mentioning 2016 and 2019. The team adds that this does not mean those versions are safe from future bugs. Microsoft will simply stop testing them. Buying Period 2 ESU late only brings you up to this October release.

So from the November Patch Tuesday onward, an Exchange 2019 server will be exposed to every new Exchange flaw with no fix and no CVE that even names your version. For organisations in the UAE and Saudi Arabia that kept mail on their own servers for data-residency reasons, the cloud route may not be acceptable either. We looked at those rules in our data-residency guide. That leaves two real paths: an in-place upgrade to Exchange SE, or a move to another mail server you host yourself.

If you are weighing the self-hosted route

Open-source mail servers have matured a lot. We compared the main candidates under real load in Mailcow vs Stalwart vs Mailu. Our own product, FanMail, is built on Stalwart Mail Server and runs fully on your own infrastructure, with a JMAP webmail, calendar, contacts and a built-in admin panel. It has replaced a commercial suite at 10,000-mailbox scale. It is not a drop-in Exchange clone: Outlook users who depend on every Exchange-specific feature should test with real mailboxes before committing.

Whichever way you go, this month's job is the same. Patch, add the registry fix, and put a date on the decision before November's patches arrive without you.

Frequently asked questions

Is October 2026 really the last update for Exchange 2016 and 2019?

Yes. Microsoft's release post says October 2026 is the last month Period 2 ESU customers receive new Exchange 2016 and 2019 security updates, and that later updates will be for Exchange SE only.

Should we uninstall the October Windows update if Exchange becomes unstable?

No. Microsoft recommends keeping the Windows update and setting the EpmapSavedContextCap registry value under HKLM\SOFTWARE\Microsoft\Rpc, then rebooting.

Are any of the October Exchange vulnerabilities being exploited?

Not at release. Microsoft listed none of the seven Exchange CVEs as exploited or publicly disclosed, but rated three as Exploitation More Likely, including the Critical remote code execution bug CVE-2026-70571.

We use Exchange Online with a hybrid server. Do we need to act?

Yes. Exchange Online is protected, but Microsoft says the SU must still be installed on on-premises Exchange servers, even ones used only for management.

Sources: Released: October 2026 Exchange Server Security Updates (Exchange Team, 13 October 2026) and Exchange servers may become unstable after installation of October 2026 Windows updates (Exchange Team, 16 October 2026).

Originally published on fanpino.com.

One Wildcard Certificate, Several Servers: Renewal That Doesn't Break

LinkBook — Tue, 20 Oct 2026 09:12:54 +0000

A wildcard certificate on one server is easy. You switch to the DNS-01 challenge, certbot renews it every couple of months, and you forget it exists. The trouble starts when the same *.example.com has to live on a second box, then a NAS, then a cloud VM in front of everything. Renewal works on one machine and quietly fails on the others, and you find out when a browser shows a red warning.

We run a handful of servers behind one domain, including a mail server, and went through this. This piece covers why per-host renewal breaks, the single-issuer pattern that keeps working, and the certbot hook detail that decides whether your script runs at the right moment.

Why each server renewing on its own goes wrong

Nothing in Let's Encrypt stops several hosts from requesting the same wildcard. The problems are around it.

Rate limits. Let's Encrypt allows up to 5 certificates per exact same set of identifiers every 7 days, and the limit is global: orders from every account count. Three servers, each reinstalled or tested a couple of times, can use that up in an afternoon.

Challenge delegation. This is the one behind "only the last server renews". With acme-dns, each registration returns its own random subdomain, and you point _acme-challenge.example.com at it with a CNAME. A name can have only one CNAME. When the third server registers and you update the record, the first two keep writing TXT values to subdomains the CA no longer looks at. Their renewals fail with a DNS validation error.

DNS credentials everywhere. Every host that renews needs a token that can write to your zone. Three hosts means three copies of a key that can change any record you own.

HTTP-01 does not scale either. Behind a home router or one public IP, port 80 can forward to only one machine. On one of our networks inbound port 80 was filtered upstream, and HTTP-01 renewals failed for weeks before anyone noticed. DNS-01 does not depend on any inbound port, which is why it is the right base for this.

The pattern: one issuer, then push

Pick one machine to own the certificate. It holds the DNS API token, runs certbot or acme.sh, and is the only host that talks to the CA. Every other server just receives files.

One renewal schedule to watch instead of three.
One DNS token, on one machine.
Rate limits stop being a concern, because one host renews one certificate.
Adding a fifth server is one more line in a script, not another ACME setup.

The receiving servers need a fixed path for the files, a user that can write there, and permission to reload their own web or mail server. Nothing else.

Use the deploy hook, not the post hook

Certbot has three hook types, and the certbot documentation is specific about when each one runs:

Hook	When certbot runs it	Good for
`--pre-hook`	Before each renewal attempt, only when a cert is due	Stopping a service that holds port 80
`--post-hook`	After each renewal attempt, including failed ones	Starting that service again
`--deploy-hook`	Only after a certificate was actually renewed	Copying the new files and reloading servers

Hooks only fire when a certificate is due for renewal, so the cron or systemd timer can run certbot renew often without side effects. The difference is in what happens when renewal fails. A post hook runs anyway. If your copy script lives there, a failed renewal still pushes the old files out and reloads every server, and your logs say "deployed" while the certificate gets closer to expiry. A deploy hook runs only after a successful renewal.

The deploy hook also gets two variables: $RENEWED_LINEAGE, the live directory with the new files (for example /etc/letsencrypt/live/example.com), and $RENEWED_DOMAINS, the names on the renewed certificate. Any executable file in /etc/letsencrypt/renewal-hooks/deploy/ runs as a deploy hook, so you do not have to remember the flag.

A minimal version:

#!/bin/sh
# /etc/letsencrypt/renewal-hooks/deploy/push-certs.sh
set -e
for host in web2 nas vm-proxy; do
  rsync -aL "$RENEWED_LINEAGE/" "certsync@$host:/etc/ssl/wildcard/"
  ssh "certsync@$host" 'sudo nginx -t && sudo systemctl reload nginx'
done

Three details matter. rsync -L copies the real files, because the live/ directory holds symlinks into archive/. The certsync user on each target should have an SSH key limited to that job, and sudo rights only for the nginx test and reload. And nginx -t runs before the reload, so a broken config on one host does not take the site down.

Mail servers need the same treatment. Postfix, Dovecot and most others load the certificate at start, so they keep serving the old one until reloaded. If your mail server also uses the wildcard, add its reload command to the loop. We covered the mail stack side in Mailcow vs Stalwart vs Mailu in production.

Test it without waiting two months

Run certbot renew --dry-run -v on the issuer first. It talks to the staging server and shows the real challenge error if there is one. The full log is in /var/log/letsencrypt/letsencrypt.log. One catch: according to the certbot docs, deploy hooks do not run during a dry run unless you add --run-deploy-hooks. Without it you have tested the renewal but not the distribution.

Then check what each server actually serves, not what is on its disk:

echo | openssl s_client -connect web2.example.com:443 -servername web2.example.com 2>/dev/null | openssl x509 -noout -enddate

If the dates differ between hosts, the copy or the reload failed on that one. Put that check in your monitoring with an alert at 14 days left. It costs nothing and catches the failures the hook cannot see. If the hosts run in containers, send the hook output to stdout so it ends up with the rest of your logs, as described in why container logs belong on stdout.

Shorter certificates make this mandatory

Let's Encrypt has announced the timeline for shorter lifetimes, in line with the CA/Browser Forum Baseline Requirements. On 10 February 2027 the default profile moves from 90-day to 64-day certificates. On 16 February 2028 it moves to 45 days, and the time a domain validation can be reused drops from 30 days to 7 hours. The opt-in tlsserver profile was scheduled to issue 45-day certificates from 13 May 2026.

A setup where someone copies a renewed certificate to the other servers by hand works, with some luck, at 90 days. At 45 days it fails. A single issuer with a deploy hook keeps working no matter how short the lifetime gets.

Checklist

One host requests and renews the wildcard. The DNS token exists only there.
If you use acme-dns, exactly one registration and one CNAME per name.
The copy and reload script lives in renewal-hooks/deploy/, not in a post hook.
Restricted SSH key per target, sudo only for test and reload.
nginx -t (or the mail server's check) before every reload.
Dry run with --run-deploy-hooks once after setup.
External expiry check on every hostname, alert at 14 days.

Frequently asked questions

Can every server request its own copy of the same wildcard certificate?

Yes, but each copy counts against Let's Encrypt's limit of 5 certificates per exact set of names every 7 days, across all accounts. With three or four servers and a few test runs you can hit it. One issuer avoids the problem.

Why does renewal work only on the server I set up last?

The usual cause is challenge delegation. With acme-dns, each registration gives you a new subdomain, and the _acme-challenge CNAME can point to only one of them. The servers registered earlier write their TXT records to a name the CA never checks.

Does certbot renew --dry-run test my deploy hook?

No. The certbot documentation says deploy hooks do not run during a dry run unless you add --run-deploy-hooks. Pre and post hooks do run.

Do I need to change anything for 45-day certificates?

If renewal and distribution are fully automated, no. If any step is manual, yes: Let's Encrypt's default profile moves to 64-day certificates on 10 February 2027 and to 45 days on 16 February 2028, so a manual copy every two months stops being enough.

A bot enumerated every payment URL on our site and our logs never saw it

LinkBook — Tue, 15 Sep 2026 08:22:36 +0000

We migrated a customer panel's nginx container last week for an unrelated SEO fix (a trailing-slash redirect bug), rebuilt it, and redeployed. Routine stuff.

A few days later I was pulling GA4 numbers to check on organic traffic and noticed something odd: roughly a quarter of our "Direct" channel sessions were landing on /app/customer/login?returnTo=/payment/<plan>/<period> for a huge spread of plan slugs - not two or three, but nearly every pricing tier across every product we sell. Same browser fingerprint every time: (not set), desktop, Germany. Zero engagement on all of them.

That's not a customer. That's a script walking through every checkout URL it could find, probably scraped off our own pricing page.

I went to pull the nginx access log to find the source IP and got nothing. The container had been rebuilt days earlier for that unrelated SEO fix, and access_log was pointed at a file inside the container (/var/log/nginx/access.log), not stdout. The rebuild wiped it. docker logs showed nothing because there was nothing on stdout to show. Our Promtail/Loki pipeline, which only scrapes the Docker logging driver, had never seen a single line of it either.

The bot's IP is gone. There's no way to get it back now.

The actual fix was two lines:

access_log /dev/stdout main;
error_log /dev/stderr warn;

nginx treats /dev/stdout and /dev/stderr as regular files it can write to, so this isn't some special integration, it just means the same log lines that used to sit in a file now go through the container's stdout/stderr, which docker logs already reads and which our existing Promtail config was already scraping. Nothing else changed.

The lesson wasn't really about the bot. Low-rate GET enumeration of public pricing URLs isn't a very interesting threat on its own. It was that we had a monitoring pipeline that looked complete (Promtail, Loki, Grafana, the works) with one container quietly opting out of it, because its base nginx config still defaulted to writing log files the way it would on a bare-metal box from ten years ago. If this container gets rebuilt again next month, whatever it's doing then, we'll actually be able to see it.

If you're running nginx in Docker and haven't checked, it's worth a look at your own access_log/error_log directives. A monitoring stack is only as complete as the containers that actually write to it.

The Docker Compose Restart Policy Gotcha That Cost Us Two Weeks of Silent Downtime

LinkBook — Sun, 30 Aug 2026 10:26:50 +0000

We run a dozen+ services in plain docker compose on a single box — no Swarm, no Kubernetes, just docker-compose.prod.yml and restart: always on everything. Or so I thought.

What actually happened

A host reboot (unrelated maintenance) brought everything back up except three containers. Nobody noticed for two weeks, because the services that stayed down were dev-only instances and one internal monitoring stack — nothing customer-facing paged anyone.

I found it by accident, running docker ps -a for something unrelated:

docker ps -a --format '{{.Names}}\t{{.Status}}' | grep -iE "exited"

Three containers, all Exited (0) from two weeks ago.

The root cause

docker compose down (run once, on purpose, weeks earlier, for an unrelated fix) stops and removes containers. When they got recreated on the next up -d, a few of them had drifted from the rest of the fleet: no explicit restart policy in their compose block, which means Docker defaults to restart: "no".

restart: "no" means exactly what it says — if the container exits for any reason (a clean down, a host reboot, an OOM kill), Docker will never bring it back. Not on daemon restart, not ever, until someone runs up -d again by hand.

The other ~90% of our services had restart: always explicitly set, so a reboot was invisible for them. These three were added later by someone (me) who forgot the line.

The one-liner that would've caught it immediately

for c in $(docker ps -aq); do
  policy=$(docker inspect --format '{{.HostConfig.RestartPolicy.Name}}' $c)
  [ "$policy" = "no" ] && echo "$(docker inspect --format '{{.Name}}' $c): $policy"
done

Now it's a step in our monthly infra audit, not something we find by accident.

The second half of this problem: zero-downtime redeploys

This same fleet has another sharp edge worth knowing about if you're running plain compose in production: docker compose up -d <service> for a redeploy briefly stops the old container before the new one is healthy — a few seconds of 502s behind Traefik/nginx.

We ended up writing a small blue-green script instead of reaching for Swarm/K8s just for this one problem:

start a second container alongside the live one (docker compose run -d --no-deps --name <container>_green <service> — has to be run, not up, since up always reconciles to one container per service)
poll its healthcheck until healthy
remove the old container, rename the new one into its place, restore restart: always
on healthcheck timeout, remove the new container and leave the old one untouched — a safe no-op, not a rollback of something that already happened

Script's public if useful: https://github.com/Fanpino/deploy-bluegreen

Neither of these is a novel idea. But "explicit restart policy on every service" and "compose up -d isn't zero-downtime" are the kind of things that are obvious in hindsight and invisible until they cost you two weeks.