acme.sh on a Server Where certbot Will Not Install

Four sites on one legacy box went dark. Not slow, not 500 - dark, as in the TLS handshake failed before a request existed. They had been down for five days before anyone noticed, because nothing was watching the one thing that mattered.

The cause was mundane: certificate renewal lived inside a control panel, the panel's licence expired, and its daily renewal cron died with it. Seven more certificates on the same machine were scheduled to expire over the following three weeks.

The distro was years past end of life, so certbot would not install cleanly and the vendor packages were long gone. This is the recipe that fixed it in under thirty minutes without touching a single vhost file, and the monitoring that makes sure it cannot happen quietly again.

Why acme.sh and not certbot

certbot is a Python application with a dependency tree. On a current distro that is fine. On an end-of-life one it is the whole problem: the system Python is too old for modern cryptography wheels, pip wants a compiler you do not want on a production box, and the snap route needs a daemon the distro never shipped.

acme.sh is a shell script. It needs bash and either curl or wget. That is the entire dependency list. It talks to the ACME API over HTTPS via curl, so even an ancient OpenSSL is not in the signing path, and bash 4.2 is new enough.

There is also a pleasant irony worth knowing if you are escaping a panel: several commercial panels use acme.sh under the hood. You are not replacing the mechanism, you are taking it out from under something that stopped running.

Install, and fix the default CA

curl https://get.acme.sh | sh -s email=[e-mail]
source ~/.bashrc

Then immediately do the step most guides skip. Since version 3, acme.sh defaults to ZeroSSL, not Let's Encrypt, and ZeroSSL requires account registration with EAB credentials. If you wanted Let's Encrypt, say so once:

acme.sh --set-default-ca --server letsencrypt
acme.sh --register-account -m [e-mail]

Skip this and your first issue attempt fails with a registration error that looks like a network problem.

Issue without touching the vhosts

The goal is zero changes to the web server configuration, because on a box like this the configs are generated by something you no longer control and a hand edit will be overwritten.

Two things make that possible.

One: use webroot validation into the directory the server already serves. Check first whether a path is already wired up - panels usually leave one behind:

curl -s -o /dev/null -w '%{http_code}\n' \
  http://yourdomain.dev/.well-known/acme-challenge/test

Anything other than a connection error means the path is reachable and you only need to find which directory on disk it maps to. Then:

acme.sh --issue -d yourdomain.dev -d www.yourdomain.dev \
  -w /var/www/letsencrypt

Two: write the certificate to the filenames the server already expects. Do not point the config at acme.sh's directory; point acme.sh at the config's filenames. --install-cert does exactly that, and the reload hook makes it live:

acme.sh --install-cert -d yourdomain.dev \
  --fullchain-file /etc/ssl/sites/yourdomain.dev.crt \
  --key-file      /etc/ssl/sites/yourdomain.dev.key \
  --reloadcmd     "nginx -s reload && apachectl graceful"

The content of the files changes, the paths never do, and nothing in the web server configuration is edited.

Expired certificates do not block renewal

The natural worry: every site redirects HTTP to HTTPS, and HTTPS is broken, so how does validation reach the challenge file?

It works. Let's Encrypt follows the redirect to HTTPS and does not validate the certificate during the HTTP-01 challenge. A long-expired certificate, a self-signed one, a hostname mismatch - none of it stops issuance. You do not need to unwind the redirects first, which is good, because on a generated config you could not anyway.

Automate it, then check for a rival

acme.sh --install-cronjob
crontab -l | grep acme

That second line is the important one, and it is the mistake we made. The panel's renewal cron was disabled - and a second panel cron doing the same job at a different hour was sitting four lines further down, missed on the first pass. Two renewal mechanisms racing over the same files is a problem that surfaces weeks later as a certificate that is valid on disk and stale in the server's memory.

Read the whole crontab. Comment out, do not delete, so you can see what used to be there:

crontab -l > /root/crontab.bak-$(date +%Y%m%d)

acme.sh renews at 60 days by default, so a failed run has 30 days of retries before anything is visible to users. That is a generous margin and exactly why a silent failure can run for months.

Monitor the certificate that is served, not the file on disk

This is the part that would have caught the original outage, and almost nobody sets it up.

Checking the file is not enough: it tells you that renewal wrote something, not that the running process picked it up. A reload that silently failed leaves a fresh file on disk and an expired certificate in memory. Check the handshake:

#!/usr/bin/env bash
# days until the served certificate expires, per domain
for d in "$@"; do
  end=$(echo | openssl s_client -servername "$d" -connect "$d":443 2>/dev/null \
        | openssl x509 -noout -enddate 2>/dev/null | cut -d= -f2)
  if [ -z "$end" ]; then echo "CRITICAL $d: no TLS handshake"; continue; fi
  days=$(( ( $(date -d "$end" +%s) - $(date +%s) ) / 86400 ))
  [ "$days" -lt 25 ] && echo "WARN $d: $days days left" || echo "OK $d: $days days"
done

Run it daily across every hostname the box serves and alert under 25 days. Two properties make it the right check: it fails loudly when the handshake fails at all, which is the real outage, and because it reads what the server hands a client, a reload that did not happen cannot hide behind a renewed file.

Add a tail of the renewal log to the same job and you have the whole story in one alert:

tail -n 5 /var/log/acme-renew.log

The result

Eleven certificates reissued in one pass, all valid for the next three months, four dead sites back up, renewal owned by a 200 KB shell script with a cron entry we can read. No vhost file was modified; only the bytes inside the certificate files changed.

The wider lesson is not about ACME clients. It is that the expiry of a thing is not the failure; the silence is. The renewal had been broken for roughly two months before the first site went down, and the only reason anyone found out was a site going dark in front of users. A twenty-line handshake check would have raised it the first week.

If you are putting a new box together rather than rescuing an old one, Caddy does all of this with no configuration at all - we wrote that stack up in Caddy and Docker in production. This recipe is for the machine you inherited, the one where the config is generated, the distro is frozen, and the only acceptable change is the smallest one.

---

Originally published at news.rozoom.app.