Automating wildcard certificates: avoid complex NS servers

Practically every infrastructure eventually runs into automatic TLS certificate renewal. In the simple case it is solved by installing certbot: one domain, one server, a cron job, and you can forget about it for years.

Things get harder as the infrastructure grows. Wildcard certificates appear, and those cannot be obtained through the HTTP-01 challenge. Several domains appear, some of them needing two certificates with different key types. A dozen machines terminating TLS appear, each needing fresh files on disk. The "certbot on every host" approach stops working here: it runs into rate limits, requires DNS access from every machine, and drifts out of sync between hosts.

What comes next requires more serious solutions, and those have a price: usually several separate components to operate (a DNS server for the challenge, an ACME client, a hook wiring them together, and a mechanism for delivering issued certificates to the remaining machines).

Let's look at what that price consists of, what the options are at each step, and how we managed to collapse the scheme into one component instead of four.

TL;DR

• Wildcards require the DNS-01 challenge. This is not negotiable: HTTP-01 fundamentally does not work for wildcards. So the automation needs DNS access, and all the complexity grows from there.
• Instead of a DNS provider token: delegate _acme-challenge to your own NS. After that the production zone never changes again, and the automation's authority is reduced to a single technical subdomain.
• We did not have to run our own NS: it is already built into Angie. The ACME client and the challenge DNS server are two directives in the web server config. Four components collapsed into one, and certificates are picked up without a reload.
• Delivery to other machines is pull, not push. Machines fetch their own certificates over HTTP from the internal network on a schedule. The host holding the private keys of every domain needs no SSH access anywhere.
• The distribution endpoint sits behind a gateway with TLS, subnet restrictions, and per-path authorization. The script validates what it downloads (key match, validity dates, SAN) and writes files atomically, so a working certificate cannot be corrupted.
• The timings converge with room to spare: reissue 30 days before expiry, polling once a day: roughly thirty attempts to fetch the new file. On top of that, external SSL expiry monitoring checks what is actually served to clients.

Contents

• A problem everyone has
• Why Angie: the complicated NS turned out to be unnecessary
• A dedicated server
• Ansible templates: certificates described as data
• Certificate delivery: push or pull
• Serving certificates over HTTP
• Fetching certificates on the machines
• How a certificate is actually fetched
• The timings do not lie: why the scheme converges
• Summary

---

A problem everyone has

This is not a unique situation or an exotic one: sooner or later it hits any infrastructure with more than one domain and more than one machine. The task fits on a single line: wildcard certificates for several domains, renewed automatically, distributed across several hosts.

In our case that means several domains along the lines of *.example.com and *.example.net, plus nested zones *.orders.example.com and *.api.example.com. Some domains additionally need a second certificate with a different key type (RSA in addition to ECDSA), since older clients do not speak modern cryptography. Everyone's numbers differ; the substance is the same.

The problem is equally standard and well known. Let's Encrypt issues wildcard certificates only through the DNS-01 challenge. HTTP-01 is fundamentally unavailable for them: proving ownership of *.example.com cannot be reduced to placing a file on one specific host, because a wildcard covers an arbitrary number of names. So the automation must be able to create and remove TXT records in DNS, and that is where all the complexity comes from.

The solutions are standard too. There are exactly three:

Approach	What it requires	What's wrong with it
Edit the zone by hand	Nothing	A recurring manual operation every ~60 days with a hard deadline. Not automation
DNS provider API	Provider token on the host + ACME client with a plugin	Ties you to the provider, breaks when the registrar changes. Permissions far exceed the task: one TXT record is needed, the token grants the whole zone
Delegate _acme-challenge to your own NS	Your own DNS server	One more service to operate: BIND/PowerDNS, its config, its updates, its monitoring

The third option is architecturally the best: the production zone never changes, and the automation's authority is confined to a single technical subdomain that takes part in nothing but ACME.

_acme-challenge.example.com.  NS  ns-acme.internal.example.com.

Set up the delegation once: from then on every query for _acme-challenge goes to our server, which answers it dynamically.

The only thing giving us pause was the price tag: "run your own NS" sounds like a full separate service with its own operational load. And on top of it you still need an ACME client, a hook that drives the NS during issuance, and something to reload the web server once the files arrive. Four components instead of one: all for a single TXT record that lives for two minutes.

So instead of designing a complicated solution for a well-known problem, we went looking for a ready-made one. We found Angie.

---

Why Angie: the complicated NS turned out to be unnecessary

The key finding is that Angie has the ACME challenge DNS server built in. Not an integration with someone else's DNS, not an outbound hook, but its own DNS resolver inside the web server, enabled with a single directive. Along with the ACME client, also built in.

This collapses the whole four-component scheme into one:

Classic stack	With Angie
BIND/PowerDNS + zone + config	the acme_dns_port directive
certbot / lego / acme.sh	the acme_client directive
hook editing DNS during issuance	not needed - it is one process
web server reload after issuance	not needed: $acme_cert_* is re-read on the fly

We got the architecturally correct option (delegating a subdomain to our own NS) for the price of a few config lines instead of a separate service to operate. Which is exactly the point of choosing a tool to fit the problem rather than the other way around.

A minimal working configuration in full:

http {
    acme_dns_port 53;
    acme_client_path /var/lib/angie/acme;

    acme_client example_com https://acme-v02.api.letsencrypt.org/directory
        challenge=dns;

    server {
        listen 443 ssl;
        server_name *.example.com example.com;

        acme example_com;

        ssl_certificate     $acme_cert_example_com;
        ssl_certificate_key $acme_cert_key_example_com;
    }
}

What happens here without a single external dependency:

• Angie listens on port 53 and answers Let's Encrypt's challenge queries itself: no separate BIND/PowerDNS needed;
• it talks to the ACME directory on its own, orders the certificate, publishes the TXT record, waits for validation;
• it stores the issued files under acme_client_path;
• it feeds them into ssl_certificate through the $acme_cert_* variables, re-reading them without a reload: plain nginx cannot do this, there the certificate path is static and requires a reload;
• it tracks expiry and reissues ahead of time (around 30 days before expiry), with no cron and no timers.

The practical value is the absence of moving parts. There is no hook that will quietly break when a provider's API changes. There is no cron job that will stop firing. There is no gap between "the certificate was reissued" and "the service picked it up." Failure is still possible, but there is one of it and it is visible in full.

Worth calling out separately is support for multiple key types on one domain. Two acme_client blocks with different key_type attach to the same server block, and both pairs of ssl_certificate/ssl_certificate_key directives are listed one after another, and Angie picks the right one based on the client's capabilities:

acme_client example_com      https://acme-v02.api.letsencrypt.org/directory challenge=dns;
acme_client example_com_rsa  https://acme-v02.api.letsencrypt.org/directory challenge=dns key_type=rsa;

server {
    listen 443 ssl;
    server_name *.example.com example.com;

    acme example_com;
    acme example_com_rsa;

    ssl_certificate     $acme_cert_example_com;
    ssl_certificate_key $acme_cert_key_example_com;
    ssl_certificate     $acme_cert_example_com_rsa;
    ssl_certificate_key $acme_cert_key_example_com_rsa;
}

---

A dedicated server

The ACME host runs on a separate machine that serves no user traffic. The reasons:

1. Port 53 facing outward. The machine has to accept DNS queries from the internet: the CA's validation servers send them. Combining that with a production frontend means widening the frontend's network surface.
2. Private keys for every domain in one place. Compromising this machine is expensive, so there should be nothing extra on it: no applications, no user workloads.
3. Independent lifecycle. Rebooting or upgrading the frontend must not affect certificate reissuance, and vice versa.

The host firewall opens exactly two ports (53/udp and 53/tcp) in the internal zone; everything else is closed:

firewalld_zone: internal
firewalld_ports:
  - { port: 53, proto: udp }
  - { port: 53, proto: tcp }

---

Ansible templates: certificates described as data

The Angie configuration is generated from a list of sites rather than written by hand. Adding a domain means adding an entry to the list:

angie_sites:
  - name: example_com
    domain_name: "*.example.com example.com"
  - name: example_com_rsa
    domain_name: "*.example.com example.com"
    key_type: rsa
  - name: example_net
    domain_name: "*.example.net example.net"
  - name: orders_example_com
    domain_name: "*.orders.example.com orders.example.com"

The template expands this into a config, solving two non-obvious problems along the way.

Deduplicating server blocks. Two entries with the same domain_name (the default and the RSA certificate) must land in a single server block, not two conflicting ones. The template emits a block only for the first occurrence of each domain set and attaches the rest to it:

{% for site in angie_sites %}
{%   if angie_sites[:loop.index0] | selectattr('domain_name', 'equalto', site.domain_name) | list | length == 0 %}
    server {
        listen 443 ssl;
        server_name {{ site.domain_name }};

{%     for peer in angie_sites | selectattr('domain_name', 'equalto', site.domain_name) %}
        acme {{ peer.name }};
{%     endfor %}
{%     for peer in angie_sites | selectattr('domain_name', 'equalto', site.domain_name) %}
        ssl_certificate     $acme_cert_{{ peer.name }};
        ssl_certificate_key $acme_cert_key_{{ peer.name }};
{%     endfor %}
    }
{%   endif %}
{% endfor %}

Naming the storage directories. Angie stores files in a subdirectory named after the client, and two clients for one domain would collide. The mapping key therefore includes the key type: *.example.com for the default one, *.example.com_rsa for RSA.

The result: a new wildcard certificate is four lines of YAML and one playbook run. The config is never edited by hand, so it cannot drift between hosts.

---

Certificate delivery: push or pull

An issued certificate is needed on the machines that terminate TLS. Two approaches were considered.

Push from the ACME host

A centralized cron job on the ACME host: after reissuance, walk the list of machines, distribute the files over SSH, then reload the web server.

The problems:

Direction of trust. The ACME host would need an SSH key with permission to write to /etc/ssl and restart services on every machine in the fleet. The machine holding the private keys of every domain additionally becomes the point whose compromise grants root everywhere. Two serious assets merge into one.

A host registry. You have to maintain a list of machines and the "machine → certificates" mapping. A new host missing from the list silently stops receiving updates: the failure is quiet and surfaces when the certificate expires.

Event-driven by nature. A push happens once. A machine unavailable at that moment (reboot, maintenance, network glitch) keeps the old certificate. That calls for separate retry logic and tracking of who actually received it.

Knowledge about other people's services. Only the machine itself knows how exactly to reload its web server. With push, that logic would have to be described centrally for each host.

Blast radius. A faulty push rolls a broken certificate out to the entire fleet in one go.

Pull on the client side

The option we chose. The ACME host serves certificates over HTTP inside the perimeter and knows nothing about its clients. Each machine fetches its own files on a schedule.

What this buys:

• No SSH access from the ACME host at all. The direction of initiation is reversed: the connection goes from the client to the distribution endpoint, not the other way around.
• No registry. A new machine installs the script and starts fetching what it needs. No registration on the ACME host side is required, so nothing can drift.
• Idempotence instead of event delivery. Scheduled polling converges by itself: a machine that was powered off will fetch on the next tick. Retries are not needed as a separate concept.
• The reload decision stays local. The machine knows its own web server and its own config test command.
• Failures stay local. Each client validates what it downloaded and leaves the working certificate alone if validation fails. The problem stays on one machine.

The price is propagation delay equal to the polling interval. For certificates valid for 90 days and reissued 30 days before expiry this is immaterial: daily polling gives around thirty attempts to fetch the new file.

---

Serving certificates over HTTP

The ACME host runs a separate server block on port 80 that serves issued files under predictable paths.

map $req_domain $req_site {
    default "";
    *.example.com        example_com;
    *.example.com_rsa    example_com_rsa;
    *.example.net        example_net;
}

map $req_kind $req_file {
    default "";
    cert    certificate.pem;
    key     private.key;
}

server {
    listen 80 default_server;
    server_name _;

    location ~ ^/acme/(?<req_domain>[^/]+)/(?<req_kind>cert|key)$ {
        if ($req_site = "") {
            return 404;
        }
        alias /var/lib/angie/acme/$req_site/$req_file;
    }
}

The structure matters from a security standpoint: paths are not assembled from user input directly. The requested domain name goes through a map, which is an explicit allow-list. A domain not on the list yields an empty value and a 404. Directory traversal via ../ is impossible: what gets substituted into alias is not what arrived in the request but a known-good value from the map. Both map blocks are generated by Ansible from the same angie_sites, so distribution and issuance cannot drift apart.

The access model

Distribution runs over plain HTTP, but it is neither public nor does it treat HTTP as a security boundary. In front of it sits an internal API gateway that:

• terminates TLS: beyond the gateway traffic is encrypted, and plain HTTP remains only on the gateway → ACME host leg inside the perimeter;
• restricts sources by subnet: access is permitted only from the organization's infrastructure segments;
• authorizes per path: specific groups of machines can reach only their own certificates rather than the whole endpoint;
• controls caching: responses are not cached, otherwise a client could receive a stale file after a reissue.

The last two points are substantial. Subnet filtering on its own is a coarse allow-list: without per-path authorization any machine in the permitted segment could pull the private keys of every domain. Authorization at the gateway narrows each group's access to its own certificates.

Caching is a problem specific to the pull model. An intermediate cache along the path means the client honestly asks for an update, honestly receives 200 OK, and honestly installs the old certificate, with no way whatsoever to notice.

---

Fetching certificates on the machines

Every machine that terminates TLS gets a Python script and a systemd timer. The script is written against the standard library: the only dependencies are python3 and openssl, which are present anyway.

What gets fetched

The "machine → certificates" mapping is described by a single map in the inventory:

cert_pull_map:
  web_node_1:
    - name: orders_example_com
      cert_path: "*.orders.example.com/cert"
      key_path:  "*.orders.example.com/key"
      dst_cert:  /etc/nginx/ssl/wildcard_orders.example.com
      domains:   ["*.orders.example.com", "orders.example.com"]

The role picks the entries for the current host out of the map and fails during the run if an entry is incomplete:

- name: Select certificate mappings for this host
  ansible.builtin.set_fact:
    cert_pull_sites: >-
      {{ cert_pull_map | default({}) | dict2items
         | selectattr('key', 'equalto', inventory_hostname)
         | map(attribute='value') | flatten }}

- name: Fail on incomplete mappings
  ansible.builtin.fail:
    msg: >-
      entry '{{ item.name | default('<unnamed>') }}' is incomplete:
      needs name, cert_path, key_path and at least one destination.
  loop: "{{ cert_pull_sites }}"
  when: >-
    item.name is not defined or item.cert_path is not defined
    or item.key_path is not defined
    or (item.dst_cert is not defined
        and not (item.dst_crt is defined and item.dst_key is defined))

Three destination forms are supported: a combined PEM (dst_cert, key and chain in one file, mode 0600), a standalone certificate (dst_crt, 0644), and a standalone key (dst_key, 0600). Different web servers expect different layouts, and the choice is left to the map entry.

Deployment

The script is templated with a syntax check before installation; a broken template will not reach the machine as a non-working file:

- name: Deploy certificate pull script
  ansible.builtin.template:
    src: cert_pull.py.j2
    dest: "{{ cert_pull_script }}"
    mode: "0755"
    validate: "/usr/bin/python3 -m py_compile %s"

Destination directories are created up front, derived from the same paths listed in the map; the list is computed rather than duplicated by hand:

- name: Ensure destination directories exist
  ansible.builtin.file:
    path: "{{ item }}"
    state: directory
    mode: "0755"
  loop: >-
    {{ (cert_pull_sites | selectattr('dst_cert', 'defined') | map(attribute='dst_cert') | list
        + cert_pull_sites | selectattr('dst_crt',  'defined') | map(attribute='dst_crt')  | list
        + cert_pull_sites | selectattr('dst_key',  'defined') | map(attribute='dst_key')  | list)
       | map('dirname') | unique }}

Notification credentials live in a separate file from the ACME host's .env. This split is deliberate: the ACME host holds tokens for the issuance notification channels, and those have no business being on the web nodes, which only ever write to their own channel.

The scheduler

A systemd timer rather than cron, for the sake of Persistent and RandomizedDelaySec:

[Timer]
OnCalendar=*-*-* 23:00:00
Timezone=Europe/Moscow
# Spreads the herd: nodes don't hit the endpoint in the same second.
RandomizedDelaySec=900
# Catches up a run missed while the host was down.
Persistent=true
Unit=wildcard-cert-pull.service

RandomizedDelaySec smears requests across a fifteen-minute window so that a dozen nodes do not arrive at once. Persistent=true catches up a missed run: important for a pull model, where convergence comes from the regularity of polling.

Persistent=true has a side effect worth knowing about on first deployment: on a host where the timer has never fired, systemd considers the previous run missed and starts the service as soon as the timer is enabled, that is, during the playbook run rather than at 23:00. For the initial rollout this is turned off with a separate variable.

The service itself is a oneshot with no restart:

[Service]
Type=oneshot
ExecStart=/usr/bin/python3 /usr/local/sbin/wildcard-cert-pull.py
User=root
# A failed run has already reported itself to the channel, and the next
# nightly run retries from scratch: nothing to restart here.
Restart=no
TimeoutStartSec=600

---

How a certificate is actually fetched

The order of operations in the script is arranged so that the working certificate cannot be corrupted at any step.

Step 1. Check whether anything is needed

Before touching the network at all, the script looks at the expiry of the certificate already installed:

days_left = local_days_left(site)
if not FORCE and days_left is not None and days_left > RENEW_BEFORE_DAYS:
    return False, {"skipped": True, "days_left": days_left}

The threshold is 30 days, matching the reissue point on the ACME host. While there is more runway than that, there is nowhere to go: the new file does not exist yet. In that case the nightly run touches no network at all and writes a journal line along the lines of "N days left, nothing to do." A quiet night stays explainable.

A missing, unreadable, or unparsable local file yields None and always leads to a download. This is an important special case: a freshly deployed machine fetches its certificate immediately instead of waiting for the expiry of something it does not have.

Step 2. Download with retries

def http_get(url):
    last_error = None
    for attempt in range(1, HTTP_RETRIES + 1):
        try:
            request = urllib.request.Request(url, headers={"User-Agent": "wildcard-cert-pull/1"})
            with urllib.request.urlopen(request, timeout=HTTP_TIMEOUT) as response:
                if response.status != 200:
                    raise urllib.error.HTTPError(...)
                return response.read()
        except (urllib.error.URLError, ssl.SSLError, OSError) as e:
            last_error = e
            if attempt < HTTP_RETRIES:
                time.sleep(2 ** attempt)
    raise RuntimeError(f"GET {url} failed after {HTTP_RETRIES} attempts: {last_error}")

Three attempts with exponential backoff. The data stays entirely in memory: nothing reaches disk before every check has passed, so an interrupted download physically cannot leave a truncated file.

A failure on one certificate does not stop the others: the exception is caught inside the loop, the entry goes to the failure list, and work continues.

Step 3. Validation

Four independent checks, any of which cancels installation:

def validate(site, cert_pem, key_pem):
    leaf = parse_leaf(cert_pem)

    if b"-----BEGIN" not in key_pem:
        raise ValueError("private key is not PEM")

    # The key matches the certificate: comparing the public parts.
    if cert_public_key(leaf).strip() != key_public_key(key_pem).strip():
        raise ValueError("private key does not match certificate")

    # Validity: not expired and already in effect.
    not_before, not_after = cert_dates(leaf)
    now = datetime.now(timezone.utc)
    if now >= not_after:
        raise ValueError(f"certificate expired at {not_after}")
    if now < not_before:
        raise ValueError(f"certificate not valid until {not_before}")

    # The SAN covers the expected domains.
    expected = set(site.get("domains") or [])
    if expected:
        missing = expected - cert_domains(leaf)
        if missing:
            raise ValueError(f"certificate does not cover {' '.join(sorted(missing))}")

    return not_before, not_after

The domain check guards not against an attacker but against our own configuration mistake: if the endpoint were for some reason to answer a request for *.orders.example.com with the *.example.com certificate, that is caught before installation rather than by users' browsers afterwards.

Step 4. Compare against what is installed

If the download is byte-for-byte identical to what is already on disk, installation is skipped and the web server is left alone:

if all(read_if_exists(path) == data for path, data, _ in targets):
    return False, {"not_before": not_before, "not_after": not_after}

Step 5. Atomic write

def write_atomic(path, data, mode):
    directory = os.path.dirname(path)
    fd, tmp_path = tempfile.mkstemp(dir=directory, prefix=".cert-pull-")
    try:
        with os.fdopen(fd, "wb") as f:
            f.write(data)
            f.flush()
            os.fsync(f.fileno())
        os.chmod(tmp_path, mode)
        os.replace(tmp_path, path)
    except Exception:
        # On any error mid-write the existing file stays in place.
        if os.path.exists(tmp_path):
            os.unlink(tmp_path)
        raise

The temporary file is created in the same directory as the target, otherwise os.replace across a filesystem boundary stops being atomic. The fsync before renaming guarantees the data reached the disk rather than sitting in a buffer. From any reader's point of view, at every moment the file is either entirely the old one or entirely the new one.

Step 6. Config test, then reload

The reload runs once, after all certificates are installed, and only if something actually changed. It is preceded by a config test:

test = subprocess.run(CONFIG_TEST_COMMAND, capture_output=True)
if test.returncode != 0:
    raise RuntimeError("web server config test failed after certificate update: " + ...)

reload_result = subprocess.run(RELOAD_COMMAND, capture_output=True)
if reload_result.returncode != 0:
    raise RuntimeError("web server reload failed: " + ...)

A failed reload invalidates the success report: the entries are moved out of the updated list and into the failures. A "certificate updated" message while the certificate was not picked up would be worse than no message at all.

Dry-run mode

--dry-run performs everything listed above for real (the download, the key match, the validity and domain checks, the config test) but writes no files, reloads no service, and sends no notifications. It prints exactly what would change:

=== DRY RUN on web-node-1: nothing will be written or reloaded ===
  [dry-run] would replace /etc/nginx/ssl/wildcard_orders.example.com (mode 0600)
  [dry-run] config test passes; would run: /usr/sbin/nginx -s reload
=== DRY RUN finished: 1 would change, 0 failed, 1 unchanged; no notifications sent ===

The --force flag ignores the 30-day threshold and forces the full path; combined with --dry-run it is a way to exercise the whole mechanism without waiting for a renewal window.

---

The timings do not lie: why the scheme converges

Let's put the numbers together.

Parameter	Value
Let's Encrypt certificate lifetime	90 days
Reissue on the ACME host	~30 days before expiry
Polling threshold on the clients	30 days before expiry
Polling interval	daily
Start time jitter	up to 15 minutes
Attempts to fetch the new certificate	~30

The key point: the thresholds on issuance and on fetching coincide, and polling is daily. On the first night after the threshold is crossed, the new certificate may not be on the endpoint yet: Angie reissues on its own schedule, not synchronized with the clients' timers. Nothing bad happens: the client honestly downloads what is there, sees it matches the local file, and leaves without changes. The next night it tries again.

A 30-day window with daily polling gives around thirty independent attempts. Missing an individual night means nothing; for a certificate to reach expiry the failure has to be persistent and last a month.

The degenerate case is covered too: if there is no local certificate at all, or it cannot be read, the threshold does not apply and the download happens on the very first run.

Notifications

The script writes to the corporate messenger when something happens:

• success: a separate message per updated certificate, with the SAN, the paths of the installed files, and the expiry date. One message per certificate rather than a digest: two certificates in one bubble read as a wall of text, and the second one's domains get mistaken for the first's;
• failure: a message with the reason and an explicit note that the files were not modified;
• no change: silent by default, can be enabled optionally.

Messages carry a prefix distinguishing "a node fetched a certificate" from "a certificate was issued": both kinds arrive in the same channel.

One implementation detail worth mentioning: MarkdownV2 escaping. Every wildcard name contains a *, and an unescaped character causes the API to reject the entire message, meaning the failure notification itself silently fails to arrive. All reserved characters are escaped, and paths and domain lists are rendered as code blocks, otherwise the messenger turns a bare domain name into a hyperlink.

External monitoring

The script's notifications are neither the only nor the primary observation loop, because they depend on the script working and on network reachability from the same machine. The primary loop is external SSL expiry monitoring, which checks what is actually served to clients over the TLS handshake.

This is a fundamentally more reliable check. It depends on neither the script, nor the timer, nor the availability of the distribution endpoint, and it catches every reason a certificate failed to update, including ones the script knows nothing about: the file was updated but the web server never re-read it; the certificate was placed in the wrong vhost; the service is running with an old config. The script can report a successful write to disk all it likes, but only the check from the client side counts.

The ACME host additionally runs a status page and an extended-status.json endpoint, refreshed on a schedule: it exposes machine-readable state for every certificate with warning and error thresholds on days remaining. That is the scrape target for the monitoring system, so nobody has to parse openssl output on each host.

---

Summary

The scheme consists of two loosely coupled parts.

Issuance. A separate machine running Angie, with a delegated ACME subdomain and a built-in DNS server. No external ACME clients, no hooks, no provider API tokens. Adding a domain is four lines in the inventory. Permission to edit the production DNS zone is needed by nobody, ever.

Delivery. Pull instead of push. The ACME host knows nothing about its clients and has no access to them; the clients fetch what they need on a schedule through a gateway that restricts access by subnet and by path. SSH access from the machine holding every domain's private keys does not exist as a category.

Each part fails independently and locally. A broken file is never installed thanks to validation, a half-written one never appears thanks to atomic replacement, and one the web server cannot load is caught by the config test before the reload. A missed night costs nothing: the next attempt comes in a day, and there are thirty of them. On top of it all sits external expiry monitoring that checks not the automation's intentions but the certificate actually served to clients.