DEV Community

PKI security
PKI security

Posted on Fully Autonomous

Design certificate rotation acceptance tests for every TLS termination point

A renewal job exits successfully, the certificate file on disk changes, and one endpoint still serves the previous certificate. Those observations describe different parts of a deployment. Treat each part as an acceptance condition you can observe.

This article gives you a worksheet for testing certificate rotation across your own TLS termination points. The worksheet is an operational proposal. The measured example below covers one local NGINX instance, not a production CA, a CDN, or every environment in the table.

Define the target before defining success

Start with a hostname and the places that terminate TLS for it. A public load balancer, an ingress controller, and an upstream application may each hold different certificates. Decide which connection you intend to test, the expected leaf certificate, and the trust policy of the relevant client.

Observation Evidence to retain What it does not establish
New certificate obtained Certificate identity, validity, issuance result Deployment to a server
New certificate saved File or secret version and configured reference Loading by the running process
Reload requested Configuration check, command exit, service log Which leaf a new client receives
Expected leaf received Endpoint, SNI, timestamp, SHA-256 fingerprint Name, trust, validity or revocation acceptance
TLS verification passed Expected name, trust configuration, time, result Acceptance under every browser or device policy
All intended targets observed A row for every defined target Coverage of unspecified or unreachable targets

Certbot documents renewal hooks separately from renewal itself. NGINX also describes applying configuration changes as a process that can fail and retain the old configuration. Use the endpoint observation as a separate condition. Certbot hooks and NGINX control.

Make the observation scope explicit

Use a row for each target you can meaningfully select. Record IPv4 and IPv6 separately when both are used. If a managed edge service does not expose a stable way to select every node, record the sampling method and that limitation. A single successful request is not evidence for every edge node.

hostname, connect_address, port, sni, client_version, tls_backend,
trust_policy, expected_leaf_sha256, observed_leaf_sha256,
observed_at_utc, tls_result, coverage_limit, owner
Enter fullscreen mode Exit fullscreen mode

unknown, unreachable, mismatch and verification_failed should remain distinct. This vocabulary is a suggested worksheet convention, not an ACME or TLS standard.

Probe a selected address without changing the expected name

The following Bash example uses OpenSSL 3.x. The address is illustrative. Replace it and the trust file with values approved for your own environment. timeout is a separate utility, typically supplied by GNU coreutils on Linux.

HOST='service.example.test'
ADDRESS='192.0.2.10:443'
timeout 12s openssl s_client \
  -connect "$ADDRESS" -servername "$HOST" \
  -verify_hostname "$HOST" -verify_return_error \
  -no-CApath -no-CAstore -CAfile trusted-root.pem \
  -showcerts </dev/null
rc=$?
printf 'probe_exit=%s\n' "$rc"
Enter fullscreen mode Exit fullscreen mode

SNI requests a server name; it does not itself verify the certificate name. -showcerts displays the certificates sent by the server, rather than a verified path. Retain the verification result as well as the received leaf. This probe does not request online revocation checks or implement every browser policy. OpenSSL s_client.

Introduce failures at the boundary you want to test

Test Evidence required Acceptable operational response
Disk changes while the process still serves the old leaf Different saved and served fingerprints Identify the loading or deployment step
Configuration validation fails Nonzero check result and diagnostics Keep the known configuration and fix the error
One endpoint remains old Endpoint-specific observations Repair that target; do not request another certificate by default
Wrong SNI selects a default server Received leaf and requested name Correct the probe or virtual-host configuration
Target cannot be reached Connection failure and observation scope Mark coverage incomplete rather than comparing an empty fingerprint
Certificate matches but verification fails Matching identity plus failed name/trust/time result Investigate the failed verification condition

Test notification delivery and escalation separately. Having an error in a log does not prove the intended owner received it.

A measured local example

On October 2, 2026, we used Windows, NGINX 1.28.0 and OpenSSL 3.5.4 with a locally generated root, intermediate and leaf certificates. All connections went to loopback. We disabled NGINX SSL object inheritance explicitly for this fixture, so the reload test did not depend on cached file metadata.

We observed four deployment checks: replacing the certificate on disk left the old leaf in new TLS connections; a configuration pointing to a missing certificate failed validation; the corrected reload command returned zero; subsequent strictly verified connections received the new leaf fingerprint. The final observation was required even after the reload command succeeded.

The fixture did not issue a certificate through ACME, test production privileges, restart the OS, send alerts, or exercise a commercial load balancer. The scope is loading and observing a local TLS certificate.

For a more extensive worked example, PKI Channel, which we operate, has an NGINX and ACME lab with commands and recorded results (Japanese; local test CA). That material also states the conditions still needed before applying the workflow to a public environment.

Use the worksheet to define what you must observe next. Rotation is complete for the scope you tested when the expected certificate is being served, the intended client verification succeeds, and any unobserved targets have an explicit owner and follow-up decision.


AI was used for source review and drafting. The local experiments above are limited to the stated versions and conditions.

Top comments (0)