An expired certificate is the most embarrassing outage in production. No code changed, no deploy happened — yet browsers show a red warning, API clients refuse to connect, and your uptime graph flatlines. I have seen a checkout page go down on a Friday night because nobody owned renewal.
Let's Encrypt certificates last 90 days by design. That is a feature, not a bug: short lifetimes force automation. Certbot is the automation. The difference between "we use Certbot" and "certificates never expire" is everything around Certbot — hooks, timers, monitoring, and knowing the failure modes before they hit you at 2 AM.
This is the setup I run in production, and the one I would defend in a DevOps interview.
How ACME Actually Works
Certbot speaks ACME (Automatic Certificate Management Environment) to Let's Encrypt. The flow is simple:
- Certbot proves you control the domain (a "challenge").
- Let's Encrypt issues a certificate valid for 90 days.
- Certbot installs it and (with a hook) reloads your web server.
- A timer renews it automatically when under 30 days remain.
There are two challenge types that matter, and picking the wrong one is the most common beginner mistake:
-
HTTP-01: Let's Encrypt fetches
http://yourdomain/.well-known/acme-challenge/<token>. Port 80 must be reachable from the internet. Simple, but it cannot issue wildcard certificates. -
DNS-01: Certbot creates a
_acme-challengeTXT record via your DNS provider's API. This is the only way to get a wildcard (*.example.com), and it works even when port 80 is closed or the server is behind a firewall.
If you need a wildcard, or your server is not publicly reachable on port 80, you need DNS-01. Decide this before you issue your first certificate — switching challenge types later means re-issuing.
The Right Way to Issue Your First Certificate
Standard site behind NGINX
sudo apt install certbot python3-certbot-nginx
sudo certbot --nginx -d example.com -d www.example.com
The --nginx plugin edits your NGINX config, adds the certificate paths, and sets up the HTTP-to-HTTPS redirect. It is the fastest path, but know what it does: it modifies your server config. In production I prefer certonly so my config stays mine.
Production preference: certonly + webroot
sudo certbot certonly --webroot -w /var/www/html -d example.com -d www.example.com
Then point NGINX at the files yourself:
ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
Note that /etc/letsencrypt/live/ holds symlinks to the latest files in archive/. Never copy the real files elsewhere — always reference the live symlinks, or renewal will silently leave your server on the old certificate.
Wildcard with DNS-01 (Cloudflare example)
sudo apt install python3-certbot-dns-cloudflare
sudo certbot certonly --dns-cloudflare \
--dns-cloudflare-credentials ~/.secrets/cloudflare.ini \
-d example.com -d "*.example.com"
The credentials file needs a scoped API token (Zone:DNS:Edit for that zone only), and must be locked down:
chmod 600 ~/.secrets/cloudflare.ini
A leaked DNS API token is a full domain takeover. Treat it like a production secret, because it is one.
Renewal: The Part Everyone Gets Wrong
Certbot only renews certificates with less than 30 days left. Test renewal before you trust it:
sudo certbot renew --dry-run
--dry-run uses the Let's Encrypt staging server, so it never counts against rate limits. Run it after every config change. If it fails here, it will fail at 3 AM when the timer fires.
systemd timer vs cron
On Debian/Ubuntu, the Certbot package ships a timer that runs twice a day:
systemctl status certbot.timer
systemctl enable --now certbot.timer
That is enough for most servers. But the timer alone does not reload NGINX after renewal. Without a reload, NGINX keeps serving the old certificate from memory until the next restart — the classic "renewal worked but the site still shows expired" bug.
Fix it with a deploy hook, which runs only when a certificate actually renews:
sudo certbot renew --deploy-hook "systemctl reload nginx"
Or make it permanent per-certificate in /etc/letsencrypt/renewal/example.com.conf:
[renewalparams]
deploy_hook = systemctl reload nginx
You can also drop scripts in /etc/letsencrypt/renewal-hooks/deploy/ — every executable there runs after a successful renewal. Pre and post hooks live in renewal-hooks/pre/ and renewal-hooks/post/.
If you use cron instead of the timer (containers, minimal images), use this and nothing cleverer:
0 3 * * * certbot renew --quiet --deploy-hook "systemctl reload nginx"
Randomize the minute across servers so you don't hammer the ACME API at the same second.
Monitoring: Trust, But Verify Expiry
Automation without monitoring is hope. Check what you actually have:
sudo certbot certificates
openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null \
| openssl x509 -noout -dates
In production, alert on expiry, don't just check it:
- 21 days: warning — renewal should already have happened.
- 14 days: page someone — the timer or hook is broken.
- 7 days: incident — renew manually now and find the root cause after.
Good tools for this: Prometheus blackbox_exporter (probe_ssl_earliest_cert_expiry), ssl_exporter, or a simple daily cron that emails you. The point is that a human hears about it before the browser warning, not from a customer.
Rate Limits: The Trap That Blocks Your Friday Deploy
Let's Encrypt rate limits are real, and hitting them during an incident is miserable:
- 50 certificates per registered domain per week.
- 5 duplicate certificates per week (same exact set of domains).
- 5 failed validations per account, per hostname, per hour.
That is why --dry-run (staging) exists, and why you test against staging first when scripting:
sudo certbot certonly --staging --webroot -w /var/www/html -d example.com
If you burn the duplicate limit debugging in production, you wait a week. Debug in staging.
Production Pitfalls Checklist
These are the failures I actually see, in order of frequency:
- Port 80 blocked. HTTP-01 needs port 80 reachable. A firewall rule "we only need 443" breaks renewal silently.
- NGINX not reloaded after renewal. Certificate renewed, server still serving the old one. Deploy hooks fix this.
-
Certbot's NGINX plugin vs manual config. The plugin rewrites your server block. If you manage NGINX with Ansible/Terraform, use
certonly— or your next config deploy overwrites Certbot's changes. -
Permissions on private keys.
/etc/letsencrypt/live/*/privkey.pemis root-only. Your app server should read it via the web server process, not by copying keys around with loose permissions. - Mixed content after the redirect. HTTPS works, but the page loads HTTP assets and browsers block them. Fix asset URLs; don't blame the certificate.
-
Forgetting the staging environment. Test renewals with
--dry-run. Always. -
Wildcard issued, but the apex domain forgotten.
*.example.comdoes not coverexample.com. Always request both:-d example.com -d "*.example.com". - One server, many services. A certificate on the load balancer does not secure the backend. Decide where TLS terminates and document it.
What Hiring Managers Actually Ask
If you are targeting remote DevOps or sysadmin roles, this topic comes up because it reveals how you think about operations:
- "Your certificate expires in 3 days and renewal is failing — walk me through it." (Answer: check
certbot certificates, runrenew --dry-runverbosely, check port 80, check the timer, check hooks, fall back to manual issuance in staging first.) - "How do you handle certificates across 50 servers?" (Answer: centralize with DNS-01, or use a short-lived internal CA / ACME client like step-ca, or terminate TLS at the load balancer and automate there — never 50 separate cron jobs you can't see.)
- "How do you know before it expires?" (Answer: monitoring and alerting on expiry, not a calendar reminder.)
The answer they want is never "I run certbot once." It is the system around it: automation, hooks, monitoring, and a tested failure path.
The Smart Manager Routine
My monthly routine takes ten minutes:
-
certbot certificates— anything under 30 days that hasn't renewed is a bug. -
certbot renew --dry-run— prove the renewal path still works. - Check the timer:
systemctl status certbot.timer— is it actually enabled? - Check expiry alerts fired (or better, check they would fire — test the alert).
- Review the deploy hook — did the last renewal actually reload the server?
Do that, and SSL expiry stops being a class of outage you can have. Certificates are infrastructure: boring, automated, monitored, and never a 2 AM surprise.
Your turn: How do you manage certificates in production — Certbot, a managed load balancer, or an internal CA? What broke for you the first time? Tell me in the comments.

Top comments (0)