DEV Community

Cover image for Let
Billy Walker for Core Solutions

Posted on Originally published at coresolutions.ltd

Let

Most certificate outages start the same way. Someone renewed a certificate by hand, long enough ago that no one remembers doing it, the calendar reminder went to a person who has since moved teams, and the first anyone hears of it is a browser warning on a Monday morning. Let's Encrypt used to soften that with an expiry email. It stopped sending them in June 2025, and certificate lifetimes are now shrinking on a schedule the whole industry has signed up to.

On Kubernetes, the standard answer is cert-manager: a controller that requests certificates from Let's Encrypt, stores them as Secrets, and renews them before they expire. This guide sets it up from scratch for both Ingress and Gateway API, and explains what each piece does, so that when something goes wrong you know where to look. (This is about certificates for traffic arriving from outside; encryption between services inside the cluster is a different job, covered in zero-trust networking in Kubernetes with Istio service mesh.)

How cert-manager gets a certificate

You tell cert-manager two things. A ClusterIssuer says where certificates come from: for Let's Encrypt, an ACME server URL, an account, and how to prove you own a domain. An annotation on your Ingress or Gateway says which hostnames need a certificate, and cert-manager creates a Certificate resource from it.

From there it works down a short chain of objects, and debugging mostly means finding the one that's stuck:

  1. CertificateRequest: a signing request for a fresh private key.
  2. Order: the conversation with Let's Encrypt for those hostnames.
  3. Challenge: one per hostname, where cert-manager proves it controls the domain.
  4. The signed certificate and key land in a Secret as tls.crt and tls.key, ready for your ingress controller or gateway to serve.

Let's Encrypt offers two ways to pass a challenge: serve a token over HTTP on the domain (HTTP-01), or publish it in a DNS TXT record (DNS-01). We'll start with HTTP-01, which needs no extra credentials.

Before you start

You'll need:

  • a Kubernetes cluster, with kubectl and helm pointed at it
  • an ingress controller (kubectl get ingressclass) or a Gateway API implementation (kubectl get gatewayclass) already serving traffic
  • a domain you control, with a DNS record for your test hostname pointing at that controller's public address
  • port 80 open to the internet, because Let's Encrypt always makes HTTP-01 checks on port 80

If you're picking an ingress controller fresh, skip ingress-nginx: the Kubernetes project retired it in March 2026. The examples use traefik as the class name; swap in your own.

Install cert-manager

helm upgrade --install cert-manager oci://quay.io/jetstack/charts/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --set crds.enabled=true
Enter fullscreen mode Exit fullscreen mode

If you'll use Gateway API, add --set config.gatewayAPI.enabled=true, and install the Gateway API CRDs first: cert-manager only looks for them at startup.

A few details trip up people following older guides. The OCI chart is the source of truth; the old charts.jetstack.io repository still works but lags each release by a few hours. crds.enabled replaces the deprecated installCRDs flag. And without --version you get the latest release, which is fine for a first run, but pin it in your own config so upgrades happen when you choose.

kubectl get pods -n cert-manager should show three pods: the controller, a webhook that validates your cert-manager resources, and a cainjector. If your first kubectl apply fails with a webhook error, it's still starting; try again in a moment.

Start with Let's Encrypt staging

Staging has the same API as production and far looser rate limits, and its certificates aren't browser-trusted. That makes it the place to get the plumbing wrong without locking yourself out of production.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: platform@example.com
    privateKeySecretRef:
      name: letsencrypt-staging-account-key
    solvers:
      - http01:
          ingress:
            ingressClassName: traefik
Enter fullscreen mode Exit fullscreen mode
  • privateKeySecretRef is where cert-manager stores your Let's Encrypt account key, which it generates and registers on first use. It's your account identity, separate from any certificate's key.
  • email is optional. Since Let's Encrypt stopped sending expiry warnings, it's no longer a safety net.
  • ingressClassName sets the class on the temporary Ingress cert-manager creates to answer a challenge. Leave it out and every ingress controller in the cluster may try to serve that challenge, which fails confusingly as soon as you run more than one.

Apply it, then run kubectl get clusterissuer letsencrypt-staging. READY turns True once Let's Encrypt has accepted the account; if it doesn't, kubectl describe usually points at a typo in the URL or no outbound internet access.

Issue your first certificate with an Ingress

Add one annotation and a tls block to an ordinary Ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: demo
  namespace: demo
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-staging
spec:
  ingressClassName: traefik
  tls:
    - hosts: [demo.example.com]
      secretName: demo-tls
  rules:
    - host: demo.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: demo
                port:
                  number: 80
Enter fullscreen mode Exit fullscreen mode

cert-manager sees the annotation and creates a Certificate named after secretName. Watch the chain unfold:

kubectl get certificate,order,challenge -n demo
Enter fullscreen mode Exit fullscreen mode

Behind that, cert-manager:

  1. starts a small solver pod and a temporary Ingress that serves the token at http://demo.example.com/.well-known/acme-challenge/<token>
  2. fetches that URL itself first, a self-check that catches mistakes like DNS pointing elsewhere before they count as failed validations
  3. tells Let's Encrypt it's ready, and Let's Encrypt fetches the same URL from several locations around the internet, so "it works from my laptop" proves very little
  4. writes the certificate into demo-tls and tidies up the solver

When the Certificate shows READY True, check what you got:

kubectl get secret demo-tls -n demo -o jsonpath='{.data.tls\.crt}' \
  | base64 -d | openssl x509 -noout -issuer -enddate
Enter fullscreen mode Exit fullscreen mode

The issuer starts with (STAGING), so the browser warning is expected. If it stays READY False, run kubectl describe on the certificate, order and challenge in turn. The Challenge is where most first failures sit, usually as a 404 (DNS points at the wrong load balancer) or a timeout (port 80 closed). The cmctl CLI shows the whole chain in one go: cmctl status certificate demo-tls -n demo.

Move to production

Copy the staging issuer, name it letsencrypt-prod, set server to https://acme-v02.api.letsencrypt.org/directory, and give it its own privateKeySecretRef. Then change the annotation on the Ingress to letsencrypt-prod. cert-manager notices the Secret came from a different issuer and fetches a trusted certificate on its own; there's nothing to delete.

Production has real rate limits, which is why staging comes first. The one people hit is 5 certificates for the same set of hostnames every 7 days: delete and recreate a certificate a few times while debugging and you're locked out for days. Failed validations are capped too, at 5 per hostname per account per hour.

Using Gateway API instead

With Gateway API, the routing that one Ingress did is split in two:

  • The Gateway owns the listeners and terminates TLS, so the certificate and the cert-manager annotation go here.
  • An HTTPRoute sends a hostname to a Service. It never touches the certificate; the Gateway has decrypted the traffic before the route sees it.

That split suits a common setup: one shared Gateway owned by the platform team, with app teams attaching routes from their own namespaces.

First, the issuer's solver needs to answer challenges through the Gateway rather than an Ingress. Swap the solvers block in both issuers for:

solvers:
  - http01:
      gatewayHTTPRoute:
        parentRefs:
          - name: public
            namespace: gateway
            kind: Gateway
Enter fullscreen mode Exit fullscreen mode

Then the shared Gateway:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: public
  namespace: gateway
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-staging
spec:
  gatewayClassName: traefik
  listeners:
    - name: http
      hostname: demo.example.com
      port: 80
      protocol: HTTP
    - name: https
      hostname: demo.example.com
      port: 443
      protocol: HTTPS
      tls:
        mode: Terminate
        certificateRefs:
          - name: demo-tls
      allowedRoutes:
        namespaces:
          from: All
Enter fullscreen mode Exit fullscreen mode

The http listener is where Let's Encrypt's check arrives; cert-manager attaches a temporary route there for the exact challenge path, so it wins over any catch-all redirect. The https listener serves real traffic, and from: All lets routes from other namespaces attach (on a shared platform, narrow that with a namespace Selector). cert-manager creates the demo-tls certificate in the gateway namespace, because a listener reads its Secret from the Gateway's own namespace. It skips any listener without a hostname, mode: Terminate and a same-namespace certificateRefs, which is the usual reason a Gateway gets no certificate.

The app team's HTTPRoute has no TLS settings at all:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: demo
  namespace: demo
spec:
  parentRefs:
    - name: public
      namespace: gateway
      sectionName: https
  hostnames: [demo.example.com]
  rules:
    - backendRefs:
        - name: demo
          port: 80
Enter fullscreen mode Exit fullscreen mode

sectionName: https attaches it to the HTTPS listener only. Watch issuance with kubectl get certificate,challenge -n gateway, then switch the annotation to letsencrypt-prod as before.

Every new hostname means another pair of listeners. Past a handful of apps, the usual move is one wildcard listener for *.example.com with a single wildcard certificate, and each team's route claims a hostname under it. Wildcards need DNS-01.

When HTTP-01 doesn't fit: DNS-01

Use DNS-01 for wildcard certificates, which Let's Encrypt only issues this way, and for hostnames the internet can't reach on port 80, such as an internal dashboard behind a VPN. cert-manager proves control by writing a TXT record at _acme-challenge.<your-domain>, so it needs API credentials for your DNS provider. Cloudflare, Route 53, Google Cloud DNS, Azure DNS and a few others are built in, with webhooks for the rest.

Solvers take selectors, so you can keep HTTP-01 as the default and send just the wildcard through DNS-01:

solvers:
  - http01:
      ingress:
        ingressClassName: traefik
  - dns01:
      cloudflare:
        apiTokenSecretRef:
          name: cloudflare-api-token
          key: api-token
    selector:
      dnsNames: ['*.example.com']
Enter fullscreen mode Exit fullscreen mode

The solver with a matching selector wins, and everything else falls through to HTTP-01. Anyone who can read that token's Secret can rewrite your DNS, so scope the token to one zone. If your cluster resolves your domain through an internal DNS view, the self-check looks in the wrong place and waits forever; the controller flags --dns01-recursive-nameservers and --dns01-recursive-nameservers-only point it at public resolvers instead.

How renewal works

cert-manager renews once two-thirds of a certificate's lifetime has passed and swaps the new one into the same Secret. Ingress controllers and gateways generally pick that up without a restart; an app that reads a mounted certificate once at startup won't.

You don't need to tune that default, and it matters more each year. The CA/Browser Forum, which sets the rules for publicly trusted certificates, agreed in 2025 to step maximum lifetimes down, and Let's Encrypt published its own timeline:

From What changes
15 March 2026 Public TLS certificates capped at 200 days
10 February 2027 Let's Encrypt default drops from 90 days to 64
15 March 2027 Industry cap drops to 100 days
16 February 2028 Let's Encrypt default drops to 45 days
15 March 2029 Industry cap drops to 47 days

At 45 days, a failed renewal leaves you about 15 days to notice instead of 30. You can rehearse that now: Let's Encrypt's tlsserver profile already issues 45-day certificates. Create a second issuer with profile: tlsserver under acme, move one low-risk hostname onto it, and any renewal problem shows up while everything else still has a 90-day buffer. (There's also a shortlived profile at 160 hours, which only makes sense once the alerting below is in place.)

One more renewal feature to know about: ACME Renewal Information (ARI) lets Let's Encrypt tell clients when to renew, for example after a mass revocation, and ARI-driven renewals are exempt from rate limits. At the time of writing, cert-manager supports it as an alpha feature behind the ACMEUseARI feature gate.

What to alert on before you forget it exists

A failed renewal is quiet. The old certificate keeps serving for weeks, so nothing looks wrong until it expires. A common cause is an unrelated change months later: the hostname moves behind a CDN, a WAF rule blocks unfamiliar paths, or DNS points at a new load balancer, and the next challenge can't reach the solver. cert-manager records the failure on a Challenge no one is looking at.

cert-manager exports Prometheus metrics, and two alerts cover it:

# Not Ready (pair with for: 1h)
certmanager_certificate_ready_status{condition="True"} == 0

# Renewal overdue by more than a day
time() - certmanager_certificate_renewal_timestamp_seconds > 86400
  and certmanager_certificate_renewal_timestamp_seconds > 0
Enter fullscreen mode Exit fullscreen mode

The second measures time since renewal was due, so it works at any certificate lifetime and won't need retuning as lifetimes shrink. Add both the day your first production certificate goes green, while you still remember cert-manager is there.

Top comments (0)