DEV Community

Cover image for Mutual TLS for Internal APIs: Private CAs, Rotation & Pinning Trade-offs
Muhammad Ali Mughal
Muhammad Ali Mughal

Posted on

Mutual TLS for Internal APIs: Private CAs, Rotation & Pinning Trade-offs

#security #webdev #privacy #devops

Public TLS proves a browser can trust your edge. Mutual TLS proves each internal service can trust the caller on the other side of the wire. That second hop is where private CAs, short-lived client certificates and painful rotations live. Pin the wrong thing and a routine renew locks every pod out of the mesh. Skip rotation and a leaked laptop cert is a permanent backdoor.

I wrote this with help from AI tools and checked every openssl command, nginx snippet and rotation step myself. I build PixelValut, a set of free browser-based developer and network tools, and this is Part 6 — the finale — of Web PKI & Certificate Forensics in Practice. Part 1 covered leaked private keys and safe rotation. Part 2 walked X.509 by hand. Part 3 covered revocation and short-lived certs. Part 4 covered Certificate Transparency monitoring. Part 5 covered ACME renewal failures. Here we close the series on the private side of the fence.

📚 Series Roadmap — Web PKI & Certificate Forensics in Practice

Related: Zero-Trust Reverse Proxy Hardening in Nginx · Webhook Security in Production

How do you run mutual TLS for internal APIs safely?

Direct answer: Stand up a private CA (or intermediate) that issues short-lived client and server certificates, automate dual-cert rotation so old and new overlap, verify peers against the CA bundle rather than pinning leaf SPKIs unless you have a strong reason, and treat every service identity as a secret with an expiry alert. mTLS without rotation is worse than no mTLS.

Two linked metal padlocks on a dark surface, AI-generated

Private CA vs public CA for service identity

Public CAs are the wrong tool for east-west traffic. You do not want every microservice hostname in CT logs, and Let's Encrypt rate limits will fight you the moment you issue hundreds of short-lived client certs.

Concern Public CA (ACME) Private CA (internal)
Trust domain Browsers / public clients Only your fleet
CT logging Required for trusted roots Usually off / private
Issuance volume Rate-limited Your policy
Revocation CRL/OCSP/short-lived (Part 3) CRL, short-lived, or CA replace
Identity model DNS names you control SPIFFE IDs, SANs, or CN conventions
Blast radius of CA key leak Huge (public trust) Contained to your mesh

A common pattern: root CA offline, intermediate online in a secrets manager or step-ca / Vault PKI, leaf certs with 24–72 hour lifetimes for clients and a few days for internal servers.

Bootstrap a minimal private CA (lab shape)

Do not use this as-is in production without HSM or hardened key storage. It is the shape you need to understand the chain.

# Root (keep offline after issuance)
openssl genrsa -out root.key 4096
openssl req -x509 -new -nodes -key root.key -sha256 -days 3650 \
  -subj "/CN=Internal Root CA" -out root.crt

# Intermediate
openssl genrsa -out int.key 4096
openssl req -new -key int.key -subj "/CN=Internal Intermediate CA" -out int.csr
openssl x509 -req -in int.csr -CA root.crt -CAkey root.key -CAcreateserial \
  -out int.crt -days 1825 -sha256 -extfile <(printf "basicConstraints=critical,CA:TRUE,pathlen:0\nkeyUsage=critical,keyCertSign,cRLSign")

# Server leaf for payments.internal
openssl genrsa -out payments.key 2048
openssl req -new -key payments.key -subj "/CN=payments.internal" -out payments.csr
openssl x509 -req -in payments.csr -CA int.crt -CAkey int.key -CAcreateserial \
  -out payments.crt -days 7 -sha256 \
  -extfile <(printf "subjectAltName=DNS:payments.internal,DNS:payments.svc.cluster.local\nextendedKeyUsage=serverAuth\n")

# Client leaf for billing-worker
openssl genrsa -out billing-worker.key 2048
openssl req -new -key billing-worker.key -subj "/CN=billing-worker" -out billing-worker.csr
openssl x509 -req -in billing-worker.csr -CA int.crt -CAkey int.key -CAcreateserial \
  -out billing-worker.crt -days 2 -sha256 \
  -extfile <(printf "subjectAltName=URI:spiffe://prod.example/ns/billing/sa/worker\nextendedKeyUsage=clientAuth\n")

cat int.crt root.crt > ca-bundle.crt
Enter fullscreen mode Exit fullscreen mode

Verify what you think you issued:

openssl verify -CAfile ca-bundle.crt payments.crt
openssl x509 -in billing-worker.crt -noout -text | egrep 'DNS:|URI:|Extended Key|Not After'
# Quick served-cert check against a live endpoint (hostname must match)
openssl s_client -connect payments.internal:8443 -servername payments.internal \
  -cert billing-worker.crt -key billing-worker.key -CAfile ca-bundle.crt </dev/null 2>/dev/null \
  | openssl x509 -noout -subject -dates
Enter fullscreen mode Exit fullscreen mode

When you are debugging a handshake from a laptop, the same chain inspection habits from Part 2 apply: SANs, EKU, and pathlen mistakes show up here first. PixelValut's SSL checker is aimed at public endpoints; for private hosts, prefer openssl s_client on a jump host that can reach the mesh.

Nginx and Envoy: require and forward client identity

Nginx terminating mTLS:

server {
  listen 8443 ssl;
  server_name payments.internal;

  ssl_certificate     /certs/payments.crt;
  ssl_certificate_key /certs/payments.key;
  ssl_client_certificate /certs/ca-bundle.crt;
  ssl_verify_client on;
  ssl_verify_depth 2;

  location / {
    proxy_set_header X-Client-Cert-DN $ssl_client_s_dn;
    proxy_set_header X-Client-Verify $ssl_client_verify;
    proxy_pass http://payments_upstream;
  }
}
Enter fullscreen mode Exit fullscreen mode

Envoy-style intent (abbreviated):

transport_socket:
  name: envoy.transport_sockets.tls
  typed_config:
    "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
    require_client_certificate: true
    common_tls_context:
      tls_certificates:
        - certificate_chain: { filename: /certs/payments.crt }
          private_key: { filename: /certs/payments.key }
      validation_context:
        trusted_ca: { filename: /certs/ca-bundle.crt }
        match_typed_subject_alt_names:
          - san_type: URI
            matcher: { prefix: "spiffe://prod.example/" }
Enter fullscreen mode Exit fullscreen mode

Clients must present a cert with clientAuth EKU. Servers need serverAuth. Mixing those up is a classic "works in curl with -k, fails in the mesh" bug.

Rotation without an outage

The safe pattern is dual trust: for a window, both the old and new CA intermediates (or both leaf generations) are accepted.

  1. Issue new intermediate; append it to every trust bundle (old + new).
  2. Roll workloads to mount both server certs if the stack supports dual certificates, or roll pods onto new leaves while clients still trust the old intermediate.
  3. Confirm handshakes with the new leaf using openssl s_client.
  4. Remove the old intermediate from bundles only after the last old leaf expires.
  5. If a leaf is compromised, revoke or wait out short lifetime — same trade-offs as Part 3. Short-lived client certs make "just wait" viable. 
# Bundle that trusts both intermediates during overlap
cat int-new.crt int-old.crt root.crt > ca-bundle.crt
# Hash the bundle so deploys fail if it silently changes
sha256sum ca-bundle.crt
# Or generate a printable fingerprint for runbooks
openssl x509 -in int-new.crt -noout -fingerprint -sha256
Enter fullscreen mode Exit fullscreen mode

Fingerprint and hash workflows pair well with a local hash generator when you are comparing artifacts across laptops and CI. For webhook-style caller auth where mTLS is unavailable, the HMAC patterns in webhook security are the fallback — not a substitute inside the mesh.

When pinning helps vs hurts

Pin target Helps when Hurts when
CA / intermediate SPKI You want to reject a replaced intermediate you did not authorize You forget to dual-pin during intermediate rotation
Leaf SPKI High-risk static appliance with rare rotation Any automated renew — pin breaks on every new leaf
Public key via HPKP-style browser pins (Don't; HPKP is dead for browsers) Same
SPIFFE ID allowlist Identity is the SAN URI, certs rotate freely You pin CN strings that ops renames

Prefer trust-bundle + SAN/SPIFFE allowlists over leaf pins. If you must pin, pin the intermediate SPKI and automate a dual-pin window. The nginx zero-trust hardening notes cover complementary edge controls (rate limits, request smuggling guards) that still matter once mTLS is on.

Diagnostic checklist

  1. Is the root offline and the online issuer only an intermediate with pathlen:0?
  2. Do server leaves have serverAuth and client leaves clientAuth?
  3. Are SANs (DNS or SPIFFE URI) exact matches for what verifiers check?
  4. Is leaf lifetime short enough that stolen certs expire before a human notices?
  5. Does rotation use an overlap window where old and new intermediates are both trusted?
  6. Are trust bundles mounted read-only and hashed in deploy checks?
  7. Do you alert on handshake failures and on certs within N hours of expiry?
  8. Is leaf pinning avoided except for rare static appliances?
  9. Can you revoke or quarantine a single client identity without rotating the whole fleet?
  10. Have you tested openssl s_client with a deliberately wrong client cert and confirmed rejection?

FAQ

Can I use Let's Encrypt certificates for internal mTLS?

Technically for DNS-resolvable names, practically no: CT exposure, rate limits and public trust are the wrong model. Use a private CA.

Is SPIFFE required?

No, but URI SANs with a stable trust domain beat ad-hoc CNs once you have more than a handful of services.

How short should client certificates be?

Hours to a few days if automation is solid. Longer lifetimes push you back toward CRL/OCSP complexity.

Does mTLS replace application auth?

No. It authenticates the calling workload. Users, tenants and object-level authorization still belong in the app.

What breaks first during a bad rotation?

Usually clients that pinned a leaf, or sidecars that cached the old trust bundle. Monitor handshake error rates by code.

Should API gateways terminate mTLS or pass it through?

Often terminate at the gateway, map client identity to an internal header or JWT, and re-encrypt to upstream with mesh mTLS. Avoid forwarding raw client cert headers without authenticating the gateway hop.

Series wrap — Web PKI & Certificate Forensics in Practice

Across six parts we moved from proving a private key was yours, through reading certificates by hand, revocation and short-lived certs, Certificate Transparency monitoring, ACME automation failures, and finally private mTLS with rotation and pinning trade-offs. The through-line is the same: treat certificates as production dependencies with owners, alerts and playbooks — not as files someone copied onto a server in 2019.

Follow & Connect

Top comments (0)