At ten past nine on a Thursday in May every partner calling our inbound order gateway started failing their TLS handshakes with certificate has expired. The certificate in the cluster was valid for another two months. cert-manager had renewed it thirty days before expiry, exactly as configured, and the Secret held the new one.
The gateway is an nginx deployment we change perhaps twice a year. Its manifest mounted the certificate and key from that Secret into /etc/nginx/certs, one file at a time, using subPath, because the directory also held a file baked into the image and mounting the whole Secret there would have hidden it.
Kubernetes keeps mounted Secrets up to date. The kubelet writes the new contents into a fresh hidden directory and switches a symlink to it, so a process that opens the file again sees the new version within a minute or so. A subPath mount does not take part in this. It binds the one file that existed when the container started, and the documentation says plainly that a container using a Secret through subPath does not receive updates. Our pods had started eighty one days earlier. They still had the certificate the Secret held on that day, and that certificate had now expired.
It would not have been enough without subPath either. nginx reads its certificates when it starts or reloads, and nothing reloaded it. We had two reasons for the same failure and had never met either, because every other service is redeployed weekly and simply starts with whatever the Secret holds.
A rollout restart fixed it in two minutes. The outage had lasted twenty three, most of it spent proving that the certificate really was valid.
The gateway now mounts the whole Secret as its own directory, with no subPath, and the file from the image moved elsewhere. A reloader watches the Secrets each deployment uses and restarts it when they change. An admission policy rejects subPath mounts of Secrets and ConfigMaps unless the pod is annotated to say who reviewed it. And our certificate alert no longer reads the Secret: a probe connects to every TLS endpoint we expose and alerts on the expiry date of the certificate it is actually served, three weeks ahead.
Renewing a certificate and serving it are two different events. We had been monitoring the first one.
– Sergey Shinder
Top comments (1)
Thanks for writing this up. This is a super common failure condition across certificate management. I've seen this happen in IIS and Apache too.
It's one of the reasons we built monitoring directly into CertKit. Every time we deploy a certificate to Kubernetes (or anywhere else) we do a TLS handshake to the endpoint to make sure the expected certificate actually landed and is being used.