In March a new depot opened and its PCs could not install our dispatch client. Our setup script installs the version pinned for that site, after checking with Get-AuthenticodeSignature that the MSI's signature is valid. It was not valid. Neither was the signature on any other installer in our artifact store, every version back to 2021.
Our code signing certificate had expired the day before. We had bought a replacement weeks earlier, and the release pipeline was already signing new builds with it, so the expiry date was in the calendar as done.
The pipeline signed with signtool sign, a certificate and a digest, and never asked for a timestamp. A signature without one can only be checked against the certificate as it is today, and today the certificate was expired, so every file it had ever signed stopped verifying at once. A signature that carries an RFC 3161 timestamp from a timestamp authority is different. The timestamp is countersigned evidence that the signature was made while the certificate was valid, and Windows goes on accepting it after the certificate expires. One missing option, /tr with a timestamp server and /td sha256, was the difference between a certificate that ages out of new work and one that takes the old work with it.
Our rollback plan was a list of pinned versions in that store. For two days we had no version we could install, current or old, except the newest build.
We re-signed every installer we still support with the new certificate, this time with a timestamp, and the depot was running by the afternoon. The pipeline now signs with /tr and /td, then runs signtool verify /pa /v and fails unless the output shows a timestamp. A weekly job verifies every stored artifact, not just the latest. Certificate expiry dates are monitored with a reminder ninety days ahead, and the reminder includes checking what the old certificate signed. And the setup script, if verification fails, says why, instead of only saying that it stopped.
We treated the certificate as a tool for making new releases. Without a timestamp, it was also the only thing keeping the old ones trusted.
– Sergey Shinder
Top comments (0)