In September a customer's security team sent us a scan of the container image for our command line tool, the one we publish openly so partners can automate their bookings with us. It listed a critical OpenSSL advisory that had been fixed upstream three weeks earlier. Our image was supposed to be rebuilt every night for exactly that reason. The newest tag in the registry was eleven weeks old.
The tool lives in a public GitHub repository. Next to the release workflow sits a nightly one, on a schedule trigger, that rebuilds the image on the same base, lets the package manager pick up security updates, runs the smoke tests and pushes the result. It had worked for two years, and nobody looked at it, because it was meant to need nobody.
When I opened the Actions tab, the workflow carried a banner saying it had been disabled because the repository had seen no activity. GitHub documents the rule: in a public repository, scheduled workflows are switched off automatically after sixty days without activity. The tool had been stable since its spring release. Nobody had pushed, opened a pull request or touched an issue, and a quiet repository was exactly what we wanted. The schedule did not fail. It simply stopped producing runs, and a workflow that does not run sends no failure notices.
We had read the nightly job as our own cron. It was a feature of somebody else's platform, with that platform's rules about when it is worth running.
We re-enabled it the same morning and the patched image was out within the hour. The schedule now lives in a private operations repository, which that rule does not cover, and builds the public repository's code from there. The last step of each run calls a heartbeat check in our monitoring, outside GitHub, which pages if no run has reported in for twenty six hours. A separate job reads the registry every morning and alerts when the newest image is more than three days old, because the age of what customers pull is the number we actually care about. And the image's labels carry its build date, so a customer's scanner shows it next to the advisory.
A schedule that depends on activity stops when you finish. For a job whose purpose was to keep a finished thing patched, that was the worst moment it could have picked.
– Sergey Shinder
Top comments (0)