An engineer scrolling a failed build on a Monday afternoon found our registry deploy token sitting in the log in plain sight. Not the whole log. One line, inside a curl command, printed by a step that had been running unchanged for fourteen months.
Masking in CI is string replacement. The runner holds a list of registered secret values and rewrites any exact match before a line reaches the log. It is a good mechanism and it is precisely as clever as that description sounds. Our deploy step read the token from the secret store, combined it with a username, base64 encoded the pair for a basic auth header, and then ran with set -x, which echoes every command it executes. The value the runner was watching for never appeared. A transformation of it did, and the runner had never been told the two were related.
There was a second copy in the same log. When the registry rejected a push, it returned the request headers in its error body, and we logged the whole response object.
So the token had been readable for fourteen months by anyone with read access to the repository, which in our organisation is everybody, and our log retention was set to four hundred days. Rotation was the easy half of the afternoon. Working out what to do about the next one took longer.
Three things came out of it. Any value derived from a secret is registered with the runner explicitly before it is used, which is one line of add-mask, and steps that touch credentials never run with shell tracing on; a lint rule on our workflow files enforces the second half. Responses from authenticated calls are logged by status code and a whitelist of fields rather than wholesale. And the long-lived token is gone entirely: the deploy job assumes a role through OIDC and gets a credential that expires in fifteen minutes, so a future leak is a fifteen minute problem rather than a fourteen month one.
We also run a scheduled scan of the last week of logs for our providers' known key prefixes and for high entropy strings. It found one more, in a different repository, belonging to a service nobody had owned since March.
Masking protects the exact string you handed over. Your pipeline spends its whole life turning that string into other strings.
– Sergey Shinder
Top comments (0)