The message arrived through a generic contact address, which is how these things usually reach you rather than through your carefully designed alerting. Someone had found an access key in a repository of ours, in a file committed three years earlier by a contractor who had long since finished the engagement. The key still worked. It had never been rotated, because nobody had ever known it existed, and it had read access to a storage account that held rather more than anyone would have guessed from the repository's name.
The part that stays with me is not the mistake itself. Committing a secret is an ordinary human error and every engineer I respect has done it at least once. What stayed with me was how completely our controls had missed it. We had scanning on the pipeline, so anything committed after we turned it on would have been caught. History was never scanned. We had a rotation policy, but it applied to credentials in the vault, and this one had never been in the vault, so it was not on any list to rotate. Every control we had was pointed at the process we intended people to follow, and none of them were pointed at what had actually happened before those controls existed.
Deleting the file, incidentally, does almost nothing. The commit is still in the history, the fork somebody made is still out there, and any mirror that ever cloned the repository has it. Once a secret has been public, the only real remediation is rotation, and rotation is where you find out how many things quietly depend on a credential nobody documented. Ours took most of a day and involved two systems that we discovered by breaking them.
So we scanned the full history of every repository, which is a tedious afternoon and produced a list nobody wanted to see. We also changed the question we ask during audits. Not "do we have a secrets policy" but "when was each of these credentials last rotated, and who would notice if it were used from an unfamiliar address". The second question is much harder, and it is the only one that would have found this.
A control that only covers what you do from now on leaves the whole of your past unguarded.
– Serguey Shinder
Top comments (0)