The key went into a config file
on a Tuesday,
by accident,
and it was committed.
Somebody noticed on Thursday.
They deleted the line,
pushed the fix,
and told the channel it was sorted.
It was not sorted.
Git does not forget.
The old commit is still there,
reachable by hash,
in every clone,
in every fork,
on every laptop that pulled that week.
The key is still valid.
That is the only fact that matters.
A secret is not a file.
It is something the world now knows,
and you cannot un-know it
by editing the place it was written.
There is one fix,
and it is not a commit.
Rotate the key.
Make the old value stop working.
Everything else is tidying.
Then go and look at where else it went,
because a secret that reached a repository
reached other places on the way.
Build logs.
CI output, which is often public.
An error message
with the whole request in it.
A backup of that repository
in object storage
under different rules.
A screenshot pasted into a ticket.
Somebody's shell history.
Now the harder question.
How long would it take you
to rotate that key this afternoon?
Most teams cannot say,
because they have never done it.
The value lives in four places,
two of them undocumented,
one of them a service
that reads it only at start up,
and nobody is certain
which customers break
while the old one dies.
That uncertainty is why teams hesitate.
Hesitation is how a two hour problem
becomes a three week one.
Rotation is a procedure,
not a heroic act.
Write it down for every secret you hold.
Where it lives.
Who reads it.
What restarts.
How you confirm the old one is dead.
Then do it once when nothing is wrong,
on a quiet Wednesday,
so the first rehearsal
is not the incident.
A scanner that finds keys in commits
is worth having.
The ability to replace one within the hour
is worth far more.
– Serguey Asael Shinder
Top comments (0)