You probably are here because you leaked a secret somewhere and want to get straight to rotating the secret. If you are a solo developer or you know for sure you are the only user of the secret and understand what rotating the secret might disrupt, start here: Rotate the secret and store the new credential safely.
If you work in a team and are not sure who uses this secret, what it gives access to, or what outages might occur from rotating it, then please read on.
When a secret leak occurs, no matter what form it takes, it is likely your first instinct to rotate the affected secret. While this is undoubtedly one of the steps you should likely take in an incident, it should not be the first. And it definitely should not be done without understanding what rotating the key or password will affect. Let's take a closer these steps to do when a secret incident occurs:
- Understand what the secret unlocks.
- Figure out how critical any associated data or system is.
- Check your logs for unauthorized use of the secret.
- Determine if any data or service access has been leaked.
- Find out what will break when you rotate the secret.
- Rotate the secret and store the new credential safely.
- Fix any broken workflows or production deployments.
- Review the incident and create an action plan to avoid further secret leakage.
If you are familiar with other incident response plans, like this one from CISA, you might notice this bears some resemblance. Those general guidelines steps of Preparation, Identification, Containment, Eradication, Recovery, and Lessons Learned are a great starting point for understanding incident response, but in this article, we will provide a more detailed path for the specific scenario of leaked credentials.
Breathe
Before you do anything, it is important to breathe and stay calm. You are not the first person to push a secret somewhere it didn't belong, and you certainly won't be the last. If you panic, you are more prone to start making more mistakes and possibly make the situation worse. When people panic, they also tend to start thinking in terms of hiding their mistakes and trying to lay blame elsewhere.
Instead, it is time to follow your team's incident response plan. Ideally, you and your security team have made a step-by-step plan to gather data and evaluate the secret and what systems it impacts, see if the credential has been used in an unauthorized way, and then rotate the secret in a way that causes the least disruption.
Let's walk through the elements of a good secrets incident response plan in more detail. If you don't already have a plan or feel your current plan is missing a step or two, we encourage you to use this resource as a guide in improving your internal plans.
What does the secret unlock?
The first thing you want to understand is what data and services an attacker could access if they had this secret. In the very interconnected world of DevOps, this might not be very straightforward, but it is a vital first step toward determining the best course of action.
Not all secrets are created equal. Some secrets will give an attacker a path to access more and more systems and scripts containing more secrets, such as what happened when Uber got breached. Conversely, a hacked API token may lead to a single S3 bucket containing nothing of value.
Depending on the complexity and size of your organization or project, this can be a very fast or a very slow process. If you are the developer, this info is likely already known, and you might finish this step as soon as you link to the proper documentation. Don't second-guess yourself. If, on the other hand, the process takes a while, while it can make you anxious, it is still best to understand the situation thoroughly before proceeding to the rotation step.
Once you have a good idea of the data and systems involved, it is time to move on to the next step, asking how sensitive that data is.
Is the associated data or system critical?
All organizations consume and produce a lot of data, and we are all creating more every day. Some of it, like any customer personal identifying information, PII, or Payment Card Industry data, PCI, is highly regulated, and any unauthorized access must be remediated with great urgency. Some data, though, is far less critical, such as images you licensed from a stock photo site or blog drafts en route to being publicly published on your website.
If you are unsure of the full ramifications of the leaked credential, it is still not time to panic. It is better to err on the side of caution and treat any secret leak as critical until you have the data to support a lesser severity. And just like in the previous step, if it takes a while to gather, that is OK, don't get in a hurry. Likewise, if you know already and it is documented clearly, it is OK to move on rapidly.
Once you understand the severity and scope of the data and service access involved, you can appropriately alert the correct teams and escalate the incident response appropriately.
Has the secret been used?
The next question is if the secret has been used, and if so, by whom? The answer to this question is in the logs. This question might need to be answered by the security team or by the DevOps team in charge of that particular system. If there is no unauthorized use detected, it is still important to move ahead with the remediation plan. Understanding if it was used can help you further scope the impact of the incident, as a data breach might trigger a whole other incident response, perhaps a regulatory-driven one, within your organization.
We know attackers are primarily motivated by financial gain, as reported in the Verizon DBIR earlier this year, showing over 80% of attacks originated from organized crime. They are after data and machine resources to sell or mine for cryptocurrency. If no one has yet used a leaked credential, it still might be possible to avoid any data leaks or unforeseen cloud services bills if you act appropriately.
Knowing if a credential was used is an important step in determining the next bit of data we need to gather.
Has any data been leaked?
If someone has accessed a database or data server, it is safe to assume that they are going to use that information in some way. Most of the time, this will be leaked somewhere online, but it might not be immediately obvious where.
In the worst circumstances, the attackers will tell you, as in ransomware attacks. But ideally, you want to know before the attacker is in the position to tell you. The answer again lies in your logs. Examine your logs for any exports of data or evidence that any data sources have been accessed.
You can quickly check if the secret has been leaked on public GitHub by using HasMySecretLeaked from GitGuardian. This free service lets you safely create a partial hash of any real credential you use. You can then submit part of that hashed secret to quickly check if the fingerprint matches any of the more than 20 million secrets we have identified from our years of research into public GitHub commits.
Public monitoring for sensitive keys and proprietary data is also a good move in the long term as well. Services like DarkOwl can be very helpful for looking for your data on the dark web. Services like GitGuardian Public Monitoring can help ensure you are alerted to any data or specific strings leaked onto public GitHub.
Knowing that you have a data leak on your hands will also escalate the situation and likely trigger a wider response in your organization. Otherwise, it is time to start your plan for rotating the secret, but don't revoke anything just yet.
What breaks if you rotate the secret?
In the interconnected world of modern DevOps environments, a small change in one part of the environment can impact the entire production system. You can inadvertently break the whole CI/CD process if you unexpectedly remove a crucial API key. This was the reality thousands of CircleCI users woke up to earlier this year after an attacker gained access to customer accounts and credentials.
It is vital to understand the impact of a secret rotation, as sometimes the costs of downtime will be higher than the potential risks some types of secret leaks can bring. It is vital to reach out to any teams that might be impacted and give them a heads-up that there is an incident response in progress. While no one likes downtime, sometimes it is unavoidable.
In an ideal world, you and your teams will have eliminated your long-lived credentials and automated the rotation process. This is exactly what we have seen at organizations in the 'elite' tier of the Secrets Management Maturity Model. In this case, there will not be much of an impact, if any at all, for rotating a secret, and you can proceed without any fear.
Now that the right teams are informed about the scope and possible impact of secrets to be rotated, it is time to do what your gut likely told you to do in the first place: revoke and replace that secret.
Rotate the secret
Rotating a secret means invalidating the old secret and replacing it with a new one. There are several things to consider when rotating a secret.
Revoke the secret
How you revoke any specific secret will depend on the system itself and the secrets management policies you have in place. For some systems like Slack, you can simply call a revocation endpoint. For others, you might need to log into the admin interface. It is always going to be best to refer to the service provider's documentation, but we have also gathered some instructions within our API Best Practices Document.
Plan to not leak the new secret
If you rotate a secret and simply hardcode the new credential again, you will be right back where you started. Before you revoke anything, you need to have a plan to store and access the new secret properly.
If you are already using a vault system like Vault by Hasicorp, AWS Secrets Manager, or Azure Key Vault, and a single occurrence of the secret was just accidentally pasted into your code, then you are ready to proceed. Otherwise, you must figure out a safe way to store and reference secrets and refactor your code appropriately.
There is a chance the secret already exists in your shared vault system. If it is, it will save time, as the code is already likely refactored to call the credential programmatically. It is a good idea to notify the team or person who added that secret to let them know what is going on and about the rotation in progress.
Create a new secret and store it safely
Now that the old secret is revoked, you must replace it ASAP. Is this something you can do on your own, or do you need approvals or involvement from another team? If you have been communicating along the way to this step, then it should be straightforward to get the buy-in or needed credentials from the appropriate team members.
This is where all the previous work of refactoring your code and implementing a secrets manager will pay off to make this the least stressful step of the whole incident. Multiple systems, such as AWS, offer an automated secrets rotation path, which would streamline the whole remediation process.
This is also a time to reflect on if a long-lived password is necessary in your setup. Wherever possible, it is a good idea to replace password access with IAM role-based access control. By leveraging tools like managed service identities on Azure, you can eliminate the need for passwords altogether.
Should you rewrite your git history?
This is an optional step that many organizations take to clean up their code. While an invalid credential poses no immediate threat from an attacker, it can help them quickly map out your environment. If an adversary does gain access to your code base, then any credentials, valid or not, make it clear what systems and data storage are in use.
But rewriting the history of a shared repository does require a bit of coordination and some git skills. Please refer to our guide and cheat sheet for rewriting your git history and removing files permanently.
As you replace your secrets and optionally rewrite your repo's history, be aware this might interrupt your regularly scheduled workflows.
Fix any broken workflows
Now that you have rotated the secret and safely stored the new one in a proper secrets manager, it is time to remediate any workflows that might have been affected. This can take the form of expected outages or unexpected broken builds. Depending on how resilient your processes are, some automated build steps or deployment scripts might need to be rerun.
If you have been communicating with the other teams who might be affected, they can often help with this part of the process. While no one likes clearing a blocked process, knowing it is coming and being ready will help the day go smoothly for everyone.
In the best-case scenario, you will not suffer any downtime or interrupt any part of your CI/CD system. This is also behavior we see in elite performers along the Secrets Management Maturity Model.
Establish or update your action plan
After the incident is resolved, but before you move on from the event, it is essential to reflect on what you did, how it went, and where you can improve. If you follow agile methods, you would phrase it as time for a retrospective.
If you are going through a secret leak incident for the first time, it is time to formalize your incident response plan. Feel free to use this article as an outline to establish your own processes for information gathering and communication. Your security team should be involved, as well as your DevOps team. Make sure you communicate the plan far and wide to all other developers who might be in this situation one day.
If you have an established plan, then this is an excellent time to reflect on what parts worked well and what could be improved. All plans need to be updated to match reality from time to time. For example, if you have listed a specific contact who is no longer with the company, it may be time to update that to a departmentally distributed email address that will not grow stale. If there are steps you left out of the original plan that were needed in the actual event, it is a good idea to list those.
Any action plan should be seen as a living document, revisited often, and updated periodically.
Prevent leaked secrets by shifting left
All the steps we have listed assume you have shared a plaintext secret inside a shared repository. While you can be alerted immediately to such leaks by using the GitGuardian Secrets Detection Platform, it would be far better to never commit or push the secret in the first place. This is precisely what you can do with ggshield, the CLI for GitGuardian.
Any developer using git can quickly set up a pre-commit git hook, local to each project or globally on their machine, to automatically catch any secrets before a commit is made. The hook will inform the developer about the type of secret, if it is valid, and where it is in the code. These local incidents do not send off alarms but instead, help guide the right course of action to keep your secrets secret.
Don't panic and follow your plan
Everyone should customize their incident response plan based on organizational realities, but we hope our guide will get you moving in the right direction.
Accidents will happen, and every developer eventually pushes something they should not have. If you panic, this will only make the situation worse. Revoking a secret without knowing the ramifications can turn a minor incident into a full-blown outage. Stay calm and stick to your plan.
Top comments (0)