Fuck up story; rotate committed secret(s) on multiple Heroku apps

Introduction

I have read a lot of exciting, helpful and inspiring articles/posts on dev.to and have felt like giving something back for a long time. Additionally a colleague of mine have been pushing me to writing a blog post for a long time (thank you, Jonathan) so I thought this fuck up and how it was solved, might as well be helpful, interesting and inspiring for others.

This is my first post. ever. Please don't hesitate to give any kind of feedback!

The Fuck Up

As the title indicates, I committed and pushed a secret to git. I had to run some code locally relying on this secret and as I were eager to resolve the problem I was working on, I eventually forgot all about the secret.... So I staged the file files, wrote a commit message, committed them and pushed it all to GitHub, including the secret and boom! Our secret was in the git history!

Removing the git commit history

When one accidentally commit and push one or more secrets to their git repository, they might consider in the stress (like I did) to just remove the commit from the git history.
Github has some interesting documentation about doing this using a tool BFG repo-cleaner, but they also recommend to not rely on this method, if the commit was actually pushed to Github:

Warning: Once you have pushed a commit to GitHub, you should consider any data it contains to be compromised. If you committed a password, change it! If you committed a key, generate a new one.

So let's not attempt to resolve this fuck up by going in that direction.

The approach

This happened in a private repository, so while it was a big concern we did not have to revoke/remove the secret right away but could take some time to consider how we could rotate it without having downtime for any users.
If it had been in a public repository, it is very likely the secret should be revoked/removed immediately.

We had two problems we needed to solve:

1. Rotate the secret in all our Heroku apps.
2. Avoid downtime for any users while doing it.

While the solution to 1) is rather straight forward the solution to 2) requires a bit more consideration and might be very different from use case to use case, if necessary at all.

Our solution to 2) was to add support for handling multiple secrets in a module of ours that does some authorization. Because of purpose and how the module works, we couldn't just make a hard switch to the new secret - we had to have both secrets being active for a while before we removed the old secret. By adding support for multiple secrets we could avoid the risk of any users being locked out.

1. Add support for multiple secrets in our module.
2. Write a script that adds a new secret in our Heroku apps.
   • For an existing secret FOO with the new secret.
   • Create a new secret FOO_OLD with the old secret.
3. Write another script that removes the old secret in our Heroku apps once we are ready to do so.

Rotating the secret on Heroku

To rotate the secrets I use Heroku's CLI to both find (prod) apps with the secret and actually rotate the secret on all those apps.

If you have not previously used the Heroku CLI you need to install it and login in first. They have a getting started guide in the documentation.

Get a list of apps to rotate the secret of

First; Find all the Heroku apps with the help of the command: heroku apps -A --json.

• -A returns all teams
• --json returns result as json.

const childProcess = require('child_process');
const { promisify } = require('util');

const exec = promisify(childProcess.exec);

const { stdout: apps } = await exec('heroku apps -A --json');

Second; Filter the list of apps for the ones you want to update - for our case we wanted to update only production apps. We have a naming convention for production apps, so we could filter the apps based on this convention by the property app.name like:

const prodAppRegex = /^FOO$/;
const isProdApp = (app) => app.name.test(prodAppRegex); // could also use a simple string comparison if fit your needs
const prodApps = apps.filter(isProdApp);

We have the same secret on our staging apps with the same name/key, so to avoid overwriting the staging secrets we did this. If you only have one environment you probably don't have to do this.

Third; Filter the remaining list for the ones that actually has the environment variable set. If you do not use the same name for the environment variable on all apps, you might have to find a slightly different approach.

const appsWithSecret = []; // list of apps that has the environment variable(s) to rotate
for (const app of JSON.parse(apps)) { // returned as a string
  const { name, ...otherPropsYouMightNeed } = app;

  const { stdout: config } = await exec(`heroku config -a ${name} --json`);

  const isMatch = ([key, value]) => key === '<env variable key>';
  // if any app could have multiple env variable matches
  // to rotate, you should use .filter instead
  const match = Object.entries(config).find((isMatch));

  if (!match) {
    continue;
  }

  appsWithSecret.push({ name, envVariable: match, otherPropsYouMightNeed });
}

Rotate the secret on the list of apps

Get app secret

As expressed earlier I like to include some dry run functionality when I write scripts like this that delete, update or create important things like rotating secrets, to verify the results before actually executing it.

We use each app name along with the config variable key to get the config variable.

async function getAppSecret(appName, configVar) {
  const { stdout } = await exec(`heroku config:get ${configVar} -a ${appName}`); // -a or --app

  // returns a string of the value
  return stdout;
}

Notice that we actually already stored the app secrets in appsWithSecrets, so you could skip this or do this instead of the filter part where we push the relevant apps to appsWithSecrets.

Update the secret on an app

Again, we use each app name along with the config variable key but also include the new value that we want to update the config variable to.

async function setAppSecret(appName, configVar, newValue) {
  const { stdout: result } = await exec(`heroku config:set ${configVar}=${newValue} -a ${appName}`); // -a or --app

  // returns a string like:
  // Setting <configVar> and restarting ⬢ <appName>... done, <new app version>
  // <configVar>: newValue
  return result;
}

Update all apps with new secret

const DRY_RUN = true; // set to false when you want to execute it
const { NEW_VALUE } = process.env; // you can set this when running your script like: "NEW_VALUE=FOO node ./yourScript.js"

for (const app of appsWithSecret) {
  const { name, envVariable } = app;
  const [key, secret] = envVariable;


  if (DRY_RUN) {
    const appSecret = await getAppSecret(name, key);
    // could verify "secret" === "appSecret"
    // console.log('is same secret?', appSecret === secret)
  } else {
    const resultOldKey = await setAppSecret(appName, `${key}_old`, secret);
    const resultNewKey = await setAppSecret(appName, key, NEW_SECRET);
  }
}

Unset the old secret in all apps

We use a similar approach/code as when we add the new secret but we slightly change our isMatch function, to find the "old" key:

  const isMatch = ([key, value]) => key === '<env variable key>_old'; // we postfixed "_old" in previous steps when also adding new secrets in each app

Then we can unset each app's old secret when we are ready to do so (in relation to avoiding downtime):

async function unsetAppSecret(appName, configVar) {
  const { stdout: result } = await exec(`heroku config:unset ${configVar} -a ${appName}`);

  // returns a string like:
  // Unsetting <configVar and restarting ⬢ <appName>... done, <new app version>
  return result;
}

You can find a full code example in this gist.

Take aways

• Do not temporarily store secret(s) in code of files that are not ignored by git. Human mistakes happen and you might forget all about the secret for a very short time period.
• Prepare your modules, components, etc. for secret rotation. While it will hopefully not be necessary to do it because of a compromised secret, it is better to already be able to quickly rotate it on a long list of apps if it happens.
• If necessary for the purpose of the module, consider adding support for using multiple secrets to avoid having downtime while doing the rotation.
• When you already have a way of rotating secrets, why not do it on a regular basis - e.g. every half year, yearly, etc.?
• Do not consider removing git history as a proper solution for secrets accidentally committed.
• Consider reviewing your changes locally, before you decide to commit, stage and push it to the remote host. If I had done this, I might have noticed that I still had the secret stored in my code and could have avoided "the disaster" (quite a lot of time spend on doing the rotation).
• Consider how you use secrets and how you rely on such for authorization across apps/services/modules. If you are using the same secrets everywhere, it might be slightly smelly already....