Do Not Commit Secrets!

Picture this: every team I've had the pleasure (or sometimes, the pain) of working with has had one thing in common – someone, somewhere, committing secrets to Git.

In my recent adventures as a Technical Lead, CTO, and all-around code wizard, I've witnessed firsthand the struggles of distinguishing between public and private keys. And you know what's even trickier? Accidentally spilling those private keys and secrets onto Git for all to see. It's like leaving your front door wide open in the middle of the night and wondering why your house got burgled.

I've seen .env files with more secrets than my ex-wife and Axios requests spilling the beans on sensitive credentials.

Now, you might be thinking, "What's the harm? Can't we just commit another version without the credentials?" Ah, if only it were that simple.

Well, let's conduct a little experiment, shall we? Create a local Git repository (assuming Git is already in your system)

raul@mbp:~|⇒  mkdir fake-git-repo
raul@mbp:~|⇒  cd fake-git-repo
raul@mbp:~/fake-git-repo|⇒  git init
Initialized empty Git repository in /Users/raul/fake-git-repo/.git/
raul@mbp:~/fake-git-repo|master ⇒  touch .env

• Add some super secret values to the .env file

raul@mbp:~/fake-git-repo|master ⇒  vi .env

Add something like this:

AWS_ACCESS_KEY_ID = 'AKJEGCKAQNSGFN2N68''
AWS_SECRET_ACCESS_KEY = 'mC4gRDpo/o+M1uOfTWgdfgqMqYe38eGmt'
~
~
~
:wq

Save the changes by typing wq (write and quit)

• Now commit those changes! although, you shouldn't!

raul@mbp:~/fake-git-repo|master⚡ ⇒  git add .
raul@mbp:~/fake-git-repo|master⚡ ⇒  git commit -m "let's commit something we shouldn't commit"
[master (root-commit) 2be5583] let's commit something we shouldn't commit
 1 file changed, 2 insertions(+)
 create mode 100644 .env

• Now, for the sake of the story, let's assume you follow a Code Review procedure and someone caught you committing these secrets. They told you it's not safe. You immediately said, "OK! I'll remove them," so you go to vi, remove those values from the file, and commit again without the secrets.

AWS_ACCESS_KEY_ID =
AWS_SECRET_ACCESS_KEY =
~
~
~
:wq

• Let's use git to double-check our changes before commit

diff --git a/.env b/.env
index 90912b1..3c569d6 100644
--- a/.env
+++ b/.env
@@ -1,2 +1,2 @@
-AWS_ACCESS_KEY_ID = 'AKJEGCKAQNSGFN2N68''
-AWS_SECRET_ACCESS_KEY = 'mC4gRDpo/o+M1uOfTWgdfgqMqYe38eGmt'
+AWS_ACCESS_KEY_ID =
+AWS_SECRET_ACCESS_KEY =

If you pay attention, the lines with a - are the ones deleted, and the ones with + are the added ones. So we're all good! We removed the values of our super secrets! Yay!

Now, commit the safer .env file.

raul@mbp:~/fake-git-repo|master⚡ ⇒  git commit -a -m 'removes secrets'
[master 61a83f4] removes secrets
 1 file changed, 2 insertions(+), 2 deletions(-)

If you open the file now, secrets are no longer there

raul@mbp:~/fake-git-repo|master ⇒  cat .env
AWS_ACCESS_KEY_ID =
AWS_SECRET_ACCESS_KEY =

You're Wrong!

You might believe you're all safe now. But the whole purpose of Git is to keep track of changes and allow us to go back in the history of our files if necessary. Let's investigate the history of our fake-repository and .env file.

• Use git log -p -2.
  • -p means patch, and when used along with log, it will show what changed (removed, added, modified).
  • -2 tells the log to show the latest 2 commits.

commit 61a83f419b174f0750d7dab56c9e0469b0c27784 (HEAD -> master)
Date:   Fri Feb 16 20:57:26 2024 +1300

    removes secrets

diff --git a/.env b/.env
index 90912b1..3c569d6 100644
--- a/.env
+++ b/.env
@@ -1,2 +1,2 @@
-AWS_ACCESS_KEY_ID = 'AKJEGCKAQNSGFN2N68''
-AWS_SECRET_ACCESS_KEY = 'mC4gRDpo/o+M1uOfTWgdfgqMqYe38eGmt'
+AWS_ACCESS_KEY_ID =
+AWS_SECRET_ACCESS_KEY =

commit 2be5583e6c94c7b70e0f4232fea5ff1fb13603a8
Date:   Fri Feb 16 20:46:59 2024 +1300

    let's commit something we shouldn't commit

If you take a deep look, AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY appear twice. But one with a - and another with a +, meaning that the line with - was removed and the one with + was added.

So, we could see what was the old value (with our secrets) which we thought we safely removed from the public!

Now, you might ask...why do I care so much about this?

Real-Life Example from GitHub

If the previous example doesn't seem scary, let's head over to github.com and see if secrets are being leaked or not, and why we should be worried.

Here's a direct link for you to search for commits on GitHub public repositories:

https://github.com/search?q=remove+secrets&type=commits

You'll see a lot of repositories with commits named remove secrets

If you click on some of them, you might be able to see files like this:

That's actually `google cloud computing secrets! I blurred the values, and I hope this person already rotated this certificate or they could be in serious problems having people accessing their Google Cloud services and probably incurring charges.

Look at this other example where someone tried to remove AWS secrets, MySQL DB connection strings, etc.

Summary

I hope this helps people understand why there's a Tech Lead, Senior Dev, etc., asking you not to commit secrets.

I've come to realize we usually tell others what not to do without giving a reason of why it shouldn't be done.

Hopefully, this blog will help me too, to teach some of the developers I work with. And if in the meantime, someone else can learn something, well, you're welcome!