Accidentally Committed Secrets? A Simple Git Fix Is Not Enough!

It happens to the best of us. A moment of distraction, and suddenly we've committed a .env file, an API key, or another secrets file to our Git repository. If you think simply removing the file and committing again will fix the issue, think again. The file will still exist in your Git commit history, posing a security risk.

In this article, we’ll explore how to effectively remove secrets from Git history and mitigate potential security risks.

The Problem

When we commit a file to a Git repository and then remove it in a subsequent commit, the file isn't actually deleted from the repository's history. Git is designed to maintain a complete historical record, which means anyone with access to our repository can:

1. Browse through previous commits
2. View the contents of those commits, including our secrets
3. Extract sensitive information with simple Git commands

# For example, someone could do this to see our secrets:
git checkout <commit-hash-with-secrets>
cat path/to/secrets/file

The Solution

Scenario 1: Haven't Pushed to Remote Yet

If the commit with secrets is still only in our local repository. This is the easiest scenario to fix. :

• Use git reset to undo the commit
• Remove the sensitive data
• Commit again with the sanitized files

# Undo the last commit, but keep the changes in our working directory 
git reset --soft HEAD~1 

# Alternatively, if we want to discard the changes completely 
git reset --hard HEAD~1

Scenario 2: Already Pushed to Remote

To truly remove sensitive information, we need to rewrite our Git history to completely eliminate the file from all commits. Here's how to do it effectively:

Step 1: Identify the Secret Files
First, clearly identify which files contain sensitive information that needs to be removed.

Step 2: Remove the Files from Git History

1) Using git filter-branch

git filter-branch --force --index-filter "git rm --cached --ignore-unmatch PATH_TO_SECRET_FILE" --prune-empty --tag-name-filter cat ----all

Replace PATH_TO_SECRET_FILE with the path to file containing secrets.

For replacing specific strings (like API keys) while keeping the file:

git filter-branch --tree-filter "find . -type f -name '*.config' -exec sed -i 's/YOUR_API_KEY/PLACEHOLDER_KEY/g' {} \;" HEAD

2) Using filter-repo

git-filter-repo isn't built-in to git itself
First, install git-filter-repo:

# For Python users
pip install git-filter-repo

# For macOS
brew install git-filter-repo

Then run:

git filter-repo --path path/to/secrets/file --invert-paths

for more info regarding git-filter-repo refer this

Step 3: Push those Changes
After cleaning history, we need to force push the changes:

git push origin --force --all

CAUTION: Force Pushing Is Irreversible ⚠️

Additional Recommendations

• Revoke any exposed API keys or tokens
• Add sensitive files to .gitignore: Ensure secret files are never tracked.
• Use Environment Variables: Store secrets in environment variables instead of committing them.
• Use Git Hooks: Automate pre-commit hooks to prevent secret files from being added.
• Use GitHub’s Secret Scanning If you're using Github or use Secret push protection for Gitlab.

Note: Remember that if your secret was exposed for any period of time, even briefly, you should consider it compromised and rotate it immediately.