DEV Community

Haven Messenger
Haven Messenger

Posted on • Originally published at havenmessenger.com

The Commit You Reverted Is Still There

A credential goes into a config file during debugging. Six minutes later someone notices, commits a fix, and the diff on the pull request is clean. The credential is still in the repository, still served over HTTPS to anyone who can construct the right URL, and on a public repository it was very likely collected by an automated scanner before the fix was written. The reason has nothing to do with carelessness and everything to do with how Git stores things.

Git does not store diffs. It stores whole objects, addressed by the hash of their content. When you commit a file, Git writes a blob holding the complete file contents, a tree mapping filenames to blobs, and a commit pointing at a tree and at its parent commits. The diff you see in a review interface is computed on demand by comparing two trees. It is a view, not a storage format.

This is why every intuition borrowed from editing a document fails here. Removing a line in a later commit creates a new blob without that line. It does not touch the old blob, which remains stored, intact, and reachable from the commit that introduced it.

You can watch it happen in any repository:

  • git log -p -- path/to/config prints every version of that file that was ever committed, including the ones later deleted.
  • git rev-list --objects --all enumerates every object reachable from any reference.
  • git cat-file -p <sha> prints any of them by hash.

The three moves that feel like removal and are not

What you did What happened to the secret
git revert A new commit was added that applies the inverse change. The original commit and its blob are untouched and still in the branch's history.
git commit --amend then force push Your branch reference now points somewhere else. The original commit object still exists on the server as an unreachable object, and on the major hosts it stays fetchable by its hash.
Deleting the file in a later commit Nothing at all. The file is absent from the current tree and present in every earlier one.

The second row is the one that surprises experienced developers. Locally, an unreachable object is eventually removed by garbage collection. On a hosted service it usually is not, at least not on any timetable you control. GitHub documents this directly: after a force push, commits remain accessible by their hash, and removing them requires contacting support rather than pushing again.

There is a further wrinkle in fork networks. Repositories in a fork relationship share an object store, so an object pushed to one of them can be requested from another by hash. Security research published in 2024 demonstrated that this makes commits from deleted forks and from closed pull requests retrievable from the upstream repository. Deleting your fork does not delete what you pushed to it.

The operative fact. A hash is not a secret. It appears in pull request timelines, in CI logs, in webhook payloads and in the events firehose that hosts publish publicly. Anyone holding it can request the object directly, and no branch needs to point at it.

How fast the collection happens

Public repository activity is published as a continuous event stream, and scraping it for freshly pushed credentials is a mature, automated business. The realistic assumption for a public push is that any credential in it was extracted, tested against the relevant API, and filed within minutes.

There is a defensive version of the same speed. Major hosts run secret scanning on pushes and participate in partner programmes with credential issuers: a pattern matching a cloud provider key or a payment processor token triggers a notification to that provider, which in some cases revokes or quarantines the credential automatically. If you have ever pushed an access key and received a revocation email before you had finished typing the fix, that is what happened.

Treat that as a backstop rather than a control. It covers well known credential formats from participating vendors. It does not cover your own service's session tokens, a database connection string, an SSH private key, or the passphrase in a comment.

The response order that matters

  1. Rotate the credential. First, before any history work, before writing the incident note. Everything else on this list is cleanup; this is the only step that changes what an attacker can do with what they already have.
  2. Check what the credential did. Pull the access logs for the window between the push and the rotation. This is the step most often skipped, and it is the one that distinguishes an exposure from an intrusion.
  3. Then consider rewriting history. git filter-repo is the current tool for this, and BFG Repo-Cleaner remains a reasonable choice for the simple case. Both rewrite every affected commit, which changes every downstream hash, which means every collaborator has to re-clone or reset. Coordinate it, or someone pushes the old history back a day later.
  4. Ask the host to expire unreachable objects. On a hosted service this is a support request. Without it, the rewrite changed what the branch points at and not what the server will serve.
  5. Write it down. A short note recording which credential, which window, what the logs showed and what changed afterwards is what turns one incident into one fewer future incident.

Steps three and four are worth doing and they are not a substitute for step one. Every clone taken before the rewrite still contains the original object, and you have no inventory of those clones.

Prevention, in order of how much it actually buys

These are not equivalent, and the ordering matters more than the list.

  • Have no long-lived credential to leak. Workload identity federation, where CI exchanges a short-lived signed assertion for a token valid for minutes, removes the static secret from the repository and from the CI configuration at the same time. It is more setup than a stored key, and afterwards there is no long-lived credential in the tree for anyone to find.
  • Inject at runtime from a secret manager. The application reads from the environment or from an authenticated fetch. Nothing secret is in the tree, so nothing secret can be committed from it.
  • Enable server-side push protection. A hook on the receiving end blocks the push before the object reaches the shared store. This is the control that works, because it does not depend on the developer's local configuration.
  • Run a pre-commit scanner as well. Tools such as gitleaks and trufflehog catch most of it before it leaves the machine, with the caveat that --no-verify exists and gets used under deadline pressure. Keep the local hook for the convenience of catching things early, and do not count it as the control.
  • Prefer scanners that verify. Entropy heuristics generate noise on base64 test fixtures and miss structured low-entropy secrets. A scanner that attempts a live authentication with a candidate credential produces findings you can triage in order.

One item deserves separating out because it is so commonly misunderstood. Adding a path to .gitignore does nothing to a file Git is already tracking. The ignore list governs untracked files. If the file was committed once, it stays tracked until git rm --cached removes it from the index, and the earlier blobs remain in history regardless.

Private repositories are not a control

The reasoning that a private repository makes committed secrets acceptable fails at several joints at once. Private repositories get forked internally, cloned to contractor laptops, backed up to third-party services, mirrored for CI, opened to a wider team on reorganisation, and occasionally made public by accident. Repositories also get open-sourced years later by people who did not write the history and do not know what is in it.

CI logs are the parallel leak path and they follow the same rule: a build script running under set -x echoes its arguments, and a shared build log retains them long after the pipeline has finished. Masking in the CI system helps only for values it knows are secret.

The formulation to work from: anything committed to a repository should be assumed to be permanent and eventually public. A content-addressed store keeps every object it has ever been given, and a working repository accumulates clones faster than anyone tracks them.

Originally published at havenmessenger.com

Top comments (0)