Secrets in Git and how to recover

The push went through, CI was green, and later somebody notices AKIA... in a config file. The instinct is to delete the file and force-push. That instinct is wrong, and the order of the next ten minutes decides what this incident costs.

The first ten minutes

1. Treat the secret as compromised the moment it left your machine. Public repositories are crawled by automated scanners within seconds, and leaked cloud keys are tested against provider APIs continuously.
2. Do not rewrite history yet. A rewrite is disruptive and requires coordination, and it protects nothing while the credential still works.
3. Record the facts: which secret, which commit SHA, which branch, when it was pushed, and whether the repository was ever public.
4. Rotate or revoke it before anything else, then read the provider's audit log for activity you cannot explain.
5. Plan the cleanup afterwards, when the credential is already dead and you are no longer racing an attacker.

Why git rm and force-push are not enough

Deleting a file in a new commit does not delete the blob. It remains in every earlier commit, and anyone can check it out. Removing the commit itself barely helps either:

• Reflogs and unreachable objects. A force-push moves a branch pointer; the old commits survive in the local reflog and on the remote until garbage collection. Forges can still serve them by SHA, and cached commit views and open pull request diffs keep rendering them.
• Forks and clones. Every fork is a separate repository that you cannot force-push, and every laptop that cloned the repo keeps the blob in .git.
• Downstream artefacts. CI job logs, build artefacts, Docker image layers, published packages and cached tarballs. A package release that shipped the secret cannot be quietly edited; it has to be deprecated and republished.
• The copies you forgot. A screenshot in a ticket, a wiki page, a chat paste, or a .env file on a server that was itself committed.

History rewriting is hygiene. Rotation is remediation. Only one of the two actually closes an incident.

Rotate the credential before you touch history

Rotate means a new credential in place and the old one destroyed, not the same key with a new label.

# AWS: list keys, create a replacement, deploy it, then destroy the old one
aws iam list-access-keys --user-name ci-deployer
aws iam create-access-key --user-name ci-deployer
aws iam update-access-key --user-name ci-deployer \
    --access-key-id AKIAOLDKEY --status Inactive
# only after the replacement is confirmed working in production:
aws iam delete-access-key --user-name ci-deployer --access-key-id AKIAOLDKEY

# Was the leaked key ever used?
aws cloudtrail lookup-events \
    --lookup-attributes AttributeKey=AccessKeyId,AttributeValue=AKIAOLDKEY

Rotate everything the secret unlocked, not just the secret: database passwords, webhook signing secrets, OAuth client secrets, SSH deploy keys, TLS certificates, and any other token stored in the same file. Personal access tokens cannot be rotated in place — revoke them and issue new ones.

Then shrink the blast radius for next time: least-privilege policies, short-lived credentials issued through OIDC instead of long-lived keys, and separate secrets per environment so a staging leak cannot touch production.

Rewriting history with git filter-repo

git filter-branch is officially discouraged because it is slow and easy to misuse. Use git filter-repo, ideally on a fresh mirror clone:

pip install git-filter-repo
git clone --mirror git@github.com:acme/app.git app.git
cd app.git
cat > ../expressions.txt <<'EOF'
literal:AKIAIOSFODNN7EXAMPLE==>REDACTED
regex:(?i)api[_-]?key\s*=\s*\S+==>api_key=REDACTED
EOF
git filter-repo --replace-text ../expressions.txt
git remote add origin git@github.com:acme/app.git
git push --force --all
git push --force --tags

To drop a file from every commit instead of redacting its text:

git filter-repo --path config/secrets.yml --invert-paths

The BFG Repo-Cleaner is the Java alternative, and it is faster on very large repositories:

java -jar bfg.jar --replace-text passwords.txt app.git
cd app.git
git reflog expire --expire=now --all
git gc --prune=now --aggressive
git push --force

Two practical warnings. git filter-repo removes the origin remote as a safety measure, so re-add it before pushing. And a rewrite changes every commit SHA from the leak onwards, so each collaborator must re-clone or hard-reset; a stale clone pushed later resurrects the blob you just removed.

After the rewrite: the loose ends

• Ask the forge to purge cached views. GitHub documents a support request for exactly this, because the old commit can remain reachable by SHA even after a force-push.
• Delete stale branches and tags that still point at the old history, then tell the team to re-clone rather than merge.
• Revoke anything the secret was copied into: CI variables, deployment scripts, monitoring dashboards, shared password entries.
• If a package was published containing the secret, deprecate that version and publish a new one. Unpublishing is not reliably available.
• Write a two-line postmortem: how the secret got committed, and which control would have caught it. That is the only part that prevents a repeat.

Preventing the next one

• Ignore the usual suspects: .env, *.pem, *.key, *.p12, credentials.json, .aws/. Remember that .gitignore does not untrack a file that is already committed — that needs git rm --cached plus a rotation.
• Ship a .env.example with placeholder names, so nobody has to guess what the file should contain.
• Scan before the commit, not after. A pre-commit hook running gitleaks or trufflehog blocks the mistake at the only moment it is cheap to fix.
• Enable provider-side push protection. GitHub secret scanning with push protection rejects a push containing a recognised credential, which catches the file nobody meant to add.
• Keep secrets in a manager — Vault, AWS Secrets Manager, SOPS with age, or a cloud key vault — and inject them at deploy time as environment variables. Nothing sensitive belongs in the repository at all.
• Scan CI output too. A debug set -x or a verbose env dump leaks credentials that were stored correctly everywhere else.

Incident checklist, in order

1. Revoke or rotate the credential, and assume it has already been used.
2. Read the provider audit log for unexplained activity and for what the credential could reach.
3. Rotate everything stored alongside it.
4. Rewrite history with git filter-repo or BFG, then force-push all branches and tags.
5. Ask the forge to purge cached views, delete stale branches, and have everyone re-clone.
6. Invalidate downstream copies: CI variables, artefacts, published packages, documentation.
7. Add the detection you were missing, and put rotation on a schedule from now on.

Anything short of step one is tidying up while the door is still open.