Secrets Management: Keeping Keys, Passwords, and Tokens Out of the Wrong Hands

⚡ TL;DR: Keys, passwords, and tokens never belong in Git, logs, or screenshots. Learn to inject secrets at runtime, store them in a vault, rotate on a schedule with least privilege, and scan to catch leaks before attackers do.

Contents

• The 3 a.m. cloud bill
• What counts as a secret
• How secrets should actually flow
• Where NOT to put secrets (and where to)
• Reading a secret the right way
• Rotation and least-privilege
• Detecting leaked secrets
• Common mistakes that cost hours (or accounts)
• Takeaways
• Where to go next

The 3 a.m. cloud bill

A developer pushes a quick fix on a Friday evening. Buried in the diff is a config file with an AWS access key, committed "just to get it working." The repo is public. Within minutes, automated bots scraping GitHub find the key. By Saturday morning the account has spun up hundreds of GPU instances mining cryptocurrency, and the bill reads five figures. The developer did not get hacked through some clever exploit. They handed over the key themselves and pushed it to the internet.

This is not a rare horror story. Scanners watch public Git pushes in real time, and a leaked credential is often abused before the developer has even noticed it is gone. Secrets are the single highest-leverage thing to protect, because one leaked key can bypass every other control you built.

📌 Who this is for: Developers and ops folks who have ever pasted an API key into a config file, dropped a password into a .env, or wondered where credentials are supposed to live. No prior security background needed, we start from "what is a secret" and end with rotation and leak detection.

What counts as a secret

A secret is any value that grants access or proves identity and would cause harm if a stranger held it: API keys, database passwords, OAuth tokens, TLS private keys, signing keys, SSH keys, webhook signing secrets. The defining trait is simple, if it leaks, someone can act as you. Treat anything matching that test as a secret, not as ordinary config.

A secret is only secret while exactly the systems that need it can read it, and nothing else can.

That sentence is the whole discipline. Everything below, where you store secrets, how you hand them to apps, how often you change them, who can read them, is just making that one sentence true and keeping it true.

In the real world	In tech
You don't tape your house key to the front door	Don't hardcode credentials in source or commit them to Git, that's leaving the key on the lock
You don't make 100 copies and hand them out	Grant least-privilege access, each service gets only the secrets it needs, nothing more
If a key is lost, you change the locks	Rotate the secret immediately on suspected leak; the old value stops working
A locksmith keeps the master copies in a safe	A secret manager (Vault, AWS Secrets Manager, KMS) is the safe, one guarded source of truth

Secrets behave exactly like the keys to your house.

How secrets should actually flow

The goal is that a secret never lives in your codebase, your container image, or your Git history. Instead the app fetches it at startup or runtime from a dedicated secret manager, holds it in memory, and never writes it down. Behind the scenes a rotation loop keeps swapping the stored value for a fresh one.

An app authenticates to the secret manager at startup, the secret is injected into memory at runtime, and a rotation loop quietly replaces it on a schedule.

1. App boots with no secret: The image and the repo contain zero credentials. There is nothing to leak even if the image is pulled or the repo is cloned.
2. App proves who it is: It authenticates to the secret manager using its workload identity, an IAM role, a Kubernetes service account, or a short-lived token, not a stored password.
3. Manager checks the policy: The secret manager confirms this identity is allowed to read this specific secret, then returns it over an encrypted channel.
4. Secret is injected at runtime: The value lands in the process's memory (or a tmpfs-backed env var). It is never written to disk, the image, or version control.
5. Rotation runs in the background: On a schedule, the manager issues a fresh value and expires the old one. Apps re-fetch and keep working, a leaked old copy becomes useless.

Where NOT to put secrets (and where to)

Most leaks come from picking the wrong storage location, not from sophisticated attacks. Here is how the common options actually compare on the three things that matter: how exposed the secret is, whether you can rotate it cleanly, and whether you can see who used it.

Approach	Leak risk	Rotation	Audit trail
Hardcoded in source	Severe, lives forever in Git history, images, and every clone	Painful, code change + redeploy everywhere	None
.env committed to Git	Severe, same as hardcoding, just in a different file	Painful, and old commits still hold it	None
Env var (set at deploy, not in Git)	Moderate, can leak via logs, crash dumps, /proc, child processes	Manual, redeploy to change it	Weak, no record of reads
Secret manager (Vault / AWS SM / KMS)	Low, encrypted at rest, scoped by policy, short-lived	Automated, rotate on a schedule, no redeploy	Full, every read is logged

The further down this table you go, the smaller the blast radius when something goes wrong.

⚠️ Warning: A .env file is fine for local development as long as it is in .gitignore and never committed. The danger is the moment it lands in version control, at that point it is identical to hardcoding the secret, and deleting it later does not remove it from Git history.

Reading a secret the right way

The code that consumes a secret should fetch it from the environment or a manager client, never embed the literal. The example below reads a database password from a secret manager with an env-var fallback for local dev, and fails loudly if it is missing rather than silently using a blank value.

db.py

import os
import boto3
from functools import lru_cache


@lru_cache(maxsize=1)
def get_db_password() -> str:
    # Local dev: read from an env var that is NEVER committed.
    local = os.environ.get("DB_PASSWORD")
    if local:
        return local

    # Production: fetch from AWS Secrets Manager at runtime.
    secret_id = os.environ["DB_SECRET_ID"]  # just a name, not the value
    client = boto3.client("secretsmanager")
    resp = client.get_secret_value(SecretId=secret_id)
    return resp["SecretString"]


def connect():
    password = get_db_password()
    # ... open the connection; never log `password`
    return password  # used in-memory only


# NEVER do this:
# DB_PASSWORD = "hunter2-prod-9f3a"   # hardcoded, leaks on first commit

Notice three things: the value of the secret never appears in the file (only the secret's name, DB_SECRET_ID), the result is cached so you do not hammer the manager on every call, and there is no print(password) anywhere, secrets in logs are one of the most common silent leaks.

Rotation and least-privilege

Two practices turn "we have a secret manager" into "we are actually safe." Rotation means every secret has an expiry; the manager issues a new value and retires the old one on a schedule, so a copy stolen six months ago is already dead. The gold standard is dynamic secrets, credentials minted on demand that live for minutes, not months.

Least-privilege access means each identity can read only the exact secrets it needs. The billing service has no business reading the email provider's API key. Scope policies per-service so a single compromised workload exposes one secret, not the whole vault. Pair this with the audit log: if you cannot answer "who read this secret and when," you cannot tell whether a leak has been used.

Detecting leaked secrets

Assume a secret will eventually slip through. Your job is to catch it fast. Run a secret scanner in pre-commit hooks and in CI so a credential is blocked before it ever reaches the remote, and enable your Git host's push protection (GitHub and GitLab both scan pushes for known token formats).

scan.sh

# Block secrets before they are committed (run as a pre-commit hook)
gitleaks protect --staged --verbose

# Scan full history for anything already leaked
gitleaks detect --source . --report-path leaks.json

# If a real secret is found in history, ROTATE it first,
# then purge from history (e.g. git filter-repo), deleting the
# file in a new commit does NOT remove the old value.

💡 Tip: When a secret leaks, rotate first, clean up second. Revoking the credential makes the leaked copy worthless immediately; scrubbing Git history is housekeeping that can wait minutes. Doing it in the other order leaves a live key exposed while you wrestle with history rewrites.

Common mistakes that cost hours (or accounts)

1. Committing secrets to Git. Even a deleted .env lives forever in history. Once pushed, treat the secret as compromised and rotate it, do not just remove the file.
2. Never rotating. A secret that has not changed in two years has likely been copied into someone's notes, a Slack message, or a screenshot. Set an expiry on everything.
3. Granting broad access. One "admin" policy that can read every secret turns any single compromised service into a full breach. Scope per-service.
4. Leaking secrets in logs. print(token), verbose error traces, and crash dumps quietly ship credentials to your log aggregator, which often has far weaker access controls than your vault.
5. Putting secrets in container image layers or CI variables in plaintext. Images get pushed to registries; CI logs get shared. Inject at runtime instead.

Takeaways

Secrets management in seven lines

• A secret is anything that lets someone act as you, keys, passwords, tokens, certs.
• Never hardcode secrets or commit .env files; Git history is forever.
• Store secrets in a dedicated manager (Vault, AWS Secrets Manager, cloud KMS).
• Inject at runtime into memory using workload identity, not baked into images.
• Rotate on a schedule so a stolen copy expires; prefer short-lived dynamic secrets.
• Grant least-privilege: each service reads only the secrets it needs, and every read is logged.
• Scan for leaks in pre-commit and CI; if one leaks, rotate first, then clean history.

Where to go next

Secrets management is one pillar of a secure delivery pipeline. Build the surrounding picture next:

• Start with the fundamentals: What Is Application Security? frames where secrets fit among the broader threats.
• Then extend the discipline to your build inputs: Securing the Software Supply Chain covers dependencies, provenance, and signing.
• Practice the delivery mechanics in the CI/CD lab, where injected secrets and pipeline variables come to life.
• See how this fits a full role on the DevOps Engineer path.

---

Originally published on TheSimplifiedTech, where this guide is interactive, with in-browser terminal labs and diagrams. Learn cloud and DevOps by doing, no videos.