Every company I've done security work with has had secrets in the wrong place. Database passwords in a .env file committed to a public repo. API keys hardcoded in a script someone forgot about. Production credentials stored in a shared Slack channel. A junior developer's laptop with access to every third-party API the company uses, because everyone shared the same .env.production over email.
Secrets management is one of those topics that feels like overhead until something goes wrong. A leaked AWS key costs thousands of dollars in minutes when an attacker spins up crypto mining instances. A leaked Stripe API key means your payment processing is compromised. A leaked database password means your users' data is accessible to someone who shouldn't have it.
The goal of a mature secrets management practice is: no human ever handles a production secret directly, secrets are rotated automatically, access is audited, and leaked secrets are detected and revoked within minutes.
The Problems with .env Files
.env files are where most teams start, and they work fine for local development when managed carefully. The problems start when practices don't scale:
Common .env failures:
-
.env.productionchecked into version control (even "private" repos) -
.envfiles shared over Slack, email, or messaging apps - Same secrets used across all environments
- No rotation, a key from two years ago that long-departed employees still have
- No access control, every developer has every secret
- No audit trail, no record of who accessed what
The principle that eliminates most of these problems: secrets should never be stored in source control, never transmitted as plaintext, and access should be role-based with an audit trail.
Local Development: Doing .env Right
For local development, .env files are acceptable when a few rules are followed. First, your .gitignore must exclude every real secrets file: .env, .env.local, .env.*.local, .env.development.local, .env.staging, .env.production, and .env.prod. Second, commit a .env.example file that lists every required key with an empty value, so new developers know exactly what secrets to obtain without any real values sitting in the repo:
# .env.example -- committed to source control
# Contains keys without values -- documents what's needed
DATABASE_URL=
REDIS_URL=
STRIPE_SECRET_KEY=
OPENAI_API_KEY=
JWT_SECRET=
SENDGRID_API_KEY=
Validating environment variables at startup
Don't let applications start with missing or malformed secrets, fail fast with a clear error. Use a schema validator like Zod to define an env schema at startup: NODE_ENV as an enum of development, staging, production; DATABASE_URL and REDIS_URL as URLs with expected protocol prefixes (postgres, redis); JWT_SECRET with a minimum length of 32 characters; STRIPE_SECRET_KEY required to start with sk_ and OPENAI_API_KEY with sk-. Parse process.env against that schema before anything else runs, print each failing field with its message, and process.exit(1) on error.
This prevents the class of bugs where an application starts successfully but fails at runtime because STRIPE_SECRET_KEY is undefined.
Secret Detection: Preventing Leaks Before They Happen
The best time to catch a committed secret is before it hits the remote repository.
Pre-commit hooks with detect-secrets
detect-secrets scans staged files for patterns matching common secret formats. Install it with pip install detect-secrets, generate an initial baseline with detect-secrets scan > .secrets.baseline, then register the hook in .pre-commit-config.yaml with the --baseline .secrets.baseline argument so existing flagged entries don't block every commit.
Gitleaks for CI scanning
Gitleaks scans your entire git history for secrets, useful for auditing repositories that predate secret scanning practices. A typical GitHub Actions workflow checks out the repo with fetch-depth: 0 for a full history scan and runs gitleaks/gitleaks-action@v2 on every push and pull request.
GitHub also has native Secret Scanning that runs automatically on public repositories and can be enabled on private repositories with Advanced Security.
If you've already committed a secret
Assume any secret that touched a remote repository is compromised. The correct response:
- Immediately revoke the secret: rotate it in the provider dashboard
- Check provider logs for unauthorized use during the exposure window
- Remove from git history using
git filter-repo(install withpip install git-filter-repo, then rungit filter-repo --replace-textwith an expression mappingACTUAL_SECRET_VALUE==>REMOVED), and coordinate agit push --force-with-leasewith your team since this rewrites shared history
Removing from history is secondary to revoking. History cleanup prevents the secret from being discoverable in old commits, but any attacker who already cloned the repo before you cleaned history may still have it.
CI/CD Secret Management
CI/CD pipelines need secrets: deployment keys, API credentials, signing certificates. The rule: secrets go in the CI/CD platform's secret store, never in pipeline configuration files or environment-specific code.
GitHub Actions
Store secrets under Settings, Secrets and variables, Actions. In the workflow file, reference them by name as ${{ secrets.DEPLOY_TOKEN }} or ${{ secrets.PRODUCTION_DATABASE_URL }} in the step's env block, never inline in the run script. GitHub Actions secrets are encrypted at rest, masked in logs, and only available to the workflow; they're never written to disk or exposed to pull requests from forks.
Environment-scoped secrets
GitHub supports environment-specific secrets that require manual approval for deployments to protected environments. Set environment: production on the deploy job, and each ${{ secrets.DATABASE_URL }} reference resolves to the production environment's version. Configure the production environment to require a reviewer's approval before secrets are exposed. This creates a human checkpoint before production deployments.
HashiCorp Vault
For organizations that need centralized secret management with dynamic credentials, fine-grained access control, and full audit logging, HashiCorp Vault is the standard.
Core concepts
Static secrets: Key-value pairs stored in Vault. Applications authenticate and retrieve the secret value. Better than .env files because access is controlled and audited.
Dynamic secrets: Vault generates short-lived credentials on demand. An application requests a database credential, Vault creates a PostgreSQL user with a 1-hour TTL, returns the credentials, and deletes the user when the TTL expires. The application never handles a long-lived password.
Policies: Define what a principal (a role, a service) can read, write, or manage. Fine-grained: the web-app role can read from secret/app/web/* but not secret/app/admin/*.
Basic setup and usage
For local experimentation, run vault server -dev, set VAULT_ADDR to http://127.0.0.1:8200 and VAULT_TOKEN to the printed dev root token, then write and read secrets with vault kv put secret/myapp/prod database_url="postgres://..." api_key="sk-..." and vault kv get secret/myapp/prod. Dev mode is for learning only, production Vault needs proper storage backend, TLS, and authentication configuration.
Vault in application code
Use the node-vault client pointed at process.env.VAULT_ADDR, authenticating with a token in development and AppRole or Kubernetes auth in production. A small helper reads a KV v2 path and returns result.data.data (KV v2 nests the payload under data.data). At application startup, load the secrets into memory; for high-security applications, keep them in an in-memory store rather than writing them into process.env.
Dynamic database credentials
Dynamic credentials are Vault's most powerful feature for database security. Enable the database engine with vault secrets enable database, then configure a connection pointing at your PostgreSQL instance with vault write database/config/myapp-postgres, specifying the plugin name, connection URL, and allowed roles. Define a role with vault write database/roles/app-role that supplies creation_statements (a SQL template creating a role with the generated name, password, and expiration), matching revocation_statements that drop the role, a default_ttl of one hour, and a max_ttl of 24 hours.
In application code, read database/creds/app-role whenever you need a connection and Vault returns a fresh username, password, and lease duration. Wrap your connection pool in a manager that checks whether the lease is within five minutes of expiring, fetches new credentials, closes the old pool, and opens a new one with the fresh username and password before the lease actually expires.
Cloud-Native Secret Managers
If you're on a cloud provider, their native secret managers are often the right choice over self-hosted Vault.
AWS Secrets Manager
Use the @aws-sdk/client-secrets-manager package: create a SecretsManagerClient for your region, build a GetSecretValueCommand with the secret name, send it, and JSON.parse the SecretString. Two habits matter here. First, cache secrets in memory with a TTL (five minutes is a reasonable default) so you're not paying for an API call on every request. Second, let AWS handle rotation for supported services like RDS, Redshift, and DocumentDB by calling aws secretsmanager rotate-secret with a rotation Lambda ARN and a rule such as AutomaticallyAfterDays=30.
GCP Secret Manager
With @google-cloud/secret-manager, instantiate a SecretManagerServiceClient and call accessSecretVersion with the full resource name projects/${projectId}/secrets/${secretId}/versions/${version}, then read response.payload.data as a string. IAM bindings on each secret control which service accounts can access it.
Secret Rotation Patterns
Secrets must be rotated regularly and immediately on suspected compromise. The challenge: rotating a secret requires updating every place it's used simultaneously.
Zero-downtime rotation strategy
The pattern that works: support both the old and new secret simultaneously during the rotation window. Keep a SecretRotationAwareAuth class that loads jwt_secret_current and jwt_secret_previous at startup into an array. Always sign new tokens with the first entry (the current secret), but when verifying, try each secret in order and accept the token if any of them validates. This handles tokens issued before rotation without rejecting in-flight requests.
The rotation workflow:
- Generate new secret, store as
jwt_secret_current - Move old
jwt_secret_currenttojwt_secret_previous - Reload secrets in all application instances
- Wait for all tokens issued with old secret to expire
- Delete
jwt_secret_previous
1Password Secrets Automation
For teams already using 1Password, 1Password Secrets Automation bridges developer credential management and production secret injection. Install the 1Password CLI, run op signin, then start any process with op run --env-file=".env.tpl" -- node server.js. The .env.tpl file references 1Password items with paths like op://Production/PostgreSQL/connection_string and op://Production/Stripe/secret_key instead of raw values.
This approach means no secret ever touches the filesystem, op run fetches them at process start and injects them into the environment. The .env.tpl file (which contains item references, not values) can safely be committed to source control.
What Most Guides Get Wrong
Most secrets management content makes two mistakes. First, it treats Vault as the starting point rather than the destination, pushing a five-node Consul cluster on a five-person team that has one production database credential. That team needed a .env.example, a pre-commit hook, and CI-scoped secrets, not an HSM-backed secrets infrastructure with its own on-call rotation. Second, it presents secret scanning as a solved problem once gitleaks is wired into CI. Scanning after push is archaeology: the secret already touched the remote, which means you must assume it is burned and rotate it regardless of what the scan says. The detection that matters is pre-commit, where the fix costs the developer ten seconds instead of an incident review. Guides also gloss over the real bottleneck, which is not tooling but distribution: getting the right secret to the right runtime without a human copying it through a chat app. Finally, almost none of them tell you to test your revocation path. When a leaked key fires a 3 AM pager alert is the wrong time to learn that rotating your primary database password requires a full service restart.
Conclusion
Start where you are: gitignore your real .env files, commit a .env.example with empty values, and validate your environment variables at startup with a schema. Add pre-commit secret scanning and move every CI/CD credential into your platform's environment-scoped secret store, with approval gates on production. Graduate to a managed secret store (Vault, AWS Secrets Manager, GCP Secret Manager, or 1Password Secrets Automation) once you need audited, role-based access, and prefer dynamic database credentials wherever your provider supports them. Build rotation around a window where old and new secrets both verify, and rehearse revocation before you need it. Do these things in order and a leaked credential becomes a five-minute rotation instead of a company incident.
For the identity layer that sits above secrets, our two-factor authentication implementation guide covers the auth side, and our MCP server security hardening guide applies the same least-privilege discipline to AI tooling credentials. For the related threat of secrets leaking through model inputs, see our prompt injection protection guide.




Top comments (0)