Introduction
Every application needs secrets—database passwords, API keys, TLS certificates, encryption keys. How you manage these secrets can be the difference between a secure system and a catastrophic data breach.
Hardcoded secrets in code repositories are leaked constantly. Environment variables can be exposed through logs or error messages. Configuration files stored in version control are a security nightmare. Yet teams continue using these anti-patterns because proper secrets management seems complex.
In this comprehensive guide, we'll explore three leading secrets management solutions—HashiCorp Vault, AWS Secrets Manager, and SOPS—helping you choose the right approach for your security requirements.
Why Secrets Management Matters
The Cost of Leaked Secrets
Real incidents:
- Uber: $148M fine (credentials in GitHub)
- Capital One: 100M records (misconfigured IAM)
- Codecov: Supply chain attack (exposed secrets)
- Travis CI: Exposed environment variables
Average cost of data breach: $4.35M (IBM 2023)
Common Anti-Patterns
Hardcoded in Code:
# NEVER do this
AWS_ACCESS_KEY = "AKIAIOSFODNN7EXAMPLE"
AWS_SECRET_KEY = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
DB_PASSWORD = "supersecret123"
Committed to Git:
# This is searchable on GitHub
cat .env
DATABASE_URL=postgresql://user:password@localhost/db
API_KEY=sk_live_51H...
STRIPE_SECRET=whsec_...
Environment Variables Exposed:
# Error messages often dump environment
import os
print(os.environ) # All secrets exposed in logs
# Container inspect shows env vars
docker inspect <container> | grep -i password
Unencrypted ConfigMaps/Secrets:
# Kubernetes Secrets are only base64 encoded, not encrypted
kubectl get secret db-password -o yaml
# Anyone with cluster access can decode
Solution Comparison Overview
| Feature | HashiCorp Vault | AWS Secrets Manager | SOPS |
|---|---|---|---|
| Type | Centralized vault | Managed service | File encryption |
| Cost | Self-hosted: $0 Enterprise: $$$ |
$0.40/secret/month $0.05/10K API calls |
Free |
| Complexity | High | Low | Medium |
| Dynamic Secrets | ✅ Yes | ⚠️ Limited | ❌ No |
| Secret Rotation | ✅ Automatic | ✅ Automatic | ❌ Manual |
| Audit Logging | ✅ Detailed | ✅ CloudTrail | ❌ Limited |
| Multi-Cloud | ✅ Yes | ❌ AWS only | ✅ Yes |
| GitOps Friendly | ⚠️ External | ⚠️ External | ✅ Yes |
| Encryption at Rest | ✅ Yes | ✅ Yes | ✅ Yes |
| Access Control | ✅ Fine-grained | ✅ IAM-based | ⚠️ KMS-based |
HashiCorp Vault
Architecture
Application → Vault Agent → Vault Server → Storage Backend
↓
(Encrypted)
↓
Consul/etcd/S3
Core Concepts
Secrets Engines: Different types of secret storage and generation
# Key-Value secrets (static)
vault kv put secret/database/config \
username="dbuser" \
password="supersecret"
# Dynamic secrets (generated on-demand)
vault read database/creds/readonly
# Returns temporary credentials that auto-expire
Authentication Methods: How clients prove identity
# Kubernetes authentication
vault write auth/kubernetes/role/myapp \
bound_service_account_names=myapp \
bound_service_account_namespaces=production \
policies=myapp-policy \
ttl=1h
Policies: Fine-grained access control
# myapp-policy.hcl
path "secret/data/database/config" {
capabilities = ["read"]
}
path "database/creds/readonly" {
capabilities = ["read"]
}
path "secret/data/api-keys/*" {
capabilities = ["read", "list"]
}
Installation and Setup
# Vault on Kubernetes with Helm
helm repo add hashicorp https://helm.releases.hashicorp.com
helm install vault hashicorp/vault \
--set server.ha.enabled=true \
--set server.ha.replicas=3 \
--set ui.enabled=true \
--set server.dataStorage.size=10Gi
# Initialize Vault
kubectl exec vault-0 -- vault operator init
# Save unseal keys and root token securely!
# Unseal Vault (repeat on all pods)
kubectl exec vault-0 -- vault operator unseal <unseal-key-1>
kubectl exec vault-0 -- vault operator unseal <unseal-key-2>
kubectl exec vault-0 -- vault operator unseal <unseal-key-3>
Using Vault with Applications
Method 1: Vault Agent Sidecar
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp
spec:
template:
metadata:
annotations:
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/role: "myapp"
vault.hashicorp.com/agent-inject-secret-database: "secret/data/database/config"
vault.hashicorp.com/agent-inject-template-database: |
{{- with secret "secret/data/database/config" -}}
DATABASE_URL=postgresql://{{ .Data.data.username }}:{{ .Data.data.password }}@postgres:5432/mydb
{{- end }}
spec:
serviceAccountName: myapp
containers:
- name: app
image: myapp:v1.0
# Secret automatically written to /vault/secrets/database
Method 2: Vault SDK in Application
import hvac
# Authenticate with Kubernetes
client = hvac.Client(url='http://vault:8200')
with open('/var/run/secrets/kubernetes.io/serviceaccount/token') as f:
jwt = f.read()
client.auth.kubernetes.login(
role='myapp',
jwt=jwt
)
# Read secrets
secret = client.secrets.kv.v2.read_secret_version(
path='database/config'
)
db_user = secret['data']['data']['username']
db_pass = secret['data']['data']['password']
Dynamic Secrets
Vault generates short-lived credentials on-demand:
# Configure database secrets engine
vault secrets enable database
vault write database/config/postgresql \
plugin_name=postgresql-database-plugin \
allowed_roles="readonly" \
connection_url="postgresql://{{username}}:{{password}}@postgres:5432/mydb" \
username="vault" \
password="vaultpass"
# Create role for read-only access
vault write database/roles/readonly \
db_name=postgresql \
creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
default_ttl="1h" \
max_ttl="24h"
# Application requests credentials
vault read database/creds/readonly
# Returns:
# Key Value
# lease_id database/creds/readonly/abc123
# lease_duration 1h
# username v-readonly-abc123
# password A1a-generated-password
# Credentials automatically revoked after 1h
Secret Rotation
# Automatic rotation for supported systems
vault write -f database/rotate-root/postgresql
# Vault rotates its own database credentials
# For static secrets, create rotation policy
vault write sys/rotate-root/config \
rotation_period="720h" # 30 days
Strengths
Dynamic Secrets: Generate short-lived credentials
Secret Rotation: Automatic credential rotation
Fine-Grained Access: Policies per path/operation
Encryption as a Service: Use Vault to encrypt/decrypt data
# Encrypt data without storing it
vault write transit/encrypt/orders \
plaintext=$(echo "sensitive data" | base64)
vault write transit/decrypt/orders \
ciphertext="vault:v1:..."
Multi-Cloud: Works anywhere
Audit Logging: Detailed audit trail
Weaknesses
Operational Complexity: Requires HA setup, unsealing, backups
Learning Curve: Many concepts to understand
Unsealing Requirement: Manual unsealing after restarts
# After pod restart, must unseal
kubectl exec vault-0 -- vault status
# Sealed: true
# Must provide unseal keys
kubectl exec vault-0 -- vault operator unseal <key-1>
kubectl exec vault-0 -- vault operator unseal <key-2>
kubectl exec vault-0 -- vault operator unseal <key-3>
Cost: Enterprise features (DR, namespaces) require license
When to Use Vault
✓ Multi-cloud or hybrid infrastructure
✓ Need dynamic secrets
✓ Require automatic secret rotation
✓ Compliance requirements (detailed audit logs)
✓ Have dedicated platform/security team
✓ Large number of secrets (>100)
✓ Need encryption as a service
Cost
Open Source (self-hosted):
- Infrastructure: $200-500/month (HA cluster)
- Operations: 0.25 FTE = $3,000-5,000/month
Total: $3,200-5,500/month
Enterprise:
- License: $15,000-50,000/year
- Infrastructure: $200-500/month
- Operations: 0.25 FTE
Total: $4,500-9,500/month
AWS Secrets Manager
Architecture
Application → AWS SDK → Secrets Manager → KMS
↓
(Encrypted storage)
Creating Secrets
# Create secret
aws secretsmanager create-secret \
--name production/database/password \
--description "Production database password" \
--secret-string "supersecret123"
# Create with JSON structure
aws secretsmanager create-secret \
--name production/database/config \
--secret-string '{
"username": "dbuser",
"password": "supersecret123",
"host": "db.example.com",
"port": 5432
}'
Using Secrets in Applications
import boto3
import json
def get_secret():
client = boto3.client('secretsmanager', region_name='us-east-1')
response = client.get_secret_value(
SecretId='production/database/config'
)
secret = json.loads(response['SecretString'])
return secret
# Use in application
secret = get_secret()
db_url = f"postgresql://{secret['username']}:{secret['password']}@{secret['host']}:{secret['port']}/mydb"
Kubernetes Integration
External Secrets Operator:
# Install External Secrets Operator
helm install external-secrets \
external-secrets/external-secrets \
-n external-secrets-system \
--create-namespace
# Create SecretStore
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: aws-secrets-manager
namespace: production
spec:
provider:
aws:
service: SecretsManager
region: us-east-1
auth:
jwt:
serviceAccountRef:
name: external-secrets-sa
---
# Create ExternalSecret
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: database-config
namespace: production
spec:
refreshInterval: 1h
secretStoreRef:
name: aws-secrets-manager
kind: SecretStore
target:
name: database-config
creationPolicy: Owner
data:
- secretKey: username
remoteRef:
key: production/database/config
property: username
- secretKey: password
remoteRef:
key: production/database/config
property: password
Automatic Rotation
# Lambda function for rotation
import boto3
import pymysql
def lambda_handler(event, context):
service_client = boto3.client('secretsmanager')
# Get current secret
current_secret = service_client.get_secret_value(
SecretId=event['SecretId']
)
# Generate new password
new_password = generate_random_password()
# Update database
connection = pymysql.connect(
host=current_secret['host'],
user=current_secret['username'],
password=current_secret['password']
)
with connection.cursor() as cursor:
cursor.execute(
f"ALTER USER '{current_secret['username']}' IDENTIFIED BY '{new_password}'"
)
connection.commit()
# Update secret
service_client.put_secret_value(
SecretId=event['SecretId'],
SecretString=json.dumps({
'username': current_secret['username'],
'password': new_password
})
)
# Enable automatic rotation
aws secretsmanager rotate-secret \
--secret-id production/database/password \
--rotation-lambda-arn arn:aws:lambda:us-east-1:123456789:function:rotate-secret \
--rotation-rules AutomaticallyAfterDays=30
Cross-Region Replication
aws secretsmanager replicate-secret-to-regions \
--secret-id production/database/password \
--add-replica-regions Region=eu-west-1 \
--add-replica-regions Region=ap-southeast-1
Strengths
Fully Managed: Zero operational overhead
AWS Integration: Native IAM, CloudTrail, VPC endpoints
Automatic Rotation: Built-in rotation for RDS, Redshift, DocumentDB
Cross-Region Replication: Automatic failover
Compliance: SOC, PCI, HIPAA certified
Weaknesses
AWS Only: Can't use with other clouds
Cost: Expensive at scale ($0.40/secret/month + API calls)
No Dynamic Secrets: Can't generate temporary credentials
Limited Rotation: Only supports specific AWS services
When to Use Secrets Manager
✓ AWS-only infrastructure
✓ Want zero operational overhead
✓ Need automatic rotation for RDS/Redshift
✓ Require cross-region replication
✓ Small to medium number of secrets (<1000)
✓ Compliance requirements (AWS certified)
Cost
100 secrets, 1M API calls/month:
- Secrets: 100 × $0.40 = $40/month
- API calls: 1M × $0.05/10K = $5/month
Total: $45/month
1,000 secrets, 10M API calls/month:
- Secrets: 1,000 × $0.40 = $400/month
- API calls: 10M × $0.05/10K = $50/month
Total: $450/month
SOPS (Secrets OPerationS)
Architecture
Developer → SOPS → KMS/PGP → Encrypted File → Git
↓
(Encrypt/Decrypt)
↓
Application
Core Concept
SOPS encrypts files while keeping structure readable:
# Original secrets.yaml
api:
key: sk_live_51H...
secret: whsec_...
database:
password: supersecret123
host: db.example.com
# Encrypted with SOPS
api:
key: ENC[AES256_GCM,data:abc123...,iv:xyz...,tag:def...]
secret: ENC[AES256_GCM,data:uvw456...,iv:rst...,tag:ghi...]
database:
password: ENC[AES256_GCM,data:mno789...,iv:jkl...,tag:pqr...]
host: db.example.com # Not encrypted (no sensitive data)
sops:
kms:
- arn: arn:aws:kms:us-east-1:123456789:key/abc-123
created_at: "2024-01-15T10:00:00Z"
pgp:
- fingerprint: ABC123...
Installation and Configuration
# Install SOPS
brew install sops
# Or download binary
curl -LO https://github.com/mozilla/sops/releases/download/v3.8.1/sops-v3.8.1.linux.amd64
chmod +x sops-v3.8.1.linux.amd64
sudo mv sops-v3.8.1.linux.amd64 /usr/local/bin/sops
Configuration (.sops.yaml):
creation_rules:
# Production secrets (AWS KMS)
- path_regex: production/.*\.yaml$
kms: arn:aws:kms:us-east-1:123456789:key/production-key
encrypted_regex: ^(data|stringData|password|secret|key)$
# Staging secrets (different KMS key)
- path_regex: staging/.*\.yaml$
kms: arn:aws:kms:us-east-1:123456789:key/staging-key
encrypted_regex: ^(data|stringData|password|secret|key)$
# Development (PGP)
- path_regex: development/.*\.yaml$
pgp: >-
ABC123DEF456,
GHI789JKL012
encrypted_regex: ^(data|stringData|password|secret|key)$
Using SOPS
# Encrypt file
sops --encrypt secrets.yaml > secrets.enc.yaml
# Edit encrypted file (decrypts, opens editor, re-encrypts on save)
sops secrets.enc.yaml
# Decrypt file
sops --decrypt secrets.enc.yaml
# Decrypt and pipe to kubectl
sops --decrypt secrets.enc.yaml | kubectl apply -f -
GitOps with SOPS and Flux
# Flux Kustomization with SOPS decryption
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: apps
namespace: flux-system
spec:
interval: 10m
path: ./apps/production
prune: true
sourceRef:
kind: GitRepository
name: flux-system
# Decrypt SOPS-encrypted files
decryption:
provider: sops
secretRef:
name: sops-kms
---
# KMS credentials for decryption
apiVersion: v1
kind: Secret
metadata:
name: sops-kms
namespace: flux-system
type: Opaque
stringData:
AWS_ACCESS_KEY_ID: AKIAIOSFODNN7EXAMPLE
AWS_SECRET_ACCESS_KEY: wJalrXUtn...
ArgoCD with SOPS
# Install SOPS plugin
apiVersion: v1
kind: ConfigMap
metadata:
name: argocd-cm
namespace: argocd
data:
kustomize.buildOptions: --enable-alpha-plugins --enable-helm
configManagementPlugins: |
- name: sops
generate:
command: ["sh", "-c"]
args: ["sops -d secrets.yaml | kubectl apply -f -"]
Strengths
GitOps Friendly: Secrets version-controlled alongside code
Simple: Just a binary, no infrastructure
Multi-Cloud: Works with AWS KMS, GCP KMS, Azure Key Vault, PGP
Free: Open source, no licensing costs
Selective Encryption: Encrypt only sensitive fields
Auditable: Git history shows who changed what
Weaknesses
No UI: Command-line only
No Dynamic Secrets: Static secrets only
No Automatic Rotation: Manual rotation required
Key Management: Must manage KMS keys/PGP keys
Limited Audit: Only Git history, no detailed access logs
When to Use SOPS
✓ GitOps workflow
✓ Small team
✓ Want secrets in version control (encrypted)
✓ Budget-conscious (free)
✓ Simple use case (static secrets)
✓ Multi-cloud (using different KMS per cloud)
✓ Don't need dynamic secrets or rotation
Cost
SOPS:
- Software: Free
- KMS usage: ~$1/month (per key)
- Operations: Minimal
Total: ~$1-5/month
Detailed Comparison
Security Posture
Vault:
- Encryption at rest ✓
- Encryption in transit ✓
- Detailed audit logs ✓
- Fine-grained access ✓
- Secret rotation ✓
- Dynamic secrets ✓
Score: 10/10
Secrets Manager:
- Encryption at rest ✓
- Encryption in transit ✓
- CloudTrail audit ✓
- IAM access control ✓
- Automatic rotation ✓ (limited)
- Dynamic secrets ✗
Score: 8/10
SOPS:
- Encryption at rest ✓
- Encryption in transit ⚠️ (Git over HTTPS)
- Git audit logs ⚠️
- KMS access control ✓
- Secret rotation ✗
- Dynamic secrets ✗
Score: 5/10
Operational Overhead
Vault: HIGH
- Setup HA cluster
- Configure storage backend
- Implement unsealing strategy
- Backup and recovery
- Monitoring and alerting
- Regular updates
Time: 40 hours initial + 20 hours/month
Secrets Manager: NONE
- Fully managed
- No infrastructure
- Automatic updates
Time: 2 hours initial + 1 hour/month
SOPS: LOW
- Install binary
- Configure .sops.yaml
- Manage KMS keys
Time: 4 hours initial + 2 hours/month
Hybrid Approaches
SOPS + Vault
Use SOPS for GitOps, Vault for dynamic secrets:
# SOPS-encrypted Kubernetes Secret
apiVersion: v1
kind: Secret
metadata:
name: vault-config
type: Opaque
stringData:
vault-token: ENC[AES256_GCM,data:abc123...]
vault-addr: https://vault.example.com
# Application uses Vault for dynamic DB credentials
import hvac
# Vault token from SOPS-encrypted Kubernetes Secret
client = hvac.Client(
url=os.getenv('VAULT_ADDR'),
token=os.getenv('VAULT_TOKEN')
)
# Get dynamic database credentials
db_creds = client.secrets.database.generate_credentials(
name='readonly'
)
Secrets Manager + Parameter Store
Use Secrets Manager for sensitive secrets, Parameter Store for config:
# Sensitive: Secrets Manager
aws secretsmanager create-secret \
--name production/database/password \
--secret-string "supersecret"
# Non-sensitive: Parameter Store (cheaper)
aws ssm put-parameter \
--name /production/database/host \
--value "db.example.com" \
--type String
Best Practices
Principle of Least Privilege
# Vault: Minimal policy
path "secret/data/myapp/*" {
capabilities = ["read"]
}
path "database/creds/readonly" {
capabilities = ["read"]
}
# Deny everything else (implicit)
// AWS IAM: Minimal policy
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "secretsmanager:GetSecretValue",
"Resource": "arn:aws:secretsmanager:us-east-1:123456789:secret:production/myapp/*"
}
]
}
Secret Rotation
Rotation schedule:
- API keys: 90 days
- Passwords: 30-60 days
- Certificates: 90 days (Let's Encrypt)
- Encryption keys: Yearly
Audit and Monitor
# Vault: Enable audit logging
vault audit enable file file_path=/var/log/vault/audit.log
# AWS: Enable CloudTrail for Secrets Manager
aws cloudtrail create-trail \
--name secrets-audit \
--s3-bucket-name audit-logs
# Alert on secret access
aws cloudwatch put-metric-alarm \
--alarm-name high-secret-access \
--metric-name GetSecretValue \
--threshold 1000 \
--comparison-operator GreaterThanThreshold
Never Log Secrets
# Bad
logger.info(f"Connecting with password: {password}")
# Good
logger.info("Connecting to database")
# Don't log secrets
Choosing the Right Solution
Decision Tree
Need dynamic secrets or automatic rotation?
├─ Yes → Vault or Secrets Manager
│ ├─ AWS-only?
│ │ ├─ Yes → Secrets Manager
│ │ └─ No → Vault
│ └─ Have ops team?
│ ├─ Yes → Vault
│ └─ No → Secrets Manager
└─ No → SOPS or Secrets Manager
├─ Using GitOps?
│ ├─ Yes → SOPS
│ └─ No → Secrets Manager
└─ Budget?
├─ Limited → SOPS
└─ Flexible → Secrets Manager
Recommendations by Team Size
Small Team (<10 engineers):
→ SOPS or Secrets Manager
- Simple to use
- Low/no operations
- Cost-effective
Medium Team (10-50 engineers):
→ Secrets Manager or Vault
- Secrets Manager if AWS-only
- Vault if multi-cloud
- Need audit and compliance
Large Team (>50 engineers):
→ Vault
- Dynamic secrets essential
- Fine-grained access control
- Dedicated platform team
Conclusion
There's no one-size-fits-all secrets management solution:
HashiCorp Vault: Most powerful, but requires operational expertise. Choose when you need dynamic secrets, automatic rotation, and have a platform team.
AWS Secrets Manager: Fully managed, AWS-native. Choose when you're AWS-only and want zero operational overhead.
SOPS: Simple file encryption. Choose when you use GitOps, have a small team, and don't need dynamic secrets.
Remember: Any secrets management solution is better than hardcoded secrets. Start with the simplest solution that meets your security requirements, then evolve as your needs grow.
Need help implementing secrets management? InstaDevOps provides expert consulting for security architecture, secrets management, and compliance. Contact us for a free consultation.
Need Help with Your DevOps Infrastructure?
At InstaDevOps, we specialize in helping startups and scale-ups build production-ready infrastructure without the overhead of a full-time DevOps team.
Our Services:
- 🏗️ AWS Consulting - Cloud architecture, cost optimization, and migration
- ☸️ Kubernetes Management - Production-ready clusters and orchestration
- 🚀 CI/CD Pipelines - Automated deployment pipelines that just work
- 📊 Monitoring & Observability - See what's happening in your infrastructure
Special Offer: Get a free DevOps audit - 50+ point checklist covering security, performance, and cost optimization.
📅 Book a Free 15-Min Consultation
Originally published at instadevops.com
Top comments (0)