Part 7: Secrets Management — AWS Secrets Manager + External Secrets Operator + IRSA
Part of the series: Building a Production-Grade DevSecOps Pipeline on AWS
Introduction
Secrets management is one of the areas where teams most commonly take shortcuts that later become security incidents. The three anti-patterns to avoid:
The pod never holds AWS credentials. It exchanges a short-lived Kubernetes
JWT (1 hour) for temporary IAM credentials via AWS STS, then ESO fetches
the secret and creates a Kubernetes Secret.
- Secrets in Git — even in a private repo, any developer with access can read them; git history preserves them forever even after deletion
- Secrets baked into images — anyone who pulls the image gets the secrets
- Secrets in Kubernetes ConfigMaps — ConfigMaps are not encrypted by default; any pod in the namespace can read them
This pipeline uses a three-layer approach:
- AWS Secrets Manager — the source of truth; all secrets live here, encrypted with KMS
- External Secrets Operator (ESO) — a Kubernetes controller that fetches secrets from AWS and creates Kubernetes Secret objects
- IRSA (IAM Roles for Service Accounts) — pod-level AWS identity so ESO can call Secrets Manager without node-level credentials
Architecture
┌──────────────────────────────────────────────────────────────────────┐
│ SECRETS FLOW │
│ │
│ AWS Secrets Manager │
│ production/myapp/db-password: "s3cr3t-v@lue" │
│ │ │
│ │ GetSecretValue (every 1h) │
│ │ Authenticated via IRSA (OIDC token → STS → temp creds) │
│ ▼ │
│ ESO Operator Pod │
│ (external-secrets namespace) │
│ │ │
│ │ Creates/updates │
│ ▼ │
│ Kubernetes Secret: myapp-db-password │
│ (myapp namespace) │
│ │ │
│ │ Mounted as env var │
│ ▼ │
│ myapp Pod: process.env.DB_PASSWORD = "s3cr3t-v@lue" │
└──────────────────────────────────────────────────────────────────────┘
IRSA Deep Dive
IRSA (IAM Roles for Service Accounts) is the mechanism that gives individual Kubernetes pods fine-grained AWS IAM permissions without sharing credentials at the node level.
┌────────────────────────────────────────────────────────────────────┐
│ HOW IRSA WORKS │
│ │
│ 1. EKS creates an OIDC provider for the cluster │
│ URL: oidc.eks.us-east-1.amazonaws.com/id/CLUSTER_ID │
│ │
│ 2. Pod's ServiceAccount is annotated: │
│ eks.amazonaws.com/role-arn: arn:aws:iam::ACCT:role/eso-role │
│ │
│ 3. EKS projects a signed OIDC JWT into the pod at: │
│ /var/run/secrets/eks.amazonaws.com/serviceaccount/token │
│ │
│ 4. ESO SDK calls sts:AssumeRoleWithWebIdentity with that token │
│ │
│ 5. AWS validates: token signed by trusted OIDC provider? │
│ sub matches trust policy condition? │
│ │
│ 6. AWS returns temporary creds (AccessKeyId + SecretKey + Token) │
│ Valid for 1 hour, then automatically expire │
│ │
│ Result: only THIS pod in THIS namespace with THIS SA can │
│ assume the role — not any other pod on the same node │
└────────────────────────────────────────────────────────────────────┘
Why IRSA over node-level IAM?
Node-level IAM (EC2 instance profile) gives every pod on the node the same permissions. If one pod is compromised, the attacker has the permissions of every pod on that node. IRSA scopes permissions to the exact service account — the blast radius of a compromise is just that one workload.
Terraform: ESO IRSA Module
# _modules/eso-irsa/main.tf
variable "cluster_name" { type = string }
variable "env" { type = string }
variable "oidc_provider_arn" { type = string }
variable "oidc_provider" { type = string } # URL without https://
variable "account_id" { type = string }
variable "aws_region" { type = string }
locals {
# Helm fullname = {release}-{chart}
# Release name = cluster name (e.g., myapp-production-use1)
# Chart name = myapp
# So the SA name = myapp-production-use1-myapp
#
# CRITICAL: The IRSA trust policy sub claim must match this exactly.
# Getting this wrong = ESO pods cannot assume the role = secrets don't sync.
sa_name = "${var.cluster_name}-myapp"
namespace = "external-secrets"
}
resource "aws_iam_role" "eso" {
name = "${var.cluster_name}-eso"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Effect = "Allow"
Principal = { Federated = var.oidc_provider_arn }
Action = "sts:AssumeRoleWithWebIdentity"
Condition = {
StringEquals = {
"${var.oidc_provider}:aud" = "sts.amazonaws.com"
# Must match: system:serviceaccount:{namespace}:{sa-name}
"${var.oidc_provider}:sub" = "system:serviceaccount:${local.namespace}:${local.sa_name}"
}
}
}]
})
}
resource "aws_iam_role_policy" "eso" {
name = "eso-secrets-policy"
role = aws_iam_role.eso.id
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Sid = "SecretsManagerRead"
Effect = "Allow"
Action = [
"secretsmanager:GetSecretValue",
"secretsmanager:DescribeSecret",
"secretsmanager:ListSecretVersionIds"
]
# Wildcard covers all secrets for this environment
Resource = "arn:aws:secretsmanager:*:${var.account_id}:secret:${var.env}/myapp/*"
}
]
})
}
output "role_arn" { value = aws_iam_role.eso.arn }
Creating Secrets in AWS Secrets Manager
Use a structured naming convention: {env}/myapp/{secret-name}
# Development secrets (dev account)
aws secretsmanager create-secret \
--name "dev/myapp/db-password" \
--secret-string '{"password":"dev-db-secret-here"}' \
--region us-east-1 \
--profile myapp-dev-use1
# Production secrets (production account)
aws secretsmanager create-secret \
--name "production/myapp/db-password" \
--secret-string '{"password":"prod-super-secret-here"}' \
--region us-east-1 \
--profile myapp-prod-use1
# To update a secret (rotation):
aws secretsmanager update-secret \
--secret-id "production/myapp/db-password" \
--secret-string '{"password":"new-rotated-password"}' \
--region us-east-1 \
--profile myapp-prod-use1
# ESO picks up the new value within refreshInterval (1h default)
ESO Installation via ArgoCD
ESO runs as a controller in the external-secrets namespace on every cluster:
# infrastructure/eso/applicationset.yaml
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: external-secrets-operator
namespace: argocd
spec:
generators:
- list:
elements:
- cluster: myapp-dev-use1
region: us-east-1
- cluster: myapp-dev-usw2
region: us-west-2
- cluster: myapp-staging-use1
region: us-east-1
- cluster: myapp-staging-usw2
region: us-west-2
- cluster: myapp-production-use1
region: us-east-1
- cluster: myapp-production-usw2
region: us-west-2
template:
metadata:
name: "eso-{{cluster}}"
spec:
project: production
source:
repoURL: https://charts.external-secrets.io
chart: external-secrets
targetRevision: "0.9.13"
helm:
values: |
serviceAccount:
annotations:
eks.amazonaws.com/role-arn: "{{irsaRoleArn}}"
destination:
name: "{{cluster}}"
namespace: external-secrets
syncPolicy:
syncOptions: [CreateNamespace=true]
SecretStore CRD
The SecretStore tells ESO where to find secrets and how to authenticate:
# apps/myapp/templates/secretstore.yaml
{{- if .Values.externalSecrets.enabled }}
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: aws-secrets-manager
namespace: {{ .Release.Namespace }}
spec:
provider:
aws:
service: SecretsManager
region: us-east-1 # Always pull from us-east-1 (single source of truth)
auth:
jwt:
serviceAccountRef:
name: {{ include "myapp.serviceAccountName" . }}
# DO NOT add namespace: field here.
# Namespaced SecretStore rejects serviceAccountRef.namespace.
# The SA is in the same namespace as the SecretStore.
{{- end }}
Lesson learned: A namespaced
SecretStore(notClusterSecretStore) will reject the config ifserviceAccountRef.namespaceis specified. The namespace is implicit — it's the same namespace as the SecretStore itself. OnlyClusterSecretStoresupports cross-namespace references.
ExternalSecret CRD
The ExternalSecret tells ESO which secret to fetch and what Kubernetes Secret to create:
# apps/myapp/templates/external-secret.yaml
{{- if .Values.externalSecrets.enabled }}
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: myapp-secrets
namespace: {{ .Release.Namespace }}
spec:
refreshInterval: {{ .Values.externalSecrets.refreshInterval | default "1h" }}
secretStoreRef:
name: aws-secrets-manager
kind: SecretStore
target:
name: myapp-db-credentials # Name of the Kubernetes Secret to create
creationPolicy: Owner # ESO owns this secret — it will delete it if ExternalSecret is deleted
data:
- secretKey: DB_PASSWORD # Key in the Kubernetes Secret
remoteRef:
key: production/myapp/db-password # Secret name in AWS Secrets Manager
property: password # JSON key within the secret value
{{- end }}
Helm Chart Integration
# apps/myapp/values.yaml (defaults — disabled)
externalSecrets:
enabled: false
refreshInterval: 1h
irsaRoleArn: ""
# apps/myapp/values-production.yaml
externalSecrets:
enabled: true
irsaRoleArn: "" # Injected per-cluster via ApplicationSet parameter
# apps/myapp/values-dev.yaml
externalSecrets:
enabled: true
irsaRoleArn: "arn:aws:iam::557702566877:role/myapp-dev-eso"
The ApplicationSet injects the correct IRSA role ARN per cluster via parameters:
parameters:
- name: "externalSecrets.irsaRoleArn"
value: "{{irsaRoleArn}}"
Deployment Mounting the Secret
# apps/myapp/templates/deployment.yaml (relevant section)
env:
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: myapp-db-credentials # Created by ESO from ExternalSecret
key: DB_PASSWORD
Verification
# Check ExternalSecret sync status
kubectl --context prod-use1 get externalsecret -n myapp
# NAME STORE REFRESH INTERVAL STATUS READY
# myapp-secrets aws-secrets-manager 1h SecretSynced True
# Check the Kubernetes Secret was created
kubectl --context prod-use1 get secret myapp-db-credentials -n myapp
# NAME TYPE DATA AGE
# myapp-db-credentials Opaque 1 5m
# View (base64 encoded — decode to see value)
kubectl --context prod-use1 get secret myapp-db-credentials -n myapp \
-o jsonpath='{.data.DB_PASSWORD}' | base64 -d
ArgoCD false positive: ArgoCD will show the myapp Application as
OutOfSynceven when everything is working. This is because ESO writesstatus.refreshTimeto the ExternalSecret at every sync cycle. ArgoCD detects this runtime write as a diff from the Git-defined manifest. This is a known, safe false positive — the secret IS syncing correctly.
Secret Rotation
When you need to rotate a secret:
# 1. Update in AWS Secrets Manager
aws secretsmanager update-secret \
--secret-id "production/myapp/db-password" \
--secret-string '{"password":"new-rotated-value"}' \
--profile myapp-prod-use1
# 2. ESO picks it up automatically within refreshInterval (1h)
# OR force immediate refresh:
kubectl --context prod-use1 annotate externalsecret myapp-secrets -n myapp \
force-sync=$(date +%s) --overwrite
# 3. Kubernetes Secret is updated
# 4. Pods need restart to pick up new env var value:
kubectl --context prod-use1 rollout restart deployment/myapp -n myapp
Grafana Admin Secret — A Real-World Lesson
During this build, we used existingSecret: grafana-admin-secret in the kube-prometheus-stack values. The secret was created with a placeholder password YourStrongPassword123!. Grafana initialized its SQLite database with this value when the pod first started.
Later, the secret value was changed — but Grafana's database still had the old value. The env var GF_SECURITY_ADMIN_PASSWORD only sets the initial password; it cannot change an existing one.
Fix:
kubectl exec -n monitoring <grafana-pod> -c grafana -- \
grafana-cli admin reset-admin-password 'NewPassword!'
Lesson: For any tool that reads credentials once at database initialization, changing the Kubernetes Secret is not enough. You must either reset via the tool's CLI or destroy and recreate the persistent volume.
Summary
By the end of Part 7 you have:
- ✅ All secrets stored exclusively in AWS Secrets Manager (never in Git or images)
- ✅ ESO installed on all 6 clusters via ArgoCD ApplicationSet
- ✅ IRSA roles per cluster with least-privilege Secrets Manager read access
- ✅ SecretStore + ExternalSecret CRDs syncing secrets into Kubernetes
- ✅ Helm chart values.yaml integration with enabled/disabled toggle per environment
- ✅ Secret rotation workflow (update in ASM → ESO auto-refreshes)
Screenshot Placeholders
SCREENSHOT: AWS Secrets Manager console showing secrets per environment
SCREENSHOT: kubectl get externalsecret showing SecretSynced: True
SCREENSHOT: ArgoCD showing ESO app as OutOfSync/Healthy (expected false positive)
Next: Part 8 — Security Stack: Kyverno, Falco, WAF, and GuardDuty
Follow the series — next part publishes next Wednesday.
Live system: https://www.matthewoladipupo.dev/health
Runbook: Operations Guide
Source code: myapp-infra | myapp-gitops | myapp
Top comments (0)