Stop storing database passwords in secrets
A database password in a Kubernetes Secret has to be created, synced, rotated and kept out of Git. Every one of those steps is a place for it to leak.
With IAM database authentication and IRSA (IAM Roles for Service Accounts), there is no password to store. The pod proves its identity to AWS, gets a short-lived token, and uses that token as the Postgres password.
This post wires it up with Terraform on Aurora PostgreSQL 17 and EKS.
How it works
Pod (ServiceAccount backend-sa)
│ 1. projected OIDC token
▼
AWS STS verifies the token against the EKS OIDC provider
│ 2. temporary credentials for the IAM role
▼
Pod calls rds:GenerateDBAuthToken → token valid for 15 minutes
│ 3. token used as the Postgres password, over TLS
▼
Aurora asks IAM: may this role connect as this DB user?
Three things must line up:
- Trust: the IAM role trusts exactly one ServiceAccount in one namespace.
-
Permission: the role has
rds-db:connectfor one cluster and one DB user. -
Database: that Postgres user exists and is a member of
rds_iam.
If login fails, one of these three is wrong.
Prerequisites
- Aurora PostgreSQL cluster with IAM DB authentication enabled
- EKS cluster with an IAM OIDC provider
- Terraform >= 1.6, AWS CLI v2, kubectl
Confirm IAM auth is on:
aws rds describe-db-clusters --db-cluster-identifier <cluster> \
--query 'DBClusters[0].IAMDatabaseAuthenticationEnabled'
Step 1: Get the cluster resource ID
The IAM policy needs the cluster resource ID (cluster-XXXX), not its name. Using the name is the most common mistake: the policy applies cleanly and login still fails.
aws rds describe-db-clusters \
--db-cluster-identifier <cluster> \
--query 'DBClusters[0].DbClusterResourceId' \
--output text
The ARN format is:
arn:aws:rds-db:<region>:<account-id>:dbuser:<cluster-resource-id>/<db-user>
Step 2: Terraform, the IAM role
The OIDC provider normally already exists, because EKS creates it when IRSA is set up. This code reuses it.
variable "region" { default = "ap-south-1" }
variable "eks_cluster_name" { type = string }
variable "db_cluster_identifier" { type = string }
variable "db_iam_user" { default = "app_user" } # the Postgres user pods log in as
variable "app_namespace" { default = "backend" }
variable "app_service_account" { default = "backend-sa" }
data "aws_caller_identity" "current" {}
data "aws_eks_cluster" "this" {
name = var.eks_cluster_name
}
# Looks up the resource ID (cluster-XXXX) from the cluster name
data "aws_rds_cluster" "this" {
cluster_identifier = var.db_cluster_identifier
}
data "aws_iam_openid_connect_provider" "eks" {
url = data.aws_eks_cluster.this.identity[0].oidc[0].issuer
}
locals {
oidc_host = replace(data.aws_eks_cluster.this.identity[0].oidc[0].issuer, "https://", "")
# arn:aws:rds-db:<region>:<account>:dbuser:<cluster-resource-id>/<db-user>
db_connect_arn = "arn:aws:rds-db:${var.region}:${data.aws_caller_identity.current.account_id}:dbuser:${data.aws_rds_cluster.this.cluster_resource_id}/${var.db_iam_user}"
}
# WHO can assume the role: one ServiceAccount in one namespace
data "aws_iam_policy_document" "trust" {
statement {
actions = ["sts:AssumeRoleWithWebIdentity"]
principals {
type = "Federated"
identifiers = [data.aws_iam_openid_connect_provider.eks.arn]
}
condition {
test = "StringEquals"
variable = "${local.oidc_host}:sub"
values = ["system:serviceaccount:${var.app_namespace}:${var.app_service_account}"]
}
condition {
test = "StringEquals"
variable = "${local.oidc_host}:aud"
values = ["sts.amazonaws.com"]
}
}
}
resource "aws_iam_role" "backend_db" {
name = "eks-backend-aurora-iam-auth"
assume_role_policy = data.aws_iam_policy_document.trust.json
}
# WHAT the role can do: log in as one DB user, nothing else
data "aws_iam_policy_document" "db_connect" {
statement {
actions = ["rds-db:connect"]
resources = [local.db_connect_arn]
}
}
resource "aws_iam_role_policy" "db_connect" {
name = "aurora-rds-db-connect"
role = aws_iam_role.backend_db.id
policy = data.aws_iam_policy_document.db_connect.json
}
output "role_arn" {
value = aws_iam_role.backend_db.arn
}
output "db_connect_arn" {
value = local.db_connect_arn
}
To use a different DB user, change db_iam_user (for example -var db_iam_user=orders_svc). The ARN is rebuilt automatically.
Apply it, then check that terraform output db_connect_arn shows your real resource ID.
Step 3: Create the DB user (one time)
IAM only decides who may log in. What the user can do inside Postgres is controlled by normal SQL grants. Connect once as the master user and run the following.
CREATE USER app_user WITH LOGIN;
GRANT rds_iam TO app_user;
GRANT CONNECT ON DATABASE appdb TO app_user;
GRANT USAGE, CREATE ON SCHEMA public TO app_user;
ALTER DEFAULT PRIVILEGES IN SCHEMA public
GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO app_user;
ALTER DEFAULT PRIVILEGES IN SCHEMA public
GRANT USAGE, SELECT ON SEQUENCES TO app_user;
What each statement does:
| Statement | What it does |
|---|---|
CREATE USER app_user WITH LOGIN |
Creates the Postgres user. It has no password, which is intentional. |
GRANT rds_iam TO app_user |
Makes this user authenticate with IAM tokens instead of a password. After this, a password login is refused. This is the line that links Postgres to IAM. |
GRANT CONNECT ON DATABASE appdb |
Lets the user open a connection to appdb. Nothing more. |
GRANT USAGE, CREATE ON SCHEMA public |
USAGE lets the user see objects in the public schema. CREATE lets it create tables there (needed if migrations run as this user). Since PostgreSQL 15, CREATE on public is not granted by default. |
ALTER DEFAULT PRIVILEGES ... ON TABLES |
A rule for future tables: any table created later in public automatically gives app_user read and write (SELECT, INSERT, UPDATE, DELETE). It does not touch existing tables. |
ALTER DEFAULT PRIVILEGES ... ON SEQUENCES |
Same rule for sequences, the counters behind auto-increment IDs. Without it, INSERT fails with permission denied for sequence. |
Two things to know:
- Default privileges apply only to objects created by the role that ran the
ALTER. If another role creates tables, either runALTER DEFAULT PRIVILEGES FOR ROLE <that_role> ...or grant on existing tables withGRANT ... ON ALL TABLES IN SCHEMA public TO app_user;. - The app can read and write rows but cannot
DROPorTRUNCATEtables it doesn't own, create roles, or touch other databases.
Quick check:
SELECT has_database_privilege('app_user', 'appdb', 'CONNECT');
\du app_user
Step 4: Kubernetes
apiVersion: v1
kind: ServiceAccount
metadata:
name: backend-sa
namespace: backend
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::<account-id>:role/eks-backend-aurora-iam-auth
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: backend
namespace: backend
spec:
replicas: 2
selector:
matchLabels:
app: backend
template:
metadata:
labels:
app: backend
spec:
serviceAccountName: backend-sa
containers:
- name: backend
image: your-registry/backend:latest
env:
- name: AWS_REGION
value: ap-south-1
- name: DB_HOST
value: <cluster-writer-endpoint>
- name: DB_PORT
value: "5432"
- name: DB_NAME
value: appdb
- name: DB_USER
value: app_user
# no DB_PASSWORD anywhere
EKS injects AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE automatically, and the AWS SDK picks them up.
Step 5: App code
Generate a fresh token for every new connection:
import os, boto3, psycopg2
from sqlalchemy import create_engine
REGION, HOST = os.environ["AWS_REGION"], os.environ["DB_HOST"]
PORT, USER, NAME = 5432, os.environ["DB_USER"], os.environ["DB_NAME"]
rds = boto3.client("rds", region_name=REGION)
def get_conn():
token = rds.generate_db_auth_token(
DBHostname=HOST, Port=PORT, DBUsername=USER, Region=REGION
)
return psycopg2.connect(
host=HOST, port=PORT, dbname=NAME, user=USER,
password=token, sslmode="require",
)
engine = create_engine(
"postgresql+psycopg2://",
creator=get_conn,
pool_size=5,
pool_recycle=600,
pool_pre_ping=True,
)
The token is only checked when a connection is opened. Open connections keep working after the 15 minutes pass.
Step 6: Test it
First, check that the pod receives the role:
kubectl run sts-test -n backend --rm -it --restart=Never \
--image=amazon/aws-cli \
--overrides='{"spec":{"serviceAccountName":"backend-sa"}}' \
--command -- aws sts get-caller-identity
The ARN in the output should be assumed-role/eks-backend-aurora-iam-auth/....
Then do a real login:
kubectl run db-test -n backend --rm -it --restart=Never \
--image=python:3.12-slim \
--overrides='{"spec":{"serviceAccountName":"backend-sa"}}' \
--env=DB_HOST=<cluster-writer-endpoint> -- sh -c 'pip install -q boto3 psycopg2-binary && python - <<PY
import os, boto3, psycopg2
h = os.environ["DB_HOST"]; r = "ap-south-1"
t = boto3.client("rds", region_name=r).generate_db_auth_token(DBHostname=h, Port=5432, DBUsername="app_user", Region=r)
c = psycopg2.connect(host=h, port=5432, dbname="appdb", user="app_user", password=t, sslmode="require")
cur = c.cursor(); cur.execute("select current_user, version()"); print(cur.fetchone())
PY
'
Troubleshooting
| Error | Likely cause |
|---|---|
PAM authentication failed |
User missing rds_iam, token generated for a different hostname, or the rds-db:connect ARN uses the cluster name instead of the resource ID |
AccessDenied ... AssumeRoleWithWebIdentity |
Namespace or ServiceAccount in the trust policy doesn't match, or the OIDC provider is missing |
no pg_hba.conf entry ... no encryption |
IAM auth requires SSL, so use sslmode=require
|
| Works for 15 minutes, then fails | A token is cached somewhere. Generate a new one per connection |
| Connection timeout | Security group does not allow port 5432 from the EKS node or cluster security group |
| Breaks after recreating the cluster | The resource ID changed, so re-apply Terraform |
Security notes
- Scope the policy to a specific DB user. Never use
.../*. - Use one IAM role and one DB user per service. CloudTrail then shows which service logged in.
- Keep the master password in Secrets Manager (
manage_master_user_password = true) and use it only for admin tasks like Step 3. - IAM auth works best below roughly 200 new connections per second, so pool connections or use RDS Proxy.
Conclusion
There is no password to leak or rotate. Access is revoked by editing IAM, and every login is tied to a Kubernetes identity.
Top comments (0)