DEV Community

Cover image for Secrets Management in Modern DevOps: Vault, IRSA, External Secrets When to Use Each
varun varde
varun varde

Posted on

Secrets Management in Modern DevOps: Vault, IRSA, External Secrets When to Use Each

#devops #programming #irsa #linux

Secrets management failures rarely begin with malicious intent.

They begin with expediency.

An engineer hardcodes an API key “temporarily.” A .env file gets committed accidentally. A production database password gets shared in Slack during an outage because “we’ll rotate it later.” Eventually those shortcuts accumulate into a sprawling credential catastrophe hidden beneath otherwise competent infrastructure.

The uncomfortable truth is that poor secrets hygiene exists everywhere:

  • Startups
  • Scaleups
  • Enterprises
  • Banks
  • Government systems
  • Fortune 500 infrastructure

The issue is rarely ignorance. It is architectural ambiguity.

Modern DevOps teams now face multiple competing approaches:

  • Cloud-native identity systems
  • Kubernetes secret abstractions
  • Vault
  • External Secrets Operator
  • Sealed Secrets
  • Workload identity federation
  • Dynamic credentials

Choosing incorrectly creates operational fragility. Choosing well dramatically improves both security and developer experience.

This guide explains when to use each model, where each one fails, and how to evolve from common anti-patterns toward a production-grade secrets architecture without detonating existing workloads.

The Secrets Management Anti-Patterns (and Their Blast Radius)

Before discussing solutions, understand the failure modes.

Because nearly every modern secrets architecture exists to solve one of these disasters.

Anti-Pattern 1: Hardcoded Secrets in Source Code

Example

API_KEY = "sk-prod-293847239847"
Enter fullscreen mode Exit fullscreen mode

This is not merely bad practice.

It is operationally radioactive.

Once committed:

  • Git history preserves it
  • Forks replicate it
  • CI logs may expose it
  • Developers clone it locally
  • Backups persist it indefinitely

Even if deleted later.

Anti-Pattern 2: Shared Credentials

Example

prod-admin / password123
Enter fullscreen mode Exit fullscreen mode

Used by:

  • Developers
  • CI systems
  • Automation tools
  • Contractors

Result

No attribution
No least privilege
No revocation granularity
Enter fullscreen mode Exit fullscreen mode

Shared credentials eliminate accountability entirely.

Anti-Pattern 3: Long-Lived Cloud Access Keys

Example

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
Enter fullscreen mode Exit fullscreen mode

Stored inside:

  • Jenkins
  • GitHub Actions
  • Kubernetes Secrets
  • Terraform variables

Static credentials eventually leak.

The question is timing, not probability.

Anti-Pattern 4: Kubernetes Secrets Misunderstood as Encryption

Base64 encoding is not encryption.

This surprises people alarmingly often.

Example

echo "cGFzc3dvcmQ=" | base64 -d
Enter fullscreen mode Exit fullscreen mode

Outputs

password
Enter fullscreen mode Exit fullscreen mode

Kubernetes Secrets require additional controls:

  • Encryption at rest
  • RBAC
  • Admission policies
  • Audit logging

Otherwise they become plaintext credential storage with better branding.

Understanding the Modern Secrets Management Stack

Modern secrets management generally falls into four categories

Each solves different problems.

IRSA / Workload Identity: Cloud-Native Secretless Authentication

This is the most important architectural shift in modern cloud security

Stop distributing credentials.
Start distributing identity.
Enter fullscreen mode Exit fullscreen mode

Instead of giving workloads access keys

Pod → authenticated identity → temporary credentials
Enter fullscreen mode Exit fullscreen mode

No static secrets required.

AWS IRSA (IAM Roles for Service Accounts)

Pods authenticate using Kubernetes service accounts mapped to IAM roles.

Terraform IRSA Role

resource "aws_iam_role" "payment_service" {
  name = "payment-service-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"

    Statement = [{
      Effect = "Allow"

      Principal = {
        Federated = aws_iam_openid_connect_provider.eks.arn
      }

      Action = "sts:AssumeRoleWithWebIdentity"

      Condition = {
        StringEquals = {
          "${replace(
            aws_eks_cluster.main.identity[0].oidc[0].issuer,
            "https://",
            ""
          )}:sub" =
          "system:serviceaccount:payments:payment-service"
        }
      }
    }]
  })
}
Enter fullscreen mode Exit fullscreen mode

Kubernetes Service Account

apiVersion: v1
kind: ServiceAccount

metadata:
  name: payment-service
  namespace: payments

  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::ACCOUNT:role/payment-service-role
Enter fullscreen mode Exit fullscreen mode

Pods automatically receive temporary credentials.

No secrets required.

Why IRSA Is Excellent

Advantages:

  • No static AWS keys
  • Automatic credential rotation
  • IAM-native permissions
  • Short-lived credentials
  • Excellent auditability

This should be the default model for AWS-native workloads.

GCP Workload Identity Equivalent

GCP uses

Kubernetes Service Account
↔
Google Service Account
Enter fullscreen mode Exit fullscreen mode

Equivalent concept. Different implementation.

Azure Workload Identity

Azure now supports federated workload identity similarly.

The industry is converging on identity federation rather than credential distribution.

This is good.

When IRSA / Workload Identity Is NOT Enough

Cloud-native identity works beautifully for cloud APIs.

It becomes weaker when dealing with:

  • Databases
  • Third-party APIs
  • Cross-cloud systems
  • Legacy applications
  • Dynamic credential issuance
  • Multi-cluster secret orchestration

This is where Vault becomes valuable.

HashiCorp Vault: When You Need More Than Cloud-Native

Vault solves problems identity federation alone cannot.

Especially dynamic secrets.

The Core Vault Capability

Vault does not merely store secrets.

It generates them dynamically.

Example

Application requests PostgreSQL credentials
↓
Vault creates short-lived DB user
↓
Credentials expire automatically
Enter fullscreen mode Exit fullscreen mode

Massive security improvement.

Vault Kubernetes Authentication

Example

vault auth enable kubernetes
Enter fullscreen mode Exit fullscreen mode

Vault Role Example

vault write auth/kubernetes/role/payment-api \
  bound_service_account_names=payment-service \
  bound_service_account_namespaces=payments \
  policies=payment-read \
  ttl=1h
Enter fullscreen mode Exit fullscreen mode

Pods authenticate automatically via Kubernetes identity.

Dynamic Database Credentials

Example

vault read database/creds/payment-role
Enter fullscreen mode Exit fullscreen mode

Returns

{
  "username": "v-token-abc123",
  "password": "generated-secret",
  "lease_duration": 3600
}
Enter fullscreen mode Exit fullscreen mode

Credentials expire automatically after one hour.

When Vault Is the Right Choice

Use Vault when you need

Requirement Vault
Dynamic secrets Excellent
Multi-cloud support Excellent
Fine-grained audit logs Excellent
PKI management Excellent
Database credential rotation Excellent
Secret leasing Excellent

Vault Tradeoffs

Vault is operationally heavier.

You now manage:

  • HA clustering
  • Storage backend
  • Unseal process
  • Disaster recovery
  • Performance replication

Vault is powerful because it solves hard problems.

Hard problems come with operational complexity.

External Secrets Operator: The Kubernetes-Native Abstraction Layer

External Secrets Operator (ESO) is one of the cleanest Kubernetes-native abstractions available today.

Instead of storing secrets directly in Kubernetes

Kubernetes Secret
← synced from →
Vault / AWS Secrets Manager / GCP Secret Manager
Enter fullscreen mode Exit fullscreen mode

Installing ESO

helm install external-secrets external-secrets/external-secrets
Enter fullscreen mode Exit fullscreen mode

AWS Secrets Manager Example 

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret

metadata:
  name: payment-api-secret

spec:
  refreshInterval: 1h

  secretStoreRef:
    name: aws-secret-store
    kind: SecretStore

  target:
    name: payment-api-secret

  data:
  - secretKey: api-key
    remoteRef:
      key: prod/payment-api
      property: api_key
Enter fullscreen mode Exit fullscreen mode

Why ESO Is Excellent

Advantages:

  • Kubernetes-native
  • GitOps-friendly
  • Central secret backend
  • Automatic refresh
  • Cleaner operational model

ESO is often the best abstraction for Kubernetes workloads.

When ESO Is NOT Enough

ESO synchronises secrets.

It does not generate dynamic credentials.

If you need:

  • Dynamic DB users
  • Certificate issuance
  • Secret leasing
  • PKI workflows

You still need Vault or equivalent systems.

Sealed Secrets: Simple Offline Encryption for GitOps

Sealed Secrets solve a specific problem elegantly

How do you store encrypted secrets safely in Git?
Enter fullscreen mode Exit fullscreen mode

Sealed Secret Workflow

Developer creates

kubectl create secret generic app-secret
Enter fullscreen mode Exit fullscreen mode

Encrypts

kubeseal --format yaml
Enter fullscreen mode Exit fullscreen mode

Result

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
Enter fullscreen mode Exit fullscreen mode

Only the cluster controller can decrypt it.

Why Teams Love Sealed Secrets

Benefits:

  • Simple
  • GitOps-compatible
  • Easy onboarding
  • No external dependency

Where Sealed Secrets Fall Short

Limitations:

  • Static secrets only
  • No automatic rotation
  • Kubernetes-scoped
  • No dynamic credential issuance

Excellent for smaller GitOps environments.

Less ideal for enterprise-scale secret orchestration.

Secrets Rotation: The Missing Piece Most Implementations Skip

This is the most neglected part of secrets management.

Teams store secrets securely but never rotate them.

Which defeats half the purpose.

Rotation Targets

Rotate regularly:

Secret Type Rotation Frequency
API keys 30–90 days
DB credentials Dynamic preferred
TLS certificates 30–90 days
CI tokens 30 days

Vault Dynamic Rotation

Best model

Generate → use → expire automatically
Enter fullscreen mode Exit fullscreen mode

No manual rotation required.

AWS Secrets Manager Rotation

Example Lambda rotation

RotationRules:
  AutomaticallyAfterDays: 30
Enter fullscreen mode Exit fullscreen mode

Common Rotation Failure Mode

Applications caching credentials indefinitely.

Result

Secret rotated
↓
Application breaks
Enter fullscreen mode Exit fullscreen mode

Applications must reload credentials gracefully.

Audit Logging: Knowing Who Accessed What and When

Secrets access without auditing is operational blindness.

Vault Audit Logging

Enable

vault audit enable file file_path=/var/log/vault_audit.log
Enter fullscreen mode Exit fullscreen mode

Every secret request becomes traceable.

AWS CloudTrail

IRSA requests appear in CloudTrail automatically.

This is one reason identity federation is so operationally attractive.

Critical Audit Questions

You should always answer:

  • Who accessed this secret?
  • When?
  • From which workload?
  • Was it expected?
  • Was it anomalous?

Without auditability, incident response becomes guesswork.

Migration Playbook: Moving from Hard-Coded to Vault in 4 Weeks

Most organisations cannot migrate instantly.

They need staged evolution.

Week 1: Discovery

Identify:

  • .env files
  • Hardcoded credentials
  • CI secrets
  • Kubernetes Secrets
  • Shared accounts

Week 2: Centralisation

Move secrets into:

  • Vault
  • AWS Secrets Manager
  • GCP Secret Manager

Without changing applications yet.

Week 3: Kubernetes Integration

Deploy:

  • ESO
  • Vault Agent Injector
  • IRSA

Start consuming secrets dynamically.

Week 4: Rotation and Cleanup

Rotate:

  • Old credentials
  • Shared passwords
  • Long-lived tokens

Then delete legacy storage completely.

Not “later.”

Immediately.

Multi-Cloud Secrets: Managing Credentials Across AWS, Azure, and GCP

Multi-cloud secrets management becomes operationally difficult quickly.

Recommended Strategy

Use Case Recommended Tool
AWS-only IRSA + Secrets Manager
GCP-only Workload Identity + Secret Manager
Azure-only Managed Identity + Key Vault
Multi-cloud Vault

Vault becomes particularly valuable when standardising identity across clouds.

Recommended Enterprise Architecture

Kubernetes Workload
        ↓
IRSA / Workload Identity
        ↓
Vault / Cloud Secret Manager
        ↓
External Secrets Operator
        ↓
Application Runtime
Enter fullscreen mode Exit fullscreen mode

Layered abstractions create operational flexibility.

Common Secrets Management Mistakes

1. Treating Kubernetes Secrets as Secure by Default

They are not.

2. Never Rotating Credentials

Static secrets become permanent liabilities.

3. Using Shared Accounts

Breaks attribution entirely.

4. Giving Vault Excessive Permissions

Vault should broker secrets.

Not become root over everything.

5. Ignoring Audit Logs

Visibility matters as much as encryption.

Modern secrets management is no longer about hiding passwords.

It is about distributing trust safely.

The strongest DevOps environments increasingly follow several principles:

Identity over credentials
Temporary over permanent
Dynamic over static
Automated over manual
Auditable over opaque
Enter fullscreen mode Exit fullscreen mode

IRSA and workload identity eliminate entire classes of cloud credential risk.

Vault enables dynamic, short-lived infrastructure authentication.

External Secrets Operator creates elegant Kubernetes-native integration.

Sealed Secrets simplify GitOps encryption.

Each tool has a legitimate role.

The mistake is not choosing the wrong product.

The mistake is assuming one tool solves every secrets problem equally well.

Top comments (0)