DEV Community: Matthew

Appendix: Live System Output

Matthew — Thu, 28 May 2026 18:51:02 +0000

Appendix: Live System Output — Real Pipeline in Production

All output below was captured live from the running pipeline on 2026-03-08.
These are not mock outputs — they come from actual AWS infrastructure and Kubernetes clusters.

ArgoCD — All 50 Applications Across 6 Clusters

The following is the live output of argocd app list from the hub cluster (myapp-production-use1).
Every component of the pipeline is represented — security, logging, monitoring, backups, and the application itself.

$ argocd app list --output wide

NAME                                           CLUSTER                NAMESPACE         PROJECT     STATUS     HEALTH
argocd/argo-rollouts-myapp-production-use1     myapp-production-use1  argo-rollouts     production  Synced     Healthy
argocd/argo-rollouts-myapp-production-usw2     myapp-production-usw2  argo-rollouts     production  Synced     Healthy
argocd/aws-lbc-myapp-production-use1           myapp-production-use1  kube-system       production  Synced     Healthy
argocd/eso-myapp-production-use1               myapp-production-use1  external-secrets  production  OutOfSync  Healthy   ← known false positive
argocd/eso-myapp-production-usw2               myapp-production-usw2  external-secrets  production  OutOfSync  Healthy   ← known false positive
argocd/falco-myapp-dev-use1                    myapp-dev-use1         falco             production  Synced     Healthy
argocd/falco-myapp-dev-usw2                    myapp-dev-usw2         falco             production  Synced     Healthy
argocd/falco-myapp-production-use1             myapp-production-use1  falco             production  Synced     Healthy
argocd/falco-myapp-production-usw2             myapp-production-usw2  falco             production  Synced     Healthy
argocd/falco-myapp-staging-use1                myapp-staging-use1     falco             production  Synced     Healthy
argocd/falco-myapp-staging-usw2                myapp-staging-usw2     falco             production  Synced     Healthy
argocd/fluent-bit-myapp-dev-use1               myapp-dev-use1         logging           production  Synced     Healthy
argocd/fluent-bit-myapp-dev-usw2               myapp-dev-usw2         logging           production  Synced     Healthy
argocd/fluent-bit-myapp-production-use1        myapp-production-use1  logging           production  Synced     Healthy
argocd/fluent-bit-myapp-production-usw2        myapp-production-usw2  logging           production  Synced     Healthy
argocd/fluent-bit-myapp-staging-use1           myapp-staging-use1     logging           production  Synced     Healthy
argocd/fluent-bit-myapp-staging-usw2           myapp-staging-usw2     logging           production  Synced     Healthy
argocd/karpenter-myapp-production-use1         myapp-production-use1  karpenter         production  Synced     Healthy
argocd/karpenter-myapp-production-usw2         myapp-production-usw2  karpenter         production  Synced     Healthy
argocd/kyverno-myapp-dev-use1                  myapp-dev-use1         kyverno           production  Synced     Healthy
argocd/kyverno-myapp-dev-usw2                  myapp-dev-usw2         kyverno           production  Synced     Healthy
argocd/kyverno-myapp-production-use1           myapp-production-use1  kyverno           production  Synced     Healthy
argocd/kyverno-myapp-production-usw2           myapp-production-usw2  kyverno           production  Synced     Healthy
argocd/kyverno-myapp-staging-use1              myapp-staging-use1     kyverno           production  Synced     Healthy
argocd/kyverno-myapp-staging-usw2              myapp-staging-usw2     kyverno           production  Synced     Healthy
argocd/kyverno-policies-myapp-dev-use1         myapp-dev-use1         kyverno           production  Synced     Healthy
argocd/kyverno-policies-myapp-dev-usw2         myapp-dev-usw2         kyverno           production  Synced     Healthy
argocd/kyverno-policies-myapp-production-use1  myapp-production-use1  kyverno           production  Synced     Healthy
argocd/kyverno-policies-myapp-production-usw2  myapp-production-usw2  kyverno           production  Synced     Healthy
argocd/kyverno-policies-myapp-staging-use1     myapp-staging-use1     kyverno           production  Synced     Healthy
argocd/kyverno-policies-myapp-staging-usw2     myapp-staging-usw2     kyverno           production  Synced     Healthy
argocd/myapp-dev-myapp-dev-use1                myapp-dev-use1         dev               dev         OutOfSync  Healthy   ← ESO drift (expected)
argocd/myapp-dev-myapp-dev-usw2                myapp-dev-usw2         dev               dev         OutOfSync  Healthy   ← ESO drift (expected)
argocd/myapp-production-myapp-production-use1  myapp-production-use1  production        production  OutOfSync  Healthy   ← ESO drift (expected)
argocd/myapp-production-myapp-production-usw2  myapp-production-usw2  production        production  OutOfSync  Healthy   ← ESO drift (expected)
argocd/myapp-staging-myapp-staging-use1        myapp-staging-use1     staging           staging     Synced     Healthy
argocd/myapp-staging-myapp-staging-usw2        myapp-staging-usw2     staging           staging     Synced     Healthy
argocd/prometheus-myapp-production-use1        myapp-production-use1  monitoring        production  OutOfSync  Degraded  ← webhook job timeout (expected)
argocd/prometheus-myapp-production-usw2        myapp-production-usw2  monitoring        production  OutOfSync  Healthy   ← prometheus webhook drift (expected)
argocd/prometheus-myapp-staging-use1           myapp-staging-use1     monitoring        production  OutOfSync  Healthy
argocd/prometheus-myapp-staging-usw2           myapp-staging-usw2     monitoring        production  OutOfSync  Healthy
argocd/velero-myapp-dev-use1                   myapp-dev-use1         velero            production  Synced     Healthy
argocd/velero-myapp-dev-usw2                   myapp-dev-usw2         velero            production  Synced     Healthy
argocd/velero-myapp-production-use1            myapp-production-use1  velero            production  Synced     Healthy
argocd/velero-myapp-production-usw2            myapp-production-usw2  velero            production  Synced     Healthy
argocd/velero-myapp-staging-use1               myapp-staging-use1     velero            production  Synced     Healthy
argocd/velero-myapp-staging-usw2               myapp-staging-usw2     velero            production  Synced     Healthy

ArgoCD — All 6 Clusters Registered and Reachable

$ argocd cluster list

SERVER                                                                    NAME                   VERSION  STATUS
https://3C0575BCE3279BAFF3BB2D5B8444226A.gr7.us-west-2.eks.amazonaws.com  myapp-dev-usw2         1.29+    Successful
https://5079196FCF4ED5112E09CA85D7B8650F.gr7.us-west-2.eks.amazonaws.com  myapp-staging-usw2     1.29+    Successful
https://EA3C5197A0C39EA32557D04B8A2240EA.gr7.us-west-2.eks.amazonaws.com  myapp-production-usw2  1.29+    Successful
https://654498BA82E54D67E79FE325057C464B.gr7.us-east-1.eks.amazonaws.com  myapp-dev-use1         1.29+    Successful
https://6C4AB3A81EFDB980A8356D40C1590263.gr7.us-east-1.eks.amazonaws.com  myapp-staging-use1     1.29+    Successful
https://kubernetes.default.svc                                            myapp-production-use1  1.29+    Successful

All 6 clusters show Successful — the ArgoCD hub on myapp-production-use1 can communicate with all spoke clusters via VPC peering (private endpoints) and public endpoints (dev).

EKS Nodes — Live Cluster Status

Dev Clusters (public endpoints, Kubernetes 1.29)

$ kubectl --context dev-use1 get nodes

ip-10-0-15-182.ec2.internal    Ready   v1.29.15-eks-ecaa3a6
ip-10-0-22-241.ec2.internal    Ready   v1.29.15-eks-ecaa3a6
ip-10-0-27-28.ec2.internal     Ready   v1.29.15-eks-ecaa3a6

$ kubectl --context dev-usw2 get nodes

ip-10-1-28-16.us-west-2.compute.internal    Ready   v1.29.15-eks-ecaa3a6
ip-10-1-3-187.us-west-2.compute.internal    Ready   v1.29.15-eks-ecaa3a6
ip-10-1-7-181.us-west-2.compute.internal    Ready   v1.29.15-eks-ecaa3a6

Production Cluster — myapp-production-use1 (private endpoint)

$ kubectl --context prod-use1 get nodes -o wide

NAME                           STATUS  VERSION               INTERNAL-IP     INSTANCE-TYPE
ip-10-20-2-113.ec2.internal    Ready   v1.29.15-eks-ecaa3a6  10.20.2.113     t3.medium
ip-10-20-24-200.ec2.internal   Ready   v1.29.15-eks-ecaa3a6  10.20.24.200    t3.medium
ip-10-20-26-204.ec2.internal   Ready   v1.29.15-eks-ecaa3a6  10.20.26.204    t3.medium
ip-10-20-7-170.ec2.internal    Ready   v1.29.15-eks-ecaa3a6  10.20.7.170     t3.medium

All nodes: Ready, Kubernetes v1.29.15-eks-ecaa3a6, VPC private IPs in the 10.20.0.0/16 CIDR (production-use1).

Application — Running Pods in Production

$ kubectl --context prod-use1 get pods -n production

NAME                                                              READY   STATUS    RESTARTS   AGE
myapp-production-myapp-production-use1-myapp-9985ccc88-f7rxj     1/1     Running   1          11d
myapp-production-myapp-production-use1-myapp-9985ccc88-l2sh2     1/1     Running   0          11d
myapp-production-myapp-production-use1-myapp-9985ccc88-vtwsm     1/1     Running   0          11d

3 replicas running (matches minReplicas: 3 in values-production.yaml).
The pod naming convention shows the ArgoCD release name (myapp-production-myapp-production-use1) and the Helm chart (myapp).

Argo Rollouts — Canary Controller Active

$ kubectl --context prod-use1 get rollouts -n production

NAME                                           DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
myapp-production-myapp-production-use1-myapp   3         3         3            3           11d

$ kubectl --context prod-use1 get hpa -n production

NAME                                           REFERENCE                                              TARGETS     MINPODS   MAXPODS   REPLICAS
myapp-production-myapp-production-use1-myapp   Rollout/myapp-production-myapp-production-use1-myapp   <unk>/60%   3         10        3

The HPA targets the Rollout resource (not a Deployment) — this is the correct configuration for Argo Rollouts. <unknown>/60% means the metrics-server hasn't collected enough data yet; the HPA is still functional and will scale when CPU crosses 60%.

Kyverno — Admission Policies Enforced

$ kubectl --context prod-use1 get clusterpolicies

NAME                       ADMISSION   BACKGROUND   VALIDATE ACTION   READY   AGE
disallow-latest-tag        true        true         Enforce           True    34h
require-non-root           true        true         Enforce           True    34h
require-readonly-filesystem true       true         Enforce           True    34h
require-resource-limits    true        true         Enforce           True    34h
restrict-image-registry    true        true         Enforce           True    34h

5 cluster-wide policies active, all in Enforce mode (not Audit) — violations are blocked, not just logged. Ready: True means each policy's webhook is registered and functioning.

Falco — Runtime Security DaemonSet Running

$ kubectl --context prod-use1 get pods -n falco

NAME                                                         READY   STATUS    RESTARTS
falco-myapp-production-use1-5d6dr                            1/1     Running   29
falco-myapp-production-use1-jql5r                            1/1     Running   0
falco-myapp-production-use1-pbhdr                            1/1     Running   0
falco-myapp-production-use1-sjlkh                            1/1     Running   0
falco-myapp-production-use1-falcosidekick-7c56844569-h4vvk   1/1     Running   1
falco-myapp-production-use1-falcosidekick-7c56844569-qcnjh   1/1     Running   2

Falco DaemonSet: one pod per node (4 nodes = 4 Falco pods). All Running. The 29 restarts on one pod is from the initial eBPF driver loading — normal behaviour on kernel version changes.

External Secrets — Synced from AWS Secrets Manager

$ kubectl --context prod-use1 get externalsecret -n production

NAME                                                   STORE                                    REFRESH   STATUS         READY
myapp-production-myapp-production-use1-myapp-secrets   myapp-production-myapp-production-use1-  1h        SecretSynced   True

SecretSynced: True — ESO has successfully fetched production/myapp/db-password from AWS Secrets Manager and created the Kubernetes Secret. The IRSA authentication chain (OIDC token → STS → Secrets Manager) is working correctly.

Velero — Scheduled Backups Running

$ kubectl --context prod-use1 get schedules -n velero

NAME                                        STATUS    SCHEDULE    LASTBACKUP   AGE
velero-myapp-production-use1-daily-backup   Enabled   0 2 * * *   16h          34h

Daily backup schedule active. Last backup ran 16 hours ago (2 AM UTC). Backups stored in S3.

ECR — Signed Images in Registry

$ aws ecr describe-images --repository-name myapp --region us-east-1 --profile myapp-mgmt

IMAGE TAG                                                          PUSHED AT              SIZE
sha-f72053d0d5fb765bc08d8b5a8374119655997784                      2026-02-22T17:37:27Z   48.5 MB
sha256-f01790daf982...956be0c.sig                                  2026-02-22T17:40:48Z   499 B   ← Cosign signature

Two OCI artifacts per image push:

The application image tagged sha-<full-git-sha> (48.5 MB)
The Cosign signature artifact tagged sha256-<digest>.sig (499 bytes) — this is the cryptographic attestation stored in ECR, verified by Kyverno at admission time

AWS GuardDuty — Threat Detection Active

$ aws guardduty list-detectors + get-detector (production account)

Status: ENABLED
DataSources:
  - S3 Logs:          ENABLED
  - Kubernetes Audit: ENABLED
  - Malware Protection (EBS): ENABLED

$ aws guardduty list-detectors (staging account)

Status: ENABLED

GuardDuty enabled in both production and staging accounts with EKS audit log monitoring. Any kubectl exec into production pods, unusual API call patterns, or crypto mining activity will generate findings.

AWS CloudWatch — Log Groups from Fluent Bit

$ aws logs describe-log-groups --log-group-name-prefix "/eks/" --region us-east-1 (production account)

/eks/myapp-production-use1
/eks/myapp-production-use1/argocd
/eks/myapp-production-use1/external-secrets
/eks/myapp-production-use1/falco
/eks/myapp-production-use1/karpenter
/eks/myapp-production-use1/kyverno
/eks/myapp-production-use1/logging
/eks/myapp-production-use1/monitoring
/eks/myapp-production-use1/production
/eks/myapp-production-use1/velero
/eks/myapp-production-use1/argo-rollouts

One CloudWatch Log Group per Kubernetes namespace, all prefixed /eks/myapp-production-use1/. Fluent Bit DaemonSet ships logs from every container in every namespace to the corresponding log group.

Live Health Check — Public Endpoint

$ curl -s https://www.matthewoladipupo.dev/health | python3 -m json.tool

{
    "status": "healthy",
    "region": "us-east-1"
}

Production application serving HTTPS traffic. Route53 latency routing directs users to the nearest healthy region. AWS WAF WebACL inspects every request before it reaches the ALB.

Grafana — Public Dashboard

URL:      https://grafana.matthewoladipupo.dev
Username: admin

Grafana deployed on myapp-production-use1 with:

50 GiB Prometheus TSDB (15-day retention)
10 GiB Grafana persistent volume
ACM wildcard TLS certificate (*.matthewoladipupo.dev)
ALB internet-facing ingress provisioned by AWS Load Balancer Controller

Add your Grafana dashboard screenshots here

Summary Table — Component Health at Time of Writing

Component	Clusters	Status
ArgoCD hub	prod-use1	✅ Running, all 6 clusters registered
Kyverno policies	All 6	✅ 5 ClusterPolicies, Enforce mode, Ready
Falco DaemonSet	All 6	✅ One pod per node, all Running
Fluent Bit DaemonSet	All 6	✅ Synced/Healthy, CloudWatch log groups created
External Secrets	All 6	✅ SecretSynced: True
Velero schedules	All 6	✅ Daily backup at 02:00 UTC, last run 16h ago
Karpenter	prod-use1, prod-usw2	✅ Synced/Healthy
Argo Rollouts	prod-use1, prod-usw2	✅ Synced/Healthy
kube-prometheus-stack	staging+prod (4)	✅ Running (OutOfSync is known false positive)
GuardDuty	prod + staging	✅ ENABLED with EKS audit logs
ECR images	mgmt account	✅ Immutable tags, Cosign signatures present
DNS + TLS	Route53 + ACM	✅ `www.matthewoladipupo.dev` → healthy

All data captured live on 2026-03-08 from the running AWS infrastructure.

Production DevSecOps Pipeline — The Complete Day-2 Operations Runbook

Matthew — Thu, 28 May 2026 18:50:54 +0000

DevSecOps Pipeline — Completion Runbook

All code is written and pushed to GitHub. This runbook covers the remaining
operational steps: Terraform applies, GitOps ARN updates, and ArgoCD deployment.

Prerequisites

Install these tools if not already present:

# AWS CLI v2
winget install Amazon.AWSCLI

# Terraform 1.6+
winget install HashiCorp.Terraform

# Terragrunt
# Download from https://github.com/gruntwork-io/terragrunt/releases
# Place in C:\Windows\System32\ or add to PATH

# kubectl
winget install Kubernetes.kubectl

# ArgoCD CLI
winget install argoproj.argocd

AWS Profile Setup

The root terragrunt.hcl uses profiles named myapp-{env}-{region_alias}.
Configure them in ~/.aws/config:

[profile myapp-production-use1]
region = us-east-1
role_arn = arn:aws:iam::591120834781:role/AdministratorAccess
source_profile = default

[profile myapp-production-usw2]
region = us-west-2
role_arn = arn:aws:iam::591120834781:role/AdministratorAccess
source_profile = default

[profile myapp-staging-use1]
region = us-east-1
role_arn = arn:aws:iam::690687753178:role/AdministratorAccess
source_profile = default

[profile myapp-staging-usw2]
region = us-west-2
role_arn = arn:aws:iam::690687753178:role/AdministratorAccess
source_profile = default

[profile myapp-dev-use1]
region = us-east-1
role_arn = arn:aws:iam::557702566877:role/AdministratorAccess
source_profile = default

[profile myapp-dev-usw2]
region = us-west-2
role_arn = arn:aws:iam::557702566877:role/AdministratorAccess
source_profile = default

PHASE 1 — Terraform Applies

Work from the myapp-infra/ directory. Run in the order shown — capture outputs
for updating GitOps files in Phase 2.

1.1 WAF (production + staging)

# Production us-east-1
terragrunt apply --terragrunt-working-dir live/production/us-east-1/waf
# Output → webacl_arn  (copy this value)

# Production us-west-2
terragrunt apply --terragrunt-working-dir live/production/us-west-2/waf
# Output → webacl_arn  (copy this value)

# Staging (no GitOps ARN needed, but good to have)
terragrunt apply --terragrunt-working-dir live/staging/us-east-1/waf
terragrunt apply --terragrunt-working-dir live/staging/us-west-2/waf

1.2 GuardDuty (all regions — no outputs needed)

terragrunt apply --terragrunt-working-dir live/production/us-east-1/guardduty
terragrunt apply --terragrunt-working-dir live/production/us-west-2/guardduty
terragrunt apply --terragrunt-working-dir live/staging/us-east-1/guardduty
terragrunt apply --terragrunt-working-dir live/staging/us-west-2/guardduty

GuardDuty has no GitOps dependency. Alerts appear in the AWS console and
optionally in CloudWatch.

1.3 ESO IRSA for Staging

# Staging us-east-1
terragrunt apply --terragrunt-working-dir live/staging/us-east-1/eso-irsa
# Output → role_arn  (copy → used in environments/staging/applicationset.yaml)

# Staging us-west-2
terragrunt apply --terragrunt-working-dir live/staging/us-west-2/eso-irsa
# Output → role_arn  (copy → used in environments/staging/applicationset.yaml)

NOTE: The ESO operator ApplicationSet (infrastructure/eso/applicationset.yaml)
already includes staging clusters. Once ESO is running on staging and the
ExternalSecret IRSA role is set, ExternalSecrets will sync automatically.

1.4 Fluent Bit IRSA (all 6 clusters)

terragrunt apply --terragrunt-working-dir live/production/us-east-1/fluent-bit-irsa
# → role_arn for myapp-production-use1

terragrunt apply --terragrunt-working-dir live/production/us-west-2/fluent-bit-irsa
# → role_arn for myapp-production-usw2

terragrunt apply --terragrunt-working-dir live/staging/us-east-1/fluent-bit-irsa
# → role_arn for myapp-staging-use1

terragrunt apply --terragrunt-working-dir live/staging/us-west-2/fluent-bit-irsa
# → role_arn for myapp-staging-usw2

terragrunt apply --terragrunt-working-dir live/dev/us-east-1/fluent-bit-irsa
# → role_arn for myapp-dev-use1

terragrunt apply --terragrunt-working-dir live/dev/us-west-2/fluent-bit-irsa
# → role_arn for myapp-dev-usw2

1.5 Karpenter (production only)

terragrunt apply --terragrunt-working-dir live/production/us-east-1/karpenter
# Outputs:
#   controller_role_arn   → for karpenter applicationset.yaml
#   node_role_arn         → for verification (name = myapp-production-use1-karpenter-node)
#   node_instance_profile → for verification
#   interruption_queue_name → should be "myapp-production-use1-karpenter"

terragrunt apply --terragrunt-working-dir live/production/us-west-2/karpenter
# Outputs same structure for usw2

The nodeRoleName values in karpenter/nodepool-applicationset.yaml are
pre-set to myapp-production-use1-karpenter-node and myapp-production-usw2-karpenter-node.
These match what Terraform creates so no update needed there.

1.6 Velero (all 6 clusters)

# Production
terragrunt apply --terragrunt-working-dir live/production/us-east-1/velero
# → role_arn for myapp-production-use1

terragrunt apply --terragrunt-working-dir live/production/us-west-2/velero
# → role_arn for myapp-production-usw2

# Staging
terragrunt apply --terragrunt-working-dir live/staging/us-east-1/velero
# → role_arn for myapp-staging-use1

terragrunt apply --terragrunt-working-dir live/staging/us-west-2/velero
# → role_arn for myapp-staging-usw2

# Dev
terragrunt apply --terragrunt-working-dir live/dev/us-east-1/velero
# → role_arn for myapp-dev-use1

terragrunt apply --terragrunt-working-dir live/dev/us-west-2/velero
# → role_arn for myapp-dev-usw2

PHASE 2 — Update GitOps ARNs

After collecting all outputs from Phase 1, update the GitOps repo
(myapp-gitops/) and push.

2.1 Production WAF ARNs

Edit environments/production/applicationset.yaml — replace "PENDING" with
real WAF ACL ARNs from Step 1.1:

elements:
  - cluster: myapp-production-use1
    ...
    wafAclArn: "arn:aws:wafv2:us-east-1:591120834781:regional/webacl/myapp-production-use1-waf/XXXXXXXX"
  - cluster: myapp-production-usw2
    ...
    wafAclArn: "arn:aws:wafv2:us-west-2:591120834781:regional/webacl/myapp-production-usw2-waf/XXXXXXXX"

2.2 Staging ESO IRSA ARNs

Edit environments/staging/applicationset.yaml — replace "PENDING" with
role ARNs from Step 1.3:

elements:
  - cluster: myapp-staging-use1
    ...
    irsaRoleArn: "arn:aws:iam::690687753178:role/myapp-staging-use1-eso"
  - cluster: myapp-staging-usw2
    ...
    irsaRoleArn: "arn:aws:iam::690687753178:role/myapp-staging-usw2-eso"

2.3 Fluent Bit IRSA ARNs

Edit infrastructure/logging/applicationset.yaml — replace all 6 "PENDING" values:

elements:
  - cluster: myapp-production-use1  roleArn: "arn:aws:iam::591120834781:role/myapp-production-use1-fluent-bit"
  - cluster: myapp-production-usw2  roleArn: "arn:aws:iam::591120834781:role/myapp-production-usw2-fluent-bit"
  - cluster: myapp-staging-use1     roleArn: "arn:aws:iam::690687753178:role/myapp-staging-use1-fluent-bit"
  - cluster: myapp-staging-usw2     roleArn: "arn:aws:iam::690687753178:role/myapp-staging-usw2-fluent-bit"
  - cluster: myapp-dev-use1         roleArn: "arn:aws:iam::557702566877:role/myapp-dev-use1-fluent-bit"
  - cluster: myapp-dev-usw2         roleArn: "arn:aws:iam::557702566877:role/myapp-dev-usw2-fluent-bit"

TIP: Role names follow the pattern {cluster_name}-fluent-bit. Verify with
terragrunt output role_arn in each fluent-bit-irsa directory.

2.4 Karpenter Controller Role ARNs

Edit infrastructure/karpenter/applicationset.yaml — replace 2 "PENDING" values:

elements:
  - cluster: myapp-production-use1  controllerRole: "arn:aws:iam::591120834781:role/myapp-production-use1-karpenter"
  - cluster: myapp-production-usw2  controllerRole: "arn:aws:iam::591120834781:role/myapp-production-usw2-karpenter"

2.5 Velero Role ARNs

Edit infrastructure/velero/applicationset.yaml — replace all 6 "PENDING" values:

elements:
  - cluster: myapp-production-use1  roleArn: "arn:aws:iam::591120834781:role/myapp-production-use1-velero"
  - cluster: myapp-production-usw2  roleArn: "arn:aws:iam::591120834781:role/myapp-production-usw2-velero"
  - cluster: myapp-staging-use1     roleArn: "arn:aws:iam::690687753178:role/myapp-staging-use1-velero"
  - cluster: myapp-staging-usw2     roleArn: "arn:aws:iam::690687753178:role/myapp-staging-usw2-velero"
  - cluster: myapp-dev-use1         roleArn: "arn:aws:iam::557702566877:role/myapp-dev-use1-velero"
  - cluster: myapp-dev-usw2         roleArn: "arn:aws:iam::557702566877:role/myapp-dev-usw2-velero"

2.6 Slack Webhooks + Grafana Password

Edit infrastructure/monitoring/prometheus-values.yaml:

Replace both https://hooks.slack.com/services/CHANGE_ME with real Slack incoming webhook URLs
Replace change-me-grafana with a real password (or use an ExternalSecret)

2.7 Commit + Push GitOps changes

cd myapp-gitops
git add environments/ infrastructure/
git commit -m "chore: fill in real ARNs from terraform outputs"
git push origin HEAD:main

2.8 Create staging Secrets Manager secret

Run this once to seed the staging ExternalSecret:

AWS_PROFILE=myapp-staging-use1 aws secretsmanager create-secret \
  --name staging/myapp/db-password \
  --secret-string '{"password":"change-me-staging"}' \
  --region us-east-1

AWS_PROFILE=myapp-staging-usw2 aws secretsmanager create-secret \
  --name staging/myapp/db-password \
  --secret-string '{"password":"change-me-staging"}' \
  --region us-west-2

PHASE 3 — ArgoCD Setup

3.1 Bootstrap ArgoCD (App of Apps)

The argocd/ directory in myapp-gitops now contains the AppProject and a
bootstrap Application. Apply the bootstrap once — after that ArgoCD manages
itself and will also pick up the AppProject automatically.

# Point kubectl at production cluster (where ArgoCD runs)
kubectl config use-context myapp-production-use1

cd myapp-gitops

# One-time bootstrap — creates the self-managing Application
kubectl apply -f argocd/bootstrap.yaml -n argocd

# ArgoCD will now sync argocd/project-production.yaml automatically.
# Watch until it's healthy:
argocd app wait bootstrap --health

The argocd/project-production.yaml AppProject already includes every
namespace and source repo needed by all components. No kubectl patch needed.

3.2 Apply new ApplicationSets to ArgoCD

After the bootstrap Application syncs (it only manages the argocd/ directory),
apply the infrastructure ApplicationSets manually once:

cd myapp-gitops

kubectl apply -f infrastructure/eso/applicationset.yaml
kubectl apply -f infrastructure/monitoring/applicationset.yaml
kubectl apply -f infrastructure/monitoring/alert-rules-applicationset.yaml
kubectl apply -f infrastructure/logging/applicationset.yaml
kubectl apply -f infrastructure/karpenter/applicationset.yaml
kubectl apply -f infrastructure/karpenter/nodepool-applicationset.yaml
kubectl apply -f infrastructure/velero/applicationset.yaml
kubectl apply -f infrastructure/falco/applicationset.yaml
kubectl apply -f infrastructure/argo-rollouts/applicationset.yaml

After this, ArgoCD self-manages all ApplicationSets via the automated sync
on the generated Applications.

PHASE 4 — ArgoCD Sync Order (Production)

Sync in this exact order to respect CRD dependencies:

# Step 1: Prometheus stack (creates CRDs for PrometheusRule, ServiceMonitor, etc.)
argocd app sync prometheus-myapp-production-use1 prometheus-myapp-production-usw2
argocd app wait prometheus-myapp-production-use1 --health
argocd app wait prometheus-myapp-production-usw2 --health

# Step 2: Alert rules (needs Prometheus CRDs)
argocd app sync alert-rules-myapp-production-use1 alert-rules-myapp-production-usw2

# Step 3: Parallel infra components (no inter-dependency)
argocd app sync \
  fluent-bit-myapp-production-use1 fluent-bit-myapp-production-usw2 \
  velero-myapp-production-use1 velero-myapp-production-usw2 \
  falco-myapp-production-use1 falco-myapp-production-usw2

# Step 4: Karpenter controller (needs ECR access to pull image from public.ecr.aws)
argocd app sync karpenter-myapp-production-use1 karpenter-myapp-production-usw2
argocd app wait karpenter-myapp-production-use1 --health

# Step 5: Karpenter NodePools (needs Karpenter CRDs installed by Step 4)
argocd app sync karpenter-nodepool-myapp-production-use1 karpenter-nodepool-myapp-production-usw2

# Step 6: Argo Rollouts controller
argocd app sync argo-rollouts-myapp-production-use1 argo-rollouts-myapp-production-usw2
argocd app wait argo-rollouts-myapp-production-use1 --health

# Step 7: App (uses Rollout CR — needs argo-rollouts controller running)
argocd app sync myapp-production-myapp-production-use1 myapp-production-myapp-production-usw2

Staging sync (can run in parallel with production steps 3+)

argocd app sync \
  eso-myapp-staging-use1 eso-myapp-staging-usw2 \
  fluent-bit-myapp-staging-use1 fluent-bit-myapp-staging-usw2 \
  velero-myapp-staging-use1 velero-myapp-staging-usw2 \
  falco-myapp-staging-use1 falco-myapp-staging-usw2 \
  prometheus-myapp-staging-use1 prometheus-myapp-staging-usw2

# After staging ESO is healthy, ExternalSecrets will sync automatically
argocd app sync myapp-staging-myapp-staging-use1 myapp-staging-myapp-staging-usw2

PHASE 5 — Verification

Monitoring

kubectl get pods -n monitoring --context myapp-production-use1
kubectl get prometheusrule -n monitoring --context myapp-production-use1
kubectl get alertmanager -n monitoring --context myapp-production-use1
# Access Grafana: kubectl port-forward svc/kube-prometheus-stack-grafana 3000:80 -n monitoring

Logging

kubectl get pods -n logging --context myapp-production-use1
# Verify log groups were created:
AWS_PROFILE=myapp-production-use1 aws logs describe-log-groups \
  --log-group-name-prefix /eks/myapp-production-use1 --region us-east-1

Karpenter

kubectl get pods -n karpenter --context myapp-production-use1
kubectl get nodepool --context myapp-production-use1
kubectl get ec2nodeclass --context myapp-production-use1
# Trigger a scale test:
kubectl scale deploy/stress --replicas=50 -n default --context myapp-production-use1
kubectl get nodes -w --context myapp-production-use1

Velero

kubectl get pods -n velero --context myapp-production-use1
kubectl get schedule -n velero --context myapp-production-use1
# Trigger manual backup:
velero backup create manual-test --context myapp-production-use1
velero backup describe manual-test --context myapp-production-use1

Falco

kubectl get pods -n falco --context myapp-production-use1
# Check CloudWatch for events:
AWS_PROFILE=myapp-production-use1 aws logs describe-log-groups \
  --log-group-name-prefix /falco --region us-east-1

Argo Rollouts (canary deploy)

kubectl get rollout -n production --context myapp-production-use1
kubectl argo rollouts get rollout myapp-production-use1-myapp -n production \
  --context myapp-production-use1 --watch

ESO Staging

kubectl get externalsecret -n staging --context myapp-staging-use1
kubectl describe externalsecret myapp-production-use1-myapp-secrets -n staging \
  --context myapp-staging-use1

WAF

AWS_PROFILE=myapp-production-use1 aws wafv2 list-web-acls \
  --scope REGIONAL --region us-east-1 | grep myapp

GuardDuty

AWS_PROFILE=myapp-production-use1 aws guardduty list-detectors --region us-east-1
AWS_PROFILE=myapp-production-usw2 aws guardduty list-detectors --region us-west-2

Troubleshooting Notes

Issue	Fix
Karpenter fails to pull image	Ensure the node IAM role has ECR pull-through cache configured or use `public.ecr.aws` directly. Karpenter controller image is on `public.ecr.aws/karpenter/karpenter`.
Falco `modern_ebpf` not supported	Some EKS AMIs/kernel versions don't support eBPF. Fall back to `driver.kind: ebpf` or `driver.kind: module` in `infrastructure/falco/values.yaml`.
Velero backup fails	Ensure S3 bucket lifecycle rule and encryption config applied. Check IRSA trust policy `sub` matches `system:serviceaccount:velero:velero`.
Alert rules not picked up	The PrometheusRule must have label `release: kube-prometheus-stack` (already set in `alert-rules.yaml`). Verify with `kubectl get prometheusrule -n monitoring -o yaml`.
Rollout stuck at 20%	Check AnalysisTemplate — if `myapp_http_requests_total` metric doesn't exist yet (app not instrumented), the analysis will fail. Set `failureLimit: 3` or temporarily disable analysis by removing the `analysis` step from the canary steps.
Karpenter NodePool not scheduling	Verify subnet and SG tags: `aws ec2 describe-subnets --filters "Name=tag:karpenter.sh/discovery,Values=myapp-production-use1"`.

Day-2 Operations Runbook

For: Anyone operating this pipeline after initial setup is complete
Live system: https://api.matthewoladipupo.dev/health

Quick Reference

URLs and Credentials

Service	URL	Notes
Application	`https://api.matthewoladipupo.dev/health`	Public
ArgoCD UI	`http://a0c3c1ea43b294c4d8f5c2a7c514f6f2-1678928976.us-east-1.elb.amazonaws.com`	admin / see Secrets Manager
Grafana	`https://grafana.matthewoladipupo.dev`	admin / see Secrets Manager
AWS SSO Portal	`https://d-9a6757fb3c.awsapps.com/start`	IAM Identity Center

Cluster → Profile Map

Cluster	kubectl context	AWS Profile	Endpoint
`myapp-production-use1`	`myapp-production-use1`	`myapp-prod-use1`	private
`myapp-production-usw2`	`myapp-production-usw2`	`myapp-prod-usw2`	private
`myapp-staging-use1`	`myapp-staging-use1`	`myapp-staging-use1`	private
`myapp-staging-usw2`	`myapp-staging-usw2`	`myapp-staging-usw2`	private
`myapp-dev-use1`	`myapp-dev-use1`	`myapp-dev-use1`	public
`myapp-dev-usw2`	`myapp-dev-usw2`	`myapp-dev-usw2`	public

OPS-1: Start of Session

Run every time you open a new terminal. SSO tokens last 8 hours.

# 1. Authenticate
aws sso login --sso-session admin --no-browser
# → browser opens → click Allow → wait for "Successfully logged in"

# 2. Verify
aws sts get-caller-identity --profile myapp-prod-use1
# → should return Account: 591120834781

# 3. Quick health check
curl https://api.matthewoladipupo.dev/health
# → {"status":"healthy","region":"us-east-1"}

If you see Token has expired and refresh failed at any point, re-run step 1.

OPS-2: Accessing Private Clusters (Production + Staging)

Production and staging clusters have endpointPublicAccess: false. You must temporarily enable public access, do your work, then lock it back. Never leave production with a public endpoint.

# Enable (wait ~3 minutes after running this)
aws eks update-cluster-config \
  --name myapp-production-use1 \
  --region us-east-1 \
  --profile myapp-prod-use1 \
  --resources-vpc-config endpointPublicAccess=true,endpointPrivateAccess=true,publicAccessCidrs=0.0.0.0/0

# Confirm it's ready
aws eks describe-cluster \
  --name myapp-production-use1 \
  --region us-east-1 \
  --profile myapp-prod-use1 \
  --query 'cluster.resourcesVpcConfig.endpointPublicAccess'
# Must return: true

# --- Do your kubectl work here ---

# Lock back immediately after
aws eks update-cluster-config \
  --name myapp-production-use1 \
  --region us-east-1 \
  --profile myapp-prod-use1 \
  --resources-vpc-config endpointPublicAccess=false,endpointPrivateAccess=true

Replace myapp-production-use1 / myapp-prod-use1 / us-east-1 with the appropriate values for other private clusters.

OPS-3: Standard Deployment

Normal deployments are fully automated — zero manual steps required:

Developer pushes to main branch of MatthewDipo/myapp
GitHub Actions: test → Trivy scan → build → push ECR → Cosign sign → update gitops values
ArgoCD detects gitops change within 3 minutes → syncs
In production: Argo Rollout starts canary (20% traffic for 5 minutes → analysis → 100%)

Monitor progress:

# Watch rollout steps (enable public endpoint first)
kubectl get rollouts -n production --context myapp-production-use1 -w

# Detailed view (requires kubectl-argo-rollouts plugin)
kubectl argo rollouts get rollout myapp -n production \
  --context myapp-production-use1 --watch

OPS-4: Promote or Abort a Canary

Promote immediately (skip the 5-minute pause):

kubectl argo rollouts promote myapp -n production --context myapp-production-use1

Abort (shift all traffic back to stable version instantly):

kubectl argo rollouts abort myapp -n production --context myapp-production-use1

# After abort the rollout shows Degraded — retry to return to Healthy
kubectl argo rollouts retry rollout myapp -n production --context myapp-production-use1

OPS-5: Roll Back a Deployment

Option A — GitOps revert (preferred, keeps git history clean):

cd /path/to/myapp-gitops
git revert HEAD --no-edit
git push origin main
# ArgoCD auto-syncs the revert within 3 minutes

Option B — ArgoCD rollback to a previous revision:

argocd app history myapp-production-myapp-production-use1
argocd app rollback myapp-production-myapp-production-use1 <revision-number>

Option C — Emergency direct image update (bypasses GitOps, use only in outage):

# Get previous image tag from ECR
aws ecr describe-images \
  --repository-name myapp \
  --region us-east-1 \
  --profile myapp-prod-use1 \
  --query 'sort_by(imageDetails,&imagePushedAt)[-2].imageTags' \
  --output json

# Force update the rollout
kubectl argo rollouts set image myapp \
  myapp=206617159586.dkr.ecr.us-east-1.amazonaws.com/myapp:sha-<previous-sha> \
  -n production --context myapp-production-use1

# After incident: update values-production.yaml in gitops to match, then push

OPS-6: Rotate a Secret

The External Secrets Operator (ESO) syncs from AWS Secrets Manager on a 1-hour cycle.

Step 1 — Update value in Secrets Manager:

aws secretsmanager put-secret-value \
  --secret-id production/myapp/db-password \
  --secret-string '{"password":"new-value-here"}' \
  --region us-east-1 \
  --profile myapp-prod-use1

Step 2 — Force immediate ESO refresh:

# Enable public endpoint first (OPS-2)
kubectl annotate externalsecret myapp-db-password \
  force-sync=$(date +%s) -n production \
  --context myapp-production-use1 --overwrite

# Verify
kubectl get externalsecret myapp-db-password -n production \
  --context myapp-production-use1
# STATUS: SecretSynced

Step 3 — Restart pods to pick up new value:

kubectl argo rollouts restart myapp -n production --context myapp-production-use1

OPS-7: Incident Response

Falco Security Alert

# 1. Identify what triggered the alert
kubectl logs -n falco daemonset/falco --context myapp-production-use1 \
  | grep -E "Warning|Critical" | tail -20

# 2. Inspect the affected pod
kubectl describe pod <pod-name> -n production --context myapp-production-use1
kubectl logs <pod-name> -n production --context myapp-production-use1 --tail=100

# 3. Contain if confirmed malicious — delete pod (Rollout replaces with clean copy)
kubectl delete pod <pod-name> -n production --context myapp-production-use1

# 4. Preserve logs before deletion
kubectl logs <pod-name> -n production --context myapp-production-use1 \
  > /tmp/incident-$(date +%Y%m%d-%H%M%S).log

# 5. Check GuardDuty for correlated findings
aws guardduty list-findings \
  --detector-id $(aws guardduty list-detectors \
    --region us-east-1 --profile myapp-prod-use1 \
    --query 'DetectorIds[0]' --output text) \
  --region us-east-1 --profile myapp-prod-use1

Kyverno Blocked a Pod

# See the rejection reason
kubectl describe pod <pod-name> -n <namespace> --context myapp-production-use1
# Look for: "admission webhook" in Events

# List policy violations
kubectl get policyreport -n <namespace> --context myapp-production-use1

Common violations and fixes:

Policy	Violation	Fix
`block-privileged`	`privileged: true` in spec	Remove the privileged flag
`require-non-root`	Running as root	Add `runAsNonRoot: true`, `runAsUser: 1000`
`block-host-path`	hostPath volume	Replace with PVC
`require-resource-limits`	No CPU/memory limits	Add `resources.limits`
`verify-image-signature`	Image not Cosign-signed	Must go through CI/CD pipeline

Application Down

# Check pods
kubectl get pods -n production --context myapp-production-use1

# Check pod logs
kubectl logs -n production -l app=myapp --context myapp-production-use1 --tail=50

# Check rollout
kubectl get rollouts -n production --context myapp-production-use1

# Check HPA — is it maxed out?
kubectl get hpa -n production --context myapp-production-use1

# Check if Karpenter is provisioning nodes
kubectl get nodes --context myapp-production-use1 -w

OPS-8: Routine Health Check

# Application
curl https://api.matthewoladipupo.dev/health

# ArgoCD — show only non-Synced apps (empty = all good)
argocd app list | grep -v "Synced.*Healthy"

# Enable public endpoint, then:
kubectl get nodes --context myapp-production-use1
kubectl get pods -n production --context myapp-production-use1
kubectl get hpa -n production --context myapp-production-use1
kubectl get externalsecret -n production --context myapp-production-use1
kubectl get schedule -n velero --context myapp-production-use1

# Lock endpoint back

OPS-9: Restore from Velero Backup

# Enable public endpoint first (OPS-2), then:

# List available backups
kubectl get backups -n velero --context myapp-production-use1

# Restore a namespace
kubectl create -f - <<EOF
apiVersion: velero.io/v1
kind: Restore
metadata:
  name: restore-$(date +%Y%m%d-%H%M)
  namespace: velero
spec:
  backupName: <backup-name-from-list>
  includedNamespaces:
    - production
  restorePVs: true
EOF

# Watch progress
kubectl get restore -n velero --context myapp-production-use1 -w

OPS-10: Common Error Reference

Error	Cause	Fix
`Token has expired and refresh failed`	SSO session expired	`aws sso login --sso-session admin --no-browser`
`dial tcp 10.x.x.x:443: i/o timeout`	Private cluster endpoint	Enable public access temporarily (OPS-2)
`config profile (X) could not be found`	Wrong profile name	Use `myapp-prod-use1` not `production`
`unknown command "argo" for "kubectl"`	Plugin not installed	Install `kubectl-argo-rollouts` binary
`SecretSynced: False` on ExternalSecret	IRSA role or secret missing	Check IAM role exists, check secret path in Secrets Manager
Pod stuck in `Pending`	Karpenter provisioning	Wait 2 min; check `kubectl get nodeclaims`
ArgoCD app `OutOfSync` after ESO sync	ESO writes `status.refreshTime`	Known false positive — safe to ignore or force-sync

Part 10: Resilience — Karpenter, HPA, Argo Rollouts, and Velero

Matthew — Wed, 20 May 2026 08:00:00 +0000

Part of the series: Building a Production-Grade DevSecOps Pipeline on AWS

Introduction

Resilience engineering is about building systems that degrade gracefully, recover automatically, and deploy safely. This final part brings together four capabilities:

Karpenter — automatically provisions the right EC2 instances for pending pods, and removes them when no longer needed
HPA — scales pod replicas based on CPU/memory pressure
Argo Rollouts — deploys new versions with controlled canary traffic and automatic rollback on errors
Velero — backs up Kubernetes resources and PVC data to S3 for disaster recovery

┌─────────────────────────────────────────────────────────────────────┐
│  RESILIENCE LAYERS                                                  │
│                                                                     │
│  Traffic Spike                                                      │
│  └─► HPA: scale pods from 3 → 8                                     │
│       └─► Karpenter: provision new nodes to fit pending pods        │
│                                                                     │
│  New Deployment                                                     │
│  └─► Argo Rollouts: 20% canary → analysis → promote or rollback     │
│                                                                     │
│  Cluster Disaster                                                   │
│  └─► Velero: restore from S3 backup to new cluster                  │
└─────────────────────────────────────────────────────────────────────┘

Karpenter — Next-Generation Node Autoscaler

Why Karpenter over Cluster Autoscaler?

Feature	Cluster Autoscaler	Karpenter
Node selection	Pre-defined ASG instance types	Picks cheapest EC2 that fits pod requests
Spot support	Limited	Native, with fallback ordering
Speed	3–5 minutes	30–90 seconds
Consolidation	Basic	Active: moves pods to fewer nodes, terminates unused
Configuration	Per-ASG scaling groups	Declarative NodePool CRD

Karpenter provisions EC2 instances directly via the EC2 Fleet API — no Auto Scaling Group required for scaling decisions. It launches the exact instance type that fits your pending pods' resource requests, which means you pay only for what you actually need.

Installation (Production Only)

Karpenter is deployed only on production clusters where cost optimization and fast scaling matter. Dev/staging use fixed 2-node groups.

# infrastructure/karpenter/applicationset.yaml
source:
  repoURL:        https://charts.karpenter.sh
  chart:          karpenter
  targetRevision: "0.37.0"
  helm:
    values: |
      serviceAccount:
        annotations:
          eks.amazonaws.com/role-arn: "{{karpenterRoleArn}}"
      settings:
        clusterName: "{{cluster}}"
        clusterEndpoint: "{{clusterEndpoint}}"
        interruptionQueue: "{{cluster}}-interruption"
      controller:
        resources:
          requests:
            cpu: 1
            memory: 1Gi

Karpenter IAM

# _modules/karpenter/main.tf

resource "aws_iam_role_policy" "karpenter" {
  name = "karpenter-policy"
  role = aws_iam_role.karpenter.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid    = "EC2Management"
        Effect = "Allow"
        Action = [
          "ec2:RunInstances",
          "ec2:TerminateInstances",
          "ec2:DescribeInstances",
          "ec2:DescribeInstanceTypes",
          "ec2:DescribeSubnets",
          "ec2:DescribeSecurityGroups",
          "ec2:DescribeLaunchTemplates",
          "ec2:CreateLaunchTemplate",
          "ec2:CreateFleet",
          "ec2:CreateTags",
          "ec2:DescribeSpotPriceHistory"
        ]
        Resource = "*"
      },
      {
        Sid    = "EKSDescribe"
        Effect = "Allow"
        Action = [
          # REQUIRED: Karpenter needs this to discover the cluster endpoint and CA
          # Without it Karpenter cannot configure new nodes to join the cluster
          "eks:DescribeCluster"
        ]
        Resource = "arn:aws:eks:*:${var.account_id}:cluster/${var.cluster_name}"
      },
      {
        Sid    = "IAMPassRole"
        Effect = "Allow"
        Action = ["iam:PassRole"]
        Resource = var.node_role_arn
      },
      {
        Sid    = "SQSInterruption"
        Effect = "Allow"
        Action = [
          "sqs:DeleteMessage",
          "sqs:GetQueueUrl",
          "sqs:ReceiveMessage"
        ]
        Resource = aws_sqs_queue.interruption.arn
      }
    ]
  })
}

Lesson learned: eks:DescribeCluster is not optional. Without it, Karpenter cannot discover the cluster endpoint and certificate authority, so newly provisioned nodes cannot join the cluster. Pods remain Pending indefinitely. Always include this permission.

NodePool CRD

# infrastructure/karpenter/nodepools/templates/nodepool.yaml
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: general
spec:
  template:
    metadata:
      labels:
        provisioner: karpenter
    spec:
      nodeClassRef:
        apiVersion: karpenter.k8s.aws/v1beta1
        kind: EC2NodeClass
        name: general

      requirements:
        - key: kubernetes.io/arch
          operator: In
          values: [amd64]
        - key: karpenter.sh/capacity-type
          operator: In
          values: [spot, on-demand]   # Prefer spot, fall back to on-demand
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: [c, m, r]           # Compute, memory, balanced families
        - key: karpenter.k8s.aws/instance-size
          operator: NotIn
          values: [nano, micro, small] # Minimum medium instances

  limits:
    cpu: "100"
    memory: 400Gi

  disruption:
    consolidationPolicy: WhenUnderutilized   # Actively consolidate idle nodes
    consolidateAfter: 30s
    expireAfter: 720h   # Rotate nodes every 30 days (security hygiene)

EC2NodeClass CRD

apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
metadata:
  name: general
spec:
  amiFamily: AL2   # Amazon Linux 2 — EKS-optimized AMI

  # Karpenter discovers subnets and security groups by tags
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: "{{cluster}}"

  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: "{{cluster}}"

  instanceProfile: "{{cluster}}-karpenter-node"

  blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs:
        volumeSize: 100Gi
        volumeType: gp3
        encrypted: true
        kmsKeyID: "{{kmsKeyArn}}"

Verifying Karpenter

# Watch for new NodeClaims as pods scale up
kubectl --context prod-use1 get nodeclaims -w

# Check NodePool status
kubectl --context prod-use1 get nodepool general

# See which nodes Karpenter provisioned vs the initial managed node group
kubectl --context prod-use1 get nodes -L karpenter.sh/capacity-type,node.kubernetes.io/instance-type

HPA — Horizontal Pod Autoscaler

HPA watches CPU and memory metrics from the metrics-server and scales pod replicas up or down automatically.

# apps/myapp/templates/hpa.yaml
{{- if .Values.autoscaling.enabled }}
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: {{ include "myapp.fullname" . }}
  namespace: {{ .Release.Namespace }}
spec:
  scaleTargetRef:
    apiVersion: {{ if .Values.rollout.enabled }}argoproj.io/v1alpha1{{ else }}apps/v1{{ end }}
    kind: {{ if .Values.rollout.enabled }}Rollout{{ else }}Deployment{{ end }}
    name: {{ include "myapp.fullname" . }}
  minReplicas: {{ .Values.autoscaling.minReplicas }}
  maxReplicas: {{ .Values.autoscaling.maxReplicas }}
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: {{ .Values.autoscaling.targetCPUUtilizationPercentage | default 70 }}
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80
{{- end }}

Production values:

# values-production.yaml
autoscaling:
  enabled: true
  minReplicas: 3
  maxReplicas: 10
  targetCPUUtilizationPercentage: 60

Note the conditional scaleTargetRef: In production, HPA targets a Rollout resource (Argo Rollouts). In dev/staging, it targets a standard Deployment. The Helm template handles both cases via .Values.rollout.enabled.

HPA + Karpenter Interaction

Traffic spike arrives
      │
      ▼
HPA: CPU > 60% → scale pods from 3 to 8
      │
      ▼
5 new pods: Pending (not enough node capacity)
      │
      ▼
Karpenter: detects Pending pods → evaluates requests
           → finds cheapest EC2 that fits → provisions 2x m5.large
      │
      ▼ ~60 seconds
New nodes join cluster → pods schedule → Running
      │
Traffic normalizes
      │
HPA: CPU < 60% → scale pods from 8 back to 3
      │
5 pods terminate
      │
Karpenter: 2 nodes underutilized → consolidate → terminate nodes

Argo Rollouts — Canary Deployments

Argo Rollouts replaces the standard Kubernetes Deployment with a Rollout resource that supports progressive delivery strategies. In production, every deployment goes through a canary phase.

Installation (Production Only)

# infrastructure/argo-rollouts/applicationset.yaml
source:
  repoURL:        https://argoproj.github.io/argo-helm
  chart:          argo-rollouts
  targetRevision: "2.35.3"
  helm:
    values: |
      installCRDs: true
      dashboard:
        enabled: true
        service:
          type: ClusterIP

Rollout CRD (replaces Deployment in production)

# apps/myapp/templates/deployment.yaml
{{- if .Values.rollout.enabled }}
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: {{ include "myapp.fullname" . }}
  namespace: {{ .Release.Namespace }}
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      {{- include "myapp.selectorLabels" . | nindent 6 }}
  template:
    metadata:
      labels:
        {{- include "myapp.selectorLabels" . | nindent 8 }}
    spec:
      containers:
        - name: myapp
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          ports:
            - containerPort: 8080
              name: http
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              cpu: 500m
              memory: 256Mi

  strategy:
    canary:
      canaryService: {{ include "myapp.fullname" . }}-canary
      stableService: {{ include "myapp.fullname" . }}
      steps:
        - setWeight: 20          # Step 1: 20% of traffic to canary
        - pause: { duration: 5m } # Step 2: Bake for 5 minutes
        - analysis:               # Step 3: Automated metric check
            templates:
              - templateName: success-rate
            args:
              - name: service-name
                value: {{ include "myapp.fullname" . }}-canary
        - setWeight: 100         # Step 4: Promote to 100%

{{- else }}
# Standard Deployment for dev/staging
apiVersion: apps/v1
kind: Deployment
...
{{- end }}

AnalysisTemplate — Automated Promotion Gate

# apps/myapp/templates/analysis-template.yaml
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
  namespace: {{ .Release.Namespace }}
spec:
  args:
    - name: service-name

  metrics:
    - name: success-rate
      interval: 60s
      count: 5             # Run 5 measurements (5 minutes total)
      failureLimit: 1      # One failure triggers rollback

      provider:
        prometheus:
          address: http://prometheus-operated.monitoring.svc.cluster.local:9090
          query: |
            sum(
              rate(myapp_http_requests_total{
                service="{{`{{args.service-name}}`}}",
                status_code!~"5.."
              }[5m])
            )
            /
            sum(
              rate(myapp_http_requests_total{
                service="{{`{{args.service-name}}`}}"
              }[5m])
            )

      successCondition: result[0] >= 0.99   # 99%+ success rate required
      failureCondition: result[0] < 0.99

Canary Service

Traffic splitting requires two Services: stable (regular Service) and canary (routes only to canary pods).

# apps/myapp/templates/service-canary.yaml
{{- if .Values.rollout.enabled }}
apiVersion: v1
kind: Service
metadata:
  name: {{ include "myapp.fullname" . }}-canary
  namespace: {{ .Release.Namespace }}
spec:
  selector:
    {{- include "myapp.selectorLabels" . | nindent 4 }}
  ports:
    - port: 80
      targetPort: http
      protocol: TCP
      name: http
{{- end }}

Canary Deployment Walkthrough

1. CI pushes new image: sha-abc123 → myapp-gitops updated
2. ArgoCD detects diff → triggers Rollout update
3. Argo Rollouts creates new ReplicaSet with sha-abc123

   Traffic: 80% → stable (sha-xyz789), 20% → canary (sha-abc123)

4. 5-minute pause
   → Monitor Grafana: error rate on canary service?
   → Logs in CloudWatch: any exceptions?

5. AnalysisRun queries Prometheus (5 measurements × 1 min)
   → success_rate = 0.997 (99.7% success) ✓ PASS

6. setWeight: 100% → all traffic to sha-abc123
7. Old ReplicaSet (sha-xyz789) scaled to 0 after stability window

Manual Intervention

# Watch rollout progress
kubectl argo rollouts status myapp -n myapp --watch

# Manually promote (skip remaining pause/analysis steps)
kubectl argo rollouts promote myapp -n myapp

# Manually abort (rollback to stable immediately)
kubectl argo rollouts abort myapp -n myapp

# Access the Argo Rollouts dashboard
kubectl port-forward svc/argo-rollouts-dashboard -n argo-rollouts 3100:3100
# Open http://localhost:3100

Velero — Backup and Disaster Recovery

Velero backs up Kubernetes resource definitions and EBS volume snapshots to S3. If a cluster is accidentally deleted or corrupted, you can restore everything to a new cluster.

What Velero Backs Up

All Kubernetes objects (Deployments, Services, ConfigMaps, Secrets, etc.)
PersistentVolumeClaim snapshots (Prometheus data, Grafana dashboards, etc.)
Namespace structure

Installation

# infrastructure/velero/applicationset.yaml
source:
  repoURL:        https://vmware-tanzu.github.io/helm-charts
  chart:          velero
  targetRevision: "6.4.0"
  helm:
    values: |
      serviceAccount:
        server:
          annotations:
            eks.amazonaws.com/role-arn: "{{veleroRoleArn}}"
      configuration:
        backupStorageLocation:
          - name: default
            provider: aws
            bucket: "myapp-velero-{{cluster}}"
            config:
              region: "{{region}}"
        volumeSnapshotLocation:
          - name: default
            provider: aws
            config:
              region: "{{region}}"
      initContainers:
        - name: velero-plugin-for-aws
          image: velero/velero-plugin-for-aws:v1.9.0
          volumeMounts:
            - mountPath: /target
              name: plugins

Scheduled Backup

# infrastructure/velero/schedule.yaml
apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: daily-backup
  namespace: velero
spec:
  schedule: "0 2 * * *"   # 2 AM UTC daily
  template:
    includedNamespaces:
      - myapp
      - monitoring
      - argocd
    excludedResources:
      - events
      - events.events.k8s.io
    snapshotVolumes: true
    ttl: 720h   # 30 days retention
    storageLocation: default
    volumeSnapshotLocations:
      - default

Restore Procedure

# List available backups
velero backup get

# Restore from a specific backup
velero restore create --from-backup daily-backup-20260308020000

# Watch restore progress
velero restore describe <restore-name> --details

# Verify restored resources
kubectl get all -n myapp
kubectl get pvc -n monitoring

Critical reminder: An untested backup is not a backup. Run a restore drill at least quarterly into a temporary cluster. The restore procedure should be documented and rehearsed so it is not being learned for the first time during an actual outage.

PodDisruptionBudget

Karpenter's consolidation can evict pods to move them to fewer nodes. Without a PDB, it might evict too many pods at once and cause downtime.

# apps/myapp/templates/pdb.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: {{ include "myapp.fullname" . }}
  namespace: {{ .Release.Namespace }}
spec:
  minAvailable: 2   # Always keep at least 2 pods running during disruption
  selector:
    matchLabels:
      {{- include "myapp.selectorLabels" . | nindent 6 }}

With minReplicas: 3 and minAvailable: 2, Karpenter can only evict one pod at a time. The remaining two continue serving traffic while the evicted pod reschedules on a new node.

Complete Resilience Picture

NORMAL OPERATION
─────────────────
3 pods (minReplicas) on 2 managed nodes
HPA watching CPU (target: 60%)
Karpenter watching for pending pods
Velero backing up daily at 2 AM UTC

TRAFFIC SPIKE
─────────────────
CPU > 60% → HPA scales to 8 pods
2 pods pending (no capacity) → Karpenter provisions m5.large
All 8 pods running → serving traffic
Spike ends → HPA scales to 3 pods
2 pods terminate → Karpenter consolidates → terminates extra node

NEW DEPLOYMENT (production)
─────────────────
ArgoCD syncs new image → Rollout starts
20% canary → 5min bake → AnalysisRun → 100% promote
OR: error rate > 1% → automatic rollback to previous version

DISASTER RECOVERY
─────────────────
Cluster accidentally deleted → restore from Velero backup
velero restore create --from-backup <last-good-backup>
Kubernetes objects restored → EBS snapshots attached → running in ~15 min

Summary

By the end of Part 10 — and the entire series — you have:

✅ Karpenter provisioning right-sized EC2 instances on demand, consolidating when idle
✅ HPA scaling pods 3→10 based on CPU utilization, targeting a Rollout in production
✅ Argo Rollouts deploying every production change as a canary with automated Prometheus-based promotion gates
✅ Velero running scheduled daily backups with 30-day retention to S3
✅ PodDisruptionBudget preventing Karpenter from evicting too many pods at once

Series Conclusion

You have now built a complete production-grade DevSecOps platform:

Layer	What You Built
Foundation	AWS Organizations, 4 accounts, SSO, SCPs
Infrastructure	Terraform modules, Terragrunt DRY configs, 6 VPCs
Compute	6 EKS clusters (k8s 1.29) across 3 environments and 2 regions
GitOps	ArgoCD hub-spoke, 35+ ApplicationSets, automated sync
CI/CD	GitHub Actions + OIDC + Trivy + Cosign + ECR
Secrets	AWS Secrets Manager + ESO + IRSA
Security	Kyverno policies + Falco runtime detection + WAF + GuardDuty
Observability	Prometheus + Grafana + Fluent Bit + CloudWatch
Resilience	Karpenter + HPA + Argo Rollouts canary + Velero DR
Networking	Route53 latency routing + ACM TLS + ALB + NetworkPolicies

This is the platform that a growing engineering team with 50–500 developers would build and operate. Each component was chosen for a reason, wired to the others, and tested against real failures.

Screenshot Placeholders

SCREENSHOT: kubectl get hpa showing current/desired replicas scaling
Screenshot 10-3: Velero Backup Schedule" width="800" height="734">

SCREENSHOT: ArgoCD — full applications view showing all 35+ apps across 6 clusters

Thank you for following this series. Source code:

Infrastructure: github.com/MatthewDipo/myapp-infra
GitOps manifests: github.com/MatthewDipo/myapp-gitops
Application: github.com/MatthewDipo/myapp

Part 9: Observability — Prometheus, Grafana, Fluent Bit, and CloudWatch

Matthew — Wed, 13 May 2026 08:00:00 +0000

Part of the series: Building a Production-Grade DevSecOps Pipeline on AWS

Introduction

Observability answers the question: what is my system doing right now, and why?

The three pillars:

Metrics — numerical measurements over time (CPU%, request rate, error rate, latency)
Logs — structured event records from every container
Traces — request flows across services (not covered in this series, but Grafana Tempo is the natural next step)

This pipeline implements metrics with Prometheus + Grafana and logs with Fluent Bit → CloudWatch. Together they give you both real-time dashboards and historical log search without leaving the AWS ecosystem.

┌──────────────────────────────────────────────────────────────────────────┐
│  OBSERVABILITY ARCHITECTURE                                              │
│                                                                          │
│  Application Pods                                                        │
│  ├─ /metrics endpoint → ServiceMonitor → Prometheus scrape               │
│  └─ stdout/stderr → Fluent Bit DaemonSet → CloudWatch Logs               │
│                                                                          │
│  Infrastructure                                                          │
│  ├─ node-exporter (CPU, memory, disk, network per node) → Prometheus     │
│  └─ kube-state-metrics (pod state, deployment state) → Prometheus        │
│                                                                          │
│  Prometheus → Grafana (dashboards + alert rules)                         │
│  Prometheus → Alertmanager (notifications)                               │
│                                                                          │
│  Falco (security events) → stdout → Fluent Bit → CloudWatch              │
│  All containers → stdout → Fluent Bit → CloudWatch                       │
└──────────────────────────────────────────────────────────────────────────┘

kube-prometheus-stack

Rather than installing Prometheus, Grafana, and Alertmanager separately, we use the kube-prometheus-stack Helm chart. It bundles:

Prometheus Operator — manages Prometheus, Alertmanager, and PrometheusRule CRDs
Prometheus — the metrics database
Alertmanager — routes alerts to notification channels
Grafana — dashboards and visualization
kube-state-metrics — exposes Kubernetes object state as metrics
node-exporter — exposes node-level metrics (CPU, memory, disk, network)

We install it only on staging and production clusters (4 of 6) — dev clusters skip monitoring to reduce cost.

Installation via ArgoCD

# infrastructure/monitoring/applicationset.yaml
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: kube-prometheus-stack
  namespace: argocd
spec:
  generators:
    - list:
        elements:
          - cluster:        myapp-production-use1
            region:         us-east-1
            grafanaIngress: "true"
            certArn:        "arn:aws:acm:us-east-1:591120834781:certificate/9ab022c9-..."
          - cluster:        myapp-production-usw2
            region:         us-west-2
            grafanaIngress: "false"
            certArn:        ""
          - cluster:        myapp-staging-use1
            region:         us-east-1
            grafanaIngress: "false"
            certArn:        ""
          - cluster:        myapp-staging-usw2
            region:         us-west-2
            grafanaIngress: "false"
            certArn:        ""
  template:
    metadata:
      name: "prometheus-{{cluster}}"
    spec:
      project: production
      sources:
        - repoURL:        https://prometheus-community.github.io/helm-charts
          chart:          kube-prometheus-stack
          targetRevision: "61.9.0"
          helm:
            valueFiles:
              - $gitopsValues/infrastructure/monitoring/prometheus-values.yaml
            parameters:
              - name:  "grafana.ingress.enabled"
                value: "{{grafanaIngress}}"
              - name:  "grafana.ingress.annotations.alb\\.ingress\\.kubernetes\\.io/certificate-arn"
                value: "{{certArn}}"
        - repoURL:        https://github.com/MatthewDipo/myapp-gitops.git
          targetRevision: main
          ref: gitopsValues
      destination:
        name:      "{{cluster}}"
        namespace: monitoring
      syncPolicy:
        syncOptions: [CreateNamespace=true, ServerSideApply=true]
        retry:
          limit: 3
          backoff: { duration: 30s, maxDuration: 10m, factor: 2 }

Important install flag: Use --no-hooks --timeout 10m (without --wait). Pre-upgrade admission webhook Jobs consistently time out in ArgoCD, causing the sync phase to show Failed — but the actual resources (Prometheus, Grafana, Alertmanager) deploy correctly. This is a known false positive. Do not let it alarm you.

prometheus-values.yaml

prometheus:
  prometheusSpec:
    retention: 15d
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: gp2
          accessModes: [ReadWriteOnce]
          resources:
            requests:
              storage: 50Gi
    # Auto-discover ALL ServiceMonitors and PodMonitors across all namespaces
    # Without these settings Prometheus only scrapes resources with matching Helm labels
    podMonitorSelectorNilUsesHelmValues:     false
    serviceMonitorSelectorNilUsesHelmValues: false
    ruleSelectorNilUsesHelmValues:           false

alertmanager:
  alertmanagerSpec:
    storage:
      volumeClaimTemplate:
        spec:
          storageClassName: gp2
          accessModes: [ReadWriteOnce]
          resources:
            requests:
              storage: 10Gi
  config:
    global:
      resolve_timeout: 5m
    route:
      group_by: [alertname, region]
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 12h
      receiver: "null"    # Placeholder — replace with Slack/PagerDuty
    receivers:
      - name: "null"

grafana:
  admin:
    existingSecret: grafana-admin-secret
    userKey:        admin-user
    passwordKey:    admin-password
  persistence:
    enabled:          true
    storageClassName: gp2
    size:             10Gi
  sidecar:
    dashboards:
      enabled:          true
      searchNamespace:  ALL   # Pick up dashboards from all namespaces
  ingress:
    enabled:          false   # Overridden per-cluster via ApplicationSet parameter
    ingressClassName: alb
    annotations:
      alb.ingress.kubernetes.io/scheme:        internet-facing
      alb.ingress.kubernetes.io/target-type:   ip
      alb.ingress.kubernetes.io/listen-ports:  '[{"HTTPS":443}]'
      alb.ingress.kubernetes.io/ssl-redirect:  "443"
      alb.ingress.kubernetes.io/certificate-arn: ""  # Injected per-region
    hosts:
      - grafana.matthewoladipupo.dev
    path: /
    pathType: Prefix

kubeStateMetrics:
  enabled: true
nodeExporter:
  enabled: true

Grafana Public Access

Grafana is only exposed publicly on myapp-production-use1. The reasons:

Grafana uses an EBS ReadWriteOnce PVC — only one node can mount it at a time, making it inherently single-instance
EBS data is AZ-local — running a second public Grafana in usw2 would show different historical data
One public Grafana that federates data from all clusters is cleaner than four separate Grafana instances

Access URL: https://grafana.matthewoladipupo.dev

The Route53 A record points to the ALB provisioned by AWS LBC when the Ingress is applied.

Getting the Grafana Password

The Grafana admin credentials are stored in grafana-admin-secret in the monitoring namespace. Important caveat: Grafana only reads this secret on first database initialization. If the secret value changes after the pod has started, you must reset the password via grafana-cli:

kubectl exec -n monitoring <grafana-pod> -c grafana -- \
  grafana-cli admin reset-admin-password 'YourNewPassword'

ServiceMonitor for the Application

By default, Prometheus only scrapes the cluster components provided by kube-prometheus-stack. To scrape your application, add a ServiceMonitor CRD:

# apps/myapp/templates/servicemonitor.yaml
{{- if .Values.serviceMonitor.enabled }}
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: {{ include "myapp.fullname" . }}
  namespace: {{ .Release.Namespace }}
  labels:
    {{- include "myapp.labels" . | nindent 4 }}
spec:
  selector:
    matchLabels:
      {{- include "myapp.selectorLabels" . | nindent 6 }}
  endpoints:
    - port: http          # Named port on the Service
      path: /metrics
      interval: {{ .Values.serviceMonitor.interval | default "30s" }}
      scrapeTimeout: {{ .Values.serviceMonitor.scrapeTimeout | default "10s" }}
  namespaceSelector:
    matchNames:
      - {{ .Release.Namespace }}
{{- end }}

The application's /metrics endpoint returns Prometheus exposition format:

# HELP myapp_http_requests_total Total HTTP requests
# TYPE myapp_http_requests_total counter
myapp_http_requests_total{method="GET",route="/health",status_code="200"} 1423
myapp_http_requests_total{method="GET",route="/metrics",status_code="200"} 89
# HELP process_cpu_seconds_total Total user and system CPU time
...

The key setting that makes auto-discovery work: serviceMonitorSelectorNilUsesHelmValues: false in prometheus-values.yaml. Without this, Prometheus only scrapes ServiceMonitors that have the Helm release's labels — ignoring your application's ServiceMonitor.

PrometheusRule — Alert Rules

# infrastructure/monitoring/alert-rules/myapp-alerts.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: myapp-alerts
  namespace: monitoring
  labels:
    release: prometheus   # Must match what Prometheus Operator is watching
spec:
  groups:
    - name: myapp.rules
      interval: 30s
      rules:
        - alert: HighErrorRate
          expr: |
            (
              sum(rate(myapp_http_requests_total{status_code=~"5.."}[5m]))
              /
              sum(rate(myapp_http_requests_total[5m]))
            ) > 0.01
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "High error rate in myapp ({{ $value | humanizePercentage }})"
            description: "More than 1% of requests are failing with 5xx errors for 5 minutes."

        - alert: PodCrashLooping
          expr: rate(kube_pod_container_status_restarts_total{namespace="myapp"}[15m]) * 60 * 15 > 2
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "Pod {{ $labels.pod }} is crash looping"
            description: "Pod has restarted more than 2 times in 15 minutes."

        - alert: HighMemoryUsage
          expr: |
            (
              container_memory_working_set_bytes{namespace="myapp",container!=""}
              / container_spec_memory_limit_bytes{namespace="myapp",container!=""}
            ) > 0.9
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "Memory usage above 90% in {{ $labels.pod }}"

Alertmanager — Adding Slack Notifications

The current config uses a null receiver (alerts fire but go nowhere). To wire up Slack:

alertmanager:
  config:
    global:
      resolve_timeout: 5m
      slack_api_url: 'https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK'
    route:
      group_by: [alertname, region, cluster]
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 4h
      receiver: slack-critical
      routes:
        - match:
            severity: critical
          receiver: slack-critical
        - match:
            severity: warning
          receiver: slack-warning
    receivers:
      - name: slack-critical
        slack_configs:
          - channel: '#alerts-critical'
            text: |
              *Alert:* {{ .GroupLabels.alertname }}
              *Severity:* {{ .GroupLabels.severity }}
              *Cluster:* {{ .GroupLabels.cluster }}
              {{ range .Alerts }}*Description:* {{ .Annotations.description }}{{ end }}
            send_resolved: true
      - name: slack-warning
        slack_configs:
          - channel: '#alerts-warning'
            text: '{{ .GroupLabels.alertname }}: {{ range .Alerts }}{{ .Annotations.summary }}{{ end }}'
            send_resolved: true

Fluent Bit — Log Shipping to CloudWatch

Fluent Bit runs as a DaemonSet — one pod per node — and reads all container logs from /var/log/containers/*.log.

IRSA for Fluent Bit

Fluent Bit needs IAM permissions to write to CloudWatch. The key lesson: use wildcard ARNs for both log groups AND log streams.

# _modules/fluent-bit-irsa/main.tf

resource "aws_iam_role_policy" "fluent_bit" {
  name = "fluent-bit-cloudwatch"
  role = aws_iam_role.fluent_bit.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid    = "CloudWatchLogs"
        Effect = "Allow"
        Action = [
          "logs:CreateLogGroup",
          "logs:CreateLogStream",
          "logs:PutLogEvents",
          "logs:DescribeLogStreams",
          "logs:DescribeLogGroups"
        ]
        Resource = [
          # Log group operations (CreateLogGroup, DescribeLogGroups)
          "arn:aws:logs:${var.aws_region}:${var.account_id}:log-group:/eks/*",
          # Log stream operations (CreateLogStream, PutLogEvents)
          # The :* suffix is REQUIRED for stream-level permissions
          "arn:aws:logs:${var.aws_region}:${var.account_id}:log-group:/eks/*:*"
        ]
      }
    ]
  })
}

Lesson learned: Using only log-group:/eks/* (without the :* suffix) grants permissions on the log group resource but NOT on log streams within it. CreateLogStream and PutLogEvents operate on the log stream resource, which requires the :* suffix. Without this, pods get AccessDeniedException on CreateLogStream even though CreateLogGroup succeeds.

Fluent Bit Helm Values

# infrastructure/logging/fluent-bit-values.yaml
serviceAccount:
  annotations:
    eks.amazonaws.com/role-arn: "{{irsaRoleArn}}"   # Injected per-cluster

cloudWatch:
  region: "{{region}}"
  logGroupName: "/eks/{{cluster}}/$(kubernetes['namespace_name'])"
  logStreamName: "$(kubernetes['pod_name'])/$(kubernetes['container_name'])"
  autoCreateGroup: true

# Enrich log records with Kubernetes metadata
filter:
  kubernetes:
    Merge_Log: On
    Keep_Log: Off
    K8S-Logging.Parser: On
    K8S-Logging.Exclude: On

CloudWatch Log Groups Created

After Fluent Bit starts, these log groups appear in CloudWatch:

/eks/myapp-production-use1/myapp
/eks/myapp-production-use1/monitoring
/eks/myapp-production-use1/argocd
/eks/myapp-production-use1/kyverno
/eks/myapp-production-use1/falco
... (one per namespace, all clusters)

Querying Logs with CloudWatch Insights

# Find all 5xx errors in the last hour
fields @timestamp, log
| filter kubernetes.namespace_name = "myapp"
| filter log like /5[0-9][0-9]/
| sort @timestamp desc
| limit 100

# Find Falco security alerts
fields @timestamp, output
| filter kubernetes.namespace_name = "falco"
| filter priority = "Warning" or priority = "Error"
| sort @timestamp desc

# Count errors by pod
stats count(*) as errors by kubernetes.pod_name
| filter log like /ERROR/
| sort errors desc

Grafana Dashboards

Pre-built Dashboards (from kube-prometheus-stack)

These are included automatically and show up in Grafana immediately after installation:

Kubernetes / Compute Resources / Cluster — total CPU/memory across all nodes
Kubernetes / Compute Resources / Namespace — resource breakdown per namespace
Node Exporter / Nodes — per-node CPU, memory, disk, network I/O
Alertmanager / Overview — alert firing/resolved history

Importing Community Dashboards

Go to https://grafana.matthewoladipupo.dev
Dashboards → Import
Enter the dashboard ID from grafana.com:
- 1860 — Node Exporter Full (very detailed node metrics)
- 13332 — Kubernetes Pods (pod-level resource view)
- 15757 — ArgoCD (sync status, app health)

Application Dashboard

With the ServiceMonitor installed, Grafana can display your app metrics. Create a panel with:

# Request rate (requests per second)
sum(rate(myapp_http_requests_total[5m])) by (route)

# Error rate (percentage of 5xx)
sum(rate(myapp_http_requests_total{status_code=~"5.."}[5m]))
/ sum(rate(myapp_http_requests_total[5m])) * 100

# 95th percentile latency (if using histogram metric)
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

Summary

By the end of Part 9 you have:

✅ kube-prometheus-stack on 4 clusters (staging + production) with EBS persistent storage
✅ Grafana publicly accessible at https://grafana.matthewoladipupo.dev (production-use1 only)
✅ ServiceMonitor scraping myapp's /metrics endpoint every 30 seconds
✅ PrometheusRule alert rules for high error rate, crash looping, high memory
✅ Fluent Bit DaemonSet on all 6 clusters shipping logs to CloudWatch
✅ CloudWatch log groups per namespace per cluster
✅ CloudWatch Insights for ad-hoc log queries

Screenshot Placeholders

SCREENSHOT: Grafana — Kubernetes cluster overview dashboard showing node CPU and memory

SCREENSHOT: Grafana — Node Exporter dashboard showing per-node metrics

SCREENSHOT: AWS CloudWatch — Log groups showing /eks/ hierarchy from Fluent Bit

SCREENSHOT: CloudWatch Insights — query result showing application logs

Next: Part 10 — Resilience: Karpenter, HPA, Argo Rollouts, and Velero

Follow the series — next part publishes next Wednesday.
Live system: https://www.matthewoladipupo.dev/health
Runbook: Operations Guide
Source code: myapp-infra | myapp-gitops | myapp

Part 8: Security Stack

Matthew — Wed, 06 May 2026 08:00:00 +0000

Security Stack — Kyverno, Falco, WAF, and GuardDuty

Part of the series: Building a Production-Grade DevSecOps Pipeline on AWS

Introduction: Defense in Depth

No single security tool is sufficient. A WAF blocks HTTP attacks but does nothing if an attacker exploits a container escape. Kyverno blocks bad pod configurations but can't stop an attacker who is already inside a running container. Each layer catches what the others miss.

Cosign, ECR), Runtime (Falco eBPF), Admission Control (Kyverno), Cloud
Perimeter (GuardDuty, WAF), Network (VPC, Security Groups)" width="800" height="524">

An attacker must penetrate all 5 layers. A container escape attempt is caught
simultaneously by Falco (runtime), Kyverno (admission), and GuardDuty (API
anomaly detection).

┌─────────────────────────────────────────────────────────────────────┐
│  SECURITY LAYERS — each catches different attack vectors            │
│                                                                     │
│  Layer 1: SUPPLY CHAIN (Part 6)                                     │
│  ┌─────────────────────────────────────────────────────────────┐    │
│  │ Trivy: no HIGH/CRITICAL CVEs in image                       │    │
│  │ Cosign: image cryptographically signed before push          │    │
│  │ Distroless: no shell/tools available post-compromise        │    │
│  └─────────────────────────────────────────────────────────────┘    │
│                    ↓ (image passes, reaches cluster)                │
│  Layer 2: ADMISSION CONTROL (Kyverno)                               │
│  ┌─────────────────────────────────────────────────────────────┐    │
│  │ Blocks bad configs at kubectl apply / ArgoCD sync time      │    │
│  │ Pod never starts if it violates policy                      │    │
│  └─────────────────────────────────────────────────────────────┘    │
│                    ↓ (pod starts, attacker gets RCE)                │
│  Layer 3: RUNTIME DETECTION (Falco)                                 │
│  ┌─────────────────────────────────────────────────────────────┐    │
│  │ eBPF syscall monitoring — detects attacks already running   │    │
│  │ Alerts within 1 second of suspicious activity               │    │
│  └─────────────────────────────────────────────────────────────┘    │
│                    ↓ (attacker reaches HTTP layer)                  │
│  Layer 4: PERIMETER (AWS WAF)                                       │
│  ┌─────────────────────────────────────────────────────────────┐    │
│  │ Blocks SQLi, XSS, log4shell, rate limiting at ALB level     │    │
│  │ Attacker request never reaches your pod                     │    │
│  └─────────────────────────────────────────────────────────────┘    │
│                    ↓ (account-level threats)                        │
│  Layer 5: THREAT INTELLIGENCE (GuardDuty)                           │
│  ┌─────────────────────────────────────────────────────────────┐    │
│  │ ML-based: crypto mining, C2 comms, compromised credentials  │    │
│  │ Monitors CloudTrail, VPC Flow Logs, DNS queries             │    │
│  └─────────────────────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────────────────────┘

Kyverno — Admission Control

Kyverno is a Kubernetes-native policy engine. Policies are written in YAML, not a separate policy language, which makes them readable and maintainable by anyone who knows Kubernetes.

Version Selection

This matters enormously. Kyverno 3.7.x requires Kubernetes ≥ 1.30 because it uses ValidatingAdmissionPolicy (a v1 API). Our clusters run Kubernetes 1.29.

❌ Kyverno 3.7.x → CrashLoopBackOff on k8s 1.29
✅ Kyverno 3.2.6 (app version 1.12.5) → compatible with k8s 1.25–1.29

# infrastructure/kyverno/applicationset.yaml
source:
  repoURL:        https://kyverno.github.io/kyverno
  chart:          kyverno
  targetRevision: "3.2.6"

Installation Flags

helm install kyverno kyverno/kyverno \
  -n kyverno --create-namespace \
  --version 3.2.6 \
  --no-hooks \        # REQUIRED: cleanup CronJobs get ImagePullBackOff
  --wait \
  --timeout 10m

System Namespace Exclusions

Kyverno policies apply to all namespaces by default. System components like CoreDNS and kube-proxy run in kube-system and don't follow application-level security policies (they need root, they need hostPath, etc.). Exclude system namespaces in every policy:

# Applies to ALL policies below
exclude:
  any:
    - resources:
        namespaces:
          - kube-system
          - kyverno
          - cert-manager
          - external-secrets
          - argocd
          - argo-rollouts
          - monitoring
          - logging
          - falco

Policy 1: Block Privileged Containers

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-privileged-containers
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: check-privileged
      match:
        any:
          - resources:
              kinds: [Pod]
      exclude:
        any:
          - resources:
              namespaces: [kube-system, kyverno, cert-manager, external-secrets,
                           argocd, argo-rollouts, monitoring, logging, falco]
      validate:
        message: "Privileged containers are not allowed."
        # Use anyPattern in Kyverno v1.12.x (NOT validate.any)
        anyPattern:
          - spec:
              containers:
                - =(securityContext):
                    =(privileged): false
          - spec:
              containers:
                - =(securityContext): {}

API change in v1.12.x: Use validate.anyPattern not validate.any. The validate.any syntax was removed in the 1.12 API version.

Policy 2: Require Non-Root Containers

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-non-root
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: check-runasnonroot
      match:
        any:
          - resources:
              kinds: [Pod]
      exclude:
        any:
          - resources:
              namespaces: [kube-system, kyverno, cert-manager, external-secrets,
                           argocd, argo-rollouts, monitoring, logging, falco]
      validate:
        message: "Containers must run as non-root (runAsNonRoot: true)."
        anyPattern:
          - spec:
              securityContext:
                runAsNonRoot: true
          - spec:
              containers:
                - securityContext:
                    runAsNonRoot: true

Policy 3: Block hostPath Volumes

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-host-path
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: check-hostpath
      match:
        any:
          - resources:
              kinds: [Pod]
      exclude:
        any:
          - resources:
              namespaces: [kube-system, kyverno, logging, falco]
      validate:
        message: "hostPath volumes are not allowed."
        deny:
          conditions:
            any:
              - key: "{{ request.object.spec.volumes[].hostPath | length(@) }}"
                operator: GreaterThan
                value: "0"

Policy 4: Require Resource Limits

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-resource-limits
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: check-limits
      match:
        any:
          - resources:
              kinds: [Pod]
      exclude:
        any:
          - resources:
              namespaces: [kube-system, kyverno, cert-manager, external-secrets,
                           argocd, argo-rollouts, monitoring, logging, falco]
      validate:
        message: "CPU and memory limits are required on all containers."
        pattern:
          spec:
            containers:
              - resources:
                  limits:
                    memory: "?*"
                    cpu: "?*"

Policy 5: Require Signed Images

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-signed-images
spec:
  validationFailureAction: Enforce
  background: false   # Must check at admission, not retroactively
  rules:
    - name: check-image-signature
      match:
        any:
          - resources:
              kinds: [Pod]
              namespaces: [myapp]   # Only enforce on application namespaces
      verifyImages:
        - imageReferences:
            - "206617159586.dkr.ecr.us-east-1.amazonaws.com/myapp:*"
          attestors:
            - entries:
                - keys:
                    kms: "awskms:///arn:aws:kms:us-east-1:206617159586:key/YOUR_KEY_ID"

Kyverno Circular Deadlock — How to Fix It

If Kyverno's webhook configurations become corrupted (e.g., from a failed upgrade), new Kyverno pods can't start because they can't pass their own admission checks. It is a deadlock.

# Symptom: Kyverno pods stuck in Pending or CrashLoopBackOff
# Error: "failed calling webhook: the server is currently unable to handle the request"

# Fix: Delete the broken webhook configs — this temporarily disables admission control
kubectl delete validatingwebhookconfiguration kyverno-resource-validating-webhook-cfg
kubectl delete validatingwebhookconfiguration kyverno-policy-validating-webhook-cfg
kubectl delete mutatingwebhookconfiguration kyverno-resource-mutating-webhook-cfg

# Kyverno pods can now start without passing their own webhooks
# Once running, Kyverno recreates the webhook configs automatically
kubectl rollout restart deployment/kyverno -n kyverno

Falco — Runtime Threat Detection

Falco operates at the Linux kernel level using eBPF probes. It monitors every system call made by every process in every container. When a pattern matches a rule, it fires an alert within milliseconds.

┌──────────────────────────────────────────────────────────────┐
│  HOW FALCO WORKS                                             │
│                                                              │
│  Kernel syscalls (open, exec, connect, read, write...)       │
│         │                                                    │
│         │  eBPF probe (kernel module or ebpf driver)         │
│         ▼                                                    │
│  Falco engine                                                │
│  ├── Checks each syscall against rule set                    │
│  ├── Rule: "exec of sh in container → ALERT"                 │
│  └── Rule: "read /etc/shadow → ALERT"                        │
│         │                                                    │
│         │  JSON alert output to stdout                       │
│         ▼                                                    │
│  Fluent Bit (DaemonSet) picks up stdout                      │
│         │                                                    │
│         ▼                                                    │
│  CloudWatch Logs: /eks/cluster-name/falco                    │
└──────────────────────────────────────────────────────────────┘

Installation

# infrastructure/falco/applicationset.yaml
source:
  repoURL:        https://falcosecurity.github.io/charts
  chart:          falco
  targetRevision: "3.8.7"
  helm:
    values: |
      driver:
        kind: ebpf   # Modern eBPF driver (no kernel module compilation)
      falco:
        json_output: true     # JSON output for Fluent Bit parsing
        log_stderr: true
        log_level: info
      falcosidekick:
        enabled: false   # Using Fluent Bit for log shipping instead

Key Rules That Fire by Default

Rule	Trigger	Severity
Terminal shell in container	`exec` of sh/bash/zsh in container	WARNING
Read sensitive file untrusted	Read of /etc/shadow, /etc/passwd	WARNING
Write below root	Any write to / or system dirs	ERROR
Outbound Connection Not Expected	Container connects to unexpected IP	NOTICE
Privilege Escalation via setuid	setuid/setgid syscall	WARNING
Modify binary dirs	Write to /bin, /usr/bin	ERROR

Testing Falco

# In one terminal, watch Falco logs
kubectl logs -f -n falco -l app.kubernetes.io/name=falco -c falco | grep -v "Notice\|Informational"

# In another terminal, trigger a rule
kubectl exec -it -n myapp <pod-name> -- sh
# Falco fires: "Notice A shell was spawned in a container with an attached terminal"
# (Note: distroless containers have no shell — this only works if you exec into a debug container)

Custom Rule: Alert on curl/wget

# infrastructure/falco/custom-rules.yaml
- rule: Unexpected curl or wget in container
  desc: Detect curl or wget being used in a container (potential exfiltration)
  condition: >
    spawned_process and
    container and
    proc.name in (curl, wget, python, python3) and
    not proc.pname in (sh, bash)
  output: >
    Curl/wget detected in container
    (user=%user.name command=%proc.cmdline container=%container.name
     image=%container.image.repository)
  priority: WARNING
  tags: [network, mitre_exfiltration]

AWS WAF — Web Application Firewall

WAF sits in front of your ALB and inspects every HTTP request before it reaches your pods. This happens at the AWS network edge — your application code never sees malicious requests.

Terraform Module

# _modules/waf/main.tf

resource "aws_wafv2_web_acl" "main" {
  name  = "${var.env}-${var.region_alias}-web-acl"
  scope = "REGIONAL"   # For ALB (not CloudFront)

  default_action {
    allow {}   # Allow by default; rules below explicitly block
  }

  # Rule 1: AWS Managed — Common Rule Set (OWASP Top 10)
  rule {
    name     = "AWSManagedRulesCommonRuleSet"
    priority = 1
    override_action { none {} }
    statement {
      managed_rule_group_statement {
        name        = "AWSManagedRulesCommonRuleSet"
        vendor_name = "AWS"
      }
    }
    visibility_config {
      cloudwatch_metrics_enabled = true
      metric_name                = "CommonRuleSet"
      sampled_requests_enabled   = true
    }
  }

  # Rule 2: SQL Injection protection
  rule {
    name     = "AWSManagedRulesSQLiRuleSet"
    priority = 2
    override_action { none {} }
    statement {
      managed_rule_group_statement {
        name        = "AWSManagedRulesSQLiRuleSet"
        vendor_name = "AWS"
      }
    }
    visibility_config {
      cloudwatch_metrics_enabled = true
      metric_name                = "SQLiRuleSet"
      sampled_requests_enabled   = true
    }
  }

  # Rule 3: Known bad inputs (log4shell, Spring4Shell, etc.)
  rule {
    name     = "AWSManagedRulesKnownBadInputsRuleSet"
    priority = 3
    override_action { none {} }
    statement {
      managed_rule_group_statement {
        name        = "AWSManagedRulesKnownBadInputsRuleSet"
        vendor_name = "AWS"
      }
    }
    visibility_config {
      cloudwatch_metrics_enabled = true
      metric_name                = "KnownBadInputs"
      sampled_requests_enabled   = true
    }
  }

  # Rule 4: Rate limiting — 2000 req/5min per IP
  rule {
    name     = "RateLimitPerIP"
    priority = 4
    action { block {} }
    statement {
      rate_based_statement {
        limit              = 2000
        aggregate_key_type = "IP"
      }
    }
    visibility_config {
      cloudwatch_metrics_enabled = true
      metric_name                = "RateLimit"
      sampled_requests_enabled   = true
    }
  }

  visibility_config {
    cloudwatch_metrics_enabled = true
    metric_name                = "${var.env}-web-acl"
    sampled_requests_enabled   = true
  }
}

output "web_acl_arn" { value = aws_wafv2_web_acl.main.arn }

Associating WAF with Your Ingress

The WAF ACL ARN is injected per-cluster via the ApplicationSet and added as an ALB annotation:

# apps/myapp/templates/ingress.yaml
{{- if .Values.ingress.enabled }}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: {{ include "myapp.fullname" . }}
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443},{"HTTP":80}]'
    alb.ingress.kubernetes.io/ssl-redirect: "443"
    alb.ingress.kubernetes.io/certificate-arn: {{ .Values.ingress.certArn }}
    {{- if .Values.ingress.wafAclArn }}
    alb.ingress.kubernetes.io/wafv2-acl-arn: {{ .Values.ingress.wafAclArn }}
    {{- end }}

AWS GuardDuty — Threat Intelligence

GuardDuty operates at the AWS account level — it analyzes CloudTrail API logs, VPC Flow Logs, and DNS query logs using machine learning to identify threats.

# _modules/guardduty/main.tf

resource "aws_guardduty_detector" "main" {
  enable = true

  datasources {
    s3_logs {
      enable = true   # Detect unusual S3 access patterns
    }
    kubernetes {
      audit_logs {
        enable = true  # Monitor EKS audit logs for suspicious API calls
      }
    }
    malware_protection {
      scan_ec2_instance_with_findings {
        ebs_volumes {
          enable = true  # Scan EBS volumes when GuardDuty finds a threat
        }
      }
    }
  }
}

What GuardDuty Detects

Finding Type	Example
`CryptoCurrency:EC2/BitcoinTool.B!DNS`	EC2 instance querying known crypto mining pools
`UnauthorizedAccess:IAMUser/TorIPCaller`	API calls originating from Tor exit nodes
`CredentialAccess:Kubernetes/SuccessfulAnonymousAccess`	Anonymous access to Kubernetes API
`Execution:Kubernetes/ExecInKubernetes.Medium`	kubectl exec into a running pod (suspicious context)
`Exfiltration:S3/ObjectRead.Unusual`	Unusual S3 read patterns suggesting data theft

Attack Scenario: All Layers in Action

Here's how the security layers stop a real attack:

Scenario: Attacker finds a dependency with RCE vulnerability

1. Attacker discovers CVE-XXXX-1234 in one of your npm packages

   → Trivy: if the CVE is HIGH/CRITICAL, the build FAILS — image never pushed
     (Layer 1: Supply Chain)

2. If Trivy missed it (unfixed CVE) and image was pushed:
   Attacker triggers the RCE, gets command execution in the pod

   → Falco: "A shell was spawned in container myapp-abc123"
     Alert fires within 1 second to CloudWatch
     (Layer 3: Runtime Detection)

   → But wait — distroless has no /bin/sh to spawn
     Attacker needs a writable filesystem — which is also blocked
     (Layer 1: Distroless base)

3. Attacker tries to deploy a privileged pod to escape to the node:

   → Kyverno: BLOCKS the pod at admission — "Privileged containers not allowed"
     Pod never starts
     (Layer 2: Admission Control)

4. Attacker tries SQL injection via the public HTTP endpoint:

   → AWS WAF: blocks the request at the ALB
     Your pod code never executes the malicious query
     (Layer 4: Perimeter)

5. Attacker's stolen AWS key starts making API calls:

   → GuardDuty: unusual API call pattern detected
     Finding generated: "UnauthorizedAccess:IAMUser/AnomalousBehavior"
     (Layer 5: Threat Intelligence)

Summary

By the end of Part 8 you have:

✅ Kyverno 3.2.6 running on all 6 clusters (compatible with k8s 1.29)
✅ Five Kyverno policies enforcing: no privileged, no root, no hostPath, resource limits, signed images
✅ Falco DaemonSet monitoring all syscalls with eBPF driver
✅ Falco alerts flowing to CloudWatch via Fluent Bit
✅ AWS WAF WebACL with OWASP Top 10, SQLi, known bad inputs, and rate limiting
✅ GuardDuty enabled in all accounts with EKS audit log monitoring

Screenshot Placeholders

SCREENSHOT: AWS WAF console showing WebACL with managed rule groups and request metrics

SCREENSHOT: AWS GuardDuty console showing Findings summary (hopefully empty in production)

SCREENSHOT: kubectl get clusterpolicies showing all Kyverno policies as Ready

Next: Part 9 — Observability: Prometheus, Grafana, Fluent Bit, and CloudWatch

Part 7: Secrets Management

Matthew — Wed, 29 Apr 2026 08:00:00 +0000

Part 7: Secrets Management — AWS Secrets Manager + External Secrets Operator + IRSA

Part of the series: Building a Production-Grade DevSecOps Pipeline on AWS

Introduction

Secrets management is one of the areas where teams most commonly take shortcuts that later become security incidents. The three anti-patterns to avoid:

static credentials using IAM Roles for Service Accounts and OIDC token
exchange" width="800" height="370">

The pod never holds AWS credentials. It exchanges a short-lived Kubernetes
JWT (1 hour) for temporary IAM credentials via AWS STS, then ESO fetches
the secret and creates a Kubernetes Secret.

Secrets in Git — even in a private repo, any developer with access can read them; git history preserves them forever even after deletion
Secrets baked into images — anyone who pulls the image gets the secrets
Secrets in Kubernetes ConfigMaps — ConfigMaps are not encrypted by default; any pod in the namespace can read them

This pipeline uses a three-layer approach:

AWS Secrets Manager — the source of truth; all secrets live here, encrypted with KMS
External Secrets Operator (ESO) — a Kubernetes controller that fetches secrets from AWS and creates Kubernetes Secret objects
IRSA (IAM Roles for Service Accounts) — pod-level AWS identity so ESO can call Secrets Manager without node-level credentials

Architecture

┌──────────────────────────────────────────────────────────────────────┐
│  SECRETS FLOW                                                        │
│                                                                      │
│  AWS Secrets Manager                                                 │
│  production/myapp/db-password: "s3cr3t-v@lue"                        │
│         │                                                            │
│         │  GetSecretValue (every 1h)                                 │
│         │  Authenticated via IRSA (OIDC token → STS → temp creds)    │
│         ▼                                                            │
│  ESO Operator Pod                                                    │
│  (external-secrets namespace)                                        │
│         │                                                            │
│         │  Creates/updates                                           │
│         ▼                                                            │
│  Kubernetes Secret: myapp-db-password                                │
│  (myapp namespace)                                                   │
│         │                                                            │
│         │  Mounted as env var                                        │
│         ▼                                                            │
│  myapp Pod: process.env.DB_PASSWORD = "s3cr3t-v@lue"                 │
└──────────────────────────────────────────────────────────────────────┘

IRSA Deep Dive

IRSA (IAM Roles for Service Accounts) is the mechanism that gives individual Kubernetes pods fine-grained AWS IAM permissions without sharing credentials at the node level.

┌────────────────────────────────────────────────────────────────────┐
│  HOW IRSA WORKS                                                    │
│                                                                    │
│  1. EKS creates an OIDC provider for the cluster                   │
│     URL: oidc.eks.us-east-1.amazonaws.com/id/CLUSTER_ID            │
│                                                                    │
│  2. Pod's ServiceAccount is annotated:                             │
│     eks.amazonaws.com/role-arn: arn:aws:iam::ACCT:role/eso-role    │
│                                                                    │
│  3. EKS projects a signed OIDC JWT into the pod at:                │
│     /var/run/secrets/eks.amazonaws.com/serviceaccount/token        │
│                                                                    │
│  4. ESO SDK calls sts:AssumeRoleWithWebIdentity with that token    │
│                                                                    │
│  5. AWS validates: token signed by trusted OIDC provider?          │
│                    sub matches trust policy condition?             │
│                                                                    │
│  6. AWS returns temporary creds (AccessKeyId + SecretKey + Token)  │
│     Valid for 1 hour, then automatically expire                    │
│                                                                    │
│  Result: only THIS pod in THIS namespace with THIS SA can          │
│  assume the role — not any other pod on the same node              │
└────────────────────────────────────────────────────────────────────┘

Why IRSA over node-level IAM?

Node-level IAM (EC2 instance profile) gives every pod on the node the same permissions. If one pod is compromised, the attacker has the permissions of every pod on that node. IRSA scopes permissions to the exact service account — the blast radius of a compromise is just that one workload.

Terraform: ESO IRSA Module

# _modules/eso-irsa/main.tf

variable "cluster_name"      { type = string }
variable "env"               { type = string }
variable "oidc_provider_arn" { type = string }
variable "oidc_provider"     { type = string }  # URL without https://
variable "account_id"        { type = string }
variable "aws_region"        { type = string }

locals {
  # Helm fullname = {release}-{chart}
  # Release name = cluster name (e.g., myapp-production-use1)
  # Chart name = myapp
  # So the SA name = myapp-production-use1-myapp
  #
  # CRITICAL: The IRSA trust policy sub claim must match this exactly.
  # Getting this wrong = ESO pods cannot assume the role = secrets don't sync.
  sa_name   = "${var.cluster_name}-myapp"
  namespace = "external-secrets"
}

resource "aws_iam_role" "eso" {
  name = "${var.cluster_name}-eso"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Federated = var.oidc_provider_arn }
      Action    = "sts:AssumeRoleWithWebIdentity"
      Condition = {
        StringEquals = {
          "${var.oidc_provider}:aud" = "sts.amazonaws.com"
          # Must match: system:serviceaccount:{namespace}:{sa-name}
          "${var.oidc_provider}:sub" = "system:serviceaccount:${local.namespace}:${local.sa_name}"
        }
      }
    }]
  })
}

resource "aws_iam_role_policy" "eso" {
  name = "eso-secrets-policy"
  role = aws_iam_role.eso.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid    = "SecretsManagerRead"
        Effect = "Allow"
        Action = [
          "secretsmanager:GetSecretValue",
          "secretsmanager:DescribeSecret",
          "secretsmanager:ListSecretVersionIds"
        ]
        # Wildcard covers all secrets for this environment
        Resource = "arn:aws:secretsmanager:*:${var.account_id}:secret:${var.env}/myapp/*"
      }
    ]
  })
}

output "role_arn" { value = aws_iam_role.eso.arn }

Creating Secrets in AWS Secrets Manager

Use a structured naming convention: {env}/myapp/{secret-name}

# Development secrets (dev account)
aws secretsmanager create-secret \
  --name "dev/myapp/db-password" \
  --secret-string '{"password":"dev-db-secret-here"}' \
  --region us-east-1 \
  --profile myapp-dev-use1

# Production secrets (production account)
aws secretsmanager create-secret \
  --name "production/myapp/db-password" \
  --secret-string '{"password":"prod-super-secret-here"}' \
  --region us-east-1 \
  --profile myapp-prod-use1

# To update a secret (rotation):
aws secretsmanager update-secret \
  --secret-id "production/myapp/db-password" \
  --secret-string '{"password":"new-rotated-password"}' \
  --region us-east-1 \
  --profile myapp-prod-use1
# ESO picks up the new value within refreshInterval (1h default)

ESO Installation via ArgoCD

ESO runs as a controller in the external-secrets namespace on every cluster:

# infrastructure/eso/applicationset.yaml
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: external-secrets-operator
  namespace: argocd
spec:
  generators:
    - list:
        elements:
          - cluster: myapp-dev-use1
            region:  us-east-1
          - cluster: myapp-dev-usw2
            region:  us-west-2
          - cluster: myapp-staging-use1
            region:  us-east-1
          - cluster: myapp-staging-usw2
            region:  us-west-2
          - cluster: myapp-production-use1
            region:  us-east-1
          - cluster: myapp-production-usw2
            region:  us-west-2
  template:
    metadata:
      name: "eso-{{cluster}}"
    spec:
      project: production
      source:
        repoURL:        https://charts.external-secrets.io
        chart:          external-secrets
        targetRevision: "0.9.13"
        helm:
          values: |
            serviceAccount:
              annotations:
                eks.amazonaws.com/role-arn: "{{irsaRoleArn}}"
      destination:
        name:      "{{cluster}}"
        namespace: external-secrets
      syncPolicy:
        syncOptions: [CreateNamespace=true]

SecretStore CRD

The SecretStore tells ESO where to find secrets and how to authenticate:

# apps/myapp/templates/secretstore.yaml
{{- if .Values.externalSecrets.enabled }}
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets-manager
  namespace: {{ .Release.Namespace }}
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1   # Always pull from us-east-1 (single source of truth)
      auth:
        jwt:
          serviceAccountRef:
            name: {{ include "myapp.serviceAccountName" . }}
            # DO NOT add namespace: field here.
            # Namespaced SecretStore rejects serviceAccountRef.namespace.
            # The SA is in the same namespace as the SecretStore.
{{- end }}

Lesson learned: A namespaced SecretStore (not ClusterSecretStore) will reject the config if serviceAccountRef.namespace is specified. The namespace is implicit — it's the same namespace as the SecretStore itself. Only ClusterSecretStore supports cross-namespace references.

ExternalSecret CRD

The ExternalSecret tells ESO which secret to fetch and what Kubernetes Secret to create:

# apps/myapp/templates/external-secret.yaml
{{- if .Values.externalSecrets.enabled }}
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: myapp-secrets
  namespace: {{ .Release.Namespace }}
spec:
  refreshInterval: {{ .Values.externalSecrets.refreshInterval | default "1h" }}

  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore

  target:
    name: myapp-db-credentials    # Name of the Kubernetes Secret to create
    creationPolicy: Owner         # ESO owns this secret — it will delete it if ExternalSecret is deleted

  data:
    - secretKey: DB_PASSWORD       # Key in the Kubernetes Secret
      remoteRef:
        key: production/myapp/db-password   # Secret name in AWS Secrets Manager
        property: password                  # JSON key within the secret value
{{- end }}

Helm Chart Integration

# apps/myapp/values.yaml (defaults — disabled)
externalSecrets:
  enabled:         false
  refreshInterval: 1h
  irsaRoleArn:     ""

# apps/myapp/values-production.yaml
externalSecrets:
  enabled:     true
  irsaRoleArn: ""   # Injected per-cluster via ApplicationSet parameter

# apps/myapp/values-dev.yaml
externalSecrets:
  enabled:     true
  irsaRoleArn: "arn:aws:iam::557702566877:role/myapp-dev-eso"

The ApplicationSet injects the correct IRSA role ARN per cluster via parameters:

parameters:
  - name: "externalSecrets.irsaRoleArn"
    value: "{{irsaRoleArn}}"

Deployment Mounting the Secret

# apps/myapp/templates/deployment.yaml (relevant section)
env:
  - name: DB_PASSWORD
    valueFrom:
      secretKeyRef:
        name: myapp-db-credentials   # Created by ESO from ExternalSecret
        key: DB_PASSWORD

Verification

# Check ExternalSecret sync status
kubectl --context prod-use1 get externalsecret -n myapp
# NAME            STORE                 REFRESH INTERVAL   STATUS          READY
# myapp-secrets   aws-secrets-manager   1h                 SecretSynced    True

# Check the Kubernetes Secret was created
kubectl --context prod-use1 get secret myapp-db-credentials -n myapp
# NAME                   TYPE     DATA   AGE
# myapp-db-credentials   Opaque   1      5m

# View (base64 encoded — decode to see value)
kubectl --context prod-use1 get secret myapp-db-credentials -n myapp \
  -o jsonpath='{.data.DB_PASSWORD}' | base64 -d

ArgoCD false positive: ArgoCD will show the myapp Application as OutOfSync even when everything is working. This is because ESO writes status.refreshTime to the ExternalSecret at every sync cycle. ArgoCD detects this runtime write as a diff from the Git-defined manifest. This is a known, safe false positive — the secret IS syncing correctly.

Secret Rotation

When you need to rotate a secret:

# 1. Update in AWS Secrets Manager
aws secretsmanager update-secret \
  --secret-id "production/myapp/db-password" \
  --secret-string '{"password":"new-rotated-value"}' \
  --profile myapp-prod-use1

# 2. ESO picks it up automatically within refreshInterval (1h)
# OR force immediate refresh:
kubectl --context prod-use1 annotate externalsecret myapp-secrets -n myapp \
  force-sync=$(date +%s) --overwrite

# 3. Kubernetes Secret is updated
# 4. Pods need restart to pick up new env var value:
kubectl --context prod-use1 rollout restart deployment/myapp -n myapp

Grafana Admin Secret — A Real-World Lesson

During this build, we used existingSecret: grafana-admin-secret in the kube-prometheus-stack values. The secret was created with a placeholder password YourStrongPassword123!. Grafana initialized its SQLite database with this value when the pod first started.

Later, the secret value was changed — but Grafana's database still had the old value. The env var GF_SECURITY_ADMIN_PASSWORD only sets the initial password; it cannot change an existing one.

Fix:

kubectl exec -n monitoring <grafana-pod> -c grafana -- \
  grafana-cli admin reset-admin-password 'NewPassword!'

Lesson: For any tool that reads credentials once at database initialization, changing the Kubernetes Secret is not enough. You must either reset via the tool's CLI or destroy and recreate the persistent volume.

Summary

By the end of Part 7 you have:

✅ All secrets stored exclusively in AWS Secrets Manager (never in Git or images)
✅ ESO installed on all 6 clusters via ArgoCD ApplicationSet
✅ IRSA roles per cluster with least-privilege Secrets Manager read access
✅ SecretStore + ExternalSecret CRDs syncing secrets into Kubernetes
✅ Helm chart values.yaml integration with enabled/disabled toggle per environment
✅ Secret rotation workflow (update in ASM → ESO auto-refreshes)

Screenshot Placeholders

SCREENSHOT: AWS Secrets Manager console showing secrets per environment

SCREENSHOT: kubectl get externalsecret showing SecretSynced: True

SCREENSHOT: ArgoCD showing ESO app as OutOfSync/Healthy (expected false positive)

Next: Part 8 — Security Stack: Kyverno, Falco, WAF, and GuardDuty

Part 6: CI/CD Pipeline

Matthew — Wed, 15 Apr 2026 14:00:00 +0000

Part 6: CI/CD Pipeline — GitHub Actions, Trivy, Cosign, and ECR

Part of the series: Building a Production-Grade DevSecOps Pipeline on AWS

Introduction

The CI/CD pipeline is the gateway between a developer's git push and a running container in production. Every security control that can be automated should live here — not as an afterthought but as a first-class gate that blocks bad artifacts from ever reaching a cluster.

This pipeline enforces:

No HIGH or CRITICAL CVEs — Trivy blocks the build before the image is pushed
No static AWS credentials — GitHub OIDC exchanges JWT tokens for temporary STS creds
Cryptographic image provenance — Cosign signs every image with an AWS KMS key; Kyverno verifies the signature before admitting the pod
Immutable image tags — sha-<full-commit-sha>, never :latest
Audit trail — every push is logged to S3 with digest, timestamp, and caller identity

The Application (myapp)

A minimal Node.js/Express API that represents any real production service:

myapp/
├── src/
│   └── index.js        # Express app with /health and /metrics
├── Dockerfile
├── package.json
└── .github/
    └── workflows/
        └── ci.yaml

// src/index.js
const express = require('express');
const promClient = require('prom-client');

const app = express();
const register = new promClient.Registry();
promClient.collectDefaultMetrics({ register });

const httpRequestsTotal = new promClient.Counter({
  name: 'myapp_http_requests_total',
  help: 'Total HTTP requests',
  labelNames: ['method', 'route', 'status_code'],
  registers: [register],
});

app.use((req, res, next) => {
  res.on('finish', () => {
    httpRequestsTotal.inc({
      method: req.method,
      route: req.path,
      status_code: res.statusCode,
    });
  });
  next();
});

app.get('/health', (req, res) => {
  res.json({ status: 'healthy', region: process.env.AWS_REGION || 'unknown' });
});

app.get('/metrics', async (req, res) => {
  res.set('Content-Type', register.contentType);
  res.end(await register.metrics());
});

app.listen(8080, () => console.log('Listening on :8080'));

Dockerfile — Distroless Nonroot

# Stage 1: Build dependencies
FROM node:18-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production

# Stage 2: Runtime — distroless (no shell, no package manager)
FROM gcr.io/distroless/nodejs18-debian12:nonroot

WORKDIR /app

# Copy only what's needed — no node_modules dev dependencies, no source maps
COPY --from=builder /app/node_modules ./node_modules
COPY src/ ./src/
COPY package.json ./

# nonroot image runs as uid 65532 by default — no root, ever
EXPOSE 8080

CMD ["src/index.js"]

Why distroless?
The nonroot variant contains only the Node.js runtime. There is no:

/bin/sh or /bin/bash — an attacker with RCE cannot spawn an interactive shell
apt, apk, yum — cannot install additional tools
curl, wget — cannot exfiltrate data or download payloads
Any other utility that would help lateral movement

Falco still alerts on any unexpected syscalls, but the attack surface is dramatically reduced.

Pipeline Overview

including OIDC auth, Trivy scanning, Cosign signing, and ArgoCD GitOps
deployment" width="800" height="295">

The complete pipeline: no static AWS credentials anywhere. OIDC authenticates
GitHub Actions to AWS. Every image is signed with Cosign before Kyverno will
admit it to the cluster.

┌─────────────────────────────────────────────────────────────────────────┐
│  git push → main                                                        │
│                    │                                                    │
│                    ▼                                                    │
│         ┌──────────────────┐                                            │
│         │  Job 1: test     │  npm ci + npm test                         │
│         └────────┬─────────┘                                            │
│                  │ needs: test                                          │
│                  ▼                                                      │
│         ┌──────────────────┐                                            │
│         │  Job 2: scan     │  trivy image → fail on HIGH/CRITICAL       │
│         └────────┬─────────┘                                            │
│                  │ needs: scan                                          │
│                  ▼                                                      │
│         ┌──────────────────────────────────────────────┐                │
│         │  Job 3: build-push-sign                      │                │
│         │  ├─ OIDC → assume IAM role (no static keys)  │                │
│         │  ├─ docker build (distroless)                │                │
│         │  ├─ push → ECR us-east-1                     │                │
│         │  ├─ push → ECR us-west-2                     │                │
│         │  ├─ cosign sign (AWS KMS)                    │                │
│         │  └─ S3 audit log                             │                │
│         └────────┬─────────────────────────────────────┘                │
│                  │ needs: build-push-sign                               │
│                  ▼                                                      │
│         ┌──────────────────────────────────────────────┐                │
│         │  Job 4: update-gitops                        │                │
│         │  └─ patch image.tag in values-*.yaml → push  │                │
│         └──────────────────────────────────────────────┘                │
└─────────────────────────────────────────────────────────────────────────┘

Full GitHub Actions Workflow

# .github/workflows/ci.yaml
name: CI

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

env:
  ECR_REGISTRY_USE1: 206617159586.dkr.ecr.us-east-1.amazonaws.com
  ECR_REGISTRY_USW2: 206617159586.dkr.ecr.us-west-2.amazonaws.com
  IMAGE_NAME: myapp
  AWS_REGION_USE1: us-east-1
  AWS_REGION_USW2: us-west-2

permissions:
  id-token: write   # Required for OIDC
  contents: write   # Required for gitops update commit
  packages: read

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '18'
          cache: 'npm'

      - name: Install dependencies
        run: npm ci

      - name: Run tests
        run: npm test

  scan:
    needs: test
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Build image for scanning
        run: |
          docker build -t ${{ env.IMAGE_NAME }}:scan-${{ github.sha }} .

      - name: Trivy vulnerability scan
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: ${{ env.IMAGE_NAME }}:scan-${{ github.sha }}
          format: table
          exit-code: '1'              # Fail the pipeline on findings
          severity: 'HIGH,CRITICAL'   # Only fail on HIGH and CRITICAL
          ignore-unfixed: true        # Skip CVEs with no available fix

  build-push-sign:
    needs: scan
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'  # Only push on main branch, not PRs
    outputs:
      image-digest-use1: ${{ steps.push-use1.outputs.digest }}
    steps:
      - uses: actions/checkout@v4

      - name: Configure AWS credentials via OIDC
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ vars.AWS_ROLE_ARN_USE1 }}
          aws-region: ${{ env.AWS_REGION_USE1 }}
          # No static access keys — OIDC exchanges GitHub JWT for STS temp creds

      - name: Login to ECR us-east-1
        uses: aws-actions/amazon-ecr-login@v2
        with:
          region: ${{ env.AWS_REGION_USE1 }}

      - name: Login to ECR us-west-2
        uses: aws-actions/amazon-ecr-login@v2
        with:
          region: ${{ env.AWS_REGION_USW2 }}

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Build and push to us-east-1
        id: push-use1
        uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: |
            ${{ env.ECR_REGISTRY_USE1 }}/${{ env.IMAGE_NAME }}:sha-${{ github.sha }}
            ${{ env.ECR_REGISTRY_USE1 }}/${{ env.IMAGE_NAME }}:latest
          cache-from: type=gha
          cache-to: type=gha,mode=max

      - name: Copy image to us-west-2
        run: |
          DIGEST="${{ steps.push-use1.outputs.digest }}"
          docker buildx imagetools create \
            --tag ${{ env.ECR_REGISTRY_USW2 }}/${{ env.IMAGE_NAME }}:sha-${{ github.sha }} \
            ${{ env.ECR_REGISTRY_USE1 }}/${{ env.IMAGE_NAME }}@${DIGEST}

      - name: Install Cosign
        uses: sigstore/cosign-installer@v3

      - name: Sign image with AWS KMS
        env:
          COSIGN_KEY: ${{ vars.COSIGN_KMS_KEY_ARN }}
        run: |
          DIGEST="${{ steps.push-use1.outputs.digest }}"
          IMAGE="${{ env.ECR_REGISTRY_USE1 }}/${{ env.IMAGE_NAME }}@${DIGEST}"

          # Sign the image — creates an OCI attestation artifact in ECR
          cosign sign --key awskms:///${COSIGN_KEY} \
            --yes \
            ${IMAGE}

          echo "Signed: ${IMAGE}"

      - name: Write S3 audit log
        run: |
          DIGEST="${{ steps.push-use1.outputs.digest }}"
          TIMESTAMP=$(date -u +%Y-%m-%dT%H:%M:%SZ)
          CALLER=$(aws sts get-caller-identity --query Arn --output text)

          echo "{
            \"timestamp\": \"${TIMESTAMP}\",
            \"repo\": \"${{ github.repository }}\",
            \"sha\": \"${{ github.sha }}\",
            \"digest\": \"${DIGEST}\",
            \"pushed_by\": \"${CALLER}\",
            \"workflow\": \"${{ github.workflow }}\",
            \"run_id\": \"${{ github.run_id }}\"
          }" | aws s3 cp - \
            s3://${{ vars.AUDIT_BUCKET }}/ci-push-audit/${{ github.sha }}.json

  update-gitops:
    needs: build-push-sign
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'
    steps:
      - name: Checkout myapp-gitops
        uses: actions/checkout@v4
        with:
          repository: MatthewDipo/myapp-gitops
          token: ${{ secrets.GITOPS_PAT }}
          path: myapp-gitops

      - name: Update image tags
        run: |
          cd myapp-gitops
          NEW_TAG="sha-${{ github.sha }}"

          # Update all environment value files
          for FILE in apps/myapp/values-dev.yaml \
                      apps/myapp/values-staging.yaml \
                      apps/myapp/values-production.yaml; do
            sed -i "s|tag: sha-[a-f0-9]*|tag: ${NEW_TAG}|g" $FILE
            echo "Updated $FILE → ${NEW_TAG}"
          done

      - name: Commit and push
        run: |
          cd myapp-gitops
          git config user.email "ci@github.com"
          git config user.name "GitHub Actions"
          git add apps/myapp/values-*.yaml
          git commit -m "ci: update image tag to sha-${{ github.sha }}"
          git push origin main

GitHub Repository Variables (not secrets)

These are set in the GitHub UI under Settings → Secrets and variables → Actions → Variables:

Variable	Value	Why not a secret?
`AWS_ROLE_ARN_USE1`	`arn:aws:iam::206617159586:role/myapp-dev-use1-github-ci`	Not sensitive — it's a role ARN, useless without the OIDC token
`COSIGN_KMS_KEY_ARN`	`arn:aws:kms:us-east-1:206617159586:key/...`	Not sensitive — KMS key ID alone grants nothing
`AUDIT_BUCKET`	`myapp-ci-audit-206617159586`	Not sensitive

Only GITOPS_PAT (GitHub Personal Access Token to push to myapp-gitops) is a secret.

No AWS_ACCESS_KEY_ID. No AWS_SECRET_ACCESS_KEY. These should never appear in a modern CI pipeline.

Cosign Image Signing

Cosign attaches a cryptographic signature to the image in the same ECR repository. The signature is stored as a separate OCI artifact tagged with the image digest.

ECR Repository: myapp

sha-abc123def456...          ← Your application image
sha256-abc123def456...sig    ← Cosign signature (OCI artifact)
sha256-abc123def456...att    ← Cosign attestation (SBOM, etc.)

Verification (what Kyverno does at admission time):

cosign verify \
  --key awskms:///arn:aws:kms:us-east-1:206617159586:key/YOUR_KEY_ID \
  206617159586.dkr.ecr.us-east-1.amazonaws.com/myapp:sha-abc123

# Output if valid:
# Verification for 206617159586.dkr.ecr.us-east-1.amazonaws.com/myapp:sha-abc123
# The following checks were performed on each of these signatures:
#   - The cosign claims were validated
#   - The signatures were verified against the specified public key

Image Naming Convention

206617159586.dkr.ecr.us-east-1.amazonaws.com/myapp:sha-<full-40-char-git-sha>

Why sha-<full-sha> and not semantic versioning?

Traceability: Given any running pod, you can run git show <sha> to see the exact commit
Immutability: Two builds of the same SHA produce the same image (deterministic builds)
No tag collisions: v1.0.0 can be overwritten; a git SHA cannot
No :latest: :latest is the devil — it means different things at different times and breaks reproducibility

Pre-commit Hooks

Before code ever reaches GitHub, pre-commit hooks catch common issues locally:

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.5.0
    hooks:
      - id: trailing-whitespace
      - id: end-of-file-fixer
      - id: check-yaml
      - id: check-merge-conflict

  - repo: https://github.com/hadolint/hadolint
    rev: v2.12.0
    hooks:
      - id: hadolint-docker
        args: ['--ignore', 'DL3006']   # DL3006: always tag FROM image

  - repo: https://github.com/Yelp/detect-secrets
    rev: v1.5.0
    hooks:
      - id: detect-secrets
        args: ['--baseline', '.secrets.baseline']

# Install pre-commit
pip install pre-commit
pre-commit install

# Run manually against all files
pre-commit run --all-files

Pipeline Security Properties

Property	How Achieved
No secrets in Git	detect-secrets pre-commit hook
No secrets in CI	OIDC replaces static AWS keys
No HIGH/CRITICAL CVEs	Trivy scan blocks pipeline
No unverified images	Kyverno admission webhook
Immutable artifacts	ECR `IMMUTABLE` tag mutability
Cryptographic provenance	Cosign + AWS KMS signing
Full audit trail	S3 + CloudTrail
Reproducible builds	Pinned base image SHA, `npm ci`

Summary

By the end of Part 6 you have:

✅ A secure Dockerfile using distroless/nonroot with multi-stage build
✅ GitHub Actions pipeline with 4 jobs: test → scan → build/push/sign → gitops update
✅ OIDC-based AWS authentication (zero static credentials)
✅ Trivy CVE scanning blocking HIGH/CRITICAL vulnerabilities
✅ Cosign image signing with AWS KMS
✅ Automatic image tag update in myapp-gitops triggering ArgoCD sync
✅ S3 audit log for every image push
✅ Pre-commit hooks catching issues before they reach CI

Screenshot Placeholders

SCREENSHOT: GitHub Actions — workflow run showing all 4 jobs passing (green checkmarks)

SCREENSHOT: ECR repository showing images with sha- tags and scan results (no HIGH/CRITICAL)

SCREENSHOT: GitHub Actions — Job 3 logs showing "Signed: ..." cosign output

Next: Part 7 — Secrets Management: AWS Secrets Manager + External Secrets Operator + IRSA

Part 5: GitOps with ArgoCD

Matthew — Thu, 09 Apr 2026 23:50:20 +0000

5: GitOps with ArgoCD — Hub-Spoke Model

Part of the series: Building a Production-Grade DevSecOps Pipeline on AWS

Introduction

GitOps flips the traditional CI/CD model. Instead of a pipeline pushing manifests into a cluster, the cluster pulls its desired state from Git. The result:

Audit trail built-in: every cluster change is a Git commit with author, timestamp, and diff
Self-healing: ArgoCD continuously reconciles — if someone kubectl applys something manually, ArgoCD reverts it within minutes
Rollback is git revert: no special tooling, no cluster access needed
Drift detection: ArgoCD shows you exactly when a cluster diverges from what's in Git

This pipeline uses ArgoCD in a hub-spoke topology: one ArgoCD installation on myapp-production-use1 manages all six clusters.

┌─────────────────────────────────────────────────────────────────┐
│          ArgoCD HUB: myapp-production-use1                      │
│                                                                 │
│  Watches: github.com/MatthewDipo/myapp-gitops (main branch)     │
│  Manages: 6 clusters via registered cluster credentials         │
│                                                                 │
│  ApplicationSets generate Applications per cluster:             │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │  environments/production/applicationset.yaml             │   │
│  │  → prometheus-myapp-production-use1                      │   │
│  │  → prometheus-myapp-production-usw2                      │   │
│  │  → prometheus-myapp-staging-use1    (staging project)    │   │
│  │  → prometheus-myapp-staging-usw2                         │   │
│  └──────────────────────────────────────────────────────────┘   │
└────────────────────────────┬────────────────────────────────────┘
         VPC Peering (private │ endpoints)
    ┌────────────────────────┤
    │    ┌───────────────────┤
    │    │    ┌──────────────┤──────────────┐
    ▼    ▼    ▼              ▼              ▼
prod-usw2  staging-use1  staging-usw2  dev-use1  dev-usw2

Why hub-spoke over running ArgoCD in every cluster?
One ArgoCD = one UI, one set of secrets, one audit log. Running six ArgoCD instances means six independent control planes to maintain, upgrade, and monitor. The operational overhead multiplies linearly with clusters.

Step 1: Install ArgoCD on the Hub

# Enable public endpoint on prod-use1 for bootstrapping
aws eks update-cluster-config \
  --name myapp-production-use1 \
  --region us-east-1 --profile myapp-prod-use1 \
  --resources-vpc-config endpointPublicAccess=true,endpointPrivateAccess=true,publicAccessCidrs="0.0.0.0/0"

sleep 180  # Wait for propagation

# Install ArgoCD
kubectl --context prod-use1 create namespace argocd

helm repo add argo https://argoproj.github.io/argo-helm
helm install argocd argo/argo-cd \
  --namespace argocd \
  --version 6.7.3 \
  --set server.service.type=LoadBalancer \
  --set configs.params."server\.insecure"=true \
  --no-hooks \
  --wait --timeout 5m

# Get the LoadBalancer URL
kubectl --context prod-use1 get svc -n argocd argocd-server \
  -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'

# Get initial admin password
kubectl --context prod-use1 get secret -n argocd argocd-initial-admin-secret \
  -o jsonpath='{.data.password}' | base64 -d && echo

# Login via CLI
argocd login <LB_HOSTNAME> --username admin --password <PASSWORD> --insecure

Step 2: Create AppProjects

AppProjects define what each group of Applications is allowed to do — which source repos, which destination clusters, and which namespaces. They are the RBAC boundary in ArgoCD.

# argocd/project-production.yaml
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: production
  namespace: argocd
spec:
  description: Production workloads and infrastructure

  sourceRepos:
    - https://github.com/MatthewDipo/myapp-gitops.git
    - https://prometheus-community.github.io/helm-charts
    - https://charts.external-secrets.io
    - https://aws.github.io/eks-charts
    - https://kyverno.github.io/kyverno
    - https://falcosecurity.github.io/charts
    - https://vmware-tanzu.github.io/helm-charts
    - https://charts.jetstack.io
    - https://grafana.github.io/helm-charts

  destinations:
    - server: "*"       # IMPORTANT: use server: "*" not name: "*"
      namespace: "*"    # ArgoCD resolves cluster name → server URL for permission checks

  clusterResourceWhitelist:
    - group: "*"
      kind: "*"

  namespaceResourceWhitelist:
    - group: "*"
      kind: "*"

Critical lesson: Use server: "*" not name: "*" in AppProject destinations. ArgoCD resolves cluster names to server URLs when enforcing project permissions. With name: "*" alone, syncs to named clusters fail with permission errors. With server: "*", it works correctly.

Apply all three projects:

kubectl --context prod-use1 apply -f argocd/project-dev.yaml
kubectl --context prod-use1 apply -f argocd/project-staging.yaml
kubectl --context prod-use1 apply -f argocd/project-production.yaml

Step 3: Add the Private GitOps Repo

ArgoCD needs credentials to pull from a private repository:

argocd repo add https://github.com/MatthewDipo/myapp-gitops.git \
  --username MatthewDipo \
  --password <GITHUB_PAT> \
  --insecure-skip-server-verification

# Verify
argocd repo list

Step 4: Register Spoke Clusters

Each spoke cluster is registered by running argocd cluster add with the cluster's kubeconfig context. This creates a ServiceAccount and ClusterRoleBinding in the spoke cluster that ArgoCD uses to manage resources.

# Dev clusters (public endpoints — accessible directly)
argocd cluster add dev-use1 --name myapp-dev-use1 --yes
argocd cluster add dev-usw2 --name myapp-dev-usw2 --yes

# Private clusters — must temporarily enable public endpoint first
for CLUSTER in myapp-staging-use1 myapp-staging-usw2 myapp-production-usw2; do
  REGION=$(echo $CLUSTER | grep -q "use1" && echo "us-east-1" || echo "us-west-2")
  ENV=$(echo $CLUSTER | cut -d- -f2)
  PROFILE="myapp-${ENV}-${REGION//us-east-1/use1}-${REGION//us-west-2/usw2}"

  echo "Enabling public endpoint on $CLUSTER..."
  aws eks update-cluster-config --name $CLUSTER --region $REGION \
    --profile myapp-${ENV}-$(echo $REGION | sed 's/us-east-1/use1/;s/us-west-2/usw2/') \
    --resources-vpc-config endpointPublicAccess=true,endpointPrivateAccess=true,publicAccessCidrs="0.0.0.0/0"
  sleep 180

  CTX=$(echo $CLUSTER | sed 's/myapp-//;s/-use1/-use1/;s/-usw2/-usw2/')
  argocd cluster add $CTX --name $CLUSTER --yes

  echo "Locking $CLUSTER back to private..."
  aws eks update-cluster-config --name $CLUSTER --region $REGION \
    --profile myapp-${ENV}-$(echo $REGION | sed 's/us-east-1/use1/;s/us-west-2/usw2/') \
    --resources-vpc-config endpointPublicAccess=false,endpointPrivateAccess=true
done

# Verify all clusters registered
argocd cluster list

Expected output:

SERVER                          NAME                      VERSION  STATUS
https://kubernetes.default.svc  in-cluster                1.29     Successful
<spoke-endpoint>                myapp-dev-use1            1.29     Successful
<spoke-endpoint>                myapp-dev-usw2            1.29     Successful
<spoke-endpoint>                myapp-staging-use1        1.29     Successful
<spoke-endpoint>                myapp-staging-usw2        1.29     Successful
<spoke-endpoint>                myapp-production-usw2     1.29     Successful

spoke clusters via VPC Peering, with dev clusters on public endpoints" width="800" height="429">

Solid lines = VPC Peering (private). Dashed lines = public endpoints (dev only).
The hub cluster runs ArgoCD and manages 47 Applications across all 6 clusters.

Step 5: ApplicationSet — The Core of Hub-Spoke GitOps

An ApplicationSet is a controller that generates ArgoCD Applications from a template and a generator. The list generator creates one Application per element in the list — one per cluster.

# environments/production/applicationset.yaml
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: myapp-production
  namespace: argocd
spec:
  generators:
    - list:
        elements:
          - cluster:    myapp-production-use1
            region:     us-east-1
            certArn:    "arn:aws:acm:us-east-1:591120834781:certificate/9ab022c9-..."
            wafAclArn:  "arn:aws:wafv2:us-east-1:591120834781:regional/webacl/..."
            irsaRoleArn: "arn:aws:iam::591120834781:role/myapp-production-use1-eso"
          - cluster:    myapp-production-usw2
            region:     us-west-2
            certArn:    "arn:aws:acm:us-west-2:591120834781:certificate/171cac9d-..."
            wafAclArn:  "arn:aws:wafv2:us-west-2:591120834781:regional/webacl/..."
            irsaRoleArn: "arn:aws:iam::591120834781:role/myapp-production-usw2-eso"

  template:
    metadata:
      name: "myapp-production-{{cluster}}"
      labels:
        component: application
        environment: production
        region: "{{region}}"
    spec:
      project: production
      sources:
        # Source 1: the Helm chart (from gitops repo)
        - repoURL:        https://github.com/MatthewDipo/myapp-gitops.git
          targetRevision: main
          ref: gitopsValues   # Named reference — used in source 2

        # Source 2: Helm chart with values from the gitops repo
        - repoURL:        https://github.com/MatthewDipo/myapp-gitops.git
          targetRevision: main
          path:           apps/myapp
          helm:
            valueFiles:
              - $gitopsValues/apps/myapp/values.yaml
              - $gitopsValues/apps/myapp/values-production.yaml
            parameters:
              - name:  "image.tag"
                value: "sha-45d92fc5ffd4555caf35b996ed1eec4e45152dce"
              - name:  "env.AWS_REGION"
                value: "{{region}}"
              - name:  "ingress.certArn"
                value: "{{certArn}}"
              - name:  "ingress.wafAclArn"
                value: "{{wafAclArn}}"
              - name:  "externalSecrets.irsaRoleArn"
                value: "{{irsaRoleArn}}"

      destination:
        name:      "{{cluster}}"
        namespace: myapp

      syncPolicy:
        automated:
          prune:    true    # Remove resources deleted from Git
          selfHeal: true    # Revert manual kubectl changes
        syncOptions:
          - CreateNamespace=true
          - ServerSideApply=true
        retry:
          limit: 3
          backoff:
            duration:    30s
            maxDuration: 10m
            factor:      2

Key ApplicationSet concepts:

automated.prune: true — If you delete a file from Git, ArgoCD deletes the corresponding Kubernetes resource. Without this, deleted manifests leave orphaned resources.

automated.selfHeal: true — If someone runs kubectl edit and changes something, ArgoCD detects the drift and reverts it within ~3 minutes. This enforces Git as the only source of truth.

ServerSideApply: true — Required for CRDs and resources that multiple controllers manage (e.g., Prometheus operator patches its own webhook config). Without this, field ownership conflicts cause sync failures.

retry — Network blips or transient API errors shouldn't fail a deployment permanently. The retry policy handles intermittent failures automatically.

Step 6: Deploy the ApplicationSets

# Apply all ApplicationSets from the hub cluster
kubectl --context prod-use1 apply -f environments/dev/applicationset.yaml
kubectl --context prod-use1 apply -f environments/staging/applicationset.yaml
kubectl --context prod-use1 apply -f environments/production/applicationset.yaml

# Infrastructure ApplicationSets
kubectl --context prod-use1 apply -f infrastructure/eso/applicationset.yaml
kubectl --context prod-use1 apply -f infrastructure/kyverno/applicationset.yaml
kubectl --context prod-use1 apply -f infrastructure/falco/applicationset.yaml
kubectl --context prod-use1 apply -f infrastructure/monitoring/applicationset.yaml
kubectl --context prod-use1 apply -f infrastructure/logging/applicationset.yaml
kubectl --context prod-use1 apply -f infrastructure/velero/applicationset.yaml
kubectl --context prod-use1 apply -f infrastructure/karpenter/applicationset.yaml
kubectl --context prod-use1 apply -f infrastructure/argo-rollouts/applicationset.yaml

# Watch sync status
argocd app list

Sync Status and Known False Positives

After deploying, most apps show Synced/Healthy. A few will permanently show OutOfSync/Healthy — this is expected and safe:

App	Why OutOfSync	Safe?
ESO (`external-secrets-*`)	ESO writes `status.refreshTime` at runtime; ArgoCD sees this as drift	✅ Yes
`myapp-*` (with ESO)	Same ESO status drift propagates to the parent app	✅ Yes
`prometheus-*`	Prometheus operator patches its own ValidatingWebhookConfiguration at runtime	✅ Yes

Real sync failures (need action):

SyncFailed — usually a YAML error or missing CRD
Degraded — pods crashing; check kubectl describe pod
Missing — ArgoCD can't reach the cluster

Troubleshooting

Orphaned resources after a Helm release name change:

# Force prune all resources not in current Git state
argocd app sync argocd/karpenter-myapp-production-use1 --prune --force

App stuck in Progressing:

argocd app get argocd/myapp-production-myapp-production-use1
# Check the "Conditions" and "Events" sections

Cluster shows Unknown:

# Re-register the cluster (token may have expired)
argocd cluster add prod-usw2 --name myapp-production-usw2 --yes

AppProject permission denied:

# Ensure project destinations include server: "*"
# Ensure sourceRepos includes the chart repo URL
kubectl --context prod-use1 edit appproject production -n argocd

VPC Peering for Hub → Spoke Connectivity

ArgoCD hub reaches spoke private endpoints via VPC peering. Three peering connections:

prod-use1 VPC (10.20.0.0/16) ◄──────► prod-usw2 VPC   (10.21.0.0/16)
prod-use1 VPC (10.20.0.0/16) ◄──────► staging-use1 VPC (10.10.0.0/16)
prod-use1 VPC (10.20.0.0/16) ◄──────► staging-usw2 VPC (10.11.0.0/16)

After the peering is active, ArgoCD can communicate with spoke API servers on their private IPs (10.x.x.x:443) without any traffic touching the internet.

Summary

By the end of Part 5 you have:

✅ ArgoCD running on the hub cluster with a public LoadBalancer URL
✅ Three AppProjects (dev, staging, production) with correct source and destination restrictions
✅ All 5 spoke clusters registered (ArgoCD SA + ClusterRoleBinding installed in each)
✅ ApplicationSets generating Applications for all environments and infrastructure components
✅ GitOps loop closed: a commit to myapp-gitops triggers automatic sync to all clusters

Screenshot Placeholders

SCREENSHOT: ArgoCD UI — Applications view showing all apps, most Synced/Healthy
## Part

SCREENSHOT: ArgoCD UI — Cluster list showing all 6 clusters registered and Successful

SCREENSHOT: ArgoCD UI — One Application detail view showing resource tree

SCREENSHOT: GitOps Repo Structure on GitHub

Next: Part 6 — CI/CD Pipeline: GitHub Actions, Trivy, Cosign, ECR

Part 4: EKS Multi-Cluster Setup

Matthew — Wed, 01 Apr 2026 13:00:00 +0000

Part 4: EKS Multi-Cluster Setup — Six Clusters Across Two Regions

Part of the series: Building a Production-Grade DevSecOps Pipeline on AWS

Introduction

Why six clusters instead of one? The answer is isolation and resilience:

┌──────────────────────────────────────────────────────────────────────────┐
│  ONE CLUSTER (anti-pattern)                                              │
│                                                                          │
│  Dev pods → same etcd as Production pods                                 │
│  A misconfigured dev deployment can consume all cluster resources        │
│  Cluster upgrade = every environment goes down simultaneously            │
│  Cost visibility: impossible to attribute spend per environment          │
└──────────────────────────────────────────────────────────────────────────┘

┌────────────────────────────────────────────────────────────────────────────┐
│  SIX CLUSTERS (this guide)                                                 │
│                                                                            │
│  myapp-dev-use1      (us-east-1, public endpoint,  2 nodes)                │
│  myapp-dev-usw2      (us-west-2, public endpoint,  2 nodes)                │
│  myapp-staging-use1  (us-east-1, private endpoint, 2 nodes)                │
│  myapp-staging-usw2  (us-west-2, private endpoint, 2 nodes)                │
│  myapp-production-use1 (us-east-1, private endpoint, 2+ nodes + Karpenter) │
│  myapp-production-usw2 (us-west-2, private endpoint, 2+ nodes + Karpenter) │
│                                                                            │
│  Benefits:                                                                 │
│  ✓ Complete IAM isolation between environments                             │
│  ✓ Production upgrade independent of dev                                   │
│  ✓ Regional failover — us-east-1 outage → us-west-2 serves traffic         │
│  ✓ Clear cost attribution per cluster tag                                  │
└────────────────────────────────────────────────────────────────────────────┘

Cluster Overview

Cluster	Region	Endpoint	Nodes	Karpenter
`myapp-dev-use1`	us-east-1	Public	2 × t3.medium	No
`myapp-dev-usw2`	us-west-2	Public	2 × t3.medium	No
`myapp-staging-use1`	us-east-1	Private	2 × t3.medium	No
`myapp-staging-usw2`	us-west-2	Private	2 × t3.medium	No
`myapp-production-use1`	us-east-1	Private	2+ × t3.medium	Yes
`myapp-production-usw2`	us-west-2	Private	2+ × t3.medium	Yes

EKS Terraform Module

# _modules/eks/main.tf

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 20.0"

  cluster_name    = var.cluster_name
  cluster_version = "1.29"

  vpc_id                   = var.vpc_id
  subnet_ids               = var.private_subnet_ids   # Nodes always in private subnets
  control_plane_subnet_ids = var.private_subnet_ids

  # Endpoint access: private always on, public only for dev
  cluster_endpoint_private_access = true
  cluster_endpoint_public_access  = var.public_api
  cluster_endpoint_public_access_cidrs = var.public_api ? ["0.0.0.0/0"] : ["0.0.0.0/0"]
  # Note: AWS rejects empty list when public is disabled — always set to 0.0.0.0/0

  # Without this the cluster creator IAM role (your Terragrunt role) can't kubectl
  enable_cluster_creator_admin_permissions = true

  # Encrypt Kubernetes secrets in etcd with KMS
  cluster_encryption_config = {
    provider_key_arn = var.kms_key_arn
    resources        = ["secrets"]
  }

  # EKS managed add-ons (AWS manages patching)
  cluster_addons = {
    coredns = {
      most_recent = true
    }
    kube-proxy = {
      most_recent = true
    }
    vpc-cni = {
      most_recent              = true
      service_account_role_arn = module.vpc_cni_irsa.iam_role_arn
    }
    aws-ebs-csi-driver = {
      most_recent              = true
      service_account_role_arn = module.ebs_csi_irsa.iam_role_arn
    }
  }

  eks_managed_node_groups = {
    main = {
      instance_types = var.instance_types   # ["t3.medium"]
      min_size       = var.min_nodes
      max_size       = var.max_nodes
      desired_size   = var.desired_nodes

      # IMPORTANT: name_prefix has a 38 character limit.
      # "myapp-production-use1-eks-node-group-" = 39 chars → FAILS.
      # Fix: use explicit role name (IAM limit is 64 chars).
      iam_role_name            = "${var.cluster_name}-node-group"
      iam_role_use_name_prefix = false

      # Nodes need these policies to pull from ECR, write to CloudWatch, etc.
      iam_role_additional_policies = {
        AmazonSSMManagedInstanceCore = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
      }

      labels = {
        "node-type" = "general"
      }

      # Karpenter discovery tag — only needed on production node groups
      # (Karpenter uses this to find the right security group)
      taints = var.karpenter_enabled ? [] : []
    }
  }

  # Node security group — allow Karpenter to manage nodes
  node_security_group_tags = var.karpenter_enabled ? {
    "karpenter.sh/discovery" = var.cluster_name
  } : {}
}

# IRSA for VPC CNI (pod networking)
module "vpc_cni_irsa" {
  source  = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"
  version = "~> 5.0"

  role_name             = "${var.cluster_name}-vpc-cni"
  attach_vpc_cni_policy = true
  vpc_cni_enable_ipv4   = true

  oidc_providers = {
    main = {
      provider_arn               = module.eks.oidc_provider_arn
      namespace_service_accounts = ["kube-system:aws-node"]
    }
  }
}

# IRSA for EBS CSI Driver (persistent volumes)
module "ebs_csi_irsa" {
  source  = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"
  version = "~> 5.0"

  role_name             = "${var.cluster_name}-ebs-csi"
  attach_ebs_csi_policy = true

  oidc_providers = {
    main = {
      provider_arn               = module.eks.oidc_provider_arn
      namespace_service_accounts = ["kube-system:ebs-csi-controller-sa"]
    }
  }
}

# _modules/eks/outputs.tf

output "cluster_name"                { value = module.eks.cluster_name }
output "cluster_endpoint"            { value = module.eks.cluster_endpoint }
output "cluster_certificate_authority_data" {
  value = module.eks.cluster_certificate_authority_data
}
output "oidc_provider_arn"           { value = module.eks.oidc_provider_arn }
output "oidc_provider"               { value = module.eks.oidc_provider }
output "node_security_group_id"      { value = module.eks.node_security_group_id }
output "node_subnet_ids"             { value = var.private_subnet_ids }

Per-Environment Terragrunt Configs

Dev (public endpoint):

# live/dev/us-east-1/eks/terragrunt.hcl

include "root" { path = find_in_parent_folders() }
terraform { source = "../../../../_modules/eks" }

dependency "vpc" {
  config_path = "../vpc"
  mock_outputs = {
    vpc_id             = "vpc-mock"
    private_subnet_ids = ["subnet-mock1", "subnet-mock2"]
  }
}

dependency "kms" {
  config_path = "../kms"
  mock_outputs = { key_arn = "arn:aws:kms:us-east-1:123456789:key/mock" }
}

inputs = {
  cluster_name       = "myapp-dev-use1"
  vpc_id             = dependency.vpc.outputs.vpc_id
  private_subnet_ids = dependency.vpc.outputs.private_subnet_ids
  kms_key_arn        = dependency.kms.outputs.key_arn
  public_api         = true    # Dev gets public endpoint for laptop + CI access
  min_nodes          = 2
  max_nodes          = 4
  desired_nodes      = 2
  instance_types     = ["t3.medium"]
  karpenter_enabled  = false
}

Production (private endpoint + Karpenter):

# live/production/us-east-1/eks/terragrunt.hcl

include "root" { path = find_in_parent_folders() }
terraform { source = "../../../../_modules/eks" }

dependency "vpc" { config_path = "../vpc" }
dependency "kms" { config_path = "../kms" }

inputs = {
  cluster_name       = "myapp-production-use1"
  vpc_id             = dependency.vpc.outputs.vpc_id
  private_subnet_ids = dependency.vpc.outputs.private_subnet_ids
  kms_key_arn        = dependency.kms.outputs.key_arn
  public_api         = false   # Private endpoint only — no public internet access
  min_nodes          = 2
  max_nodes          = 10
  desired_nodes      = 2
  instance_types     = ["t3.medium"]
  karpenter_enabled  = true    # Karpenter manages additional nodes beyond the initial 2
}

Bootstrapping Private Clusters

Staging and production clusters have endpointPublicAccess: false. This means kubectl from your laptop or CI cannot reach the API server directly. You must temporarily enable public access, bootstrap the cluster (install ArgoCD, register spokes, etc.), then lock it back down.

# Step 1: Temporarily enable public access
aws eks update-cluster-config \
  --name myapp-production-use1 \
  --region us-east-1 \
  --profile myapp-prod-use1 \
  --resources-vpc-config \
    endpointPublicAccess=true,\
    endpointPrivateAccess=true,\
    publicAccessCidrs="0.0.0.0/0"

# Step 2: Wait 3 minutes — AWS takes time to update the Elastic Network Interfaces
sleep 180

# Step 3: Verify access
kubectl --context prod-use1 get nodes

# Step 4: Bootstrap (install ArgoCD, apply ApplicationSets, etc.)
# ... your bootstrap commands ...

# Step 5: Lock back to private-only
aws eks update-cluster-config \
  --name myapp-production-use1 \
  --region us-east-1 \
  --profile myapp-prod-use1 \
  --resources-vpc-config \
    endpointPublicAccess=false,\
    endpointPrivateAccess=true

Do not forget Step 5. A production cluster with a public API endpoint is a security risk — the API server is internet-accessible, relying solely on authentication for protection.

kubectl Context Setup

After terragrunt apply completes for each cluster, add it to your kubeconfig:

# Update kubeconfig for all 6 clusters
aws eks update-kubeconfig \
  --name myapp-dev-use1 \
  --region us-east-1 \
  --alias dev-use1 \
  --profile myapp-dev-use1

aws eks update-kubeconfig \
  --name myapp-dev-usw2 \
  --region us-west-2 \
  --alias dev-usw2 \
  --profile myapp-dev-usw2

aws eks update-kubeconfig \
  --name myapp-staging-use1 \
  --region us-east-1 \
  --alias staging-use1 \
  --profile myapp-staging-use1

aws eks update-kubeconfig \
  --name myapp-staging-usw2 \
  --region us-west-2 \
  --alias staging-usw2 \
  --profile myapp-staging-usw2

aws eks update-kubeconfig \
  --name myapp-production-use1 \
  --region us-east-1 \
  --alias prod-use1 \
  --profile myapp-prod-use1

aws eks update-kubeconfig \
  --name myapp-production-usw2 \
  --region us-west-2 \
  --alias prod-usw2 \
  --profile myapp-prod-usw2

# Verify
kubectl config get-contexts

EKS Add-ons

EKS managed add-ons are maintained by AWS — they patch security vulnerabilities in CoreDNS, kube-proxy, and vpc-cni without you having to manage Helm releases.

kube-proxy     — handles iptables rules for Service routing
coredns        — in-cluster DNS resolution
vpc-cni        — AWS VPC networking for pods (each pod gets a real VPC IP)
aws-ebs-csi-driver — allows EKS to provision EBS volumes for PersistentVolumeClaims

Why IRSA for vpc-cni and ebs-csi?
These add-ons need to call AWS APIs (EC2 for ENI management, EC2 for EBS volume ops). Without IRSA they would use the node's EC2 instance profile — giving every pod on the node those permissions. With IRSA, only the specific add-on service account has the permissions.

Fixing kubectl 401 Errors

If kubectl get nodes returns HTTP 401 Unauthorized, the IAM role you're using is not in the cluster's access entries.

# List current access entries
aws eks list-access-entries \
  --cluster-name myapp-production-use1 \
  --region us-east-1 \
  --profile myapp-prod-use1

# If your OrganizationAccountAccessRole is missing, add it:
ROLE_ARN="arn:aws:iam::591120834781:role/OrganizationAccountAccessRole"

aws eks create-access-entry \
  --cluster-name myapp-production-use1 \
  --region us-east-1 \
  --profile myapp-prod-use1 \
  --principal-arn $ROLE_ARN

aws eks associate-access-policy \
  --cluster-name myapp-production-use1 \
  --region us-east-1 \
  --profile myapp-prod-use1 \
  --principal-arn $ROLE_ARN \
  --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy \
  --access-scope type=cluster

The root cause: enable_cluster_creator_admin_permissions = true must be explicitly set in the EKS module. If it's missing, Terraform creates the cluster but the IAM role that ran Terraform doesn't get an access entry.

AWS Load Balancer Controller

The AWS Load Balancer Controller (LBC) runs in every cluster and watches for Ingress resources with ingressClassName: alb. When it sees one, it provisions an Application Load Balancer in AWS automatically.

Install via Helm (or ArgoCD) after the cluster is up:

# IRSA for LBC
eksctl create iamserviceaccount \
  --cluster myapp-production-use1 \
  --namespace kube-system \
  --name aws-load-balancer-controller \
  --attach-policy-arn arn:aws:iam::aws:policy/ElasticLoadBalancingFullAccess \
  --override-existing-serviceaccounts \
  --approve \
  --region us-east-1 \
  --profile myapp-prod-use1

# Install controller
helm repo add eks https://aws.github.io/eks-charts
helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
  -n kube-system \
  --set clusterName=myapp-production-use1 \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller \
  --set region=us-east-1 \
  --set vpcId=<VPC_ID>

In this pipeline, the LBC is deployed via ArgoCD ApplicationSet — the Helm release is version-controlled in myapp-gitops/infrastructure/aws-lbc/.

StorageClass for EBS PVCs

kube-prometheus-stack needs persistent storage for Prometheus and Grafana data. With the EBS CSI driver installed, create a StorageClass:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp2
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer   # Don't provision until pod is scheduled
reclaimPolicy: Retain                      # Don't delete EBS volume if PVC is deleted
parameters:
  type: gp2
  encrypted: "true"
  kmsKeyId: <your-kms-key-arn>

Node Security Group Tags

For Karpenter to manage node lifecycles, it needs to find the cluster's node security group. Tag it during EKS creation:

node_security_group_tags = {
  "karpenter.sh/discovery" = var.cluster_name
}

Similarly, private subnets need the discovery tag:

private_subnet_tags = {
  "karpenter.sh/discovery" = var.cluster_name
}

Verifying All Six Clusters

for CTX in dev-use1 dev-usw2 staging-use1 staging-usw2 prod-use1 prod-usw2; do
  echo "=== $CTX ==="
  kubectl --context $CTX get nodes -o wide
done

Expected output:

=== dev-use1 ===
NAME                           STATUS   ROLES    AGE   VERSION
ip-10-0-8-xx.ec2.internal      Ready    <none>   5d    v1.29.15-eks-ac2d5a0
ip-10-0-16-xx.ec2.internal     Ready    <none>   5d    v1.29.15-eks-ac2d5a0

=== prod-use1 ===
NAME                           STATUS   ROLES    AGE   VERSION
ip-10-20-8-xx.ec2.internal     Ready    <none>   5d    v1.29.15-eks-ac2d5a0
ip-10-20-16-xx.ec2.internal    Ready    <none>   5d    v1.29.15-eks-ac2d5a0

Summary

By the end of Part 4 you have:

✅ Six EKS clusters (Kubernetes 1.29) across three environments and two regions
✅ Private endpoints on staging and production (public on dev)
✅ KMS encryption for Kubernetes secrets in etcd
✅ IAM IRSA for VPC CNI and EBS CSI add-ons
✅ AWS Load Balancer Controller installed
✅ kubectl contexts configured for all six clusters
✅ Karpenter discovery tags on production node security groups and subnets

Screenshot Placeholders

SCREENSHOT: AWS EKS console showing 2 clusters running in production with ACTIVE status

SCREENSHOT: kubectl get nodes output for all 6 clusters

Next: Part 5 — GitOps with ArgoCD: Hub-Spoke Model

Part 3: Infrastructure as Code

Matthew — Wed, 25 Mar 2026 08:00:00 +0000

Part 3: Infrastructure as Code — Terraform Modules + Terragrunt

Part of the series: Building a Production-Grade DevSecOps Pipeline on AWS

Introduction

Plain Terraform works fine for a single environment. But this pipeline has 6 clusters across 3 environments and 2 regions — 18+ Terragrunt child directories. Without a DRY strategy, you end up copy-pasting the same provider, backend, and module blocks everywhere, and a single account ID change means updating 18 files.

Terragrunt solves this with two mechanisms:

include — child configs inherit the root config's provider generation and remote state
dependency — explicit ordering ensures VPC exists before EKS, KMS before EKS, etc.

The result: each child terragrunt.hcl is typically 10–30 lines of pure inputs, with all boilerplate generated automatically.

Repository Layout

myapp-infra/
├── _modules/                    # Reusable Terraform modules (no Terragrunt here)
│   ├── vpc/
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   └── outputs.tf
│   ├── eks/
│   ├── kms/
│   ├── iam/
│   ├── ecr/
│   ├── waf/
│   ├── guardduty/
│   ├── eso-irsa/
│   ├── fluent-bit-irsa/
│   ├── karpenter/
│   └── velero/
│
└── live/                        # Terragrunt wrappers — one dir per resource per env/region
    ├── terragrunt.hcl           # ROOT config (provider + backend generation)
    ├── dev/
    │   ├── us-east-1/
    │   │   ├── vpc/
    │   │   │   └── terragrunt.hcl
    │   │   ├── kms/
    │   │   │   └── terragrunt.hcl
    │   │   ├── eks/
    │   │   │   └── terragrunt.hcl
    │   │   └── iam/
    │   │       └── terragrunt.hcl
    │   └── us-west-2/
    │       └── ... (mirror)
    ├── staging/
    │   └── ...
    └── production/
        └── ...

Key principle: modules in _modules/ are pure Terraform — no Terragrunt, no state config, no provider config. They are just reusable building blocks. The live/ tree contains nothing but thin Terragrunt wrappers that call those modules with environment-specific values.

Dependency Ordering

┌─────────────────────────────────────────────────────────────┐
│ APPLY ORDER (Terragrunt resolves this from dependency graph)│
│                                                             │
│  1. kms          (no dependencies)                          │
│  2. vpc          (no dependencies)                          │
│  3. eks          (depends on: vpc, kms)                     │
│  4. iam          (depends on: eks — needs OIDC provider URL)│
│  5. eso-irsa     (depends on: eks, iam)                     │
│  6. fluent-bit-irsa (depends on: eks)                       │
│  7. karpenter    (depends on: eks, iam)                     │
│  8. velero       (depends on: eks)                          │
│  9. waf          (no dependencies)                          │
│  10. guardduty   (no dependencies)                          │
└─────────────────────────────────────────────────────────────┘

Run everything in order automatically:

cd live/production/us-east-1
terragrunt run-all apply

Terragrunt reads all dependency blocks, builds a DAG, and applies in the correct order.

Root Terragrunt Config

# live/terragrunt.hcl

locals {
  path_parts = split("/", path_relative_to_include())
  env        = local.path_parts[0]   # dev | staging | production
  region     = local.path_parts[1]   # us-east-1 | us-west-2

  account_ids = {
    dev        = "557702566877"
    staging    = "YOUR_STAGING_ACCOUNT_ID"
    production = "591120834781"
  }
  account_id = local.account_ids[local.env]

  # Region short alias for naming (avoids long names hitting IAM limits)
  region_alias = local.region == "us-east-1" ? "use1" : "usw2"

  cluster_name = "myapp-${local.env}-${local.region_alias}"
}

# Auto-generate provider.tf in every child directory
generate "provider" {
  path      = "provider.tf"
  if_exists = "overwrite_terragrunt"
  contents  = <<-EOF
    provider "aws" {
      region = "${local.region}"
      assume_role {
        role_arn = "arn:aws:iam::${local.account_id}:role/OrganizationAccountAccessRole"
      }
      default_tags {
        tags = {
          Environment = "${local.env}"
          Region      = "${local.region}"
          ManagedBy   = "Terraform"
          Project     = "myapp"
          Cluster     = "${local.cluster_name}"
        }
      }
    }
  EOF
}

# Auto-generate backend.tf — per-module state file in S3
remote_state {
  backend = "s3"
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite_terragrunt"
  }
  config = {
    bucket         = "myapp-terraform-state-${local.account_id}"
    key            = "${path_relative_to_include()}/terraform.tfstate"
    region         = "us-east-1"   # State always in us-east-1 regardless of resource region
    encrypt        = true
    dynamodb_table = "myapp-terraform-locks"
    role_arn       = "arn:aws:iam::${local.account_id}:role/OrganizationAccountAccessRole"
  }
}

VPC Module

The VPC is the network foundation everything else sits in. Each environment gets its own VPC per region — 6 VPCs total.

CIDR allocation:
  dev     us-east-1:  10.0.0.0/16
  dev     us-west-2:  10.1.0.0/16
  staging us-east-1:  10.10.0.0/16
  staging us-west-2:  10.11.0.0/16
  prod    us-east-1:  10.20.0.0/16
  prod    us-west-2:  10.21.0.0/16

# _modules/vpc/main.tf

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"

  name = var.vpc_name
  cidr = var.vpc_cidr

  azs = [
    "${var.region}a",
    "${var.region}b",
    "${var.region}c"
  ]

  # Public subnets — for NAT Gateways, Internet-facing ALBs
  public_subnets = [
    cidrsubnet(var.vpc_cidr, 8, 0),   # x.x.0.0/24
    cidrsubnet(var.vpc_cidr, 8, 1),   # x.x.1.0/24
    cidrsubnet(var.vpc_cidr, 8, 2),   # x.x.2.0/24
  ]

  # Private subnets — EKS nodes, RDS, ElastiCache
  private_subnets = [
    cidrsubnet(var.vpc_cidr, 3, 1),   # x.x.8.0/21  (2048 IPs)
    cidrsubnet(var.vpc_cidr, 3, 2),   # x.x.16.0/21
    cidrsubnet(var.vpc_cidr, 3, 3),   # x.x.24.0/21
  ]

  enable_nat_gateway     = true
  single_nat_gateway     = var.single_nat_gateway   # true for dev (cost), false for prod (HA)
  enable_vpn_gateway     = false
  enable_dns_hostnames   = true
  enable_dns_support     = true

  # Required tags for AWS Load Balancer Controller to discover subnets
  public_subnet_tags = {
    "kubernetes.io/role/elb"                    = "1"
    "kubernetes.io/cluster/${var.cluster_name}" = "shared"
  }

  private_subnet_tags = {
    "kubernetes.io/role/internal-elb"           = "1"
    "kubernetes.io/cluster/${var.cluster_name}" = "shared"
    "karpenter.sh/discovery"                    = var.cluster_name  # Karpenter node discovery
  }

  # VPC Flow Logs for network traffic auditing
  enable_flow_log                      = true
  create_flow_log_cloudwatch_log_group = true
  create_flow_log_cloudwatch_iam_role  = true
  flow_log_max_aggregation_interval    = 60
}

# _modules/vpc/outputs.tf
output "vpc_id"              { value = module.vpc.vpc_id }
output "private_subnet_ids" { value = module.vpc.private_subnets }
output "public_subnet_ids"  { value = module.vpc.public_subnets }
output "vpc_cidr_block"     { value = module.vpc.vpc_cidr_block }

Terragrunt child config:

# live/production/us-east-1/vpc/terragrunt.hcl

include "root" {
  path = find_in_parent_folders()
}

terraform {
  source = "../../../../_modules/vpc"
}

inputs = {
  vpc_name          = "myapp-production-use1"
  vpc_cidr          = "10.20.0.0/16"
  region            = "us-east-1"
  cluster_name      = "myapp-production-use1"
  single_nat_gateway = false   # HA: one NAT GW per AZ
}

KMS Module

# _modules/kms/main.tf

# Handle the AWSServiceRoleForAutoScaling chicken-and-egg problem.
# In a fresh account this SLR doesn't exist yet, so we optionally create it.
resource "aws_iam_service_linked_role" "autoscaling" {
  count            = var.create_autoscaling_slr ? 1 : 0
  aws_service_name = "autoscaling.amazonaws.com"
}

# Wait 10s for IAM to propagate before referencing it in KMS key policy
resource "null_resource" "wait_for_slr" {
  count = var.create_autoscaling_slr ? 1 : 0
  depends_on = [aws_iam_service_linked_role.autoscaling]
  provisioner "local-exec" {
    command = "sleep 10"
  }
}

resource "aws_kms_key" "main" {
  depends_on              = [null_resource.wait_for_slr]
  description             = "${var.env}-${var.region}-main"
  deletion_window_in_days = 30
  enable_key_rotation     = true

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid      = "RootFullAccess"
        Effect   = "Allow"
        Principal = { AWS = "arn:aws:iam::${var.account_id}:root" }
        Action   = "kms:*"
        Resource = "*"
      },
      {
        Sid    = "AutoScalingSLR"
        Effect = "Allow"
        Principal = {
          AWS = "arn:aws:iam::${var.account_id}:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling"
        }
        Action   = ["kms:Encrypt", "kms:Decrypt", "kms:ReEncrypt*",
                    "kms:GenerateDataKey*", "kms:DescribeKey", "kms:CreateGrant"]
        Resource = "*"
      }
    ]
  })
}

resource "aws_kms_alias" "main" {
  name          = "alias/${var.env}-${var.region_alias}-main"
  target_key_id = aws_kms_key.main.key_id
}

Critical lesson: AWSServiceRoleForAutoScaling is an account-scoped IAM entity, not region-scoped. If you're deploying to two regions in the same account, only the first region should set create_autoscaling_slr = true. The second region's KMS config uses create_autoscaling_slr = false because the SLR already exists from the first apply.

# live/production/us-east-1/kms/terragrunt.hcl
include "root" { path = find_in_parent_folders() }
terraform { source = "../../../../_modules/kms" }
inputs = {
  env                    = "production"
  region                 = "us-east-1"
  region_alias           = "use1"
  account_id             = "591120834781"
  create_autoscaling_slr = false  # Already created by us-west-2 first apply
}

EKS Module (overview — full detail in Part 4)

# _modules/eks/main.tf (abbreviated)

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 20.0"

  cluster_name    = var.cluster_name
  cluster_version = "1.29"

  vpc_id                   = var.vpc_id
  subnet_ids               = var.private_subnet_ids
  control_plane_subnet_ids = var.private_subnet_ids

  # Private endpoint — spokes only; dev gets public too
  cluster_endpoint_private_access = true
  cluster_endpoint_public_access  = var.public_api

  # Must be explicit — without this the creator role can't kubectl
  enable_cluster_creator_admin_permissions = true

  cluster_encryption_config = {
    provider_key_arn = var.kms_key_arn
    resources        = ["secrets"]
  }

  eks_managed_node_groups = {
    main = {
      instance_types = ["t3.medium"]
      min_size       = 2
      max_size       = 10
      desired_size   = 2

      # Workaround: name_prefix is limited to 38 chars.
      # Long cluster names (staging, production) overflow this limit.
      # Using explicit name bypasses the prefix (IAM name limit is 64 chars).
      iam_role_name            = "${var.cluster_name}-node-group"
      iam_role_use_name_prefix = false
    }
  }
}

VPC Peering (for ArgoCD hub-spoke)

ArgoCD on myapp-production-use1 needs to reach the private API endpoints of the 5 spoke clusters. VPC peering provides private connectivity without internet traversal.

prod-use1 (10.20.0.0/16)  ←──── VPC Peering ────► prod-usw2   (10.21.0.0/16)
prod-use1 (10.20.0.0/16)  ←──── VPC Peering ────► staging-use1 (10.10.0.0/16)
prod-use1 (10.20.0.0/16)  ←──── VPC Peering ────► staging-usw2 (10.11.0.0/16)

Dev clusters use public endpoints — no VPC peering needed.

# live/production/us-east-1/vpc-peering/terragrunt.hcl
# NOTE: vpc-peering configs CANNOT use include "root" with remote_state.
# They must define remote_state explicitly because the generate label
# conflicts with the parent. Define it inline instead.

locals {
  env    = "production"
  region = "us-east-1"
}

remote_state {
  backend = "s3"
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite_terragrunt"
  }
  config = {
    bucket         = "myapp-terraform-state-591120834781"
    key            = "production/us-east-1/vpc-peering/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "myapp-terraform-locks"
  }
}

generate "provider" {
  path      = "provider.tf"
  if_exists = "overwrite_terragrunt"
  contents  = <<-EOF
    provider "aws" {
      region = "us-east-1"
      assume_role {
        role_arn = "arn:aws:iam::591120834781:role/OrganizationAccountAccessRole"
      }
    }
    # Peer VPC is in the staging account — needs its own provider alias
    provider "aws" {
      alias  = "staging"
      region = "us-east-1"
      assume_role {
        role_arn = "arn:aws:iam::STAGING_ACCOUNT_ID:role/OrganizationAccountAccessRole"
      }
    }
  EOF
}

inputs = {
  requester_vpc_id = dependency.prod_use1_vpc.outputs.vpc_id
  accepter_vpc_id  = dependency.staging_use1_vpc.outputs.vpc_id
  # ... route table IDs, CIDR blocks, etc.
}

Running the Stack

# First apply: production us-east-1 (this region creates the AutoScaling SLR)
cd live/production/us-west-2
terragrunt run-all apply --terragrunt-non-interactive

# Second: production us-east-1 (SLR already exists, create_autoscaling_slr=false)
cd live/production/us-east-1
terragrunt run-all apply --terragrunt-non-interactive

# Check what changed before applying
cd live/staging/us-east-1
terragrunt run-all plan

# Destroy a specific module (e.g., for re-creating)
cd live/dev/us-east-1/eks
terragrunt destroy

State Management Best Practices

Each module has its own state file: {env}/{region}/{module}/terraform.tfstate.

Why not one big state file?

A corrupt or locked state file affects only one module, not the entire environment
terraform plan on EKS doesn't load/lock VPC state — faster, safer
Different engineers can work on different modules concurrently

State file key examples:

production/us-east-1/vpc/terraform.tfstate
production/us-east-1/eks/terraform.tfstate
production/us-east-1/iam/terraform.tfstate
production/us-west-2/vpc/terraform.tfstate

Common Pitfalls

Problem	Symptom	Fix
AutoScaling SLR doesn't exist	`MalformedPolicyDocumentException` on KMS create	Set `create_autoscaling_slr = true` in first region; `false` in subsequent
IAM name_prefix > 38 chars	`ValidationError: name_prefix` on node group create	Use `iam_role_name` + `iam_role_use_name_prefix = false`
VPC peering uses `include "root"`	`generate label already defined` error	Define `remote_state` block explicitly in vpc-peering configs
AWS SG description has Unicode	`Invalid description` on security group	Use plain ASCII only in SG descriptions — no arrows (→) or greater-than (>)

Summary

By the end of Part 3 you have:

✅ DRY Terragrunt root config (provider + backend auto-generated from path)
✅ VPC module with public/private subnets, NAT gateways, flow logs
✅ KMS module handling the AutoScaling SLR chicken-and-egg problem
✅ Dependency graph ensuring correct apply order
✅ Per-module S3 state isolation
✅ VPC peering between production hub and all spoke VPCs

Next: Part 4 — EKS Multi-Cluster: Six Clusters Across Two Regions

Part 2: AWS Foundation

Matthew — Wed, 18 Mar 2026 09:00:00 +0000

Part 2: AWS Foundation — Organizations, SSO, and Account Setup

Part of the series: Building a Production-Grade DevSecOps Pipeline on AWS

Introduction

The foundation of any serious AWS deployment is a well-structured multi-account setup. Running everything in one AWS account is the equivalent of putting all your files on a single server with no access controls — the blast radius of any mistake or breach is your entire infrastructure.

In this part we set up AWS Organizations with four accounts, configure AWS IAM Identity Center (SSO) for human access, and establish the IAM trust relationships that allow GitHub Actions to deploy to our clusters without static credentials.

Why Multi-Account?

┌─────────────────────────────────────────────────────────────────────┐
│  SINGLE ACCOUNT (anti-pattern)                                      │
│                                                                     │
│  Dev workloads ─────────────────────────────┐                       │
│  Staging workloads ─────────────────────────┤── Same IAM boundary   │
│  Production workloads ──────────────────────┘   Same VPC space      │
│                                                  Same billing       │
│  Risk: dev engineer accidentally deletes production RDS             │
│  Risk: security incident in dev reaches production secrets          │
└─────────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────────┐
│  MULTI-ACCOUNT (this guide)                                         │
│                                                                     │
│  Dev Account:         strong IAM isolation, cheap instance sizes    │
│  Staging Account:     production-like, but no real data             │
│  Production Account:  SCPs block destructive operations             │
│  Management Account:  no workloads, only ECR + SSO + billing        │
│                                                                     │
│  Benefit: IAM permissions are account-scoped                        │
│  Benefit: Service Control Policies (SCPs) protect production        │
│  Benefit: Separate billing per environment                          │
│  Benefit: VPC IP space per account (no CIDR conflicts)              │
└─────────────────────────────────────────────────────────────────────┘

Step 1: Create AWS Organizations

AWS Console → AWS Organizations → Create Organization

AWS Organizations gives you a single management pane for all accounts, consolidated billing, and — crucially — Service Control Policies (SCPs) that act as guardrails even for account root users.

After creating the organization, note your Organization ID (format: o-xxxxxxxxxx).

Step 2: Create Member Accounts

Navigate to AWS Organizations → AWS Accounts → Add an AWS Account.

Create three accounts:

Account Name	Purpose	Example ID
`myapp-dev`	Development environment	`557702566877`
`myapp-staging`	Staging environment	(your value)
`myapp-production`	Production environment	`591120834781`

Important: Use + email aliases to reuse your existing email. If your email is you@gmail.com, use you+aws-dev@gmail.com, you+aws-staging@gmail.com, etc. Gmail (and most providers) deliver these to the same inbox.

Tip: Record each account ID immediately. You will reference them throughout this series.

SCREENSHOT: AWS Organizations showing all 4 accounts in their OUs

Step 3: Organize Accounts into OUs

Organizational Units (OUs) let you apply different SCPs to groups of accounts.

AWS Console → AWS Organizations → AWS Accounts → Root

Create OUs:
  Root
  ├── Management (leave root account here)
  ├── Workloads
  │   ├── Dev        (move myapp-dev here)
  │   ├── Staging    (move myapp-staging here)
  │   └── Production (move myapp-production here)

Step 4: Apply Service Control Policies

SCPs are JSON IAM policies attached to OUs. They define the maximum permissions any principal in that OU can ever have — even the account root user cannot exceed them.

Apply this SCP to the Production OU to prevent accidental deletion of critical resources:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyDangerousEKSOperations",
      "Effect": "Deny",
      "Action": [
        "eks:DeleteCluster",
        "eks:DeleteNodegroup"
      ],
      "Resource": "*",
      "Condition": {
        "StringNotEquals": {
          "aws:PrincipalTag/AllowDestructive": "true"
        }
      }
    },
    {
      "Sid": "DenyRDSDelete",
      "Effect": "Deny",
      "Action": [
        "rds:DeleteDBInstance",
        "rds:DeleteDBCluster"
      ],
      "Resource": "*"
    },
    {
      "Sid": "RequireRegions",
      "Effect": "Deny",
      "NotAction": [
        "iam:*",
        "sts:*",
        "route53:*",
        "cloudfront:*",
        "waf:*",
        "acm:*",
        "support:*",
        "health:*"
      ],
      "Resource": "*",
      "Condition": {
        "StringNotEquals": {
          "aws:RequestedRegion": [
            "us-east-1",
            "us-west-2"
          ]
        }
      }
    }
  ]
}

The region restriction SCP ensures no resources are accidentally created outside your approved regions. IAM, STS, Route53, and ACM are excluded because they are global services.

Step 5: Configure AWS IAM Identity Center (SSO)

AWS IAM Identity Center (formerly AWS SSO) lets your team log in with a single set of credentials across all accounts. It is far superior to creating IAM users in every account.

AWS Console → IAM Identity Center → Enable

Steps:
1. Choose identity source: "Identity Center directory" (built-in, no external IdP needed)
2. Create users for each team member
3. Create Permission Sets (these become IAM roles in each account)
4. Assign users to accounts with appropriate Permission Sets

Create two Permission Sets:

AdministratorAccess (for platform engineers):

Name: AdministratorAccess
Session duration: 8 hours
Managed policy: AdministratorAccess

ReadOnlyAccess (for developers / auditors):

Name: ReadOnlyAccess
Session duration: 8 hours
Managed policy: ReadOnlyAccess

Assign to accounts:

myapp-dev:         you → AdministratorAccess
myapp-staging:     you → AdministratorAccess
myapp-production:  you → AdministratorAccess
Management:        you → AdministratorAccess

Configure the AWS CLI for SSO:

# Run on your local machine
aws configure sso

# When prompted:
SSO session name: admin
SSO start URL: https://your-id.awsapps.com/start
SSO region: us-east-1
SSO registration scopes: sso:account:access

# After login, name each profile:
# Profile for dev/us-east-1: myapp-dev-use1
# Profile for dev/us-west-2: myapp-dev-usw2
# etc.

Your ~/.aws/config will look like:

[sso-session admin]
sso_start_url = https://your-id.awsapps.com/start
sso_region = us-east-1
sso_registration_scopes = sso:account:access

[profile myapp-prod-use1]
sso_session = admin
sso_account_id = 591120834781
sso_role_name = AdministratorAccess
region = us-east-1
output = json

[profile myapp-prod-usw2]
sso_session = admin
sso_account_id = 591120834781
sso_role_name = AdministratorAccess
region = us-west-2
output = json

[profile myapp-dev-use1]
sso_session = admin
sso_account_id = 557702566877
sso_role_name = AdministratorAccess
region = us-east-1
output = json

Authenticate:

aws sso login --sso-session admin
# Opens browser → log in → token saved locally for 8 hours

# Test:
aws sts get-caller-identity --profile myapp-prod-use1

SCREENSHOT: IAM Identity Center showing SSO portal with all accounts and permission sets

Step 6: GitHub OIDC — No Static AWS Keys in CI

This is one of the most important security decisions in the entire pipeline. Traditional CI/CD stores AWS access keys as GitHub Secrets. Those keys:

Never expire automatically
Are as powerful as the IAM user they belong to
Can be exfiltrated from logs if misconfigured

OIDC (OpenID Connect) eliminates static keys. GitHub Actions generates a short-lived JWT token for each workflow run. AWS validates this token cryptographically and exchanges it for a temporary STS credential that expires when the job ends.

┌─────────────────────────────────────────────────────────────────────┐
│  OIDC TOKEN FLOW                                                    │
│                                                                     │
│  GitHub Actions Job starts                                          │
│       │                                                             │
│       ▼                                                             │
│  GitHub generates JWT (signed by GitHub's OIDC provider)            │
│  Claims include:                                                    │
│    sub: repo:MatthewDipo/myapp:ref:refs/heads/main                  │
│    aud: sts.amazonaws.com                                           │
│       │                                                             │
│       ▼                                                             │
│  aws sts assume-role-with-web-identity                              │
│    --role-arn arn:aws:iam::ACCOUNT:role/ROLE                        │
│    --web-identity-token <JWT>                                       │
│       │                                                             │
│       ▼                                                             │
│  AWS validates JWT against GitHub's OIDC endpoint                   │
│  (https://token.actions.githubusercontent.com)                      │
│       │                                                             │
│       ▼                                                             │
│  Returns: AccessKeyId + SecretAccessKey + SessionToken              │
│  (valid for 1 hour maximum, then automatically expire)              │
└─────────────────────────────────────────────────────────────────────┘

Terraform to create the OIDC provider (management account, us-east-1 only — it is global):

# _modules/iam/main.tf

# Fetch GitHub's OIDC thumbprint automatically
data "tls_certificate" "github" {
  url = "https://token.actions.githubusercontent.com"
}

resource "aws_iam_openid_connect_provider" "github" {
  count = var.create_github_oidc ? 1 : 0

  url = "https://token.actions.githubusercontent.com"

  client_id_list = ["sts.amazonaws.com"]

  thumbprint_list = [
    data.tls_certificate.github.certificates[0].sha1_fingerprint
  ]
}

# IAM role for GitHub Actions CI — one per cluster
resource "aws_iam_role" "github_ci" {
  name = "${var.cluster_name}-github-ci"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = {
        Federated = var.github_oidc_provider_arn
      }
      Action    = "sts:AssumeRoleWithWebIdentity"
      Condition = {
        StringEquals = {
          "token.actions.githubusercontent.com:aud" = "sts.amazonaws.com"
        }
        StringLike = {
          # Only main branch of your specific repo can assume this role
          "token.actions.githubusercontent.com:sub" =
            "repo:${var.github_user}/${var.github_repo}:ref:refs/heads/main"
        }
      }
    }]
  })
}

resource "aws_iam_role_policy" "github_ci" {
  name = "github-ci-policy"
  role = aws_iam_role.github_ci.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid    = "ECRAuth"
        Effect = "Allow"
        Action = ["ecr:GetAuthorizationToken"]
        Resource = "*"
      },
      {
        Sid    = "ECRPush"
        Effect = "Allow"
        Action = [
          "ecr:BatchCheckLayerAvailability",
          "ecr:GetDownloadUrlForLayer",
          "ecr:BatchGetImage",
          "ecr:InitiateLayerUpload",
          "ecr:UploadLayerPart",
          "ecr:CompleteLayerUpload",
          "ecr:PutImage",
          "ecr:DescribeImages",
          "ecr:ListImages"
        ]
        Resource = "arn:aws:ecr:*:${var.account_id}:repository/myapp"
      },
      {
        Sid    = "KMSCosignSign"
        Effect = "Allow"
        Action = [
          "kms:Sign",
          "kms:GetPublicKey",
          "kms:DescribeKey"
        ]
        Resource = var.cosign_kms_key_arn
      },
      {
        Sid    = "S3AuditWrite"
        Effect = "Allow"
        Action = ["s3:PutObject"]
        Resource = "${var.audit_bucket_arn}/ci-push-audit/*"
      }
    ]
  })
}

Key security insight: The Condition block in the trust policy is critical. It restricts role assumption to:

Only your specific repository (repo:MatthewDipo/myapp)
Only the main branch

A fork of your repository, or a branch other than main, cannot assume this role.

Step 7: ECR Repository in the Management Account

We store Docker images in the management account's ECR rather than per-environment accounts. This means one place to manage image lifecycle policies and one IAM policy for cross-account pull permissions.

# _modules/ecr/main.tf

resource "aws_ecr_repository" "app" {
  name                 = var.name
  image_tag_mutability = "IMMUTABLE"   # Tags cannot be overwritten

  image_scanning_configuration {
    scan_on_push = true   # ECR runs basic CVE scan on every push
  }

  encryption_configuration {
    encryption_type = "KMS"
    kms_key         = var.kms_key_arn
  }
}

resource "aws_ecr_lifecycle_policy" "app" {
  repository = aws_ecr_repository.app.name

  policy = jsonencode({
    rules = [
      {
        rulePriority = 1
        description  = "Keep last 30 tagged images"
        selection = {
          tagStatus     = "tagged"
          tagPrefixList = ["sha-"]
          countType     = "imageCountMoreThan"
          countNumber   = 30
        }
        action = { type = "expire" }
      },
      {
        rulePriority = 2
        description  = "Delete untagged images after 1 day"
        selection = {
          tagStatus   = "untagged"
          countType   = "sinceImagePushed"
          countUnit   = "days"
          countNumber = 1
        }
        action = { type = "expire" }
      }
    ]
  })
}

# Cross-account pull policy — allows dev/staging/prod accounts to pull images
resource "aws_ecr_repository_policy" "cross_account" {
  repository = aws_ecr_repository.app.name

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid    = "CrossAccountPull"
        Effect = "Allow"
        Principal = {
          AWS = [
            "arn:aws:iam::${var.dev_account_id}:root",
            "arn:aws:iam::${var.staging_account_id}:root",
            "arn:aws:iam::${var.prod_account_id}:root"
          ]
        }
        Action = [
          "ecr:GetDownloadUrlForLayer",
          "ecr:BatchGetImage",
          "ecr:BatchCheckLayerAvailability"
        ]
      }
    ]
  })
}

IMMUTABLE tags mean that once you push sha-abc123, that tag forever points to that exact image digest. No one can silently overwrite an existing tag with a different image — a subtle but important supply chain security control.

SCREENSHOT: ECR repository showing images with sha- tags and scan results

Step 8: KMS Keys for Encryption at Rest

Each environment gets its own KMS key for encrypting:

EKS Kubernetes secrets (etcd encryption)
EBS volumes (Prometheus/Grafana/Velero PVCs)
ECR images
S3 buckets (Velero backups, CI audit logs)

# _modules/kms/main.tf

resource "aws_kms_key" "main" {
  description             = "${var.env}-${var.region}-main"
  deletion_window_in_days = 30
  enable_key_rotation     = true   # Rotate annually, automatically

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid    = "Enable IAM User Permissions"
        Effect = "Allow"
        Principal = { AWS = "arn:aws:iam::${var.account_id}:root" }
        Action   = "kms:*"
        Resource = "*"
      },
      {
        Sid    = "Allow EKS"
        Effect = "Allow"
        Principal = {
          Service = "eks.amazonaws.com"
        }
        Action   = ["kms:Encrypt", "kms:Decrypt", "kms:GenerateDataKey*", "kms:DescribeKey"]
        Resource = "*"
      },
      {
        Sid    = "Allow AutoScaling"
        Effect = "Allow"
        Principal = {
          AWS = "arn:aws:iam::${var.account_id}:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling"
        }
        Action   = ["kms:Encrypt", "kms:Decrypt", "kms:ReEncrypt*", "kms:GenerateDataKey*", "kms:DescribeKey", "kms:CreateGrant"]
        Resource = "*"
      }
    ]
  })
}

resource "aws_kms_alias" "main" {
  name          = "alias/${var.env}-${var.region}-main"
  target_key_id = aws_kms_key.main.key_id
}

Lesson learned: The AWSServiceRoleForAutoScaling must exist in the account before you can reference it in the KMS key policy. In fresh accounts, create this Service Linked Role first or AWS will reject the key policy with MalformedPolicyDocumentException. See Part 3 for the Terragrunt pattern that handles this.

Step 9: Terragrunt Root Configuration

Before writing any Terragrunt configs, establish the root terragrunt.hcl that all child configs inherit:

# live/terragrunt.hcl  (root)

locals {
  # Parse path to extract env and region
  # e.g., live/production/us-east-1/eks → env=production, region=us-east-1
  path_parts = split("/", path_relative_to_include())
  env        = local.path_parts[0]
  region     = local.path_parts[1]

  account_ids = {
    dev        = "557702566877"
    staging    = "STAGING_ACCOUNT_ID"
    production = "591120834781"
    management = "MGMT_ACCOUNT_ID"
  }

  account_id = local.account_ids[local.env]
}

# Generate provider.tf in every child directory
generate "provider" {
  path      = "provider.tf"
  if_exists = "overwrite_terragrunt"
  contents  = <<EOF
provider "aws" {
  region = "${local.region}"
  assume_role {
    role_arn = "arn:aws:iam::${local.account_id}:role/OrganizationAccountAccessRole"
  }
  default_tags {
    tags = {
      Environment = "${local.env}"
      ManagedBy   = "Terraform"
      Project     = "myapp"
    }
  }
}
EOF
}

# Generate backend.tf — S3 state, DynamoDB lock table
remote_state {
  backend = "s3"
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite_terragrunt"
  }
  config = {
    bucket         = "myapp-terraform-state-${local.account_id}"
    key            = "${path_relative_to_include()}/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "myapp-terraform-locks"
    role_arn       = "arn:aws:iam::${local.account_id}:role/OrganizationAccountAccessRole"
  }
}

The generate "provider" block means you never write a provider.tf by hand. Every module automatically gets the correct AWS account and region based purely on its directory path. This is the key DRY benefit of Terragrunt.

Understanding `OrganizationAccountAccessRole`

When you create a member account via AWS Organizations, AWS automatically creates a role called OrganizationAccountAccessRole in that account. This role trusts your management account, allowing management account principals to assume it and perform actions in the member account.

This is how Terraform (running with management account credentials) deploys infrastructure into dev, staging, and production without needing separate credentials per account.

Management Account (your terminal / CI)
    │
    │ sts:AssumeRole
    ▼
arn:aws:iam::591120834781:role/OrganizationAccountAccessRole
    │
    │ (full AdministratorAccess in production account)
    ▼
Production Account resources

Step 10: Bootstrap S3 State Buckets

Before running any Terragrunt, each account needs its S3 bucket and DynamoDB table for Terraform state.

# Run this once per account (adjust account ID and profile)
for PROFILE in myapp-dev-use1 myapp-staging-use1 myapp-prod-use1; do
  ACCOUNT_ID=$(aws sts get-caller-identity --profile $PROFILE --query Account --output text)
  REGION="us-east-1"

  # Create state bucket
  aws s3api create-bucket \
    --bucket "myapp-terraform-state-${ACCOUNT_ID}" \
    --region $REGION \
    --profile $PROFILE

  # Enable versioning (lets you recover from bad applies)
  aws s3api put-bucket-versioning \
    --bucket "myapp-terraform-state-${ACCOUNT_ID}" \
    --versioning-configuration Status=Enabled \
    --profile $PROFILE

  # Enable encryption
  aws s3api put-bucket-encryption \
    --bucket "myapp-terraform-state-${ACCOUNT_ID}" \
    --server-side-encryption-configuration '{
      "Rules": [{"ApplyServerSideEncryptionByDefault": {"SSEAlgorithm": "AES256"}}]
    }' \
    --profile $PROFILE

  # Block public access
  aws s3api put-public-access-block \
    --bucket "myapp-terraform-state-${ACCOUNT_ID}" \
    --public-access-block-configuration \
      "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true" \
    --profile $PROFILE

  # Create DynamoDB lock table
  aws dynamodb create-table \
    --table-name myapp-terraform-locks \
    --attribute-definitions AttributeName=LockID,AttributeType=S \
    --key-schema AttributeName=LockID,KeyType=HASH \
    --billing-mode PAY_PER_REQUEST \
    --region $REGION \
    --profile $PROFILE

  echo "State backend ready for account $ACCOUNT_ID"
done

Summary

By the end of this part you have:

✅ AWS Organizations with four accounts (management, dev, staging, production)
✅ Service Control Policies protecting production from accidental destruction
✅ AWS SSO for human access (no IAM users with permanent credentials)
✅ GitHub OIDC provider enabling keyless CI authentication
✅ ECR repository with immutable tags, cross-account pull, and lifecycle policies
✅ KMS keys for encryption at rest in every environment
✅ Terragrunt root config that automatically derives account/region from directory path
✅ S3 + DynamoDB Terraform state backend per account

If this was useful, follow me on dev.to — I publish Part 3 next Wednesday covering the Infrastructure as Code — Terraform Modules + Terragrunt.

Questions? Drop them in the comments — I read and reply to every one.

Next: Part 3 — Infrastructure as Code: Terraform Modules + Terragrunt

Part 1: Architecture Overview

Matthew — Wed, 11 Mar 2026 09:24:20 +0000

Building a Production-Grade DevSecOps Pipeline on AWS: A Complete Guide

Series Overview: This 10-part series walks you through building a real-world, production-grade DevSecOps platform on AWS from scratch — the same architecture used at mature engineering organizations. By the end, you will have six EKS clusters, a GitOps delivery model, a hardened CI/CD pipeline, runtime security, full-stack observability, and automated disaster recovery.

Live system: Everything in this series is running in production right now.
Check it: curl https://www.matthewoladipupo.dev/health
→ {"status":"healthy","region":"us-east-1"}

This is Part 1 of a 10-part series. You can follow the full series here on dev.to.

Part 1: Architecture Overview & What We Are Building

Introduction

Most DevSecOps tutorials show you one piece of the puzzle — a Kubernetes cluster here, a CI pipeline there. This series is different. We build the entire platform end to end: infrastructure-as-code, multi-environment clusters, GitOps, security policy enforcement, runtime threat detection, secrets management, observability, canary deployments, autoscaling, and backup — all wired together the way a production engineering team would actually build it.

Every component in this guide is running live. The screenshots you will see throughout this series come from the actual deployment at matthewoladipupo.dev.

6 EKS clusters, ArgoCD hub-spoke GitOps, GitHub Actions CI/CD, and AWS
security services" width="800" height="545">The complete system: GitHub → CI/CD → ECR → ArgoCD hub → 6 EKS clusters
across 3 environments and 2 AWS regions. Every component is covered in this
10-part series.

What you will build:

Layer	Tools
Infrastructure as Code	Terraform + Terragrunt
Cloud Platform	AWS (multi-account, multi-region)
Container Orchestration	Amazon EKS (6 clusters)
GitOps Delivery	ArgoCD (hub-spoke)
CI/CD Pipeline	GitHub Actions
Container Security	Trivy, Cosign, Distroless images
Policy Enforcement	Kyverno
Runtime Security	Falco
Secrets Management	AWS Secrets Manager + External Secrets Operator
Monitoring	Prometheus + Grafana (kube-prometheus-stack)
Logging	Fluent Bit → AWS CloudWatch
Canary Deployments	Argo Rollouts
Autoscaling	Karpenter + HPA
Backup & DR	Velero + S3
Web Application Firewall	AWS WAF v2
Threat Detection	AWS GuardDuty
DNS & TLS	Route53 + ACM (wildcard cert)
Load Balancing	AWS Load Balancer Controller

High-Level Architecture

The platform follows a hub-spoke GitOps model across three environments and two AWS regions.

┌─────────────────────────────────────────────────────────────────────────────────┐
│                            AWS ORGANIZATION                                     │
│                                                                                 │
│  ┌──────────────────┐    ┌──────────────────┐    ┌──────────────────────────┐   │
│  │  Management Acct │    │  Dev Account     │    │  Staging Account         │   │
│  │  (ECR, CI Audit) │    │  557702566877    │    │                          │   │
│  │                  │    │  ┌─────────────┐ │    │  ┌────────┐ ┌────────┐   │   │
│  │  ┌───────────┐   │    │  │ EKS use1    │ │    │  │EKS use1│ │EKS usw2│   │   │
│  │  │ECR: myapp │   │    │  │ (public ep) │ │    │  │        │ │        │   │   │
│  │  └───────────┘   │    │  └─────────────┘ │    │  └────────┘ └────────┘   │   │
│  └──────────────────┘    │  ┌─────────────┐ │    └──────────────────────────┘   │
│                          │  │ EKS usw2    │ │                                   │
│                          │  │ (public ep) │ │    ┌──────────────────────────┐   │
│                          │  └─────────────┘ │    │  Production Account      │   │
│                          └──────────────────┘    │  591120834781            │   │
│                                                  │                          │   │
│                                                  │  ┌────────┐ ┌────────┐   │   │
│                                                  │  │EKS use1│ │EKS usw2│   │   │
│                                                  │  │  HUB   │ │ Spoke  │   │   │
│                                                  │  │(ArgoCD)│ │        │   │   │
│                                                  │  └────────┘ └────────┘   │   │
│                                                  └──────────────────────────┘   │
└─────────────────────────────────────────────────────────────────────────────────┘

Detailed Architecture Diagram

                              DEVELOPER WORKFLOW
                              ──────────────────
                              git push → GitHub

                                      │
                                      ▼
┌─────────────────────────────────────────────────────────────────────────────────┐
│                         GITHUB ACTIONS CI PIPELINE                              │
│                                                                                 │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────────────┐   │
│  │  Lint +  │  │  Trivy   │  │  Docker  │  │  Cosign  │  │  Push to ECR     │   │
│  │  Test    │→ │  Scan    │→ │  Build   │→ │  Sign    │→ │  (multi-region)  │   │
│  └──────────┘  └──────────┘  └──────────┘  └──────────┘  └──────────────────┘   │
│                                                                    │            │
│                                              OIDC (no static keys) │            │
│                                              IAM Roles per cluster │            │
└────────────────────────────────────────────────────────────────────┼────────────┘
                                                                     │
                                                                     ▼
                                                             myapp-gitops repo
                                                           (image tag updated)
                                                                     │
                                                                     ▼
┌─────────────────────────────────────────────────────────────────────────────────┐
│                    ARGOCD HUB  (myapp-production-use1)                          │
│                                                                                 │
│   ApplicationSets (list generators per cluster)                                 │
│   ┌─────────────────────────────────────────────────────────────────────────┐   │
│   │  environments/  infrastructure/  argocd/                                │   │
│   │  ├─ dev         ├─ monitoring    └─ project-*.yaml                      │   │
│   │  ├─ staging     ├─ logging                                              │   │
│   │  └─ production  ├─ eso                                                  │   │
│   │                 ├─ kyverno                                              │   │
│   │                 ├─ falco                                                │   │
│   │                 ├─ velero                                               │   │
│   │                 ├─ karpenter                                            │   │
│   │                 └─ argo-rollouts                                        │   │
│   └─────────────────────────────────────────────────────────────────────────┘   │
│                                                                                 │
│   Syncs to ──────────────────────────────────────────────────────────────────►  │
└──────┬──────────────────────────────────────────────────────────────────────────┘
       │  VPC Peering (private endpoints)
       ├────────────────────────────────► myapp-production-usw2
       ├────────────────────────────────► myapp-staging-use1
       ├────────────────────────────────► myapp-staging-usw2
       ├────────────────────────────────► myapp-dev-use1
       └────────────────────────────────► myapp-dev-usw2

Per-Cluster Component Stack

Every cluster runs the same security and observability baseline. The diagram below shows what runs on each cluster after bootstrapping:

┌──────────────────────────────────────────────────────────────────────┐
│                    EKS CLUSTER (per-cluster stack)                   │
│                                                                      │
│  SYSTEM NAMESPACES                                                   │
│  ┌─────────────┐  ┌──────────────────┐  ┌────────────────────────┐   │
│  │  kube-system│  │  kyverno         │  │  falco                 │   │
│  │  (aws-lbc,  │  │  (policy engine) │  │  (runtime security)    │   │
│  │   coreDNS)  │  │                  │  │                        │   │
│  └─────────────┘  └──────────────────┘  └────────────────────────┘   │
│                                                                      │
│  ┌─────────────────────┐  ┌───────────────────────────────────────┐  │
│  │  external-secrets   │  │  monitoring                           │  │
│  │  (ESO operator)     │  │  Prometheus ── Grafana ── Alertmanager│  │
│  └─────────────────────┘  └───────────────────────────────────────┘  │
│                                                                      │
│  ┌───────────────────────┐  ┌──────────────────┐  ┌─────────────┐    │
│  │  logging              │  │  velero          │  │  karpenter  │    │
│  │  (Fluent Bit DS)      │  │  (backup)        │  │  (prod only)│    │
│  └───────────────────────┘  └──────────────────┘  └─────────────┘    │
│                                                                      │
│  ┌───────────────────────┐  ┌──────────────────────────────────────┐ │
│  │  argo-rollouts        │  │  myapp (prod) / myapp (dev/staging)  │ │
│  │  (canary controller)  │  │  Rollout ──► canary ──► stable       │ │
│  └───────────────────────┘  └──────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────────────┘

Network Architecture

                        AWS REGION: us-east-1
┌───────────────────────────────────────────────────────────┐
│  VPC: production-use1  (10.20.0.0/16)                     │
│                                                           │
│  ┌──────────────────────────┐  ┌──────────────────────┐   │
│  │  Public Subnets          │  │  Private Subnets     │   │
│  │  10.20.0.0/24 (us-east-1a)  10.20.8.0/21 (use1a)   │   │
│  │  10.20.1.0/24 (us-east-1b)  10.20.16.0/21 (use1b)  │   │
│  │  10.20.2.0/24 (us-east-1c)  10.20.24.0/21 (use1c)  │   │
│  │                          │  │                      │   │
│  │  NAT Gateways            │  │  EKS Node Groups     │   │
│  │  Internet Gateway        │  │  EKS API endpoint    │   │
│  │  ALB (internet-facing)   │  │  (private only)      │   │
│  └──────────────────────────┘  └──────────────────────┘   │
└───────────────────────────────────────────────────────────┘
          │                            │
          │ VPC Peering                │
          │ (private, encrypted)       │
          ▼                            ▼
┌────────────────────┐    ┌────────────────────────────────┐
│  VPC: prod-usw2    │    │  VPC: staging-use1             │
│  (10.21.0.0/16)    │    │  (10.10.0.0/16)                │
└────────────────────┘    └────────────────────────────────┘

Internet Traffic Flow:
User → Route53 (latency routing) → ALB → AWS WAF → ALB Target Group
     → Pod (via ALB target type: ip, directly to pod IP)
     → Response back to user

CI/CD Pipeline Flow

┌─────────────────────────────────────────────────────────────────────┐
│  GitHub: MatthewDipo/myapp                                          │
│                                                                     │
│  Developer: git push origin main                                    │
└──────────────────────────────┬──────────────────────────────────────┘
                               │ triggers
                               ▼
┌─────────────────────────────────────────────────────────────────────┐
│  GitHub Actions: .github/workflows/ci.yaml                          │
│                                                                     │
│  Job 1: lint-and-test                                               │
│    └─ npm ci && npm test                                            │
│                                                                     │
│  Job 2: scan (needs: lint-and-test)                                 │
│    └─ trivy image --severity HIGH,CRITICAL                          │
│                                                                     │
│  Job 3: build-push-sign (needs: scan)                               │
│    ├─ OIDC → assume IAM role (no static AWS keys)                   │
│    ├─ docker build (distroless:nonroot base)                        │
│    ├─ docker push → ECR us-east-1 (management account)              │
│    ├─ docker push → ECR us-west-2 (management account)              │
│    ├─ cosign sign --key awskms:// (KMS signing key)                 │
│    └─ Write S3 audit log (image digest + timestamp)                 │
│                                                                     │
│  Job 4: update-gitops (needs: build-push-sign)                      │
│    └─ Patch myapp-gitops/apps/myapp/values-*.yaml (image.tag)       │
└──────────────────────────────┬──────────────────────────────────────┘
                               │ git push to myapp-gitops
                               ▼
┌─────────────────────────────────────────────────────────────────────┐
│  GitHub: MatthewDipo/myapp-gitops                                   │
│  ArgoCD detects diff → triggers sync per cluster                    │
│                                                                     │
│  Production: Argo Rollouts Canary                                   │
│    ├─ Step 1: setWeight 20% (canary gets 20% traffic)               │
│    ├─ Step 2: pause 5 minutes                                       │
│    ├─ Step 3: AnalysisRun (check error rate < 1%)                   │
│    └─ Step 4: setWeight 100% (promote to stable)                    │
│                                                                     │
│  Dev/Staging: Rolling Update (immediate)                            │
└─────────────────────────────────────────────────────────────────────┘

Security Architecture

┌─────────────────────────────────────────────────────────────────────────┐
│                       SECURITY LAYERS                                   │
│                                                                         │
│  Layer 1: SUPPLY CHAIN SECURITY                                         │
│  ┌────────────────────────────────────────────────────────────────────┐ │
│  │  Source: GitHub branch protection + required reviews               │ │
│  │  Build:  Trivy CVE scan (fails pipeline on HIGH/CRITICAL)          │ │
│  │  Image:  Distroless base (no shell, no package manager)            │ │
│  │  Sign:   Cosign + AWS KMS (cryptographic attestation)              │ │
│  │  Verify: Kyverno policy blocks unsigned images at admission        │ │
│  └────────────────────────────────────────────────────────────────────┘ │
│                                                                         │
│  Layer 2: INFRASTRUCTURE SECURITY                                       │
│  ┌────────────────────────────────────────────────────────────────────┐ │
│  │  Network:  Private EKS endpoints (staging/prod)                    │ │
│  │            VPC peering (no internet traversal between clusters)    │ │
│  │            NetworkPolicies (deny-all default, allow explicitly)    │ │
│  │  IAM:      IRSA (pod-level IAM, no node-level credentials)         │ │
│  │            OIDC for GitHub Actions (no static AWS keys in CI)      │ │
│  │  Secrets:  AWS Secrets Manager (never stored in Git)               │ │
│  │  KMS:      Envelope encryption for EKS secrets + ECR + S3          │ │
│  └────────────────────────────────────────────────────────────────────┘ │
│                                                                         │
│  Layer 3: WORKLOAD ADMISSION CONTROL (Kyverno)                          │
│  ┌────────────────────────────────────────────────────────────────────┐ │
│  │  ✗ Block privileged containers                                     │ │
│  │  ✗ Block hostPath volume mounts                                    │ │
│  │  ✗ Block containers running as root (uid=0)                        │ │
│  │  ✗ Block images without valid Cosign signature                     │ │
│  │  ✗ Block missing resource limits                                   │ │
│  │  ✓ Allow myapp from ECR with valid signature                       │ │
│  └────────────────────────────────────────────────────────────────────┘ │
│                                                                         │
│  Layer 4: RUNTIME THREAT DETECTION (Falco)                              │
│  ┌────────────────────────────────────────────────────────────────────┐ │
│  │  Alert on: shell spawned in container                              │ │
│  │  Alert on: sensitive file read (/etc/shadow, /etc/passwd)          │ │
│  │  Alert on: unexpected outbound network connection                  │ │
│  │  Alert on: privilege escalation attempts                           │ │
│  │  Output: CloudWatch Logs (via Fluent Bit)                          │ │
│  └────────────────────────────────────────────────────────────────────┘ │
│                                                                         │
│  Layer 5: PERIMETER SECURITY                                            │
│  ┌────────────────────────────────────────────────────────────────────┐ │
│  │  AWS WAF v2: Managed rules (OWASP Top 10, SQL injection, XSS)      │ │
│  │  AWS GuardDuty: Account-level threat intelligence                  │ │
│  │  ACM: TLS 1.2+ enforced, HTTP → HTTPS redirect                     │ │
│  └────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────┘

GitOps Repository Structure

Two GitHub repositories drive everything after the CI pipeline builds the image:

MatthewDipo/myapp-gitops/
│
├── argocd/
│   ├── project-dev.yaml             # AppProject: dev clusters
│   ├── project-staging.yaml         # AppProject: staging clusters
│   └── project-production.yaml      # AppProject: production clusters
│
├── environments/
│   ├── dev/
│   │   └── applicationset.yaml      # Deploys myapp to dev-use1, dev-usw2
│   ├── staging/
│   │   └── applicationset.yaml      # Deploys myapp to staging-use1, staging-usw2
│   └── production/
│       └── applicationset.yaml      # Deploys myapp to prod-use1, prod-usw2
│
├── infrastructure/
│   ├── monitoring/
│   │   ├── applicationset.yaml      # kube-prometheus-stack (4 clusters)
│   │   ├── prometheus-values.yaml
│   │   └── alert-rules/
│   │       └── applicationset.yaml  # PrometheusRule CRDs
│   ├── logging/
│   │   └── applicationset.yaml      # Fluent Bit DaemonSet (6 clusters)
│   ├── eso/
│   │   └── applicationset.yaml      # External Secrets Operator (6 clusters)
│   ├── kyverno/
│   │   └── applicationset.yaml      # Kyverno + policies (6 clusters)
│   ├── falco/
│   │   └── applicationset.yaml      # Falco DaemonSet (6 clusters)
│   ├── velero/
│   │   └── applicationset.yaml      # Velero (6 clusters)
│   ├── karpenter/
│   │   ├── applicationset.yaml      # Karpenter controller (2 prod)
│   │   └── nodepools/
│   │       └── applicationset.yaml  # NodePool + EC2NodeClass CRDs
│   └── argo-rollouts/
│       └── applicationset.yaml      # Argo Rollouts controller (2 prod)
│
└── apps/
    └── myapp/                       # Helm chart for the application
        ├── Chart.yaml
        ├── values.yaml              # Default values
        ├── values-dev.yaml
        ├── values-staging.yaml
        ├── values-production.yaml
        └── templates/
            ├── deployment.yaml      # Deployment OR Rollout (conditional)
            ├── service.yaml
            ├── service-canary.yaml  # Canary service (prod only)
            ├── ingress.yaml
            ├── hpa.yaml
            ├── networkpolicy.yaml
            ├── serviceaccount.yaml
            ├── external-secret.yaml
            ├── servicemonitor.yaml  # Prometheus scraping
            └── analysis-template.yaml

Infrastructure Repository Structure

MatthewDipo/myapp-infra/
│
├── _modules/                        # Reusable Terraform modules
│   ├── vpc/
│   ├── eks/
│   ├── kms/
│   ├── iam/
│   ├── ecr/
│   ├── waf/
│   ├── guardduty/
│   ├── eso-irsa/
│   ├── fluent-bit-irsa/
│   ├── karpenter/
│   └── velero/
│
└── live/                            # Terragrunt configurations (per env/region)
    ├── terragrunt.hcl               # Root config (provider, remote state)
    ├── dev/
    │   ├── us-east-1/
    │   │   ├── vpc/terragrunt.hcl
    │   │   ├── kms/terragrunt.hcl
    │   │   ├── eks/terragrunt.hcl
    │   │   ├── iam/terragrunt.hcl
    │   │   └── fluent-bit-irsa/terragrunt.hcl
    │   └── us-west-2/
    │       └── ... (mirror of use1)
    ├── staging/
    │   ├── us-east-1/
    │   │   ├── vpc/ kms/ eks/ iam/
    │   │   ├── waf/terragrunt.hcl
    │   │   ├── guardduty/terragrunt.hcl
    │   │   ├── eso-irsa/terragrunt.hcl
    │   │   └── fluent-bit-irsa/terragrunt.hcl
    │   └── us-west-2/
    │       └── ...
    └── production/
        ├── us-east-1/
        │   ├── vpc/ kms/ eks/ iam/
        │   ├── waf/
        │   ├── guardduty/
        │   ├── eso-irsa/
        │   ├── fluent-bit-irsa/
        │   ├── karpenter/
        │   └── velero/
        └── us-west-2/
            └── ...

AWS Account Structure

┌──────────────────────────────────────────────┐
│         AWS Organizations Root               │
│                                              │
│  ┌───────────────────────────────────────┐   │
│  │  Management / Root Account            │   │
│  │  • AWS SSO (Identity Center)          │   │
│  │  • ECR repositories (shared)          │   │
│  │  • S3 CI audit bucket                 │   │
│  │  • GitHub OIDC provider               │   │
│  └───────────────────────────────────────┘   │
│           │            │            │        │
│           ▼            ▼            ▼        │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐    │
│  │   Dev    │  │ Staging  │  │Production│    │
│  │ Account  │  │ Account  │  │ Account  │    │
│  │          │  │          │  │          │    │
│  │ 2x EKS   │  │ 2x EKS   │  │ 2x EKS   │    │
│  │ VPCs     │  │ VPCs     │  │ VPCs     │    │
│  │ KMS keys │  │ KMS keys │  │ KMS keys │    │
│  │ Secrets  │  │ Secrets  │  │ Secrets  │    │
│  │ Manager  │  │ Manager  │  │ Manager  │    │
│  └──────────┘  └──────────┘  └──────────┘    │
└──────────────────────────────────────────────┘

Technology Decision Rationale

Understanding WHY each tool was chosen matters as much as knowing HOW to configure it.

Terragrunt over plain Terraform
Terragrunt provides DRY (Don't Repeat Yourself) configuration. Without it, you would have near-identical provider, backend, and module blocks repeated across 18+ directories. Terragrunt's include and dependency blocks eliminate 90% of that duplication while keeping each environment's overrides explicit and auditable.

ArgoCD hub-spoke over fleet management per cluster
Running ArgoCD on every cluster is operationally expensive. The hub-spoke model means one ArgoCD installation manages all six clusters via VPC peering. This single pane of glass dramatically simplifies debugging — you see all cluster states in one place.

Kyverno over OPA/Gatekeeper
Kyverno policies are written in YAML and operate on the same resource schema as Kubernetes objects. OPA/Gatekeeper requires learning Rego, a purpose-built policy language. For Kubernetes-native teams, Kyverno is faster to adopt and maintain.

External Secrets Operator over Sealed Secrets
Sealed Secrets encrypts secrets and commits them to Git — meaning the encrypted value is your source of truth. ESO keeps secrets out of Git entirely: the secret lives in AWS Secrets Manager, and ESO fetches it at runtime with IRSA credentials. This is a fundamentally stronger posture because a compromised Git repo never exposes secret material.

Argo Rollouts over plain Kubernetes rolling updates
Rolling updates are binary — you either roll forward or roll back. Argo Rollouts adds weighted traffic splitting between stable and canary versions, analysis runs (automated metric-based promotion gates), and pause steps for manual inspection. A canary deployment that automatically fails back on a rising error rate is far safer than a rolling update you monitor manually.

Distroless base images
The Google distroless nonroot image contains only the application runtime and its direct dependencies — no shell (sh, bash), no package manager (apt, apk), no curl or wget. If an attacker achieves code execution inside the container, they have almost no tools available to escalate or exfiltrate. Combined with Falco alerting on shell spawning, you get both prevention and detection.

Cost Estimate

These are rough estimates for running the full stack 24/7 in AWS us-east-1 + us-west-2. Production workloads should be sized to actual usage.

Component	Approx. Monthly Cost
6x EKS cluster control planes	~$216 ($0.10/hr × 6)
12x EC2 t3.medium nodes (2 per cluster)	~$300
6x EBS volumes (gp2, 50–100GB each)	~$60
NAT Gateways (2 per VPC, 6 VPCs)	~$250
ALBs (production + monitoring)	~$30
Route53 hosted zone + queries	~$5
ACM (free)	$0
ECR storage	~$5
CloudWatch logs (Fluent Bit)	~$15
S3 (Velero backups, CI audit)	~$10
AWS WAF	~$15
GuardDuty	~$10
Secrets Manager	~$2
Total	~$918/month

Cost reduction tip: For a demo/learning setup, use 1 node per cluster, t3.small instances, and skip the second region. This brings the cost to approximately $200–300/month.

Prerequisites

Before starting Part 2, ensure you have the following:

Tools to install:

# AWS CLI v2
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip && sudo ./aws/install

# Terraform 1.6+
brew install terraform   # or download from terraform.io

# Terragrunt 0.54+
brew install terragrunt  # or download from terragrunt.gruntwork.io

# kubectl
curl -LO "https://dl.k8s.io/release/$(curl -sL https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

# Helm 3
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

# ArgoCD CLI
curl -sSL -o argocd https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64
sudo install -m 555 argocd /usr/local/bin/argocd

# Cosign
curl -sSfL https://github.com/sigstore/cosign/releases/latest/download/cosign-linux-amd64 \
  -o cosign && sudo install -m 0755 cosign /usr/local/bin/cosign

# Trivy
sudo apt-get install wget apt-transport-https gnupg lsb-release
wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | sudo apt-key add -
echo "deb https://aquasecurity.github.io/trivy-repo/deb $(lsb_release -sc) main" \
  | sudo tee /etc/apt/sources.list.d/trivy.list
sudo apt-get update && sudo apt-get install trivy

AWS accounts needed:

AWS Organization root account (or a management account)
Three member accounts: dev, staging, production
AWS SSO (IAM Identity Center) configured

GitHub:

Two repositories: myapp (application code) and myapp-gitops (manifests)
A Personal Access Token for ArgoCD to pull from the GitOps repo

Domain name:

A registered domain you control (we use matthewoladipupo.dev)
Nameservers pointed to Route53

Series Roadmap

Part	Title
Part 1	Architecture Overview (this article)
Part 2	AWS Foundation: Organizations, SSO, and Account Setup
Part 3	Infrastructure as Code: Terraform Modules + Terragrunt
Part 4	EKS Multi-Cluster: Six Clusters Across Two Regions
Part 5	GitOps with ArgoCD: Hub-Spoke Model
Part 6	CI/CD Pipeline: GitHub Actions, Trivy, Cosign, ECR
Part 7	Secrets Management: AWS Secrets Manager + ESO + IRSA
Part 8	Security Stack: Kyverno, Falco, WAF, GuardDuty
Part 9	Observability: Prometheus, Grafana, Fluent Bit, CloudWatch
Part 10	Resilience: Karpenter, HPA, Argo Rollouts, Velero
Live Data	Appendix
Codes	Runbook

Screenshot Placeholders

SCREENSHOT: ArgoCD UI showing all 6 clusters registered and all ApplicationSets synced

SCREENSHOT: Grafana dashboard — Node CPU/Memory overview across production clusters

SCREENSHOT: GitHub Actions workflow showing all steps passing (lint → scan → build → sign → push → gitops update)

SCREENSHOT: AWS ECR showing signed image with cosign attestation tag

If this was useful, follow me on dev.to — I will publish Part 2 next Wednesday covering the AWS Organizations + IAM Identity Center setup.
Questions? Drop them in the comments — I read and reply to every one.

Next up: Part 2 — AWS Foundation: Organizations, SSO, and Account Setup

Source code: myapp-infra | myapp-gitops | myapp
Runbook: Operations Guide — every operational procedure for this pipeline.