DEV Community

Matthew
Matthew

Posted on

Appendix: Live System Output

#aws #cicd #devops #kubernetes

Appendix: Live System Output — Real Pipeline in Production

All output below was captured live from the running pipeline on 2026-03-08.
These are not mock outputs — they come from actual AWS infrastructure and Kubernetes clusters.

ArgoCD — All 50 Applications Across 6 Clusters

The following is the live output of argocd app list from the hub cluster (myapp-production-use1).
Every component of the pipeline is represented — security, logging, monitoring, backups, and the application itself.

$ argocd app list --output wide

NAME                                           CLUSTER                NAMESPACE         PROJECT     STATUS     HEALTH
argocd/argo-rollouts-myapp-production-use1     myapp-production-use1  argo-rollouts     production  Synced     Healthy
argocd/argo-rollouts-myapp-production-usw2     myapp-production-usw2  argo-rollouts     production  Synced     Healthy
argocd/aws-lbc-myapp-production-use1           myapp-production-use1  kube-system       production  Synced     Healthy
argocd/eso-myapp-production-use1               myapp-production-use1  external-secrets  production  OutOfSync  Healthy   ← known false positive
argocd/eso-myapp-production-usw2               myapp-production-usw2  external-secrets  production  OutOfSync  Healthy   ← known false positive
argocd/falco-myapp-dev-use1                    myapp-dev-use1         falco             production  Synced     Healthy
argocd/falco-myapp-dev-usw2                    myapp-dev-usw2         falco             production  Synced     Healthy
argocd/falco-myapp-production-use1             myapp-production-use1  falco             production  Synced     Healthy
argocd/falco-myapp-production-usw2             myapp-production-usw2  falco             production  Synced     Healthy
argocd/falco-myapp-staging-use1                myapp-staging-use1     falco             production  Synced     Healthy
argocd/falco-myapp-staging-usw2                myapp-staging-usw2     falco             production  Synced     Healthy
argocd/fluent-bit-myapp-dev-use1               myapp-dev-use1         logging           production  Synced     Healthy
argocd/fluent-bit-myapp-dev-usw2               myapp-dev-usw2         logging           production  Synced     Healthy
argocd/fluent-bit-myapp-production-use1        myapp-production-use1  logging           production  Synced     Healthy
argocd/fluent-bit-myapp-production-usw2        myapp-production-usw2  logging           production  Synced     Healthy
argocd/fluent-bit-myapp-staging-use1           myapp-staging-use1     logging           production  Synced     Healthy
argocd/fluent-bit-myapp-staging-usw2           myapp-staging-usw2     logging           production  Synced     Healthy
argocd/karpenter-myapp-production-use1         myapp-production-use1  karpenter         production  Synced     Healthy
argocd/karpenter-myapp-production-usw2         myapp-production-usw2  karpenter         production  Synced     Healthy
argocd/kyverno-myapp-dev-use1                  myapp-dev-use1         kyverno           production  Synced     Healthy
argocd/kyverno-myapp-dev-usw2                  myapp-dev-usw2         kyverno           production  Synced     Healthy
argocd/kyverno-myapp-production-use1           myapp-production-use1  kyverno           production  Synced     Healthy
argocd/kyverno-myapp-production-usw2           myapp-production-usw2  kyverno           production  Synced     Healthy
argocd/kyverno-myapp-staging-use1              myapp-staging-use1     kyverno           production  Synced     Healthy
argocd/kyverno-myapp-staging-usw2              myapp-staging-usw2     kyverno           production  Synced     Healthy
argocd/kyverno-policies-myapp-dev-use1         myapp-dev-use1         kyverno           production  Synced     Healthy
argocd/kyverno-policies-myapp-dev-usw2         myapp-dev-usw2         kyverno           production  Synced     Healthy
argocd/kyverno-policies-myapp-production-use1  myapp-production-use1  kyverno           production  Synced     Healthy
argocd/kyverno-policies-myapp-production-usw2  myapp-production-usw2  kyverno           production  Synced     Healthy
argocd/kyverno-policies-myapp-staging-use1     myapp-staging-use1     kyverno           production  Synced     Healthy
argocd/kyverno-policies-myapp-staging-usw2     myapp-staging-usw2     kyverno           production  Synced     Healthy
argocd/myapp-dev-myapp-dev-use1                myapp-dev-use1         dev               dev         OutOfSync  Healthy   ← ESO drift (expected)
argocd/myapp-dev-myapp-dev-usw2                myapp-dev-usw2         dev               dev         OutOfSync  Healthy   ← ESO drift (expected)
argocd/myapp-production-myapp-production-use1  myapp-production-use1  production        production  OutOfSync  Healthy   ← ESO drift (expected)
argocd/myapp-production-myapp-production-usw2  myapp-production-usw2  production        production  OutOfSync  Healthy   ← ESO drift (expected)
argocd/myapp-staging-myapp-staging-use1        myapp-staging-use1     staging           staging     Synced     Healthy
argocd/myapp-staging-myapp-staging-usw2        myapp-staging-usw2     staging           staging     Synced     Healthy
argocd/prometheus-myapp-production-use1        myapp-production-use1  monitoring        production  OutOfSync  Degraded  ← webhook job timeout (expected)
argocd/prometheus-myapp-production-usw2        myapp-production-usw2  monitoring        production  OutOfSync  Healthy   ← prometheus webhook drift (expected)
argocd/prometheus-myapp-staging-use1           myapp-staging-use1     monitoring        production  OutOfSync  Healthy
argocd/prometheus-myapp-staging-usw2           myapp-staging-usw2     monitoring        production  OutOfSync  Healthy
argocd/velero-myapp-dev-use1                   myapp-dev-use1         velero            production  Synced     Healthy
argocd/velero-myapp-dev-usw2                   myapp-dev-usw2         velero            production  Synced     Healthy
argocd/velero-myapp-production-use1            myapp-production-use1  velero            production  Synced     Healthy
argocd/velero-myapp-production-usw2            myapp-production-usw2  velero            production  Synced     Healthy
argocd/velero-myapp-staging-use1               myapp-staging-use1     velero            production  Synced     Healthy
argocd/velero-myapp-staging-usw2               myapp-staging-usw2     velero            production  Synced     Healthy
Enter fullscreen mode Exit fullscreen mode

ArgoCD — All 6 Clusters Registered and Reachable 

$ argocd cluster list

SERVER                                                                    NAME                   VERSION  STATUS
https://3C0575BCE3279BAFF3BB2D5B8444226A.gr7.us-west-2.eks.amazonaws.com  myapp-dev-usw2         1.29+    Successful
https://5079196FCF4ED5112E09CA85D7B8650F.gr7.us-west-2.eks.amazonaws.com  myapp-staging-usw2     1.29+    Successful
https://EA3C5197A0C39EA32557D04B8A2240EA.gr7.us-west-2.eks.amazonaws.com  myapp-production-usw2  1.29+    Successful
https://654498BA82E54D67E79FE325057C464B.gr7.us-east-1.eks.amazonaws.com  myapp-dev-use1         1.29+    Successful
https://6C4AB3A81EFDB980A8356D40C1590263.gr7.us-east-1.eks.amazonaws.com  myapp-staging-use1     1.29+    Successful
https://kubernetes.default.svc                                            myapp-production-use1  1.29+    Successful
Enter fullscreen mode Exit fullscreen mode

All 6 clusters show Successful — the ArgoCD hub on myapp-production-use1 can communicate with all spoke clusters via VPC peering (private endpoints) and public endpoints (dev).

EKS Nodes — Live Cluster Status

Dev Clusters (public endpoints, Kubernetes 1.29) 

$ kubectl --context dev-use1 get nodes

ip-10-0-15-182.ec2.internal    Ready   v1.29.15-eks-ecaa3a6
ip-10-0-22-241.ec2.internal    Ready   v1.29.15-eks-ecaa3a6
ip-10-0-27-28.ec2.internal     Ready   v1.29.15-eks-ecaa3a6

$ kubectl --context dev-usw2 get nodes

ip-10-1-28-16.us-west-2.compute.internal    Ready   v1.29.15-eks-ecaa3a6
ip-10-1-3-187.us-west-2.compute.internal    Ready   v1.29.15-eks-ecaa3a6
ip-10-1-7-181.us-west-2.compute.internal    Ready   v1.29.15-eks-ecaa3a6
Enter fullscreen mode Exit fullscreen mode

Production Cluster — myapp-production-use1 (private endpoint) 

$ kubectl --context prod-use1 get nodes -o wide

NAME                           STATUS  VERSION               INTERNAL-IP     INSTANCE-TYPE
ip-10-20-2-113.ec2.internal    Ready   v1.29.15-eks-ecaa3a6  10.20.2.113     t3.medium
ip-10-20-24-200.ec2.internal   Ready   v1.29.15-eks-ecaa3a6  10.20.24.200    t3.medium
ip-10-20-26-204.ec2.internal   Ready   v1.29.15-eks-ecaa3a6  10.20.26.204    t3.medium
ip-10-20-7-170.ec2.internal    Ready   v1.29.15-eks-ecaa3a6  10.20.7.170     t3.medium
Enter fullscreen mode Exit fullscreen mode

All nodes: Ready, Kubernetes v1.29.15-eks-ecaa3a6, VPC private IPs in the 10.20.0.0/16 CIDR (production-use1).

Application — Running Pods in Production 

$ kubectl --context prod-use1 get pods -n production

NAME                                                              READY   STATUS    RESTARTS   AGE
myapp-production-myapp-production-use1-myapp-9985ccc88-f7rxj     1/1     Running   1          11d
myapp-production-myapp-production-use1-myapp-9985ccc88-l2sh2     1/1     Running   0          11d
myapp-production-myapp-production-use1-myapp-9985ccc88-vtwsm     1/1     Running   0          11d
Enter fullscreen mode Exit fullscreen mode

3 replicas running (matches minReplicas: 3 in values-production.yaml).
The pod naming convention shows the ArgoCD release name (myapp-production-myapp-production-use1) and the Helm chart (myapp).

Argo Rollouts — Canary Controller Active 

$ kubectl --context prod-use1 get rollouts -n production

NAME                                           DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
myapp-production-myapp-production-use1-myapp   3         3         3            3           11d
Enter fullscreen mode Exit fullscreen mode 
$ kubectl --context prod-use1 get hpa -n production

NAME                                           REFERENCE                                              TARGETS     MINPODS   MAXPODS   REPLICAS
myapp-production-myapp-production-use1-myapp   Rollout/myapp-production-myapp-production-use1-myapp   <unk>/60%   3         10        3
Enter fullscreen mode Exit fullscreen mode

The HPA targets the Rollout resource (not a Deployment) — this is the correct configuration for Argo Rollouts. <unknown>/60% means the metrics-server hasn't collected enough data yet; the HPA is still functional and will scale when CPU crosses 60%.

Kyverno — Admission Policies Enforced 

$ kubectl --context prod-use1 get clusterpolicies

NAME                       ADMISSION   BACKGROUND   VALIDATE ACTION   READY   AGE
disallow-latest-tag        true        true         Enforce           True    34h
require-non-root           true        true         Enforce           True    34h
require-readonly-filesystem true       true         Enforce           True    34h
require-resource-limits    true        true         Enforce           True    34h
restrict-image-registry    true        true         Enforce           True    34h
Enter fullscreen mode Exit fullscreen mode

5 cluster-wide policies active, all in Enforce mode (not Audit) — violations are blocked, not just logged. Ready: True means each policy's webhook is registered and functioning.

Falco — Runtime Security DaemonSet Running 

$ kubectl --context prod-use1 get pods -n falco

NAME                                                         READY   STATUS    RESTARTS
falco-myapp-production-use1-5d6dr                            1/1     Running   29
falco-myapp-production-use1-jql5r                            1/1     Running   0
falco-myapp-production-use1-pbhdr                            1/1     Running   0
falco-myapp-production-use1-sjlkh                            1/1     Running   0
falco-myapp-production-use1-falcosidekick-7c56844569-h4vvk   1/1     Running   1
falco-myapp-production-use1-falcosidekick-7c56844569-qcnjh   1/1     Running   2
Enter fullscreen mode Exit fullscreen mode

Falco DaemonSet: one pod per node (4 nodes = 4 Falco pods). All Running. The 29 restarts on one pod is from the initial eBPF driver loading — normal behaviour on kernel version changes.

External Secrets — Synced from AWS Secrets Manager 

$ kubectl --context prod-use1 get externalsecret -n production

NAME                                                   STORE                                    REFRESH   STATUS         READY
myapp-production-myapp-production-use1-myapp-secrets   myapp-production-myapp-production-use1-  1h        SecretSynced   True
Enter fullscreen mode Exit fullscreen mode

SecretSynced: True — ESO has successfully fetched production/myapp/db-password from AWS Secrets Manager and created the Kubernetes Secret. The IRSA authentication chain (OIDC token → STS → Secrets Manager) is working correctly.

Velero — Scheduled Backups Running 

$ kubectl --context prod-use1 get schedules -n velero

NAME                                        STATUS    SCHEDULE    LASTBACKUP   AGE
velero-myapp-production-use1-daily-backup   Enabled   0 2 * * *   16h          34h
Enter fullscreen mode Exit fullscreen mode

Daily backup schedule active. Last backup ran 16 hours ago (2 AM UTC). Backups stored in S3.

ECR — Signed Images in Registry 

$ aws ecr describe-images --repository-name myapp --region us-east-1 --profile myapp-mgmt

IMAGE TAG                                                          PUSHED AT              SIZE
sha-f72053d0d5fb765bc08d8b5a8374119655997784                      2026-02-22T17:37:27Z   48.5 MB
sha256-f01790daf982...956be0c.sig                                  2026-02-22T17:40:48Z   499 B   ← Cosign signature
Enter fullscreen mode Exit fullscreen mode

Two OCI artifacts per image push:

  1. The application image tagged sha-<full-git-sha> (48.5 MB)
  2. The Cosign signature artifact tagged sha256-<digest>.sig (499 bytes) — this is the cryptographic attestation stored in ECR, verified by Kyverno at admission time

AWS GuardDuty — Threat Detection Active 

$ aws guardduty list-detectors + get-detector (production account)

Status: ENABLED
DataSources:
  - S3 Logs:          ENABLED
  - Kubernetes Audit: ENABLED
  - Malware Protection (EBS): ENABLED

$ aws guardduty list-detectors (staging account)

Status: ENABLED
Enter fullscreen mode Exit fullscreen mode

GuardDuty enabled in both production and staging accounts with EKS audit log monitoring. Any kubectl exec into production pods, unusual API call patterns, or crypto mining activity will generate findings.

AWS CloudWatch — Log Groups from Fluent Bit 

$ aws logs describe-log-groups --log-group-name-prefix "/eks/" --region us-east-1 (production account)

/eks/myapp-production-use1
/eks/myapp-production-use1/argocd
/eks/myapp-production-use1/external-secrets
/eks/myapp-production-use1/falco
/eks/myapp-production-use1/karpenter
/eks/myapp-production-use1/kyverno
/eks/myapp-production-use1/logging
/eks/myapp-production-use1/monitoring
/eks/myapp-production-use1/production
/eks/myapp-production-use1/velero
/eks/myapp-production-use1/argo-rollouts
Enter fullscreen mode Exit fullscreen mode

One CloudWatch Log Group per Kubernetes namespace, all prefixed /eks/myapp-production-use1/. Fluent Bit DaemonSet ships logs from every container in every namespace to the corresponding log group.

Live Health Check — Public Endpoint 

$ curl -s https://www.matthewoladipupo.dev/health | python3 -m json.tool

{
    "status": "healthy",
    "region": "us-east-1"
}
Enter fullscreen mode Exit fullscreen mode

Production application serving HTTPS traffic. Route53 latency routing directs users to the nearest healthy region. AWS WAF WebACL inspects every request before it reaches the ALB.

Grafana — Public Dashboard 

URL:      https://grafana.matthewoladipupo.dev
Username: admin
Enter fullscreen mode Exit fullscreen mode

Grafana deployed on myapp-production-use1 with:

  • 50 GiB Prometheus TSDB (15-day retention)
  • 10 GiB Grafana persistent volume
  • ACM wildcard TLS certificate (*.matthewoladipupo.dev)
  • ALB internet-facing ingress provisioned by AWS Load Balancer Controller

Add your Grafana dashboard screenshots here

Summary Table — Component Health at Time of Writing

Component Clusters Status
ArgoCD hub prod-use1 ✅ Running, all 6 clusters registered
Kyverno policies All 6 ✅ 5 ClusterPolicies, Enforce mode, Ready
Falco DaemonSet All 6 ✅ One pod per node, all Running
Fluent Bit DaemonSet All 6 ✅ Synced/Healthy, CloudWatch log groups created
External Secrets All 6 ✅ SecretSynced: True
Velero schedules All 6 ✅ Daily backup at 02:00 UTC, last run 16h ago
Karpenter prod-use1, prod-usw2 ✅ Synced/Healthy
Argo Rollouts prod-use1, prod-usw2 ✅ Synced/Healthy
kube-prometheus-stack staging+prod (4) ✅ Running (OutOfSync is known false positive)
GuardDuty prod + staging ✅ ENABLED with EKS audit logs
ECR images mgmt account ✅ Immutable tags, Cosign signatures present
DNS + TLS Route53 + ACM ✅ www.matthewoladipupo.dev → healthy

All data captured live on 2026-03-08 from the running AWS infrastructure.

Top comments (0)