This is a technical breakdown of a migration we ran for a global B2B service (strict NDA, so no names). The setup was unusual: a release cycle of one day or less, no CTO or architect on the client side, a junior dev team, and acceptable downtime measured in minutes. Our side: two part-time DevOps engineers.
The short business version with numbers lives in the original case study. Here's the engineering side: what was breaking, what stack we put together, and where things bit us.
The Problem & Stack
The starting point was a legacy setup on AWS ECR/ECS. ECS itself isn't the problem – but here everything went wrong at once:
- CI/CD pipelines were failing constantly. Every failure blocked developers, and at "release once a day" velocity, that meant hours of patching instead of shipping features.
- Zero observability – troubleshooting meant log archaeology.
- No fast, safe rollback: reverting was manual and terrifying.
Context made it worse: the client had burned through 8 DevOps teams in the six months before we came on board. The problem wasn't just the technology – no technical owner was making foundational decisions. All communication ran through the CEO and CMO, with dozens of micro-calls per day.
The classic dilemma at kick-off (Discovery): quick win vs. rewrite – patch the existing ECR/ECS and get to go-live faster, or rewrite everything from scratch on Amazon EKS. We made the case that running on the old architecture from day one put production stability at risk, along with the trust of the earliest users. The client decided not to take that risk.
The key decision that took 90% of the stress off the table: we stood up EKS in parallel with the old system and only cut over once the new environment was ready – no "big bang" at 2 am. Traffic was shifted gradually at the load balancer/DNS level: start with a small percentage on the new EKS, monitor error rate and latency in Datadog, and scale up to 100%. The old ECS stayed as a hot fallback the whole time; we only shut it down after several days of stable 100% traffic on the new setup.
The Architecture Solution
EKS instead of ECS. We needed an ecosystem for GitOps, fine-grained node control, and service mesh – on ECS, that means workarounds or third-party tools. The foundation: 2 isolated EKS clusters. Splitting environments gives you blast radius isolation and cleaner access control.
Mongo Atlas via VPC Peering. State lives in Mongo Atlas; we set up VPC Peering between our VPC and the Atlas VPC – database traffic goes over a private channel, not the public internet.
Karpenter instead of Cluster Autoscaler. FinOps was critical here, and Karpenter gives you mixed node groups (Spot + On-Demand) and bin-packing out of the box. It provisions nodes against actual pod req/limits rather than pre-sliced ASGs.
Istio instead of bare Ingress. We needed mTLS between services, traffic control, and mesh-level observability – all in one layer.
ArgoCD instead of CI push-deploys. GitOps gives you exactly what we were building toward: rollback in git = rollback in the cluster.
Datadog APM instead of Prometheus/Grafana. We deliberately went against the default here – more on that below.
Implementation Details & Code Blocks
1. IaC: Everything Through Terraform, Zero Manual Clicks
All infrastructure is code: no kubectl apply by hand in production, everything goes through Terraform and ArgoCD. We'll skip the basic cluster skeleton (standard terraform-aws-modules/eks/aws) and focus on IRSA (IAM Roles for Service Accounts). Without proper IRSA, neither Karpenter nor External Secrets can reach the AWS API with least-privilege access – and dropping credentials into pod env vars in production under ISO 27001 isn't an option.
The role for Karpenter, bound to its service account via the cluster's OIDC provider:
# example for this post: real ARNs/policies are broader
data "aws_iam_policy_document" "karpenter_assume" {
statement {
actions = ["sts:AssumeRoleWithWebIdentity"]
principals {
type = "Federated"
identifiers = [module.eks.oidc_provider_arn]
}
condition {
test = "StringEquals"
variable = "${module.eks.oidc_provider}:sub"
values = ["system:serviceaccount:karpenter:karpenter"]
}
}
}
resource "aws_iam_role" "karpenter" {
name = "karpenter-controller"
assume_role_policy = data.aws_iam_policy_document.karpenter_assume.json
}
The same pattern (AssumeRoleWithWebIdentity + a condition scoped to a specific serviceaccount) we used for External Secrets as well – this is exactly the kind of thing that gets credited during security hardening for ISO 27001.
2. Karpenter: Mixed Spot/On-Demand + Graviton
The tastiest part from a FinOps perspective. We handed node management entirely to Karpenter, let it mix Spot with On-Demand, and moved to Graviton ARM64 (r8g) – roughly 20% cheaper than x86. The key in the NodePool is getting the requirements right; otherwise, Karpenter either only picks expensive instances or dumps critical workloads onto Spot:
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
name: default
spec:
template:
spec:
requirements:
- key: karpenter.sh/capacity-type
operator: In
values: ["spot", "on-demand"] # mixed
- key: karpenter.k8s.aws/instance-family
operator: In
values: ["r8g"] # r8g = Graviton ARM64, specifying arch is redundant
disruption:
consolidationPolicy: WhenEmptyOrUnderutilized # bin-packing
consolidateAfter: 1m
limits:
cpu: "200" # example for this post, real limit depends on your workload profile
consolidationPolicy: WhenEmptyOrUnderutilized is the bin-packing that trimmed our node count by 30–35%: Karpenter periodically repacks pods onto fewer nodes and terminates the rest. We intentionally skip the arch (arm64) requirement: the r8g family only exists on Graviton, so Karpenter infers the architecture from instance-family. A redundant constraint just clutters the manifest.
Pitfall #1. Spot nodes die without warning. On a stateful workload or long-lived connections without graceful shutdown, you'll see 5xx on every interruption. Fix it with terminationGracePeriodSeconds, PodDisruptionBudget, and pinning critical pods to On-Demand via nodeAffinity. Don't throw everything onto Spot just because it's cheap.
3. ArgoCD: Rollback = git revert
GitOps delivered the biggest qualitative leap: deployment and rollback became git operations, not cluster operations.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: payment-api
spec:
project: default
source:
repoURL: git@github.com:client/k8s-manifests.git
targetRevision: HEAD
path: apps/payment-api
destination:
server: https://kubernetes.default.svc
namespace: payment
syncPolicy:
automated:
prune: true
selfHeal: true # manual changes in the cluster get rolled back to git state
selfHeal: true is both a safety net and a pitfall. Safety net: any manual change in the cluster (that classic kubectl edit under incident pressure) automatically reverts to git state. Pitfall: if someone edits a resource by hand to "quickly put out the fire," ArgoCD silently rolls it back – and they have no idea why their fix "disappeared." The cure is discipline: everything goes through a PR, and for emergencies, temporarily disable auto-sync on the specific application. Rolling back a bad release now looks like this:
git revert <bad-commit-sha>
git push origin main
# ArgoCD picks up the change and restores the cluster to the previous state in seconds
That's what gave us MTTR −82%: rollbacks stopped being late-night incidents.
4. Istio: mTLS and Traffic Control
Service mesh covered mTLS between services and traffic control in one layer. Enforcing strict mTLS at the namespace level:
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
name: default
namespace: payment
spec:
mtls:
mode: STRICT # encrypted traffic between pods only
Pitfall #2. Enabling STRICT mTLS cluster-wide will immediately take down any services not yet in the mesh. Roll it out namespace by namespace, starting with PERMISSIVE, and on two isolated clusters, do it independently – don't try to flip the switch globally all at once.
5. Why Datadog Instead of Prometheus/Grafana
The most counterintuitive call here. The default DevOps instinct is Prometheus + Grafana (we support both, plus OpenTelemetry). But given the context – junior team, no CTO, relentless pace – the value of a tool isn't its raw power; it's how fast a developer with no DevOps background can find the root cause on their own. Datadog APM gave us exactly the developer-facing interface where a junior opens a trace and immediately sees which service is dragging, no PromQL spelunking required. A deliberate trade-off: slightly more money for SaaS in exchange for a significant drop in troubleshooting overhead. For this client, it paid off.
Pitfall #3. Secrets sprawl across Helm charts fast. We killed that immediately with External Secrets Operator + AWS Secrets Manager, and handed TLS certificates to cert-manager. Secrets pull from Secrets Manager into the cluster declaratively, no plaintext in git:
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: payment-db
namespace: payment
spec:
refreshInterval: 1h
secretStoreRef:
name: aws-secrets-manager
kind: ClusterSecretStore
target:
name: payment-db-creds # ESO drops a ready k8s Secret here
data:
- secretKey: mongo-uri
remoteRef:
key: prod/payment/mongo-uri
Otherwise, a month later, you have secrets in three places, and nobody knows which one is current.
Security as a Stack, Not a Checkbox
ISO 27001 at the end isn't marketing – it's the result of a specific stack. On the perimeter: CloudFront (CDN) and AWS WAF filtering traffic before it reaches the cluster; inside, SecurityHub and Inspector run continuous vulnerability monitoring (CVE scanning in images, misconfiguration detection in AWS resources). Access is controlled via RBAC, external traffic comes in through the AWS Load Balancer Controller (ALB/NLB), and at the pod level – IRSA with least-privilege, mTLS via Istio, and secrets via ESO. It's the combination, not any single tool, that got the client through ISO 27001 and laid the foundation for SOC 2.
Culture: Why Infrastructure Won't Save You Without Code Review
This isn't about YAML, but technically, it saved production more than anything else. With no architect and no CTO, we introduced strict, close code review on every PR – that alone cut hours of patching and downtime. The lesson I'd carve in stone: if you're joining a project with no technical owner, push the client to hire an architect from day one. Until then, code review is your only gatekeeper.
Separately – close collaboration with the client's AI team. The validation service is tied directly to their ML models, and coordinating at the infrastructure level (right resources for inference, workload isolation, control over what deploys and when) saved the business from real reputational damage more than once. The takeaway: when infrastructure serves ML workloads, DevOps can't live in a vacuum from the people writing the models.
Engineering Metrics
What we got in numbers after cutting over to the new setup:
- Deployment frequency ×4 – through full GitOps automation via ArgoCD.
- *MTTR −82% *– rollback via git revert, zero-downtime deploys.
- Compute costs: up to −80% on Spot instances via Karpenter; −30–35% node count reduction via auto-consolidation; ~−20% from moving to Graviton ARM64. Combined: roughly $45,600/year in savings.
- 99.7% uptime, Change Failure Rate down 4%.
- Infrastructure went through full security hardening, earning the client ISO 27001 and a foundation for SOC 2.
The non-technical takeaway: infrastructure went from a blocker to a driver. Developers stopped waiting on deploys and stopped fearing rollbacks.
Discussion
The most controversial call in this project, for me, was going with Datadog over the usual Prometheus/Grafana stack – purely for the junior team's sake. It works, but it's vendor lock-in, and it costs money.
Question for the readers: how do you balance self-hosted observability (Prometheus/Grafana/OTel) vs. SaaS like Datadog when the people actually staring at dashboards are junior? Do you push for self-hosted for control and cost, or do you trade interface convenience for faster MTTR?
And for anyone running Spot through Karpenter in production: what's your real interruption rate, and how do you handle stateful workloads? Would love to compare numbers.
Top comments (0)