DEV Community: Alpacked

The Self-Hosted LLM Breakeven Point Isn't 2M Tokens a Day. It's a Ratio.

Alpacked — Sun, 23 Aug 2026 15:17:58 +0000

Self-hosting an LLM is not automatically cheaper. This is the single most common mistake in these calculations: people price the GPU, compare it against last month's API bill, and walk away with a number that has almost nothing to do with what they'll actually spend.

Savings show up under two conditions - when you've calculated the breakeven point for your specific workload, and when you've split traffic between a local model and an API instead of attempting a wholesale migration.

Enterprise spending on model APIs more than doubled in six months - from $3.5 billion in late 2024 to $8.4 billion by mid-2025, according to Menlo Ventures. Inference in production, not model development, is what drove it.

TL;DR

A production instance on a consumer GPU costs roughly $850/month, and more than half of that is an engineer, not hardware.
Breakeven isn't a token volume. It's $850 ÷ (your blended API price × 30) - which lands anywhere from 2.3M to 94M tokens/day depending on which tier you're displacing.
The "2M tokens/day" rule of thumb is correct for replacing frontier calls and badly wrong for anything cheaper.
The architecture that works is hybrid: local for high-volume and regulated traffic, API for hard reasoning.

Why teams start doing this math

Three pressures, usually arriving in this order:

Cost scales linearly with volume. There's no ceiling. The product grows, the bill grows with it. For an API that's both a feature (you don't pay for idle capacity) and a problem (you can't fix a budget).
Data leaves your perimeter. For HIPAA or PCI-DSS, an enterprise agreement is not a substitute for controlling your own infrastructure. Every prompt physically travels somewhere else.
Network round-trips. 50–200ms per request, which you feel in any real-time UX.

What a local instance actually costs

The figure $850/month shows up throughout this post as the cost of one production instance on a consumer GPU. Here's where it comes from - because in most write-ups this number is either pulled from thin air or collapsed into "the price of a graphics card."

A few notes on each:

Capex. The RTX 5090 carries an official MSRP of $1,999, but GDDR7 shortages kept street prices at $3,500–4,000+ through early 2026. Add a 1,200W+ PSU and serious cooling for its 575W TDP, plus ~$1,200 for CPU, RAM, and chassis. Check current pricing before you build this into a business case - the memory market moves in weeks.
Electricity. Substitute your own rate. The spread between jurisdictions is severalfold, not a rounding error. An RTX 4090 at 450W still costs $50+/month for the card alone.
Operations. This is the largest line item and the one everyone underestimates. An MLOps engineer who updates models, debugs CUDA errors, and keeps monitoring honest costs more than all the hardware combined.

More than half of that total isn't silicon or electricity - it's a person. If nobody on the team has that skill set, self-hosting isn't cheaper. It just moves the cost into a budget line that doesn't exist yet.

Why this number matters
The important property of $850 is that it's fixed. It doesn't move until the server hits its throughput ceiling. Everything below is a comparison between a flat line and a linear one.

The breakeven point isn't a volume - it's a ratio

The industry rule of thumb says self-hosting becomes competitive above 2 million tokens per day. The rule works, but it's incomplete: it says nothing about what you're replacing. That turns out to be the dominant variable.

Step 1 - Blended price per tier

At a 70% input / 30% output split:

That 70/30 split is doing real work here, so adjust it. Chat and summarization workloads skew further toward input and land cheaper; agentic loops that generate long tool calls skew toward output and land considerably more expensive.

Step 2 - Monthly bill by volume

The formulas:
monthly API cost = volume in M/day × price per 1M × 30 breakeven volume = infrastructure cost ÷ (price per 1M × 30)

Step 3 - Where "2M" actually comes from

Plug $850 into the second formula and the answer splits three ways:

Against the cheap tier - ~94M tokens/day
Against mid tier - ~5.7M tokens/day
Against frontier - ~2.3M tokens/day

There it is. The 2M rule is the breakeven for displacing frontier calls, not calls in general. If you're running classification through the cheapest tier available, a local server will essentially never pay for itself - and that's a correct, useful conclusion, not a failure.
It also explains why published estimates disagree by an order of magnitude. They're pricing different substitutions.

Two things that shift the math

Consolidation. That fixed $850 doesn't serve one workload. A single node can run an internal helpdesk bot, a document analysis pipeline, and a coding assistant at the same time. Three applications at 700K tokens/day each look uneconomical in isolation; together they're 2.1M on one server at the same fixed cost. Fine-tuning works the same way - with a provider, every training run is a separate charge; on your own hardware it's capacity you've already paid for.

Payback period. At a steady 10M+ tokens/day against the mid tier:

That lands squarely inside the 6–12 month range usually quoted for 10M+ workloads.

Architecture: hybrid, not migration

Moving everything local is the wrong goal. GPT-5.6, Claude Opus 5, and Gemini 3.1 Pro (as of publication - this list will age before you finish reading) simply aren't available for self-hosting. The question isn't what replaces them. It's what you stop sending to them.

What goes local

Simple, high-volume, and sensitive traffic:
Ticket classification and routing
Form and document data extraction
Boilerplate response generation
Routine autocomplete and code review in the IDE

A support team handling hundreds of near-identical tickets a day is the archetype: predictable volume, straightforward tasks, and frequently customer PII that can't go anywhere. What counts as "local-grade" has shifted a lot, too - a well-quantized 32B now handles work that needed a 70B two years ago.

What stays on the API

Low volume, high cost of being wrong:
Complex contract or document analysis
Unfamiliar codebases
Genuinely novel architectural decisions
Anything where multi-step reasoning quality is the product

A legal team reviewing contracts once a week is the archetype. The call is expensive and entirely justified.

What determines your savings

Published cases land anywhere from −33% to −83%, and the commonly cited range for teams that reach a hybrid setup is 40–70% versus a fully API-dependent stack. Where you fall inside that spread comes down to two things:

How much of your traffic is even allowed to leave the perimeter.
Which tier it was hitting before the move.

Displacing a cheap tier yields a smaller percentage for the same engineering effort - see the arithmetic above. The spread between two projects usually says more about their compliance posture than about their engineering.

VRAM: the constraint everything else measures against

The baseline rule: 0.5GB of VRAM per billion parameters at 4-bit quantization. If the model doesn't fit entirely in VRAM, the system falls back to CPU and throughput drops 10–100x. The gap between 30 TPS and 1 TPS is the gap between a working system and a broken one.

Sizing

Treat the speed column as a conservative floor for single-stream generation through Ollama. On an optimized serving stack the same model classes go several times faster - see the inference server table below.

Bandwidth, not TFLOPS

For inference, memory bandwidth decides:

RTX 5090 - 1.79 TB/s GDDR7, 32GB, ~213 tok/s on 8B models
RTX 4090 - 24GB, 128 tok/s. The 5090 is 67% ahead
RTX 3090 - still earns its place used, for the same 24GB at a fraction of the price

CUDA remains the default. AMD (ROCm) and Intel (IPEX-LLM) will run basic inference, but with noticeably more configuration friction: a large share of models, optimized kernels, and serving frameworks are NVIDIA-first or NVIDIA-only, and some models simply won't start outside CUDA.

Quantization and a starting model

Q4_K_M is the default. Going from Q8 to Q4 nearly halves memory for quality loss you're unlikely to notice. On newer hardware, AWQ and NVFP4 (Blackwell) push the quality/memory tradeoff further.
A sensible starting model is Qwen3.5-9B (~6.6GB, 262K native context) - it fits on a consumer card with room to spare. Check the actual size of whatever tag you pull before you deploy; it varies meaningfully between quantizations.

Implementation

Step 1 - Dev setup: Ollama + Open WebUI

yaml
# docker-compose.yml
services:
  ollama:
    image: ollama/ollama:latest
    ports:
      # Ollama listens on all interfaces by default - bind to loopback,
      # expose it only through a gateway or VPN
      - "127.0.0.1:11434:11434"
    volumes:
      - ollama_models:/root/.ollama   # models survive restarts and image updates
    environment:
      - OLLAMA_MAX_LOADED_MODELS=1    # >1 on a single card almost always means OOM
      - OLLAMA_KEEP_ALIVE=30m         # unload the model after idle
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    ports:
      - "127.0.0.1:3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    volumes:
      - webui_data:/app/backend/data
    depends_on:
      - ollama

volumes:
  ollama_models:
  webui_data:

Two things worth knowing before you run this:

Resource floor. Budget at least 8GB RAM for containers and 50+ GB of disk for models. Ollama detects hardware on its own: GPU if there is one, CPU if there isn't, just slower.
The migration lever. Ollama exposes an OpenAI-compatible API on port 11434. Anything built on the OpenAI SDK switches to the local endpoint by changing one variable. No code changes.

Step 2 - Production: vLLM
Ollama handles up to 4 concurrent requests. That's fine for development and testing. The moment real simultaneous users show up is the moment to move to vLLM.

Those TPS ranges span multiple model sizes and GPU generations - the low end is larger models on an RTX 4090, the high end is 8B on a 5090. Use them to compare tools against each other, not as a benchmark. Measure your own numbers on your own workload.

A note on the multipliers in marketing material: published PagedAttention figures range from 19x to 24x over HuggingFace Transformers depending on benchmark, model, and dataset. The metric that actually matters is P99 under your real concurrency. A reasonable expectation from vLLM on a typical profile: under 100ms at 128 concurrent requests.

Ray Serve and TGI win for teams already living in the corresponding ML stack. Without that stack, they add complexity without a payoff.

bash
docker run --gpus all --ipc=host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -p 127.0.0.1:8000:8000 \
  vllm/vllm-openai:latest \
  --model "$MODEL_REPO" \                # Hugging Face repo id
  --served-model-name local-default \    # stable name for clients, decoupled from repo id
  --host 0.0.0.0 --port 8000 \
  --api-key "$VLLM_API_KEY" \            # basic auth
  --quantization awq \                   # only if you actually have AWQ weights
  --gpu-memory-utilization 0.90 \        # 0.95+ hits OOM on KV-cache spikes
  --max-model-len 16384 \                # the main lever on VRAM consumption
  --max-num-seqs 64 \                    # ceiling on concurrent sequences
  --tensor-parallel-size 1                # >1 only for multi-GPU

Three flags people trip over:

--ipc=host is mandatory. Without it, shared memory is capped at the default 64MB and the process dies during batching.
--gpu-memory-utilization counts against total card memory, not free memory. If anything else is already resident on the GPU, dial it down manually.
--max-num-seqs is your real concurrency ceiling. vLLM removes the hard 4-request limit, but concurrency isn't unbounded - it's this flag plus whatever VRAM is left for KV-cache. Capacity planning starts there.

Step 3 - The router

The simplest router that works is a static map from task type to model. The temptation to put a classifier in front of inference is strong, but a classifier is one more model call, one more source of latency, and one more thing that can fail. A static map can be debugged with grep and never lies to you.

The more consequential decision is what happens when the local instance goes down, and one policy for all traffic doesn't work here:

Ordinary tasks fail over to the API. A slightly slower response, and that's the end of it.
Regulated traffic fails hard - queue it or return 503. Automatically falling back to an external API with sensitive data is precisely the incident this whole architecture was built to prevent.

python
import os
import httpx
from openai import (
    OpenAI,
    APIConnectionError,
    APITimeoutError,
    InternalServerError,
    RateLimitError,
)

LOCAL = OpenAI(
    base_url="http://127.0.0.1:8000/v1",
    api_key=os.environ["VLLM_API_KEY"],
    timeout=httpx.Timeout(30.0, connect=2.0),
    max_retries=1,
)
# Example uses an OpenAI-compatible provider. For Claude or Gemini,
# use that vendor's SDK or a proxy layer with an OpenAI-compatible endpoint.
FRONTIER = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

LOCAL_TASKS = {"classify", "extract", "summarize_ticket", "autocomplete"}

LOCAL_MODEL = os.environ["LOCAL_MODEL"]        # must match --served-model-name
FRONTIER_MODEL = os.environ["FRONTIER_MODEL"]

# Deliberately narrow: connection, timeout, 5xx, 429.
# A 400 or 401 is a bug in our config, not a local outage - let it surface.
TRANSIENT = (APITimeoutError, APIConnectionError, InternalServerError, RateLimitError)


def route(task: str, messages: list, regulated: bool = False):
    # `regulated` is set by the calling service: PCI-DSS for payment data,
    # HIPAA for PHI. Different rules, one thing in common:
    # this traffic does not leave the perimeter.
    if not (regulated or task in LOCAL_TASKS):
        return FRONTIER.chat.completions.create(model=FRONTIER_MODEL, messages=messages)

    try:
        return LOCAL.chat.completions.create(model=LOCAL_MODEL, messages=messages)
    except TRANSIENT:
        if regulated:
            raise          # no fallback: a refusal beats a leak
        return FRONTIER.chat.completions.create(model=FRONTIER_MODEL, messages=messages)

Catching a bare Exception here would defeat the point - a config error would quietly route to the API disguised as a local outage.

Things that will bite you

The silent CPU fallback

Ollama doesn't crash when a model won't fit in VRAM. It offloads layers to CPU and starts serving 1 TPS instead of 30. Alerts stay quiet because the service is technically healthy, so the first signal you get is users complaining.
Check after every model update:

bash
ollama ps    # PROCESSOR column should read "100% GPU", not "47%/53% CPU/GPU"
nvidia-smi --query-gpu=memory.used,memory.total --format=csv

Context window isn't free

Doubling the window barely dents throughput but does cost memory: the KV-cache grows linearly with context length, so 8K→16K doubles it. As a share of total consumption, that's typically +15–20% for a 9B-class model with Q4 weights - but run the numbers for your own model, because the fraction changes with model size and window length.

Maxing out --max-model-len and getting a silent CPU fallback is a well-established way to kill production on a Friday evening. Time is better spent on RAG (pgvector, Weaviate, Elasticsearch): memory stays predictable and the retrieved context is more relevant than a bloated window ever is. A smaller model with a solid pipeline routinely beats a larger one with no context - faster, and on cheaper hardware.

Permissions in the vector store

The common mistake is treating it as one shared index for all users. A query from one user legitimately surfaces another user's confidential document, and technically no rule was broken. "Who can retrieve which document" is a question to answer before deployment, not after.

The fix is document-level permissions, in one of two shapes:

Separate indexes per role, or
Filtering retrieval results by the requesting user's rights before context reaches the model

MoE math doesn't work like dense math

The temptation is to size VRAM by active parameters. A 235B MoE with 22B active costs roughly what a dense 22B costs to run - but every expert weight still has to be resident in memory. The win is quality per unit of compute, not memory savings.

*Network isolation
*
The inference server shouldn't be reachable from outside without a specific reason: private subnet, access through an internal gateway or VPN. Ollama listening on all interfaces by default is convenient for local testing and unacceptable in production.

When not to self-host

Being honest here is more useful than being enthusiastic. Three scenarios where self-hosting clearly loses:

Low or unpredictable volume. Below 1M tokens/day the API is cheaper at every tier - the infrastructure never pays for itself. With spiky traffic you have to size the local server for peak, which means it idles most of the time. APIs don't have that problem.
You need frontier models. The gap between top proprietary models and the best open weights keeps narrowing, but for genuinely hard multi-step reasoning it's still there.
No MLOps expertise. Self-hosted inference isn't set-and-forget. Models need updating, CUDA errors arrive at the worst possible moment, and performance degrades as load patterns shift. Without someone who can handle that, maintenance overhead eats the token savings faster than you'd expect - and in the $850 breakdown above, that line item is more than half the total.

Open question

The most debatable piece of this setup is the router itself. A static map is transparent and free, but it can't tell a simple ticket from a hard one inside the same category, so some requests go local when they'd be better off escalated.

So: how do you split traffic between local and frontier models?

Hardcoded rules?
A dedicated router model at the front door?
Confidence-based escalation, where the local LLM hands off on its own? And at your volumes, does the classifier overhead actually pay for itself?

Full TCO breakdown, quantization format comparison, RAG and fine-tuning architectures, and the complete breakeven tables are in the full self-hosted LLM guide, which this post is drawn from.

Our Experience Migrating from AWS ECS to EKS: 4x Faster Releases on a 1-Day SDLC

Alpacked — Mon, 17 Aug 2026 07:34:43 +0000

This is a technical breakdown of a migration we ran for a global B2B service (strict NDA, so no names). The setup was unusual: a release cycle of one day or less, no CTO or architect on the client side, a junior dev team, and acceptable downtime measured in minutes. Our side: two part-time DevOps engineers.

The short business version with numbers lives in the original case study. Here's the engineering side: what was breaking, what stack we put together, and where things bit us.

The Problem & Stack

The starting point was a legacy setup on AWS ECR/ECS. ECS itself isn't the problem – but here everything went wrong at once:

CI/CD pipelines were failing constantly. Every failure blocked developers, and at "release once a day" velocity, that meant hours of patching instead of shipping features.
Zero observability – troubleshooting meant log archaeology.
No fast, safe rollback: reverting was manual and terrifying.

Context made it worse: the client had burned through 8 DevOps teams in the six months before we came on board. The problem wasn't just the technology – no technical owner was making foundational decisions. All communication ran through the CEO and CMO, with dozens of micro-calls per day.

The classic dilemma at kick-off (Discovery): quick win vs. rewrite – patch the existing ECR/ECS and get to go-live faster, or rewrite everything from scratch on Amazon EKS. We made the case that running on the old architecture from day one put production stability at risk, along with the trust of the earliest users. The client decided not to take that risk.

The key decision that took 90% of the stress off the table: we stood up EKS in parallel with the old system and only cut over once the new environment was ready – no "big bang" at 2 am. Traffic was shifted gradually at the load balancer/DNS level: start with a small percentage on the new EKS, monitor error rate and latency in Datadog, and scale up to 100%. The old ECS stayed as a hot fallback the whole time; we only shut it down after several days of stable 100% traffic on the new setup.

The Architecture Solution

EKS instead of ECS. We needed an ecosystem for GitOps, fine-grained node control, and service mesh – on ECS, that means workarounds or third-party tools. The foundation: 2 isolated EKS clusters. Splitting environments gives you blast radius isolation and cleaner access control.

Mongo Atlas via VPC Peering. State lives in Mongo Atlas; we set up VPC Peering between our VPC and the Atlas VPC – database traffic goes over a private channel, not the public internet.

Karpenter instead of Cluster Autoscaler. FinOps was critical here, and Karpenter gives you mixed node groups (Spot + On-Demand) and bin-packing out of the box. It provisions nodes against actual pod req/limits rather than pre-sliced ASGs.

Istio instead of bare Ingress. We needed mTLS between services, traffic control, and mesh-level observability – all in one layer.

ArgoCD instead of CI push-deploys. GitOps gives you exactly what we were building toward: rollback in git = rollback in the cluster.
Datadog APM instead of Prometheus/Grafana. We deliberately went against the default here – more on that below.

Implementation Details & Code Blocks

1. IaC: Everything Through Terraform, Zero Manual Clicks

All infrastructure is code: no kubectl apply by hand in production, everything goes through Terraform and ArgoCD. We'll skip the basic cluster skeleton (standard terraform-aws-modules/eks/aws) and focus on IRSA (IAM Roles for Service Accounts). Without proper IRSA, neither Karpenter nor External Secrets can reach the AWS API with least-privilege access – and dropping credentials into pod env vars in production under ISO 27001 isn't an option.
The role for Karpenter, bound to its service account via the cluster's OIDC provider:

# example for this post: real ARNs/policies are broader
data "aws_iam_policy_document" "karpenter_assume" {
  statement {
    actions = ["sts:AssumeRoleWithWebIdentity"]
    principals {
      type        = "Federated"
      identifiers = [module.eks.oidc_provider_arn]
    }
    condition {
      test     = "StringEquals"
      variable = "${module.eks.oidc_provider}:sub"
      values   = ["system:serviceaccount:karpenter:karpenter"]
    }
  }
}

resource "aws_iam_role" "karpenter" {
  name               = "karpenter-controller"
  assume_role_policy = data.aws_iam_policy_document.karpenter_assume.json
}

The same pattern (AssumeRoleWithWebIdentity + a condition scoped to a specific serviceaccount) we used for External Secrets as well – this is exactly the kind of thing that gets credited during security hardening for ISO 27001.

2. Karpenter: Mixed Spot/On-Demand + Graviton

The tastiest part from a FinOps perspective. We handed node management entirely to Karpenter, let it mix Spot with On-Demand, and moved to Graviton ARM64 (r8g) – roughly 20% cheaper than x86. The key in the NodePool is getting the requirements right; otherwise, Karpenter either only picks expensive instances or dumps critical workloads onto Spot:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: default
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]   # mixed
        - key: karpenter.k8s.aws/instance-family
          operator: In
          values: ["r8g"]                  # r8g = Graviton ARM64, specifying arch is redundant
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized   # bin-packing
    consolidateAfter: 1m
  limits:
    cpu: "200"   # example for this post, real limit depends on your workload profile

consolidationPolicy: WhenEmptyOrUnderutilized is the bin-packing that trimmed our node count by 30–35%: Karpenter periodically repacks pods onto fewer nodes and terminates the rest. We intentionally skip the arch (arm64) requirement: the r8g family only exists on Graviton, so Karpenter infers the architecture from instance-family. A redundant constraint just clutters the manifest.

Pitfall #1. Spot nodes die without warning. On a stateful workload or long-lived connections without graceful shutdown, you'll see 5xx on every interruption. Fix it with terminationGracePeriodSeconds, PodDisruptionBudget, and pinning critical pods to On-Demand via nodeAffinity. Don't throw everything onto Spot just because it's cheap.

3. ArgoCD: Rollback = git revert

GitOps delivered the biggest qualitative leap: deployment and rollback became git operations, not cluster operations.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: payment-api
spec:
  project: default
  source:
    repoURL: git@github.com:client/k8s-manifests.git
    targetRevision: HEAD
    path: apps/payment-api
  destination:
    server: https://kubernetes.default.svc
    namespace: payment
  syncPolicy:
    automated:
      prune: true
      selfHeal: true   # manual changes in the cluster get rolled back to git state

selfHeal: true is both a safety net and a pitfall. Safety net: any manual change in the cluster (that classic kubectl edit under incident pressure) automatically reverts to git state. Pitfall: if someone edits a resource by hand to "quickly put out the fire," ArgoCD silently rolls it back – and they have no idea why their fix "disappeared." The cure is discipline: everything goes through a PR, and for emergencies, temporarily disable auto-sync on the specific application. Rolling back a bad release now looks like this:

git revert <bad-commit-sha>
git push origin main
# ArgoCD picks up the change and restores the cluster to the previous state in seconds

That's what gave us MTTR −82%: rollbacks stopped being late-night incidents.

4. Istio: mTLS and Traffic Control

Service mesh covered mTLS between services and traffic control in one layer. Enforcing strict mTLS at the namespace level:

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: payment
spec:
  mtls:
    mode: STRICT   # encrypted traffic between pods only

Pitfall #2. Enabling STRICT mTLS cluster-wide will immediately take down any services not yet in the mesh. Roll it out namespace by namespace, starting with PERMISSIVE, and on two isolated clusters, do it independently – don't try to flip the switch globally all at once.

5. Why Datadog Instead of Prometheus/Grafana

The most counterintuitive call here. The default DevOps instinct is Prometheus + Grafana (we support both, plus OpenTelemetry). But given the context – junior team, no CTO, relentless pace – the value of a tool isn't its raw power; it's how fast a developer with no DevOps background can find the root cause on their own. Datadog APM gave us exactly the developer-facing interface where a junior opens a trace and immediately sees which service is dragging, no PromQL spelunking required. A deliberate trade-off: slightly more money for SaaS in exchange for a significant drop in troubleshooting overhead. For this client, it paid off.

Pitfall #3. Secrets sprawl across Helm charts fast. We killed that immediately with External Secrets Operator + AWS Secrets Manager, and handed TLS certificates to cert-manager. Secrets pull from Secrets Manager into the cluster declaratively, no plaintext in git:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: payment-db
  namespace: payment
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: payment-db-creds   # ESO drops a ready k8s Secret here
  data:
    - secretKey: mongo-uri
      remoteRef:
        key: prod/payment/mongo-uri

Otherwise, a month later, you have secrets in three places, and nobody knows which one is current.

Security as a Stack, Not a Checkbox

ISO 27001 at the end isn't marketing – it's the result of a specific stack. On the perimeter: CloudFront (CDN) and AWS WAF filtering traffic before it reaches the cluster; inside, SecurityHub and Inspector run continuous vulnerability monitoring (CVE scanning in images, misconfiguration detection in AWS resources). Access is controlled via RBAC, external traffic comes in through the AWS Load Balancer Controller (ALB/NLB), and at the pod level – IRSA with least-privilege, mTLS via Istio, and secrets via ESO. It's the combination, not any single tool, that got the client through ISO 27001 and laid the foundation for SOC 2.

Culture: Why Infrastructure Won't Save You Without Code Review

This isn't about YAML, but technically, it saved production more than anything else. With no architect and no CTO, we introduced strict, close code review on every PR – that alone cut hours of patching and downtime. The lesson I'd carve in stone: if you're joining a project with no technical owner, push the client to hire an architect from day one. Until then, code review is your only gatekeeper.

Separately – close collaboration with the client's AI team. The validation service is tied directly to their ML models, and coordinating at the infrastructure level (right resources for inference, workload isolation, control over what deploys and when) saved the business from real reputational damage more than once. The takeaway: when infrastructure serves ML workloads, DevOps can't live in a vacuum from the people writing the models.

Engineering Metrics

What we got in numbers after cutting over to the new setup:

Deployment frequency ×4 – through full GitOps automation via ArgoCD.
*MTTR −82% *– rollback via git revert, zero-downtime deploys.
Compute costs: up to −80% on Spot instances via Karpenter; −30–35% node count reduction via auto-consolidation; ~−20% from moving to Graviton ARM64. Combined: roughly $45,600/year in savings.
99.7% uptime, Change Failure Rate down 4%.
Infrastructure went through full security hardening, earning the client ISO 27001 and a foundation for SOC 2.

The non-technical takeaway: infrastructure went from a blocker to a driver. Developers stopped waiting on deploys and stopped fearing rollbacks.

Discussion

The most controversial call in this project, for me, was going with Datadog over the usual Prometheus/Grafana stack – purely for the junior team's sake. It works, but it's vendor lock-in, and it costs money.

Question for the readers: how do you balance self-hosted observability (Prometheus/Grafana/OTel) vs. SaaS like Datadog when the people actually staring at dashboards are junior? Do you push for self-hosted for control and cost, or do you trade interface convenience for faster MTTR?

And for anyone running Spot through Karpenter in production: what's your real interruption rate, and how do you handle stateful workloads? Would love to compare numbers.