DEV Community: Suresh Babu Narayanan

The Kubernetes bill nobody looks at until finance does

Suresh Babu Narayanan — Mon, 10 Aug 2026 01:00:05 +0000

Kubernetes does not send you a warning before it quietly costs you three times what it should. It just runs every pod exactly as requested, every replica exactly as configured and the bill arrives a month later as a single number nobody can immediately explain.

I have spent ten years across AWS, Azure, and Kubernetes (AKS, EKS, ARO) and the pattern behind most "why is our cloud bill so high" conversations is almost never one dramatic mistake. It is four small, boring, easy-to-miss things, quietly compounding across every service in the cluster.

Here they are - with a free calculator and a set of scripts at the end so you can check your own cluster in the next ten minutes.

The core thing to understand: requests, not usage, drive cost

This is the single most important idea in this article, so it goes first: Kubernetes bills you for what you ask for, not what you use.

When a pod requests 500m CPU and 512Mi memory, the scheduler reserves that exact capacity on a node - permanently, for as long as the pod runs - whether the pod is idle or maxed out. A node's size, and therefore its cost, is determined by the sum of requests scheduled onto it, not by real-time usage graphs.

This is why a Grafana dashboard showing "12% average CPU utilization" and a cloud bill that keeps climbing aren't in tension. They're both true, and they're describing the same problem from two different angles.

Mistake 1 - requests set once, in a hurry, and never revisited

Someone writes the first version of a deployment manifest. They don't know the real usage pattern yet, because the service doesn't exist yet, so they guess generously - "let's give it 1 CPU and 1Gi, we can tune later." It ships. It works. Nobody tunes it later, because it's working, and "working" doesn't create a ticket.

Multiply that single generous guess across every microservice a team ships, and you get a cluster where every workload is sized for a worst case that rarely arrives.

The fix isn't guessing better - it's measuring. Run kubectl top pods against your actual traffic for a week or two, then compare it to requests. The gap is your first real answer to "where does our cloud spend actually go."

kubectl top pods --all-namespaces --sort-by=cpu

Compare that against:

kubectl get pods --all-namespaces -o custom-columns=\
NAME:.metadata.name,CPU_REQ:.spec.containers[*].resources.requests.cpu,MEM_REQ:.spec.containers[*].resources.requests.memory

Mistake 2 - orphaned storage and empty load balancers

These are the sneakiest of the four, because they don't show up in CPU or memory dashboards at all. They're not compute problems - they're forgotten housekeeping.

A PersistentVolumeClaim survives its pod. Delete the deployment, forget the PVC existed, and the underlying disk keeps being billed every month, silently, until someone audits storage specifically. A LoadBalancer-type Service can end up with zero backing endpoints - the pods behind it were scaled down or deleted, but the cloud load balancer it provisioned is still running and still billing, whether or not a single packet reaches it.

Neither shows up as "high CPU" or "high memory." They show up as a line item nobody remembers approving.

# PVCs not currently mounted by any pod
kubectl get pvc --all-namespaces

# LoadBalancer services worth checking for zero endpoints
kubectl get svc --all-namespaces --field-selector spec.type=LoadBalancer

(The toolkit at the end automates the actual "is this one orphaned" comparison - doing it by hand across a big cluster gets tedious fast.)

Mistake 3 - non-prod environments running like it's prod

Dev and staging namespaces get created by copying the closest working example - which is usually the production manifest, replica count and all. Nobody consciously decided "this dev environment should run 3 replicas of everything, 24 hours a day, 7 days a week, including the entire weekend when literally nobody is looking at it." It just inherited that shape from prod and nobody revisited the decision.

Run at business hours only (roughly 45 hours/week) instead of continuously (168 hours/week), and a non-prod environment can cost well under half of its always-on price - often closer to a quarter - for identical infrastructure. A scheduled scale-to-zero is usually a single CronJob:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: scale-down-dev
spec:
  schedule: "0 19 * * 1-5"   # 7pm weekdays
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: kubectl
              image: bitnami/kubectl
              command: ["kubectl", "scale", "deployment", "--all", "--replicas=0", "-n", "dev"]
          restartPolicy: OnFailure

(Pair it with a matching scale-up CronJob for 8am, obviously - nobody wants to explain to the team why dev is down every morning.)

Mistake 4 - no autoscaling, so peak-sizing runs all day

Without a HorizontalPodAutoscaler, a deployment runs at whatever fixed replica count someone typed once - usually sized for the worst moment it needs to handle, running continuously even during the other 20 hours a day when traffic is a fraction of that peak.

The fix is one resource, and it pays for itself continuously once it exists:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-service
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-service
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

What this actually costs - a worked example

Take a genuinely ordinary microservice: 500m CPU and 512Mi memory requested, 3 replicas, running at roughly 35% real average utilization - not a horror story, just a typical unreviewed service. At AWS on-demand list pricing, that's roughly $35-40/month requested, of which roughly $23-26/month sits unused at that utilization level.

One service, ~$25/month wasted. Doesn't sound alarming. Now multiply it across the 40, 80, or 200 microservices a mid-sized platform team actually runs, and the number stops being trivial - it becomes the kind of line item a finance team asks about by name.

I built a small calculator so you can plug in your own numbers instead of trusting mine: PodTab → - type in requests, replicas, and real usage percentage, get an itemized monthly and annual figure.

Check your own cluster in the next ten minutes

Everything above is diagnosable with kubectl alone - no new tooling, no agent installed in your cluster, nothing sent anywhere. I packaged the checks I run into three small scripts: one flags over-provisioned pods against real usage, one finds orphaned PVCs and empty load balancers, one finds deployments with no autoscaler and non-prod namespaces quietly running at prod scale.

Free version: build the checks yourself from the kubectl commands above - everything you need is in this article. Packaged version, if you'd rather not assemble it: Kubernetes Cost Audit Toolkit →.

Either way - go check. The waste in most clusters isn't dramatic. It's four small, boring things, quietly compounding, waiting for someone to actually look.

I'm a DevOps engineer with 13 years across AWS, Azure, Kubernetes, Terraform and Ansible. Previously: Terraform state mistakes and a self-healing pipeline with Prometheus. I also maintain CronPort, a free cron expression converter, and sell a Terraform AWS Starter Kit.

10 Hard-Earned Lessons from a Year of Building with LLMs in Production

Suresh Babu Narayanan — Wed, 05 Aug 2026 05:35:38 +0000

AI Twitter (and LinkedIn, and every tech newsletter) makes LLM engineering look like a solved problem: wire up an API call, add a system prompt, ship it. Anyone who's actually taken an LLM feature from prototype to production knows that's about 10% of the job. The other 90% is the unglamorous stuff - the failure modes nobody demos, the costs nobody mentions in the launch tweet, the "it worked great until real users touched it" moments.

Here are ten lessons that took me longer to learn than they should have, and that I wish someone had told me before I started.

Your prompt is a piece of software, not a paragraph

Early on, prompts feel like writing - you tweak the wording until the output looks right, and move on. That mindset falls apart fast once a prompt has to survive contact with real, messy user input.

Treat prompts like code: version them, test them against a fixed set of inputs before every change, and track regressions. A prompt that works beautifully on your five test cases and silently breaks on the sixth isn't done - it's untested.

Evaluation is the actual hard part

Building the feature is fast. Knowing whether it's good is slow. Most teams underinvest here because eval work doesn't produce a demo-able artifact - it produces confidence, and confidence doesn't screenshot well.

The fix isn't complicated, just unglamorous: build a small, real eval set from actual (or realistic) inputs early, and re-run it every time you touch a prompt, model, or pipeline step. Even 30-50 well-chosen examples beat vibes-based testing by a wide margin.

Retrieval quality matters more than model choice

If you're building anything RAG-adjacent, it's tempting to assume a better model will fix bad answers. In practice, most "the AI is wrong" bugs I've debugged were retrieval bugs wearing a model-quality costume - the right chunk of context simply never made it into the prompt.

Before swapping models, check what your retrieval step is actually returning. Log it. Read it. It's usually the culprit.

Structured output is worth the upfront pain

Free-form text output feels faster to build, until you're three regex patterns deep trying to parse a response that almost-but-not-quite follows the format you asked for. Investing in structured output (function calling, JSON schemas, whatever your provider supports) early saves you from a category of bugs that otherwise creeps in right before launch.

Latency is a feature, not an afterthought

Users tolerate a slow first response far less than a slow search result, because they expect a "chat" to feel conversational. Streaming responses, showing intermediate steps for agentic workflows, and setting realistic timeouts aren't polish - they're core to whether the feature feels usable at all.

Cost scales in ways that surprise you

A demo costing a few cents per call feels irrelevant, right up until it's running at production volume with retries, multi-step agent loops, and users who send far more requests than your test users ever did. Model cost, retry cost, and "the agent looped four times before finishing" cost are three different line items - track them separately, or the invoice will surprise you.

Guardrails aren't optional, even for "internal tools"

It's tempting to skip input/output filtering on internal or low-stakes tools. Then someone pastes something unexpected into the input field, or an agent with tool access does something you didn't anticipate, and you're debugging an incident instead of shipping a feature. Basic guardrails scale down easily; retrofitting them after an incident doesn't.

Agentic workflows fail differently than single-shot calls

A single bad response is annoying. A multi-step agent that makes a wrong decision early and confidently builds on it for five more steps is a much harder problem - the failure compounds instead of staying contained.

If you're building agents (and it feels like everyone is right now), invest early in step-level logging and the ability to see where a chain went wrong, not just that it did.

MCP and tool ecosystems are changing the integration story fast

The Model Context Protocol and similar standards are quietly solving a problem that used to eat weeks of custom integration work per tool. If you haven't looked at how MCP-style tool servers work recently, it's worth revisiting — the ecosystem is moving fast enough that assumptions from six months ago are already outdated.

The best LLM feature is often the smallest one

The instinct is to build the most impressive agentic pipeline possible. The features that actually stick with users tend to be narrower and more reliable: one job, done well, with predictable behavior - not a general-purpose agent that's occasionally magical and occasionally wrong in ways users can't predict.

None of this is meant to be discouraging - it's the opposite. LLM engineering is still young enough that most of these lessons come from hands-on scar tissue rather than established best practice. If you're early in this space, the gap between "the demo worked" and "it works reliably for real users" is exactly where the interesting engineering happens.

What's the lesson that took you the longest to learn? I'd genuinely like to know - drop it in the comments.

I'm a DevOps engineer with 10 years across AWS, Azure, Kubernetes, Terraform and Ansible. Previously: I built a self-healing pipeline with Prometheus and 50 lines of Python. I also maintain CronPort, a free cron expression converter for crontab, Kubernetes, GitHub Actions, AWS EventBridge and Terraform.

10 years in DevOps, 2 weeks on DEV, and a deploy queue that humbled me

Suresh Babu Narayanan — Tue, 28 Jul 2026 05:14:44 +0000

Hi. I'm Suresh - DevOps/automation engineer, 10 years in, currently deep in AWS, Azure, Kubernetes, Terraform, and Ansible for a living. I joined DEV two weeks ago. I've published two articles. I have no idea what I'm doing with this platform yet, and I'm having a genuinely great time finding out.

This isn't a "here's what I learned" listicle. It's more like - pull up a chair, let me tell you what the last two weeks looked like from the inside, because some of it was funnier than I expected infrastructure writing to be.

Why start now, after 10 years of not writing

Short version: I got tired of solving the same problem quietly, for the fifth company, and having literally nothing to show for it except my own memory. Thirteen years of pattern-matching - "oh, this is the local-state-on-a-laptop problem again," "oh, this is the giant-shared-state-file problem again" - and none of it existed anywhere I could point to. It just lived in my head, useful to exactly one person.

So: write it down. Worst case, nobody reads it and I've still got a clean writeup of things I already knew. Best case, someone googling the exact 2am problem I've already solved five times finds it and skips the pain.

What actually happened in two weeks

I built a self-healing Prometheus + Alertmanager pipeline that restarts nginx on its own, wrote it up, and - this is the part nobody warns you about - spent longer getting the cover image and Dev.to formatting right than writing the actual article. Nobody tells you that going in.

Then I wrote about the five Terraform state mistakes I've watched happen (and caused, once, early on - we don't need to relive that one in detail). Somewhere in the middle of publishing that one, I also deployed a small free tool, and this is the actual highlight of my two weeks:

The deploy got stuck. Queued. For twenty minutes. Then I retried, and it queued again. Turns out it was a known platform-side issue completely outside my control - nothing wrong with my file, nothing wrong with my setup, just a stuck queue with no ETA. Thirteen years of infrastructure experience, and my solution was the same one I'd tell anyone else: stop waiting, fail over. Dragged the exact same file onto a different host. Live in about ninety seconds.

The irony was not lost on me that I'd just published an article about self-healing infrastructure the same week my own deploy needed a very manual, very human intervention to heal itself. Infrastructure has a sense of humor. It is not always kind.

The actual surprises, ranked

Writing the article is maybe 40% of the work. Cover image, tags, formatting, screenshots, re-reading it four times to make sure it actually sounds like me and not like a template - that's the other 60%. I did not budget for this.
People are nicer than I expected. I came in braced for the stereotypical "well actually" comment section. Hasn't happened yet. Might just be early days, but so far it's been genuinely constructive.
The "did I really live this or does it just sound made up" test is real. I made myself a rule early on: if I can't answer a specific follow-up question about something I wrote, I don't publish it as my own story. Turns out that's a surprisingly high bar when you're staring at a blank editor at 11pm.
I have way more stories than I thought. Thirteen years apparently generates a lot of "oh, that reminds me of the time..." - I just never had a reason to write any of them down before.

Where this is going

More of the same, honestly. Real problems I've actually hit, real fixes, occasionally a tool if I get annoyed enough at doing something manually more than twice. No hot takes on whatever's trending this week unless I've genuinely got something to add.

AMA, kind of

Two weeks isn't enough runway for a proper "ask me anything," but I'll happily answer anything about:

DevOps/platform engineering as a 10-year career (AWS, Azure, Kubernetes, the lot)
What it's actually like starting a technical writing habit from zero, in real time, badly, alongside a full-time job
The self-healing pipeline or Terraform state posts, if you've read either
Whether infrastructure really does have a sense of humor, or if I'm just tired

Drop a comment - genuinely, I'll answer everything. Two weeks in, this is still the part I didn't expect to enjoy as much as I do.

Five Terraform state mistakes I keep seeing after 10 years (and the fixes)

Suresh Babu Narayanan — Mon, 27 Jul 2026 00:56:48 +0000

Terraform's state file is the most important file most teams never think about - right up until the day it thinks about them.

I've spent thirteen years in infrastructure, the last several deep in Terraform across AWS and Azure, and the same five state mistakes keep appearing. Not in beginner projects - in real companies, with real change boards, usually discovered at the worst possible moment. Every one of them is boring to fix before it happens and expensive to fix after.

Here they are, with the fixes I now bake into every project on day one.

Mistake 1 - Local state on someone's laptop

The default terraform.tfstate sitting in the project folder works perfectly for exactly one person, for exactly as long as their laptop lives. Then a second engineer joins, or the laptop dies, and you discover that the state file is the infrastructure's memory. Lose it, and Terraform looks at your cloud account like it's never met it before.

I've seen this exact scenario play out early in a project, before anyone had gotten around to setting up a proper backend. State living locally, on one engineer's laptop, "for now." Then a re-clone, or a laptop swap, or just someone else needing to run a plan - and the moment of "wait, where's the state file?" that makes everyone's stomach drop slightly. Nothing was permanently lost in the end, but it's the kind of near-miss that makes you migrate to remote state on every project since, on day one, no exceptions.

The fix: remote state in S3, from the first commit, with versioning and encryption on. The bucket is a five-minute bootstrap:

resource "aws_s3_bucket" "tf_state" {
  bucket = var.state_bucket_name
  lifecycle {
    prevent_destroy = true # deleting your state bucket is a very bad day
  }
}

resource "aws_s3_bucket_versioning" "tf_state" {
  bucket = aws_s3_bucket.tf_state.id
  versioning_configuration {
    status = "Enabled"
  }
}

That prevent_destroy line has saved more weekends than any monitoring tool I've deployed.

Mistake 2 - No state locking

Two engineers run terraform apply at the same time. Both read the same state, both write different endings to it. The result is a state file that matches neither reality nor anyone's intentions - Terraform's version of a git merge conflict, except the conflict is your production VPC.

The fix used to be a DynamoDB table, and half the tutorials online still say so. But since Terraform 1.11, the S3 backend can lock natively:

terraform {
  backend "s3" {
    bucket       = "yourco-tfstate"
    key          = "prod/terraform.tfstate"
    region       = "ap-southeast-2"
    use_lockfile = true   # S3-native locking - no DynamoDB table needed
    encrypt      = true
  }
}

One less resource to bootstrap, pay for, and explain to auditors. If you're pinned below 1.11, keep DynamoDB - but put a version upgrade on the roadmap, because this is genuinely simpler.

![How S3-native locking resolves the race - the second apply waits, nothing corrupts]

The second apply doesn't corrupt anything or silently overwrite - it just waits its turn.

Mistake 3 - One giant state file for everything

Dev, staging, prod, the DNS zones, that one Lambda from 2023 - all in a single state. Every plan takes minutes and touches everything. Every mistake has maximum blast radius. Every junior engineer's experiment holds a loaded reference to prod.

I've worked in environments where dev, staging, and prod infrastructure lived far too close together - sometimes sharing state, sometimes just sharing enough blast radius that a plan in one environment made everyone nervous about the others. Plans that should take seconds stretch out uncomfortably long when Terraform has to reason about everything at once, and every review turns into "wait, what else does this touch?" It's a slow tax that nobody notices being introduced and everybody complains about once it's there.

The fix: one state file per environment, minimum. Same code, different backends:

envs/
├── dev/    → key = "dev/terraform.tfstate"    (2 AZs, NAT off - cheap)
└── prod/   → key = "prod/terraform.tfstate"   (3 AZs, NAT on)

The objection is always "but then dev and prod can drift apart." They can - and that's a code review problem, which is a much better problem than "dev corrupted prod's state," which is a résumé problem.

Mistake 4 - Treating state as too sacred to touch (or not sacred enough)

Two opposite failure modes, same root cause: nobody on the team actually knows the state manipulation commands.

The not sacred enough version: someone opens terraform.tfstate in an editor and hand-edits JSON. I have watched this happen. It works right up until it catastrophically doesn't.

The too sacred version: a resource needs renaming, and because nobody trusts themselves with state surgery, the team does a destroy-and-recreate of a production database instead. The scheduled-downtime email goes out for what should have been a zero-downtime metadata change.

The fix: learn three tools, in this order -

# renaming/refactoring a resource without touching the real thing:
terraform state mv aws_instance.web aws_instance.web_server

# adopting an existing resource into Terraform (1.5+, plan-reviewable):
import {
  to = aws_s3_bucket.legacy
  id = "my-existing-bucket"
}

# removing something from state WITHOUT destroying it (1.7+):
removed {
  from = aws_instance.snowflake
  lifecycle { destroy = false }
}

The import and removed blocks are the underrated ones - they turn state surgery into a reviewable pull request instead of a hero typing commands into prod at 6pm.

I've had to reach for terraform import and manual state inspection more than once - usually to bring an existing resource under management without Terraform deciding the "safe" move was to destroy and recreate it. That instinct - trusting state mv and import over letting Terraform tear something down "because state says so" - is something you only really learn by almost letting the destroy happen once and catching it in the plan output first.

Mistake 5 - Secrets in state, state in Git

Here's the uncomfortable truth many teams learn during their first security review: Terraform state stores resource attributes in plaintext - including database passwords, and anything a provider returns. Which means the day someone commits terraform.tfstate to the repo "just as a backup," those secrets are in Git history forever.

The fix is layered:

.gitignore the state files on day one - *.tfstate and *.tfstate.* (backup files too)
Encrypt the state bucket (KMS server-side encryption in the bootstrap above)
Prefer ephemeral values and secret-manager references over raw secret strings in configuration, so fewer secrets land in state at all
Treat state bucket access as production access - IAM-restricted, logged

If you're checking one thing after reading this article, make it git log --all -- "*.tfstate" on your oldest repo. I hope it comes back empty. It often doesn't.

Patterns you'll probably recognise, even if the stories above don't quite match

State mistakes tend to rhyme across companies more than people expect. A few more I'd bet you've seen at least one of:

terraform.tfstate committed to Git "just this once" - and now it's in history forever
Two people running apply in the same environment at the same time, no lock, no warning until the diff looks wrong
A renamed resource treated as delete-then-create instead of a state mv, because nobody knew the command existed
Local state surviving purely because nobody's laptop has died yet - not a policy, just luck
State quietly holding secrets nobody remembered were sensitive, until a security review found them

If even one of these lands, you already know exactly which of the five fixes above to reach for first.

The pattern behind all five

None of these mistakes are exotic. They're all the same mistake wearing five costumes: treating state as an implementation detail instead of what it actually is - the single source of truth that makes Terraform work at all.

The teams that avoid the 2am state incidents aren't smarter. They just made five boring decisions on day one: remote backend, locking, per-env isolation, learned state surgery, and secrets hygiene. Thirty minutes of setup, total.

I packaged the thirty minutes

Because I kept setting up the same foundations on every new engagement, I turned my day-one layout into a small kit: bootstrap for a versioned/encrypted/locked state bucket, per-environment structure with S3-native locking, a multi-AZ VPC module, provider-level default_tags so untagged resources are impossible, and pre-commit hooks (fmt, validate, tflint) so bad code never reaches CI.

The free version of the knowledge is this article - you can build all of it yourself from what's above. If you'd rather skip the assembly, the kit is here: Terraform AWS Starter Kit - and the repo layout is documented in the listing so you know exactly what you're getting.

Either way: go check where your state lives. Today, ideally.

My nginx died on a Saturday, so I taught it to fix itself

Suresh Babu Narayanan — Fri, 17 Jul 2026 06:26:44 +0000

A few months ago I set myself a weekend challenge: could I make a service restart itself when it dies, using only free open-source tools, with zero manual steps in between?

Not because restarting nginx is hard. It's one command. But I've spent thirteen years in infrastructure, and the pattern I keep seeing is this: the alert fires at 2am, a human wakes up, squints at a phone, VPNs in, and runs the one command. The gap between "we knew" and "we acted" is where the pain lives. I wanted to close that gap on my own laptop, end to end, so I properly understood every link in the chain - not just the theory.

Here's the whole thing: Prometheus notices nginx is down, Alertmanager fires a webhook, and a small Python service restarts the container. Total cost: $0. Total code I had to write: about 50 lines.

What we're building

nginx  →  nginx-exporter  →  Prometheus  →  Alertmanager  →  responder (Flask)  →  docker restart nginx
         (status→metrics)    (scrape+rule)   (route alert)     (receive+act)

Five containers, one docker-compose file. The full repo is at the end - everything below is copy-pasteable, but I'd genuinely suggest typing the alert rule and the responder yourself. That's where the learning is.

Prerequisites: a laptop with Docker, basic YAML tolerance, and about an hour. It took me longer the first time, and I'll show you exactly where I lost the time, because that part taught me more than the parts that worked.

Step 1 - nginx and its exporter

nginx doesn't speak Prometheus natively, so we pair it with the official exporter, which reads nginx's stub_status page and republishes it as metrics - including the one we care about: nginx_up, which is 1 when nginx answers and 0 when it doesn't.

The nginx config just adds a status listener:

server {
    listen 8081;
    location /stub_status {
        stub_status;
        access_log off;
    }
}

And in docker-compose, the exporter points at it:

nginx-exporter:
  image: nginx/nginx-prometheus-exporter:1.1.0
  command:
    - --nginx.scrape-uri=http://nginx:8081/stub_status

Why an exporter instead of just letting Prometheus scrape nginx directly? Because Prometheus's own up metric tells you whether the scrape target answered - and if you scrape the exporter, up stays 1 even while nginx is dead, because the exporter itself is fine. nginx_up is the truth. This distinction bit me once in a real environment years ago, and it's the kind of thing you only internalise by getting it wrong.

Step 2 - Prometheus and the alert rule

Prometheus scrapes the exporter every 5 seconds (aggressive, but this is a demo and I wanted fast feedback loops):

global:
  scrape_interval: 5s
  evaluation_interval: 5s

The alert rule is four lines that matter:

- alert: NginxDown
  expr: nginx_up == 0
  for: 30s
  labels:
    severity: critical
    action: auto_restart

The for: 30s is doing more work than it looks like. Without it, a single blipped scrape - a container pausing for GC, a network hiccup - fires the alert and restarts a perfectly healthy service. With it, nginx has to be continuously down for 30 seconds before anything happens. Every flapping-alert horror story I've heard traces back to someone skipping this line. In production I'd set it longer; for a demo, 30 seconds keeps the feedback loop tight.

The action: auto_restart label is a habit from working with event-driven automation at scale: route on labels, not alert names. Today one alert triggers a restart; later, twenty alerts can share the same remediation path just by carrying the label.

Step 3 - Alertmanager, and where I lost an hour

Alertmanager's job is routing: this alert goes to that webhook.

route:
  receiver: auto-remediation
  group_by: ["alertname"]
  group_wait: 10s
  repeat_interval: 5m

receivers:
  - name: auto-remediation
    webhook_configs:
      - url: http://responder:5000/alert
        send_resolved: true

This is where my Saturday went sideways. My first version had the webhook URL pointing at localhost:5000 - which inside the Alertmanager container means Alertmanager itself, not my responder. Prometheus showed the alert firing, Alertmanager showed it received, and… nothing happened. No errors anywhere obvious, because from Alertmanager's point of view nothing was wrong; it was dutifully delivering webhooks into the void.

The fix is obvious in hindsight - use the compose service name, responder:5000 - but I mention it because "alert fires, action doesn't happen, no visible error" is exactly the failure mode you must know how to debug before trusting automation to act for you. docker logs alertmanager and a curl from inside the container found it in the end.

The other setting worth a sentence: repeat_interval: 5m means if the alert keeps firing, the webhook re-fires every 5 minutes. That sounds helpful until you realise it means your restart action will retry forever against a service that's never coming back. Hold that thought.

Step 4 - the responder: 50 lines of Flask

The receiving end is deliberately boring - a Flask app that validates the alert and restarts the container via the Docker API:

@app.route("/alert", methods=["POST"])
def alert():
    payload = request.get_json(silent=True) or {}
    for a in payload.get("alerts", []):
        name = a.get("labels", {}).get("alertname")
        if name != "NginxDown" or a.get("status") != "firing":
            continue
        if not allowed_to_restart():
            log.warning("circuit breaker OPEN - escalate to a human")
            continue
        client.containers.get("nginx").restart(timeout=10)
        restart_history.append(time.time())

The part I care about is allowed_to_restart() - a circuit breaker that refuses more than 3 restarts in 10 minutes:

def allowed_to_restart() -> bool:
    now = time.time()
    while restart_history and now - restart_history[0] > WINDOW_SECONDS:
        restart_history.pop(0)
    return len(restart_history) < MAX_RESTARTS

Why bother, in a toy? Because auto-remediation without a stop condition isn't self-healing, it's self-harm with good intentions. If nginx is crashing because of a broken config or a full disk, restarting it in a loop fixes nothing and hides the real problem behind a wall of "remediation succeeded" logs. Three strikes, then a human gets paged. In a real setup this is where you'd escalate to Slack or PagerDuty; in the demo it just logs loudly.

Step 5 - kill it and watch it heal

The payoff. With everything running:

docker stop nginx

Then I sat and watched three windows: the Prometheus alerts page, docker logs -f responder, and docker ps.

The sequence: the alert went pending at the next scrape, firing after the 30-second hold, Alertmanager delivered after its 10-second group wait, and the responder log printed restarted container 'nginx'. Then - my favourite line ever - docker ps showing nginx with an uptime of a few seconds while everything else showed twenty minutes.

End to end, dead-to-healed took on my machine. Nobody touched anything.

I ran it a few more times just to enjoy it, which promptly tripped my own circuit breaker. Satisfying to be refused by my own safety rail.

The irony that wrote itself

Here's the part I didn't plan. The same week I finished this write-up, I deployed a small side project - a cron expression converter - to a well-known static hosting platform. Single HTML file, simplest deployment imaginable.

The deployment got stuck in "Queued". For twenty minutes. Then the retry got stuck too. It turned out to be a known platform-side queue jam that community threads report lasting anywhere from hours to over a day, and there was nothing I could do from my side but wait or fail over. I dragged the same file onto a different host and was live in ninety seconds.

Which is the whole point of this article, really. Everything fails: my nginx, their deploy queue, your thing next Tuesday. The difference between a bad day and a non-event isn't preventing failure - it's how much of the response you've already built before it happens. My pipeline had a restart path ready; my deployment had a second host ready. Same principle, different layer.

Where I'd take this next

If I were promoting this beyond the laptop, in rough order:

Deduplication and idempotency - Alertmanager retries webhooks; the responder should handle the same alert twice gracefully (it mostly does now, but "mostly" is doing heavy lifting).
Escalation - after the circuit breaker opens, page a human with the restart history attached, so they arrive with context instead of a cold start.
Audit trail - every automated action written somewhere durable. When automation acts on your behalf, "what did it do and when" must never be a mystery.
Smarter checks than up/down - restart-on-death is the "hello world" of remediation. The interesting versions act on latency, error rates, and saturation before the service dies.

There are, of course, mature event-driven automation platforms that do all of this at enterprise scale with rulebooks and workflow engines. Having now built the primitive version by hand, I understand far better what those platforms are actually doing under the hood - which was the point of the weekend.

Try it yourself

The full repo - compose file, configs, responder, and a README with the exact break-it steps - is here: https://github.com/suredares/Self-healing-Demo

Clone it, run docker compose up -d --build, then docker stop nginx and watch your infrastructure quietly refuse to stay broken.

If you build it and something behaves differently on your setup, tell me in the comments - genuinely curious what breaks on other machines. And if you've built auto-remediation that went wrong in an interesting way, I definitely want to hear that story.

*I'm a DevOps engineer with 10 years across AWS, Azure, Kubernetes, Terraform and Ansible, writing about automation that actually works. I also built CronPort - a free tool that converts cron expressions between crontab, Kubernetes, GitHub Actions, AWS EventBridge and Terraform.
I also wrote up Five Terraform state mistakes I keep seeing

I also sell a Terraform AWS Starter Kit if you want production-ready foundations without the setup day.

If it saved you time, you can buy me a coffee Buymeacoffee*