DEV Community: Suresh Babu Narayanan

10 years in DevOps, 2 weeks on DEV, and a deploy queue that humbled me

Suresh Babu Narayanan — Tue, 28 Jul 2026 05:14:44 +0000

Hi. I'm Suresh - DevOps/automation engineer, 10 years in, currently deep in AWS, Azure, Kubernetes, Terraform, and Ansible for a living. I joined DEV two weeks ago. I've published two articles. I have no idea what I'm doing with this platform yet, and I'm having a genuinely great time finding out.

This isn't a "here's what I learned" listicle. It's more like - pull up a chair, let me tell you what the last two weeks looked like from the inside, because some of it was funnier than I expected infrastructure writing to be.

Why start now, after 10 years of not writing

Short version: I got tired of solving the same problem quietly, for the fifth company, and having literally nothing to show for it except my own memory. Thirteen years of pattern-matching - "oh, this is the local-state-on-a-laptop problem again," "oh, this is the giant-shared-state-file problem again" - and none of it existed anywhere I could point to. It just lived in my head, useful to exactly one person.

So: write it down. Worst case, nobody reads it and I've still got a clean writeup of things I already knew. Best case, someone googling the exact 2am problem I've already solved five times finds it and skips the pain.

What actually happened in two weeks

I built a self-healing Prometheus + Alertmanager pipeline that restarts nginx on its own, wrote it up, and - this is the part nobody warns you about - spent longer getting the cover image and Dev.to formatting right than writing the actual article. Nobody tells you that going in.

Then I wrote about the five Terraform state mistakes I've watched happen (and caused, once, early on - we don't need to relive that one in detail). Somewhere in the middle of publishing that one, I also deployed a small free tool, and this is the actual highlight of my two weeks:

The deploy got stuck. Queued. For twenty minutes. Then I retried, and it queued again. Turns out it was a known platform-side issue completely outside my control - nothing wrong with my file, nothing wrong with my setup, just a stuck queue with no ETA. Thirteen years of infrastructure experience, and my solution was the same one I'd tell anyone else: stop waiting, fail over. Dragged the exact same file onto a different host. Live in about ninety seconds.

The irony was not lost on me that I'd just published an article about self-healing infrastructure the same week my own deploy needed a very manual, very human intervention to heal itself. Infrastructure has a sense of humor. It is not always kind.

The actual surprises, ranked

Writing the article is maybe 40% of the work. Cover image, tags, formatting, screenshots, re-reading it four times to make sure it actually sounds like me and not like a template - that's the other 60%. I did not budget for this.
People are nicer than I expected. I came in braced for the stereotypical "well actually" comment section. Hasn't happened yet. Might just be early days, but so far it's been genuinely constructive.
The "did I really live this or does it just sound made up" test is real. I made myself a rule early on: if I can't answer a specific follow-up question about something I wrote, I don't publish it as my own story. Turns out that's a surprisingly high bar when you're staring at a blank editor at 11pm.
I have way more stories than I thought. Thirteen years apparently generates a lot of "oh, that reminds me of the time..." - I just never had a reason to write any of them down before.

Where this is going

More of the same, honestly. Real problems I've actually hit, real fixes, occasionally a tool if I get annoyed enough at doing something manually more than twice. No hot takes on whatever's trending this week unless I've genuinely got something to add.

AMA, kind of

Two weeks isn't enough runway for a proper "ask me anything," but I'll happily answer anything about:

DevOps/platform engineering as a 10-year career (AWS, Azure, Kubernetes, the lot)
What it's actually like starting a technical writing habit from zero, in real time, badly, alongside a full-time job
The self-healing pipeline or Terraform state posts, if you've read either
Whether infrastructure really does have a sense of humor, or if I'm just tired

Drop a comment - genuinely, I'll answer everything. Two weeks in, this is still the part I didn't expect to enjoy as much as I do.

Five Terraform state mistakes I keep seeing after 13 years (and the fixes)

Suresh Babu Narayanan — Mon, 27 Jul 2026 00:56:48 +0000

Terraform's state file is the most important file most teams never think about - right up until the day it thinks about them.

I've spent thirteen years in infrastructure, the last several deep in Terraform across AWS and Azure, and the same five state mistakes keep appearing. Not in beginner projects - in real companies, with real change boards, usually discovered at the worst possible moment. Every one of them is boring to fix before it happens and expensive to fix after.

Here they are, with the fixes I now bake into every project on day one.

Mistake 1 - Local state on someone's laptop

The default terraform.tfstate sitting in the project folder works perfectly for exactly one person, for exactly as long as their laptop lives. Then a second engineer joins, or the laptop dies, and you discover that the state file is the infrastructure's memory. Lose it, and Terraform looks at your cloud account like it's never met it before.

I've seen this exact scenario play out early in a project, before anyone had gotten around to setting up a proper backend. State living locally, on one engineer's laptop, "for now." Then a re-clone, or a laptop swap, or just someone else needing to run a plan - and the moment of "wait, where's the state file?" that makes everyone's stomach drop slightly. Nothing was permanently lost in the end, but it's the kind of near-miss that makes you migrate to remote state on every project since, on day one, no exceptions.

The fix: remote state in S3, from the first commit, with versioning and encryption on. The bucket is a five-minute bootstrap:

resource "aws_s3_bucket" "tf_state" {
  bucket = var.state_bucket_name
  lifecycle {
    prevent_destroy = true # deleting your state bucket is a very bad day
  }
}

resource "aws_s3_bucket_versioning" "tf_state" {
  bucket = aws_s3_bucket.tf_state.id
  versioning_configuration {
    status = "Enabled"
  }
}

That prevent_destroy line has saved more weekends than any monitoring tool I've deployed.

Mistake 2 - No state locking

Two engineers run terraform apply at the same time. Both read the same state, both write different endings to it. The result is a state file that matches neither reality nor anyone's intentions - Terraform's version of a git merge conflict, except the conflict is your production VPC.

The fix used to be a DynamoDB table, and half the tutorials online still say so. But since Terraform 1.11, the S3 backend can lock natively:

terraform {
  backend "s3" {
    bucket       = "yourco-tfstate"
    key          = "prod/terraform.tfstate"
    region       = "ap-southeast-2"
    use_lockfile = true   # S3-native locking - no DynamoDB table needed
    encrypt      = true
  }
}

One less resource to bootstrap, pay for, and explain to auditors. If you're pinned below 1.11, keep DynamoDB - but put a version upgrade on the roadmap, because this is genuinely simpler.

![How S3-native locking resolves the race - the second apply waits, nothing corrupts]

The second apply doesn't corrupt anything or silently overwrite - it just waits its turn.

Mistake 3 - One giant state file for everything

Dev, staging, prod, the DNS zones, that one Lambda from 2023 - all in a single state. Every plan takes minutes and touches everything. Every mistake has maximum blast radius. Every junior engineer's experiment holds a loaded reference to prod.

I've worked in environments where dev, staging, and prod infrastructure lived far too close together - sometimes sharing state, sometimes just sharing enough blast radius that a plan in one environment made everyone nervous about the others. Plans that should take seconds stretch out uncomfortably long when Terraform has to reason about everything at once, and every review turns into "wait, what else does this touch?" It's a slow tax that nobody notices being introduced and everybody complains about once it's there.

The fix: one state file per environment, minimum. Same code, different backends:

envs/
├── dev/    → key = "dev/terraform.tfstate"    (2 AZs, NAT off - cheap)
└── prod/   → key = "prod/terraform.tfstate"   (3 AZs, NAT on)

The objection is always "but then dev and prod can drift apart." They can - and that's a code review problem, which is a much better problem than "dev corrupted prod's state," which is a résumé problem.

Mistake 4 - Treating state as too sacred to touch (or not sacred enough)

Two opposite failure modes, same root cause: nobody on the team actually knows the state manipulation commands.

The not sacred enough version: someone opens terraform.tfstate in an editor and hand-edits JSON. I have watched this happen. It works right up until it catastrophically doesn't.

The too sacred version: a resource needs renaming, and because nobody trusts themselves with state surgery, the team does a destroy-and-recreate of a production database instead. The scheduled-downtime email goes out for what should have been a zero-downtime metadata change.

The fix: learn three tools, in this order -

# renaming/refactoring a resource without touching the real thing:
terraform state mv aws_instance.web aws_instance.web_server

# adopting an existing resource into Terraform (1.5+, plan-reviewable):
import {
  to = aws_s3_bucket.legacy
  id = "my-existing-bucket"
}

# removing something from state WITHOUT destroying it (1.7+):
removed {
  from = aws_instance.snowflake
  lifecycle { destroy = false }
}

The import and removed blocks are the underrated ones - they turn state surgery into a reviewable pull request instead of a hero typing commands into prod at 6pm.

I've had to reach for terraform import and manual state inspection more than once - usually to bring an existing resource under management without Terraform deciding the "safe" move was to destroy and recreate it. That instinct - trusting state mv and import over letting Terraform tear something down "because state says so" - is something you only really learn by almost letting the destroy happen once and catching it in the plan output first.

Mistake 5 - Secrets in state, state in Git

Here's the uncomfortable truth many teams learn during their first security review: Terraform state stores resource attributes in plaintext - including database passwords, and anything a provider returns. Which means the day someone commits terraform.tfstate to the repo "just as a backup," those secrets are in Git history forever.

The fix is layered:

.gitignore the state files on day one - *.tfstate and *.tfstate.* (backup files too)
Encrypt the state bucket (KMS server-side encryption in the bootstrap above)
Prefer ephemeral values and secret-manager references over raw secret strings in configuration, so fewer secrets land in state at all
Treat state bucket access as production access - IAM-restricted, logged

If you're checking one thing after reading this article, make it git log --all -- "*.tfstate" on your oldest repo. I hope it comes back empty. It often doesn't.

Patterns you'll probably recognise, even if the stories above don't quite match

State mistakes tend to rhyme across companies more than people expect. A few more I'd bet you've seen at least one of:

terraform.tfstate committed to Git "just this once" - and now it's in history forever
Two people running apply in the same environment at the same time, no lock, no warning until the diff looks wrong
A renamed resource treated as delete-then-create instead of a state mv, because nobody knew the command existed
Local state surviving purely because nobody's laptop has died yet - not a policy, just luck
State quietly holding secrets nobody remembered were sensitive, until a security review found them

If even one of these lands, you already know exactly which of the five fixes above to reach for first.

The pattern behind all five

None of these mistakes are exotic. They're all the same mistake wearing five costumes: treating state as an implementation detail instead of what it actually is - the single source of truth that makes Terraform work at all.

The teams that avoid the 2am state incidents aren't smarter. They just made five boring decisions on day one: remote backend, locking, per-env isolation, learned state surgery, and secrets hygiene. Thirty minutes of setup, total.

I packaged the thirty minutes

Because I kept setting up the same foundations on every new engagement, I turned my day-one layout into a small kit: bootstrap for a versioned/encrypted/locked state bucket, per-environment structure with S3-native locking, a multi-AZ VPC module, provider-level default_tags so untagged resources are impossible, and pre-commit hooks (fmt, validate, tflint) so bad code never reaches CI.

The free version of the knowledge is this article - you can build all of it yourself from what's above. If you'd rather skip the assembly, the kit is here: Terraform AWS Starter Kit - and the repo layout is documented in the listing so you know exactly what you're getting.

Either way: go check where your state lives. Today, ideally.

I'm a DevOps engineer with 13 years across AWS, Azure, Kubernetes, Terraform and Ansible. Previously: I built a self-healing pipeline with Prometheus and 50 lines of Python. I also maintain CronPort, a free cron expression converter for crontab, Kubernetes, GitHub Actions, AWS EventBridge and Terraform.

My nginx died on a Saturday, so I taught it to fix itself

Suresh Babu Narayanan — Fri, 17 Jul 2026 06:26:44 +0000

A few months ago I set myself a weekend challenge: could I make a service restart itself when it dies, using only free open-source tools, with zero manual steps in between?

Not because restarting nginx is hard. It's one command. But I've spent thirteen years in infrastructure, and the pattern I keep seeing is this: the alert fires at 2am, a human wakes up, squints at a phone, VPNs in, and runs the one command. The gap between "we knew" and "we acted" is where the pain lives. I wanted to close that gap on my own laptop, end to end, so I properly understood every link in the chain - not just the theory.

Here's the whole thing: Prometheus notices nginx is down, Alertmanager fires a webhook, and a small Python service restarts the container. Total cost: $0. Total code I had to write: about 50 lines.

What we're building

nginx  →  nginx-exporter  →  Prometheus  →  Alertmanager  →  responder (Flask)  →  docker restart nginx
         (status→metrics)    (scrape+rule)   (route alert)     (receive+act)

Five containers, one docker-compose file. The full repo is at the end - everything below is copy-pasteable, but I'd genuinely suggest typing the alert rule and the responder yourself. That's where the learning is.

Prerequisites: a laptop with Docker, basic YAML tolerance, and about an hour. It took me longer the first time, and I'll show you exactly where I lost the time, because that part taught me more than the parts that worked.

Step 1 - nginx and its exporter

nginx doesn't speak Prometheus natively, so we pair it with the official exporter, which reads nginx's stub_status page and republishes it as metrics - including the one we care about: nginx_up, which is 1 when nginx answers and 0 when it doesn't.

The nginx config just adds a status listener:

server {
    listen 8081;
    location /stub_status {
        stub_status;
        access_log off;
    }
}

And in docker-compose, the exporter points at it:

nginx-exporter:
  image: nginx/nginx-prometheus-exporter:1.1.0
  command:
    - --nginx.scrape-uri=http://nginx:8081/stub_status

Why an exporter instead of just letting Prometheus scrape nginx directly? Because Prometheus's own up metric tells you whether the scrape target answered - and if you scrape the exporter, up stays 1 even while nginx is dead, because the exporter itself is fine. nginx_up is the truth. This distinction bit me once in a real environment years ago, and it's the kind of thing you only internalise by getting it wrong.

Step 2 - Prometheus and the alert rule

Prometheus scrapes the exporter every 5 seconds (aggressive, but this is a demo and I wanted fast feedback loops):

global:
  scrape_interval: 5s
  evaluation_interval: 5s

The alert rule is four lines that matter:

- alert: NginxDown
  expr: nginx_up == 0
  for: 30s
  labels:
    severity: critical
    action: auto_restart

The for: 30s is doing more work than it looks like. Without it, a single blipped scrape - a container pausing for GC, a network hiccup - fires the alert and restarts a perfectly healthy service. With it, nginx has to be continuously down for 30 seconds before anything happens. Every flapping-alert horror story I've heard traces back to someone skipping this line. In production I'd set it longer; for a demo, 30 seconds keeps the feedback loop tight.

The action: auto_restart label is a habit from working with event-driven automation at scale: route on labels, not alert names. Today one alert triggers a restart; later, twenty alerts can share the same remediation path just by carrying the label.

Step 3 - Alertmanager, and where I lost an hour

Alertmanager's job is routing: this alert goes to that webhook.

route:
  receiver: auto-remediation
  group_by: ["alertname"]
  group_wait: 10s
  repeat_interval: 5m

receivers:
  - name: auto-remediation
    webhook_configs:
      - url: http://responder:5000/alert
        send_resolved: true

This is where my Saturday went sideways. My first version had the webhook URL pointing at localhost:5000 - which inside the Alertmanager container means Alertmanager itself, not my responder. Prometheus showed the alert firing, Alertmanager showed it received, and… nothing happened. No errors anywhere obvious, because from Alertmanager's point of view nothing was wrong; it was dutifully delivering webhooks into the void.

The fix is obvious in hindsight - use the compose service name, responder:5000 - but I mention it because "alert fires, action doesn't happen, no visible error" is exactly the failure mode you must know how to debug before trusting automation to act for you. docker logs alertmanager and a curl from inside the container found it in the end.

The other setting worth a sentence: repeat_interval: 5m means if the alert keeps firing, the webhook re-fires every 5 minutes. That sounds helpful until you realise it means your restart action will retry forever against a service that's never coming back. Hold that thought.

Step 4 - the responder: 50 lines of Flask

The receiving end is deliberately boring - a Flask app that validates the alert and restarts the container via the Docker API:

@app.route("/alert", methods=["POST"])
def alert():
    payload = request.get_json(silent=True) or {}
    for a in payload.get("alerts", []):
        name = a.get("labels", {}).get("alertname")
        if name != "NginxDown" or a.get("status") != "firing":
            continue
        if not allowed_to_restart():
            log.warning("circuit breaker OPEN - escalate to a human")
            continue
        client.containers.get("nginx").restart(timeout=10)
        restart_history.append(time.time())

The part I care about is allowed_to_restart() - a circuit breaker that refuses more than 3 restarts in 10 minutes:

def allowed_to_restart() -> bool:
    now = time.time()
    while restart_history and now - restart_history[0] > WINDOW_SECONDS:
        restart_history.pop(0)
    return len(restart_history) < MAX_RESTARTS

Why bother, in a toy? Because auto-remediation without a stop condition isn't self-healing, it's self-harm with good intentions. If nginx is crashing because of a broken config or a full disk, restarting it in a loop fixes nothing and hides the real problem behind a wall of "remediation succeeded" logs. Three strikes, then a human gets paged. In a real setup this is where you'd escalate to Slack or PagerDuty; in the demo it just logs loudly.

Step 5 - kill it and watch it heal

The payoff. With everything running:

docker stop nginx

Then I sat and watched three windows: the Prometheus alerts page, docker logs -f responder, and docker ps.

The sequence: the alert went pending at the next scrape, firing after the 30-second hold, Alertmanager delivered after its 10-second group wait, and the responder log printed restarted container 'nginx'. Then - my favourite line ever - docker ps showing nginx with an uptime of a few seconds while everything else showed twenty minutes.

End to end, dead-to-healed took on my machine. Nobody touched anything.

I ran it a few more times just to enjoy it, which promptly tripped my own circuit breaker. Satisfying to be refused by my own safety rail.

The irony that wrote itself

Here's the part I didn't plan. The same week I finished this write-up, I deployed a small side project - a cron expression converter - to a well-known static hosting platform. Single HTML file, simplest deployment imaginable.

The deployment got stuck in "Queued". For twenty minutes. Then the retry got stuck too. It turned out to be a known platform-side queue jam that community threads report lasting anywhere from hours to over a day, and there was nothing I could do from my side but wait or fail over. I dragged the same file onto a different host and was live in ninety seconds.

Which is the whole point of this article, really. Everything fails: my nginx, their deploy queue, your thing next Tuesday. The difference between a bad day and a non-event isn't preventing failure - it's how much of the response you've already built before it happens. My pipeline had a restart path ready; my deployment had a second host ready. Same principle, different layer.

Where I'd take this next

If I were promoting this beyond the laptop, in rough order:

Deduplication and idempotency - Alertmanager retries webhooks; the responder should handle the same alert twice gracefully (it mostly does now, but "mostly" is doing heavy lifting).
Escalation - after the circuit breaker opens, page a human with the restart history attached, so they arrive with context instead of a cold start.
Audit trail - every automated action written somewhere durable. When automation acts on your behalf, "what did it do and when" must never be a mystery.
Smarter checks than up/down - restart-on-death is the "hello world" of remediation. The interesting versions act on latency, error rates, and saturation before the service dies.

There are, of course, mature event-driven automation platforms that do all of this at enterprise scale with rulebooks and workflow engines. Having now built the primitive version by hand, I understand far better what those platforms are actually doing under the hood - which was the point of the weekend.

Try it yourself

The full repo - compose file, configs, responder, and a README with the exact break-it steps - is here: https://github.com/suredares/Self-healing-Demo

Clone it, run docker compose up -d --build, then docker stop nginx and watch your infrastructure quietly refuse to stay broken.

If you build it and something behaves differently on your setup, tell me in the comments - genuinely curious what breaks on other machines. And if you've built auto-remediation that went wrong in an interesting way, I definitely want to hear that story.

*I'm a DevOps engineer with 13 years across AWS, Azure, Kubernetes, Terraform and Ansible, writing about automation that actually works. I also built CronPort - a free tool that converts cron expressions between crontab, Kubernetes, GitHub Actions, AWS EventBridge and Terraform.
I also wrote up Five Terraform state mistakes I keep seeing

I also sell a Terraform AWS Starter Kit if you want production-ready foundations without the setup day.

If it saved you time, you can buy me a coffee Buymeacoffee*