DEV Community: Neeraja Khanapure

Not in any textbook — learned this from a 3am page:

Neeraja Khanapure — Tue, 28 Apr 2026 10:34:18 +0000

LinkedIn Draft — Workflow (2026-04-28)

Not in any textbook — learned this from a 3am page:

Kubernetes rollouts: why 'pods are Ready' is the wrong promotion gate

Readiness is a node-local signal. Production health is a global one. Most rollout pipelines conflate the two — and that's where incidents come from.

Bad gate:                         Good gate:

Deploy ──▶ Pods Ready? ──▶ Done   Deploy ──▶ Pods Ready?
           (local signal)                    │
                                             ▼
                                    SLO window check
                                    (error rate + p95)
                                             │
                                    Pass ──▶ Promote
                                    Fail ──▶ Auto-rollback

Where it breaks:
▸ 100% Ready pods while P95 latency spikes — bad cache warmup, noisy neighbor, DB connection saturation.
▸ HPA reacts slower than a fast rollout — you ship overload before autoscaling catches up.
▸ Canary stuck green because metrics lack the right labels/slices to isolate the failing segment.

The rule I keep coming back to:
→ Promote only when the canary holds your SLO slice (error rate + latency) for a fixed observation window. Otherwise: auto-rollback.

How I sanity-check it:
▸ Argo Rollouts or Flagger with Prometheus gates — error rate, latency percentiles, saturation.
▸ Alert on canary-vs-baseline deltas, not absolute thresholds. Catches regressions that pass absolute checks.

Operational maturity isn't about tools — it's about designing for the failure you haven't seen yet.

Deep dive: https://neeraja-portfolio-v1.vercel.app/workflows/kubernetes-rollouts-why-pods-are-ready-is-the-wrong-promotion-gate

Strong opinions on this? Good. I want to hear the pushback.

kubernetes #reliability #devops #sre

A Friday systems thinking thread — worth sitting with:

Neeraja Khanapure — Fri, 24 Apr 2026 10:12:05 +0000

LinkedIn Draft — Insight (2026-04-24)

A Friday systems thinking thread — worth sitting with:

Observability debt is invisible until an incident makes it expensive

You can't debug what you can't slice. Teams add dashboards for years and still can't answer the two questions that matter most in an incident: which customers are affected, and which change caused it. The problem is almost never the tool — it's the label strategy.

Observability debt accumulation:

Month 1:  Service A metrics added (no ownership labels)
Month 3:  Service B metrics added (different label schema)
Month 6:  Dashboard count: 47. Useful in incident: 3.
Month 9:  P0 incident. Can't isolate by customer/version.
          Engineer guesses. Guesses wrong. +45min MTTR.

Fix: Define label schema FIRST. Instrument second.

The non-obvious part:
→ The teams that debug incidents fastest don't have more metrics — they have metrics that answer the right questions at the right cardinality. SLI-first instrumentation design is a force multiplier. Most teams instrument first and wonder why dashboards are noisy.

My rule:
→ Define your SLIs, then design labels that let you isolate by (service, env, version, customer tier) without exploding cardinality. Instrument last.

Worth reading:
▸ Brendan Gregg's USE Method + Google's RED Method for SLI-first design
▸ Prometheus label best practices — cardinality anti-patterns (prometheus.io/docs)

https://neeraja-portfolio-v1.vercel.app/insights/observability-debt-is-invisible-until-an-incident-makes-it-expensive

What's the version of this that your org gets wrong? Drop it below.

devops #sre #observability #platformengineering

A hard-earned rule from incident retrospectives:

Neeraja Khanapure — Tue, 21 Apr 2026 10:05:01 +0000

LinkedIn Draft — Workflow (2026-04-21)

A hard-earned rule from incident retrospectives:

GitOps drift: the silent accumulation that makes clusters unmanageable

GitOps promises Git as the source of truth. The reality: every manual kubectl during an incident is a lie you told your cluster and forgot to retract.

GitOps truth gap over time:

Week 1:  Git ══════════ Cluster  (clean)
Week 4:  Git ══════╌╌╌╌ Cluster  (2 manual patches)
Week 12: Git ════╌╌╌╌╌╌╌╌╌╌╌╌╌  (drift accumulates)
                         Cluster  (unknown state)

Where it breaks:
▸ Manual patches during incidents create cluster state Git doesn't know about — Argo/Flux will overwrite it silently.
▸ Secrets managed outside GitOps (sealed-secrets, Vault agent) drift independently — invisible in sync status.
▸ Multi-cluster setups multiply drift: each cluster diverges at its own pace once human intervention happens.

The rule I keep coming back to:
→ Treat every manual cluster change as a 5-minute loan. Commit it back to Git before the incident closes — or it's gone.

How I sanity-check it:
▸ Argo CD drift detection dashboard — surface out-of-sync resources before they become incident contributors.
▸ Weekly diff job: live cluster state vs Git. Opens a PR for anything untracked. Makes drift visible before it's painful.

The best platform teams I've seen measure success by how rarely product teams have to think about infrastructure.

Deep dive: https://neeraja-portfolio-v1.vercel.app/workflows/gitops-drift-the-silent-accumulation-that-makes-clusters-unmanageable

Curious what guardrails you've built around this. Drop your pattern below.

gitops #kubernetes #devops #platformengineering

Something every senior engineer learns the expensive way:

Neeraja Khanapure — Sun, 19 Apr 2026 16:53:12 +0000

LinkedIn Draft — Workflow (2026-04-19)

Something every senior engineer learns the expensive way:

Terraform DAGs at scale: when the graph becomes the hazard

Terraform's dependency graph is elegant at small scale. At 500+ resources across a mono-repo, it becomes the most dangerous part of your infrastructure.

SAFE (small module):              DANGEROUS (at scale):

[vpc] ──▶ [subnet] ──▶ [ec2]     [shared-net] ──▶ [team-a-infra]
                                          │         [team-b-infra]
                                          │         [team-c-infra]
                                          │         [data-layer]
                                  One change → fan-out destroy/create

Where it breaks:
▸ Implicit ordering assumptions survive until a refactor exposes them — usually as an unplanned destroy chain in prod.
▸ Fan-out graphs make blast radius review near-impossible. 'What does this change affect?' has no fast answer.
▸ depends_on papering over bad module interfaces — it fixes the symptom and couples the modules permanently.

The rule I keep coming back to:
→ If a module needs depends_on to be safe, the module boundary is wrong. Redesign the interface — don't paper over it.

How I sanity-check it:
▸ terraform graph | dot -Tsvg > graph.svg — visualize fan-out and cycles before every major refactor.
▸ Gate all applies with OPA/Conftest + mandatory human review on any planned destroy operations.

The difference between a senior engineer and a principal is knowing which guardrails to build before you need them.

Deep dive: https://neeraja-portfolio-v1.vercel.app/workflows/terraform-dags-at-scale-when-the-graph-becomes-the-hazard

Curious what guardrails you've built around this. Drop your pattern below.

terraform #iac #devops #sre

Not in any textbook — learned this from a 3am page:

Neeraja Khanapure — Sat, 18 Apr 2026 11:03:01 +0000

LinkedIn Draft — Workflow (2026-04-18)

Not in any textbook — learned this from a 3am page:

Kubernetes cost spikes: the usual suspects and how to find them fast

Cloud bills spike in Kubernetes for the same reasons every time. None of them are visible in the default dashboards — and most of them are invisible until month-end.

Cost leak sources (ranked by surprise factor):

1. Unset resource requests   → scheduler packs nodes → OOM → over-provision
2. Autoscaler scale-down lag → zombie nodes after traffic spike
3. Log pipelines w/o sample  → 40% of bill, 0% of dashboards
4. Idle namespaces           → dev clusters running 24/7
5. Spot interruption gaps    → fallback to on-demand, never reverted

Where it breaks:
▸ Missing resource requests let the scheduler over-pack nodes — when pods OOM, you over-provision to compensate.
▸ Cluster autoscaler adds nodes faster than it removes them. Spot interruptions leave zombie capacity for hours.
▸ Logging agents (Fluentd/Filebeat) on every node with no sampling become the largest line item nobody owns.

The rule I keep coming back to:
→ Every workload needs requests AND limits. Review autoscaler scale-down thresholds monthly. Sample logs at source, not at the sink.

How I sanity-check it:
▸ Kubecost or OpenCost — per-namespace/team attribution. Without this, no one feels accountable for the number.
▸ KEDA for event-driven workload scaling — eliminates idle replicas without sacrificing responsiveness.

The best platform teams I've seen measure success by how rarely product teams have to think about infrastructure.

Deep dive: https://neeraja-portfolio-v1.vercel.app/workflows/kubernetes-cost-spikes-the-usual-suspects-and-how-to-find-them-fast

Strong opinions on this? Good. I want to hear the pushback.

kubernetes #finops #devops #platformengineering

This is what separates teams that scale from teams that survive:

Neeraja Khanapure — Fri, 17 Apr 2026 10:00:17 +0000

LinkedIn Draft — Insight (2026-04-17)

This is what separates teams that scale from teams that survive:

Dependency updates are reliability work, not maintenance work

The orgs that get hit hardest by CVEs and supply chain incidents have one thing in common: they batch dependency updates into quarterly sprints. By then, the update is a 6-version jump, the changelog is 200 lines, and the 'quick upgrade' becomes a multi-day incident waiting to happen.

Batch update model (risky):   Continuous model (safe):

Q1: Skip                      Week 1: +2 deps (auto-PR)
Q2: Skip                      Week 2: +1 dep (auto-PR)
Q3: Skip                      Week 3: +3 deps (auto-PR)
Q4: "Upgrade sprint"          ...
    6-version jumps            Each PR: small diff, fast review
    Breaking changes           Rollback: one PR revert
    3-day debugging session    MTTR if it breaks: 10 min

The non-obvious part:
→ Automated dependency PRs (Renovate/Dependabot) with a 2-week merge SLA cost almost nothing — a 5-minute review per PR. Skipping them accumulates a compounding tax: more conflicts, larger blast radius, slower rollback. The math strongly favors continuous updates.

My rule:
→ Automate dependency PRs. Set a team policy: merge or explicitly defer within 2 weeks. Every skip is a known risk you're consciously accepting — treat it that way.

Worth reading:
▸ Renovate Bot — automerge strategies, scheduling, and grouping (docs.renovatebot.com)
▸ SLSA framework — supply chain integrity levels and provenance (slsa.dev)

https://neeraja-portfolio-v1.vercel.app/insights/dependency-updates-are-reliability-work-not-maintenance-work

If you're earlier in your career: bookmark this. It'll make more sense after your first real production incident.

devops #sre #observability #platformengineering

A system design trap I've seen catch strong teams off guard:

Neeraja Khanapure — Tue, 14 Apr 2026 09:59:40 +0000

LinkedIn Draft — Workflow (2026-04-14)

A system design trap I've seen catch strong teams off guard:

MLOps retraining in production: the guardrails matter more than the pipeline

Wiring a retraining loop is a weekend project. Making it safe in production — data drift, silent label shifts, rollback semantics — is the actual engineering problem.

Risky loop:          Safe loop:

Data ──▶ Train       Data ──▶ Version ──▶ Train
  │        │           │                    │
  ▼        ▼           ▼                    ▼
 Prod    Deploy      Validate            Shadow eval
 (no gate)          lineage              │
                                    Pass ──▶ Canary
                                    Fail ──▶ Rollback

Where it breaks:
▸ Better offline metrics / worse live KPIs — training/serving skew from feature drift you didn't catch.
▸ Unversioned training data makes RCA impossible. You can't reproduce what trained the broken model.
▸ No rollback path means every bad retrain is a production incident with a multi-hour recovery.

The rule I keep coming back to:
→ No model promotes without: versioned dataset lineage, shadow/canary evaluation against live traffic, and a tested one-click rollback.

How I sanity-check it:
▸ DVC + LakeFS for dataset versioning, MLflow/SageMaker Registry for model promotion gates.
▸ Prometheus + Grafana for drift monitoring — alert on trend, not single-point anomalies.

Operational maturity isn't about tools — it's about designing for the failure you haven't seen yet.

Deep dive: https://neeraja-portfolio-v1.vercel.app/workflows/mlops-retraining-in-production-the-guardrails-matter-more-than-the-pipeline

This is where most runbooks stop — what's your next step after this?

mlops #aiops #platformengineering #sre

Not in any textbook — learned this from a 3am page:

Neeraja Khanapure — Mon, 13 Apr 2026 01:12:51 +0000

LinkedIn Draft — Workflow (2026-04-13)

Not in any textbook — learned this from a 3am page:

On-call burnout is an alert design problem, not a schedule problem

Every team I've seen fight burnout by rotating people faster. The actual fix is almost always the same: the alerts are wrong.

Alert quality spectrum:

Noisy ◀─────────────────────────── ▶ Actionable

[cpu > 80%]  [pod restart]  [error budget burn]  [customer impact]
     │              │               │                    │
 ignore me      maybe?         investigate!          wake me up

Where it breaks:
▸ Alerts without a named owner and a runbook produce paralysis, not action — especially at 2am.
▸ Flapping alerts are the fastest path to alert blindness — engineers learn to dismiss pages before reading them.
▸ Cause-based alerts (disk full) and symptom-based alerts (latency spike) need different urgency and routing.

The rule I keep coming back to:
→ Before any alert ships: Who acts on it? What do they do? What's the cost of 30 minutes of inaction? If you can't answer all three, it's not ready.

How I sanity-check it:
▸ Weekly alert review ritual: tag every last-week page as actionable / noisy / redundant. Kill the bottom two categories.
▸ PagerDuty/OpsGenie grouping + escalation policies — reduce interrupt rate without hiding real incidents.

Reliability is a product feature. The engineers who treat it that way are the ones who get asked into the room.

Deep dive: https://neeraja-portfolio-v1.vercel.app/workflows/on-call-burnout-is-an-alert-design-problem-not-a-schedule-problem

Hiring managers: engineers who think about this in interviews are the ones worth calling back.

sre #observability #reliability #devops

Something I wish someone had told me five years earlier:

Neeraja Khanapure — Fri, 10 Apr 2026 09:52:55 +0000

LinkedIn Draft — Insight (2026-04-10)

Something I wish someone had told me five years earlier:

Distributed tracing: the gap between having it and using it in incidents

Most orgs instrument distributed traces correctly and then debug incidents with grep. The investment in tracing pays off only when your debugging workflow changes — when you start from a trace ID instead of a log query. That's a culture change, not a tooling change.

Current state (most orgs):      Target state:

Incident fires                  Incident fires
     │                               │
Grep logs ──▶ Guess service ──▶  Pull trace ID from alert
     │                               │
More grep ──▶ Find error      Trace shows full request path
     │                               │
Escalate ──▶ More engineers   Latency waterfall identifies
     │                         bottleneck in 3 minutes
MTTR: 90 min                  MTTR: 15 min

The non-obvious part:
→ Traces don't reduce MTTR on their own — runbooks that start from trace IDs do. The highest-leverage thing you can do after instrumenting is to rewrite your top 5 incident runbooks to start with 'get the trace ID from the alert, open it in Jaeger/Tempo, find the slowest span.' Engineers follow runbooks under pressure.

My rule:
→ Instrument your 3 highest-traffic endpoints first. Then rewrite one runbook to start from a trace ID. Measure incident time-to-hypothesis before and after.

Worth reading:
▸ OpenTelemetry instrumentation guides — language SDKs (opentelemetry.io/docs)
▸ Grafana Tempo + Loki correlation — trace-to-log workflow without leaving the dashboard

https://neeraja-portfolio-v1.vercel.app/insights/distributed-tracing-the-gap-between-having-it-and-using-it-in-incidents

If you're earlier in your career: bookmark this. It'll make more sense after your first real production incident.

devops #sre #observability #platformengineering

A hard-earned rule from incident retrospectives:

Neeraja Khanapure — Tue, 07 Apr 2026 09:48:10 +0000

LinkedIn Draft — Workflow (2026-04-07)

A hard-earned rule from incident retrospectives:

Incident RCA without a data-backed timeline is just a story you told yourself

Most post-mortems produce lessons that don't stick. The root cause is almost always the same: the timeline was built from memory, not from data.

Memory-based timeline:     Data-backed timeline:

T+0  "Deploy happened"     T+0:00  Deploy (Argo event)
T+?  "Errors started"      T+0:07  Error rate +0.3% (Prometheus)
T+?  "Someone noticed"     T+0:12  P95 latency 340ms→2.1s (trace)
T+?  "We rolled back"      T+0:19  Alert fired (PD)
                           T+0:31  Rollback complete (Argo)

Where it breaks:
▸ Log timestamps across services diverge by seconds without NTP — your timeline is wrong before you begin.
▸ Correlation between a deploy event and a metric spike gets missed when dashboards lack deployment markers.
▸ Contributing factors vanish from the narrative because they're hard to prove — and the same incident repeats.

The rule I keep coming back to:
→ Build the timeline from data only before the RCA meeting begins. If you can't source an event, mark it 'unverified' — not assumed.

How I sanity-check it:
▸ OpenTelemetry trace IDs as the timeline spine — they cross service boundaries with sub-millisecond precision.
▸ Grafana annotations on every deploy, config change, and scaling event — visible on every dashboard automatically.

Systems that are hard to debug were designed without the debugger in mind. Build observability in, not on.

Deep dive: https://neeraja-portfolio-v1.vercel.app/workflows/incident-rca-without-a-data-backed-timeline-is-just-a-story-you-told-yourself

This is where most runbooks stop — what's your next step after this?

sre #reliability #observability #devops

Something I keep explaining in architecture reviews:

Neeraja Khanapure — Sun, 05 Apr 2026 16:55:28 +0000

LinkedIn Draft — Workflow (2026-04-05)

Something I keep explaining in architecture reviews:

Secrets management: designing for rotation, not just storage

Most orgs solve 'where do we store secrets securely.' The teams that get paged at 2am are the ones who never solved 'how do we rotate them without downtime.'

Storage-only design:        Rotation-aware design:

Secret ──▶ Vault            Secret ──▶ Vault ──▶ Agent Injector
              │                                        │
         Pod (env var)                           Pod (file mount)
              │                                        │
         Restart to           Auto-reload ◀────── Lease renewer
         get new value        (zero downtime)

Where it breaks:
▸ Secrets as env vars require pod restarts on rotation — making rotation a deployment event with blast radius.
▸ Vault leases expiring in long-running jobs produce auth errors that look like app bugs, not infra failures.
▸ Secret sprawl across namespaces means rotation happens in 12 places — and one always gets missed.

The rule I keep coming back to:
→ Design rotation before you design storage. If you can't rotate a secret in under 10 minutes with no downtime, the design isn't production-ready.

How I sanity-check it:
▸ Vault Agent Injector or External Secrets Operator — decouple secret delivery from pod lifecycle.
▸ Monthly secret access log audit — stale consumers are how you discover forgotten service accounts before attackers do.

Reliability is a product feature. The engineers who treat it that way are the ones who get asked into the room.

Deep dive: https://neeraja-portfolio-v1.vercel.app/workflows/secrets-management-designing-for-rotation-not-just-storage

If this triggered a war story, I'd genuinely love to hear it.

security #devops #kubernetes #sre

Something I wish someone had told me five years earlier:

Neeraja Khanapure — Fri, 03 Apr 2026 09:39:52 +0000

LinkedIn Draft — Insight (2026-04-03)

Something I wish someone had told me five years earlier:

Zero-downtime deployments: what 'zero' actually requires most teams don't have

Most teams say they do zero-downtime deploys and mean 'we haven't gotten a complaint in a while.' Actually measuring it reveals the truth: connection drops, in-flight request failures, and cache invalidation spikes during rollouts that nobody's tracking because nobody defined what zero means.

What 'zero downtime' actually requires:

✓ Health checks reflect REAL readiness (not just 'process started')
✓ Graceful shutdown drains in-flight requests (SIGTERM handling)
✓ Connection draining at the load balancer (not just the pod)
✓ Rollback faster than the deploy (< 5 min, automated)
✓ SLI measurement during the rollout window (not just after)

Missing any one of these = not zero downtime. Just unmonitored downtime.

The non-obvious part:
→ The most common failure mode is passing health checks before the app is actually ready — DB connections not pooled, caches not warm, background workers not started. The pod is 'Ready' and the app is still initializing. Users see errors. Nobody's dashboard shows it because nobody's measuring error rate during the rollout window.

My rule:
→ Define 'zero downtime' with a measurable SLI: error rate < 0.1% during any 5-minute deploy window. Validate this in staging before calling it done. Measure it in production on every release.

Worth reading:
▸ Kubernetes deployment strategies — rolling, blue/green, canary with traffic splitting
▸ AWS ALB / GCP Cloud Load Balancing — connection draining configuration and health check tuning

https://neeraja-portfolio-v1.vercel.app/insights/zero-downtime-deployments-what-zero-actually-requires-most-teams-dont-have

If you're a manager reading this — it's worth asking your team where they are on this.

devops #sre #observability #platformengineering