DEV Community: Mridul Tiwari

Patch night wasn’t one incident — it was three failure modes in a row

Mridul Tiwari — Sun, 27 Sep 2026 03:13:28 +0000

Tuesday’s plan looked boring on paper: run the SSM patch runbooks across a bunch of Auto Scaling Groups, bake AMIs, promote launch templates, go home. By midnight I was still on a call with Jenkins console output in one terminal and aws autoscaling describe-auto-scaling-groups in the other. Nobody had paged us for a single catastrophic outage. What we had was worse in a different way — a pile of medium failures that kept teaching the same lesson: the platform replaces your box faster than you can trust what’s on disk.

We’re on several AWS accounts in ap-south-1, with a mix of ASG-backed app tiers, central Logstash, non-prod EKS behind private ALBs, and Jenkins still in the middle of deploy paths. The week of 19–26 Sep was a patching wave plus finishing non-prod EKS ingress cutover, logging gaps on refreshed EC2, and the usual Jenkins/Argo/dashboard tickets. Storm-on-EKS had already been written up the week before; this one was the “everything else” shift.

I’ll focus on three threads that actually changed how I work: patch automation that lied, Filebeat that looked like Logstash, and an ALB that went healthy only after we remembered how traffic really flows.

When the runbook finishes but the AMI doesn’t

We drove bulk patching through SSM documents — Patching-ASG, NewRunbook, Ami-Patch — with shared params: package excludes (java, elasticsearch, tomcat, nginx, agents, and friends), IMDSv2 metadata, no overwriting the document default TargetAmiName, dedupe by ASG. On paper that’s the right shape. In practice, a large batch of assisted runs faltered: wrong params, retries, timeouts, and duplicate AMIs for the same intent. AMIs existed, but I stopped trusting “green” in the SSM console as “safe to put in the launch template.”

My triage order became mechanical: does the AMI exist? Are there dupes for the same ASG intent? If we’re launching replacements, is it InsufficientInstanceCapacity on Graviton in one AZ — where our retry policy was same subnet, same instance type, no family hop? Is the worker even in SSM? When automation output looked suspect, manual no-reboot / standard create-image from a known-good instance was slower but honest. Automation was still worth it for param gathering and dedupe logic; validate before LT promotion was the gate we’d been skipping in spirit.

Two failures stuck in my notes. On a central Logstash ASG we hit InvalidAMIID.NotFound: the launch template (or source AMI) was deregistered while the instance kept running. Automation can’t RunInstances until you register a fresh AMI from the live node — the running box is the source of truth, and when the running instance’s LT id/version had drifted from what the ASG thought it was using, SourceAmiId had to come from the running instance AMI, not a stale LT pointer.

The other was uglier: NewRunbook timed out on an Elasticsearch master patch worker at verifySsmInstall after 20 minutes. Cloud-init on aarch64 tried to install SSM via a path that blew up with Unsupported architecture aarch64. The worker never registered; we had an orphaned EC2 until someone terminated it manually. That’s a recurring risk I’m now paranoid about: arm64 first-boot SSM install paths that don’t match the AMI family.

With patch AMIs unreliable, we fell back to in-place patching on ASG members — yum update, reboot for kernel. First time through, I learned this is not “SSH and yum” in a vacuum.

I patched what I thought was a “safe” instance — termination protection in my head, “this one shouldn’t be replaced.” The ASG still replaced it. Long patch window or a bad reboot fails the health check; the group launches from the launch template. Those replacements are not clones of the old disk. Fresh boot, often missing agents, log paths, local tuning, manual fixes from three incidents ago. We turned one patching task into config drift repair on top of Jenkins already being on fire.

The pattern we landed on: ASG Standby. One member at a time — move to Standby (out of rotation, still running), patch, reboot, validate, back to InService. It doesn’t make the ASG magic, but it cuts the odds that traffic and health checks drive a replace cycle on the box you’re mid-flight on. You still coordinate desired capacity and which AZ you’re touching.

Then the kernel trap. yum installed a new kernel; we rebooted; uname -r still showed the old kernel. On RHEL-family AMIs that often means the boot loader default never moved — grub2-set-default, read /etc/default/grub, grub2-mkconfig, fix the right BOOT entry and cmdline, reboot again, verify. “Patched and rebooted” is not “running the new kernel.” First time through that was trial and error; I didn’t capture exact timings, but it burned a chunk of the window.

Meanwhile Jenkins wasn’t a sidebar. On two consecutive patch nights (~mid-week), frontend build agents, deploy paths, and related CI failures ate 3–4 hours each night before patching could finish. Patch work, ASG churn, config repair, and CI recovery were coupled. I’d call patch night done only after agents were healthy — lesson learned the expensive way.

`harvester.running: 0` and why I stopped blaming Logstash first

Separate thread, same week: “Filebeat not sending” after an ASG instance refresh. My instinct — and I’ve seen teams do this — is to stare at Logstash or OpenSearch. The metrics told a different story: harvester.running: 0, registrar.states: 0. Filebeat wasn’t reading files. Output to Logstash is irrelevant until harvesters run.

We walked paths and permissions. Generic template inputs didn’t match where the app actually wrote logs — path mismatch. Directories at 750 and files at 640 bite when the filebeat user isn’t the app user. Fix was manual config correction after debugging; shipping came back when paths aligned. Not primarily a Logstash outage.

Logstash did throw noise if you went looking: InvalidFrameProtocolException for Beats protocol 10 and 13 on port 5044. Bytes looked like HTTP CRLF. Plain beats { port => 5044 } on Logstash; Filebeat output.logstash without SSL. Likely something doing HTTP health checks on the Beats port or random TCP to 5044 — fix on the LB side is TCP health check, not HTTP on 5044. I keep that diagnostic order now: harvesters → paths/perms → then protocol errors on the collector.

Fleet-wise, Ansible had security-group gaps — filebeat role missing on some boxes, Promtail still on others. That’s a separate fix from the one-off path correction, but it explains why refresh keeps biting the same org.

On EKS we were also splitting Alloy streams for application-namespace container logs — lines starting with [TOMCAT] to *-tomcat-logs, everything else to *-app-logs. Different layer, same theme: the pipeline is only as good as the contract at the edge.

Private ALB by security group, and the 8080 rule we almost forgot

Non-prod EKS ingress finished a move from CIDR-based private ALB to security-group inbound (alb-private-sg values). We removed legacy CIDR ingress objects and dropped alb-private.yaml from Argo valueFiles; Argo CD’s own ingress rode the same cutover. Shipped live with tcp/8080 from the ALB security group onto cluster and node security groups. No major outage was reported — which still doesn’t mean it was free.

The gotcha we’d seen on older shared ALBs came back: custom ALB SG on the ingress without the LBC-managed “traffic” SG means targets sit unhealthy / timeout until you explicitly allow tcp/8080 from the ALB SG onto cluster and node SGs. Same pattern as before; easy to forget when the manifest “looks” right in Git.

Deploy model shifted too: stopped Argo Rollouts canary for GitOps-managed apps in favor of RollingUpdate Deployment with maxUnavailable: 0 where we configured it — less ceremony, more predictable rollouts for that environment.

Smaller ALB puzzle on the same listener: one app’s host + /prometheus returned a fixed 503 “Backend action does not exist” while another app on the same ALB forwarded /prometheus fine. Missing or wrong listener rule for the first app’s metrics path — not a pod problem, a rule problem.

Storm CI got Jenkins docker build + topology deploy aligned with other container workloads; one image tag deployable to multiple topologies with entrypoint sleep infinity and jar submit on a separate path. Housekeeping that week included infra pod timezone audit (UTC vs IST) and readonly digging on why a gateway deployment scaled one pod at a time — HPA/PDB/scheduling, not glamorous but real.

What I’d do again (and what I’m watching)

This wasn’t a single root-cause postmortem; it was platform whack-a-mole with a few sharp takeaways tied to what we actually touched.

Patching: Gate automation output — dupes, timeouts, arm64 SSM workers — before LT promotion. In-place on ASGs: Standby → patch → reboot → verify uname -r and grub default → InService. Expect LT replacements to lack pet config; don’t rely on disk state — Ansible, agents, or pull-based config has to be the default. Treat Jenkins health as a dependency on patch night, not a parallel ticket you’ll “get to.”

Logs: If harvesters are zero, fix Filebeat before Logstash. HTTP on 5044 is a distraction with a real fix elsewhere.

EKS ingress: SG-based private ALB is fine if you remember the ALB SG → node/cluster SG on 8080 contract every time.

OpenSearch dedicated masters at ~97–98% OS RAM on ~16 GiB nodes with ~26–70% JVM heap looked alarming until we reconciled fixed ~10 GiB heap with OS “used” not equaling heap pressure — data nodes looked healthier on OS % because of larger RAM and mapped buffers. Critical Elasticsearch data-tier patch/reboot was prep and read-only SOP on our shift, not execution — I’m glad that stayed someone else’s careful window.

Governance housekeeping: script to copy EC2 instance governance tags to attached EBS volumes (dry-run vs --apply), tested on one instance before fleet. Prometheus got a new EKS scrape job on non-prod with the same Jenkins encrypt-secret pattern as other EKS jobs.

If I’d publish one table from the week for my own notebook, it would be symptoms vs layer: duplicate or missing patch AMI → automation/LT; fresh ASG instance missing agents/logs → replace churn + config drift; logs “not sending” with zero harvesters → Filebeat paths/perms; ALB targets timeout → SG rules on 8080, not the app. Nothing here is a new framework — it’s the kind of week where three boring layers stack up and the job is to not mistake the symptom’s loudest service for the broken contract.

Next patch wave, I’m validating AMIs before promotion, running Standby on in-place work, and checking Jenkins agents before I call the night done. The rest can stay in tickets — but those three habits would have saved us half the hours we didn’t get back.

No official Storm Helm chart — how we put Apache Storm on EKS anyway

Mridul Tiwari — Sat, 19 Sep 2026 04:30:00 +0000

The Argo CD app for Storm had been red for two days, but the symptom that actually sent me down the rabbit hole was quieter than a crash loop: init containers on the Supervisor pods, stuck forever, printing the same line over and over — waiting for nimbus …:6627. No Nimbus, no workers, no UI. Just a cluster that looked almost deployed and wasn't.

We were trying to do something the organization had never done before: run Apache Storm on Kubernetes. Not a lift-and-shift of an existing pattern — there was no pattern. Storm still lived on traditional EC2/ASG elsewhere. The ask was to move it onto the same non-prod EKS cluster we were already standing up for the other application platform: Terraform, Karpenter, Argo GitOps, the whole stack. Streaming workloads sitting with everything else instead of a separate pet fleet.

That part made sense strategically. Operationally, it was a blank page.

Why there was no paved road

Apache Storm does not ship a maintained official Helm chart. The project does not publish a credible "migrate Storm to Kubernetes" guide. Community options exist — G-Research's gresearch/storm chart with a Bitnami ZooKeeper subchart is the one people point at — but nothing we could treat as a supported product. We evaluated it and still built a thin local chart instead, same as other custom overlays in the repo: own the templates, own the values, no opaque upstream dependency.

That decision felt right and scary in equal measure. We were the first team anywhere in the org — any account, any region — putting Storm on Kubernetes. No internal reference architecture, no runbook, no "copy what prod did." Every choice about storage, networking, image choice, and GitOps layout was experimental. Before we committed, we wrote down what we were afraid of. That list turned out to be more accurate than our initial rollout plan.

We planned a phased rollout on purpose. Phase one: ZooKeeper and Nimbus only — prove the control plane and that PVCs actually bind. Phase two: Supervisors and Storm UI as stateless Deployments. Phase three: a LogViewer sidecar on supervisor pods, not a separate Deployment. Phase four: private ECR images instead of Docker Hub upstream. Phase five: Storm UI on the shared private ALB ingress pattern the rest of other already used.

One shared Storm namespace on the non-prod cluster, not one cluster per environment. Karpenter-provisioned ARM64 nodes. A StorageClass for gp3 embedded in the Storm chart itself rather than a separate Argo Application — tradeoff being that chart prune would remove the class, but we avoided yet another GitOps app for a single consumer.

The architecture looked reasonable on paper:

Component	Kind	Storage
ZooKeeper	StatefulSet	PVC (data + datalog)
Nimbus	StatefulSet	PVC
Supervisor	Deployment	emptyDir; pod IP as `storm.local.hostname`
Storm UI	Deployment	emptyDir
LogViewer	sidecar on Supervisor	shared emptyDir `/logs`

Reasonable, and completely untested in our environment.

The cascade that wasn't obvious at first

When things broke, they did not break loudly. Failure chain number one was storage, and it propagated silently.

Our chart requested a StorageClass named gp3. The EBS CSI driver on the cluster was healthy. The StorageClass did not exist. The cluster only had legacy in-tree gp2. We had flagged this exact fear before rollout — "we feared silent PVC Pending forever" — and we were right.

PVCs for ZooKeeper and Nimbus sat in Pending. Without bound volumes, those StatefulSet pods never scheduled. Without Nimbus listening on :6627, everything downstream waited. Supervisor and UI init containers blocked on the nimbus health check. Argo marked the app Degraded. The StatefulSets showed OutOfSync because pods never became ready. From the outside it looked like a Storm problem. It was a storage class problem three layers down.

That was the first lesson in how StatefulSet failures hide: init containers do not scream "your StorageClass is wrong." They scream "upstream isn't ready yet," which sends you chasing Nimbus when Nimbus was never going to start.

We fixed the missing class — embedded gp3 via EBS CSI in the chart — and the control plane could finally land. Then failure chain number two hit immediately, and this one was louder.

Nimbus crashed with:

/docker-entrypoint.sh: exec: nimbus: not found

Storm UI did the same for ui.

We had copied Helm examples that pass bare args like nimbus or ui, which works for upstream Docker Hub images. Our private ECR images use a docker-entrypoint.sh that execs the first argument as a command. They need storm nimbus and storm ui. Supervisors and the LogViewer sidecar already used the storm … form; Nimbus and UI did not. Official image docs do not apply when you own the entrypoint. We should have tested container startup before wiring the full chart, not after PVCs finally bound.

Fix was a one-line args change per deployment. The archaeology cost more time than the fix.

Argo fighting StatefulSets, and the ALB that couldn't find a port

With pods actually running, we got failure chain number three: perpetual Argo sync drift on StatefulSet volumeClaimTemplates. Kubernetes injects apiVersion, kind, volumeMode, and status after create. Our chart templates did not emit the full schema Argo expected, so the app sat OutOfSync even when the cluster was fine. This is a known footgun with GitOps and StatefulSets. We fixed it two ways: corrected the chart templates to emit the full volumeClaimTemplate schema, and added ignoreDifferences jq paths with RespectIgnoreDifferences for the immutable VCT fields Argo will never reconcile away.

That one felt familiar if you've run StatefulSets under Argo before. The ingress problem did not.

Failure chain number four was AWS Load Balancer Controller weirdness exposing Storm UI. We wanted the same private ALB pattern as other other apps. LBC returned effectively: TargetGroup port is empty. We had Instance target type with a ClusterIP service — a combination that does not work the way we had it wired. Ingress backends need numeric ports with target-type: ip, not named ports with Instance mode. We switched to target-type: ip and backend port 8080 as a number, not a named service port. Storm UI came up behind the shared ALB.

Separately, we were migrating inbound access on that ALB from CIDR allowlists to security-group-based inbound. That affects every app on the load balancer, not just Storm. We added a parallel SG-based ALB config file so CIDR and SG ingresses could coexist during cutover testing rather than replacing the inbound model on a shared ALB before every consumer was validated.

What we validated along the way

Not everything we worried about broke, but everything we worried about deserved an explicit check.

Stateful workloads on Karpenter made us nervous — ZK and Nimbus expect stable identity and disk; Karpenter nodes are ephemeral by design. For non-prod we accepted single-replica ZooKeeper with no quorum. HA in prod is still an open question.

ARM64 was non-negotiable on our Karpenter pool. Private ECR images are single-arch. If they had been amd64-only, we'd have seen scheduling failures or runtime crashes on Graviton. We verified arch before calling the rollout done.

Supervisor and LogViewer logs live on emptyDir. Pod restart means lost worker logs. Fine for non-prod exploration; scary for prod debugging. We noted it and moved on — durable log shipping is still TBD.

Topology lifecycle is also TBD. The chart runs the cluster; it does not submit topologies. How jar submit, worker scaling, and upgrades interact with Deployments versus traditional supervisor hosts — we haven't answered that yet. We kept topologies out of scope on purpose so we could fail fast on PVCs and images without dragging application deployment into the first week.

By the end of the week, all four failure chains were closed. Argo showed the Storm app healthy. We had the first Apache Storm cluster running on EKS in the organization: custom Helm, GitOps-managed, on Karpenter ARM nodes. ZooKeeper and Nimbus StatefulSets bound PVCs. Private images ran once we corrected entrypoints. Storm UI sat behind the same private ALB pattern as the rest of other.

What I'd do differently, and what I'd tell the next team

If I were starting this again, I'd run a pre-flight checklist before the first argocd app sync, not after init containers had been waiting for forty-eight hours:

StorageClass exists on the target cluster — gp3 via CSI is not the same as in-tree gp2. PVC Pending cascades silently through init containers.
Container entrypoint behavior — docker run with the exact args your Helm chart will pass, especially for private images. Do not assume args: [nimbus] from upstream examples.
Argo + StatefulSet volumeClaimTemplates — emit the full schema or configure ignoreDifferences upfront. You will hit VCT drift; budget for it in the chart, not as a week-two surprise.
AWS LBC + ClusterIP — target-type: ip and numeric backend ports. Named ports plus Instance mode gave us an empty TargetGroup port and a day of ingress debugging.

The bigger takeaway is about first-in-org workloads. No reference deployment means every assumption — ARM, storage, HA, logs, ingress models on shared infrastructure — has to be validated on the actual cluster. Writing down the fear inventory before we committed did not prevent the failures. It did prevent us from treating them as mysteries. We knew storage was a risk, entrypoints were a risk, Argo StatefulSet sync was a risk, ALB quirks were a risk. When each one fired, we recognized it instead of spiraling.

We still went forward for good reasons. Platform direction is Kubernetes; maintaining a separate Storm EC2 fleet alongside a new EKS cluster doubles operational surface — patching, AMIs, security groups, deploy pipelines. Non-prod first bounded blast radius. We reused GitOps muscle we already had with other apps, Alloy, Jenkins ingress. Phased delivery let us prove the control plane before workers and UI. Someone had to be first; better on non-prod with eyes open than a surprise prod migration later.

Prod cutover still has open questions: ZooKeeper HA with a three-node quorum on EKS, a topology submit pipeline from CI, supervisor horizontal scale and slot planning, durable log shipping, whether to adopt the community chart or keep the local one, DNS cutover from legacy Storm UI, and migrating the shared ALB fully to SG-based inbound after all apps are tested. None of those blocked non-prod stand-up. They are the next chapter.

Storm on Kubernetes is DIY. There is no official Helm chart and no migration guide. Budget time for chart authorship and image entrypoint archaeology. And when init containers say they're waiting for Nimbus, check whether Nimbus ever had a chance to start — the bug might be three layers below Storm itself.

We tagged EKS workers like app servers — and Prometheus started scraping telegraf on nodes that never had it

Mridul Tiwari — Sat, 12 Sep 2026 05:25:01 +0000

The Telegraf Down alerts started piling up on a Thursday morning, and at first glance it looked bad. Nine targets, all critical, all under the same rule name. My first instinct was a fleet-wide agent failure — something in the monitoring stack had broken overnight and we were about to page half the platform team for a problem that didn't exist.

It wasn't that. It was worse in a quieter way: one alert name, at least three unrelated root causes, and one of them was our own governance work from three days earlier finally catching up with us.

The tagging work nobody thought would touch monitoring

On September 8 we were in the middle of an org-wide AWS governance push — standard tags across prod VPCs for cost allocation, ownership, compliance, inventory. Same schema everywhere: Application, Component, businessunit, environment, techteam, Role, Criticality, and the rest. Multiple VPCs in ap-south-1, each with its own change ticket, but the rules were consistent.

We were careful about it. Add-only on instances and ASGs — never overwrite existing Name, service, backup, or legacy tags. For Component, businessunit, and techteam, if an instance already had a value, we kept it; VPC-wide defaults only filled gaps. VPC-level tags went on first (Tech Owner, ticket reference), then bulk instance tagging.

In one prod application VPC, that all went according to plan for the traditional stuff. Tomcat/Spring ASGs already had techteam set; we added environment=prod and the common governance keys without touching what was there.

The EKS application nodegroup ASG was different. Before September 8 it only carried EKS-managed tags — eks:cluster-name, eks:nodegroup-name, k8s.io/cluster/*. No techteam, no environment, no Application. The ASG-level script added environment=prod and the governance keys. techteam wasn't set at ASG level because our rule said use existing values only, and there wasn't one.

The running worker instances ended up with techteam, Application, environment, and businessunit anyway — likely from EKS nodegroup or launch template tagging tied to the same governance work, not from us replacing an old value. From a compliance perspective, that looked fine. From a monitoring perspective, we had just told Prometheus these were prod platform hosts.

Why the alert fired three days later

Central Prometheus discovers host telegraf via EC2 service discovery. The job filters on tag:environment=prod and tag:techteam matching a regex of several platform team values. Scrape port :9273.

EKS worker nodes are EC2 instances. Once they picked up environment=prod plus a matching techteam, they entered the scrape pool. But Kubernetes workers don't run host telegraf on :9273. Metrics on those nodes come from in-cluster DaemonSets or cAdvisor — not a host agent listening on the standard port.

Five EKS worker targets went permanently up=0. Telegraf Down, critical. It looked like a sudden outage. It was a discovery side effect from September 8 governance work, surfacing in bulk once alert volume crossed whatever threshold made it impossible to ignore.

That was the story I wanted to be true — one cause, one fix, done by lunch. I started grouping targets by hostname and role before touching anything, which turned out to be the only reason we didn't make things worse.

One alert, three root causes

Prometheus showed roughly nine telegraf targets down, plus one kafka exporter and one redis exporter. Same alert family, completely different problems underneath.

EKS workers (5 hosts) — tagging side effect. These nodes were never supposed to be in the EC2 SD scrape pool. They matched the tag filter after governance tagging. No telegraf process on :9273 because that's not how we monitor Kubernetes workers. False positive, unfixed at the time we captured this.

New AL2023 ASG instances (2 hosts) — real telegraf misconfiguration, unrelated to EKS. Launched a few days before the alert spike. Telegraf RPM was installed, but the config was broken: prometheus_client output was commented out, so the agent logged "no outputs found" and had nothing to expose on the scrape port. Ansible-pull had never run on those boxes to template telegraf.conf. Fixes for AL2023 telegraf existed in the ansible repo; they just hadn't been applied to new ASG instances because there was no ansible-pull cron on those hosts.

Kafka broker (:9308) — long-standing debt, not a telegraf problem. The kafka exporter service had been disabled for months. Telegraf on :9278 on the same host was actually fine. The alert rule conflated exporter health with agent health, or we were looking at the wrong port when we first opened the target list.

On top of those three, we also had a redis host with both telegraf and redis exporter down, and a dev/test host that probably shouldn't have been in the prod scrape config at all. Different owners, different fixes — all wearing the same alert costume.

I spent a while on the kafka broker before I noticed telegraf was healthy and the exporter was the dead component. Classic trap: one alert name makes you assume one failure mode. The AL2023 hosts took longer because the RPM was there — systemctl status telegraf looked plausible until you read the config and saw the commented output block. EKS was the fastest to classify once I checked tags and confirmed no listener on :9273.

The fix we wanted vs. the fixes we didn't want

For the EKS issue, the least-change path was a Prometheus relabel drop on the telegraf EC2 SD job: drop targets where eks:cluster-name exists (or an equivalent EKS tag). Roughly four lines, config reload. No governance rollback, no new DaemonSet, no alert silence.

We explicitly ruled out a few alternatives:

Remove tags from the nodegroup — rolls back compliance work for a monitoring problem. Wrong lever.
Install host telegraf as a DaemonSet on EKS — large ops burden unless we actually need host-level metrics on workers, which we don't for this scrape job.
Silence the alert — hides real failures on the AL2023 hosts and the kafka/redis issues.

For the AL2023 hosts, the real fix is ansible-pull (or equivalent) running the telegraf role so prometheus_client output gets templated correctly. That's ops work on two boxes, not a Prometheus change.

For kafka, re-enable the exporter after verifying cluster health — separate ticket, separate owner.

At the time we captured this, most of those fixes were still pending: Prometheus relabel for EKS nodes, ansible-pull on the AL2023 ASGs, kafka exporter re-enablement, and SSH access to diagnose the redis host.

What I'd check before the next bulk VPC tag

Governance tags and monitoring discovery share the same key space. We use environment and techteam for cost and compliance; Prometheus EC2 SD uses the same keys to decide what to scrape. Those two systems don't talk to each other until an alert fires.

Before bulk tagging a VPC, I'd pull the EC2 SD jobs that filter on those tag keys and ask: will EKS nodes enter this pool? Will patch-automation orphans? Dev instances that happen to have environment=prod from a template copy? Tagging makes an instance visible to discovery; it doesn't install the agent discovery expects.

EKS workers are not app EC2. Tagging them like traditional Tomcat boxes makes Prometheus expect host telegraf on :9273. The nodes did nothing wrong. The tags were correct for governance. The scrape config was written for a different kind of machine.

And when nine targets go red under one rule name, classify by host role before you fix anything. We almost treated a compliance side effect, a config drift on new AMIs, and months-old exporter debt as a single incident. They're three. The alert didn't know that. We had to.

Standing up a greenfield EKS cluster — and every mistake we stepped on

Mridul Tiwari — Sat, 05 Sep 2026 04:25:11 +0000

The Argo CD UI showed SyncFailed on karpenter-node-infra before we'd deployed a single application pod. EC2NodeClass and NodePool didn't exist in the cluster — not because the manifests were wrong, but because the CRDs had never been installed. We'd already burned an afternoon on Terraform provider pins and ARM AMI mismatches. This was supposed to be the easy part: GitOps bootstrap, Karpenter online, move on.

It wasn't.

What we were building

We weren't cloning a legacy self-managed K8s 1.21 ASG stack. This was a greenfield non-prod EKS cluster in an existing VPC — private API only, VPN/office CIDRs on the cluster security group, subnets tagged for cluster shared ownership, internal ELB, and karpenter.sh/discovery.

The compute model had two layers on purpose:

Layer 1 — a bootstrap managed node group: on-demand ARM64, AL2023, t4g.xlarge-class instances, min 1 / max 3, tainted CriticalAddonsOnly=true:NoSchedule. It runs Karpenter controller, Argo CD, EKS addons, and optionally Cluster Autoscaler scoped only to that ASG.

Layer 2 — Karpenter NodePool for application workloads, with a separate EC2NodeClass and IAM role.

We chose Karpenter over Cluster Autoscaler for app scaling, EKS Pod Identity (not IRSA) for Karpenter/LBC/CA auth, Terragrunt for backend/provider DRY, terraform-aws-modules/eks ~20.36, and a dedicated GitOps branch. Target Kubernetes version: 1.36 on create — destroy/recreate, not an upgrade ladder on an empty cluster.

That all sounded clean on paper.

Terraform got mean before the cluster existed

The first terraform plan failure was the AWS provider. We'd drifted to provider 6.x; EKS module 20.x still references launch template fields that 6.x removed. Plan blew up with errors about fields that no longer exist. Fix was blunt: pin hashicorp/aws >= 5.40, < 6.0 alongside the module version and document the upper bound next time we bump either one.

Then architecture assumptions collided with reality. We started with spot on the bootstrap pool; requirement clarified to 100% on-demand. Fine — but we'd already baked spot into early config. More annoying: we paired an ARM64 AMI with x86 instance types. AMI architecture has to match the instance family. Obvious in hindsight, expensive when you're staring at nodes that never join correctly.

The managed node group apply failed when AWS still had desired=0 while we set min_size=1. The EKS module ignores desired_size after create — so on first create you have to set min/desired/max together, or manually bump desired ≥ min once before apply. I didn't know that until apply failed.

Disk was another silent lie. Module defaults said disk_size=50, but instances came up around 20 GiB. The module default doesn't always win against the launch template path we were on. We needed explicit block_device_mappings on the bootstrap launch template — 50 GiB gp3 root — to get what we thought we'd already configured.

IAM role name_prefix exceeded 38 characters. Set a short explicit iam_role_name on the node group instead of letting Terraform generate something too long.

We pinned the AL2023 ARM AMI via data source to cluster_version, not "latest surprise AMI." Addons in order: pod-identity-agent before compute, then coredns, vpc-cni, kube-proxy, ebs-csi. Karpenter got its node role, instance profile, access entry, and controller Pod Identity association in Terraform — which almost caused a duplicate IAM role later when GitOps still pointed at IRSA from a reference cluster. More on that.

One dead end I won't repeat: CONTAINER_RUNTIME=docker on AL2023 / K8s 1.32+. Invalid. Dockershim is gone. containerd via nodeadm is the correct path on AL2023.

GitOps: where reference configs go to die

We copied a reference GitOps branch without scrubbing it. Wrong cluster names, IRSA ARNs, Karpenter version mismatched to our K8s version. I almost created a duplicate Karpenter controller IAM role — Terraform already had Pod Identity wired up, but Helm values still carried IRSA annotations from the old cluster.

Argo CD itself went on the bootstrap nodes with tolerations for CriticalAddonsOnly. Helm install first, then a Git deploy key secret — pods don't have SSH_AUTH_SOCK, so SSH agent forwarding wasn't an option.

Bootstrap order we settled on:

Argo CD on bootstrap nodes
Git deploy key
karpenter-crds app (sync wave -1) — CRDs from chart crds/ dir, ServerSideApply
Karpenter controller (skipCrds: true)
karpenter-node-infra — EC2NodeClass + NodePool manifests

That order exists because of a Helm behavior that's easy to miss: helm template skips the chart crds/ directory. The controller chart never installed NodePool/EC2NodeClass CRDs. Argo kept reporting SyncFailed until we split CRDs into their own Application.

We made it worse before we made it better. Two Argo apps owned the same CRDs, and we had argo-cd self-sync enabled. Sync deadlock. CRD ownership has to be exactly one Argo app; the controller must use skipCrds: true. And the Application for argo-cd itself should not auto-sync — that's another path to deadlock.

valueFiles: ../../helm-values/... failed with parent path escape — Argo blocked it, charts rendered empty. Fix: multi-source with ref: values, or keep values under paths Argo allows inside the chart tree.

Bootstrap node sizing bit us too. t4g.large was too small for Argo + Karpenter + all system addons. We bumped to xlarge and stopped fighting eviction loops on the control plane of our control plane.

The Karpenter vs Cluster Autoscaler confusion

Mid-build someone asked why Cluster Autoscaler wasn't adding nodes for pending app pods. Because app capacity is Karpenter's job, not CA's. We scoped CA to the bootstrap ASG only — optional, and only for that tainted pool. Pending app workloads need a healthy NodePool, EC2NodeClass, and Karpenter controller — not CA scale-out.

That question was a signal we'd documented the two-layer model in Terraform but not clearly enough in runbooks.

Version path we shouldn't have taken

On an empty cluster we walked 1.31 → 1.32 → 1.34 → 1.36 instead of targeting 1.36 upfront and destroy/recreating once. Greenfield means you pick the version before first apply, not ladder upgrades on nothing.

The repo-server red herring

After a bootstrap node recycle, Argo showed ComparisonError and repo-server connection refused. I spent time suspecting chart bugs. Stale comparison state. Once bootstrap pods were healthy again, refresh cleared it — not a chart fix, not a Git problem.

What actually worked

When we stopped fighting copied config and pinned the boring stuff:

Terraform: provider < 6.0, explicit gp3 50 GiB on launch template, min/desired/max aligned on first NG create, Pod Identity for Karpenter/LBC/CA, short IAM role names, AMI arch matched to Graviton instance types.
GitOps: skeleton branch scrubbed for cluster name, discovery tag, Pod Identity vs IRSA, Karpenter version for K8s 1.36. Single CRD owner app with ServerSideApply. Controller with skipCrds: true. No auto-sync on argo-cd self-management.
Runtime: xlarge bootstrap nodes, on-demand only, containerd via nodeadm on AL2023.

End state: private control plane, Pod Identity auth, GitOps-ready Karpenter scaling, system controllers isolated from app workloads on a repeatable greenfield pattern.

What I'd do differently next time

I'd treat the pre-apply checklist as blocking, not advisory:

Confirm on-demand vs spot, ARM vs x86, target K8s version before first terraform apply.
Size bootstrap for Argo + Karpenter + all addons — start at xlarge.
Document the EKS module desired_size ignore behavior and set all three sizing knobs together on create.
Pin AWS provider upper bound whenever we pin EKS module version.
GitOps from a skeleton, never a fork — scrub names, discovery tags, Pod Identity vs IRSA, Karpenter/K8s version alignment.
CRD ownership: one Argo app, controller skips CRDs, argo-cd app doesn't auto-sync.
valueFiles: stay inside allowed paths or use multi-source values refs.

The cluster came up fine in the end. The story isn't "EKS is hard" — it's that greenfield gives you freedom to skip legacy baggage and still step on every sharp edge if you copy someone else's YAML without reading it. We learned more from the SyncFailed CRDs and the provider 6.x plan failure than from the architecture diagram. That's probably how it should be.

When Kafka Hits 100% Disk and the Volume Won't Grow

Mridul Tiwari — Sat, 29 Aug 2026 06:44:43 +0000

The alert didn't come from consumer lag. It came from disk — two of three brokers on our production Kafka cluster reporting /data at 100%, with about 20K free on an 850G EBS volume. That's not "we should look at this tomorrow" territory. That's "something is about to stop accepting writes" territory.

We couldn't expand the volume. Not "we'd prefer not to" — the EBS volume literally wasn't modifiable at that moment. So the usual playbook — bump the disk, watch the graph flatten — was off the table. Whatever we did had to reclaim space from inside Kafka itself, without casually deleting segments on prod.

What we were looking at

This is a three-broker cluster in production. The pain was concentrated on a high-volume log topic: 40 partitions, replication factor 2, fed by Kubernetes log shippers. Three consumer groups were attached. On paper, consumption looked fine — lag was around 262, which is nothing you'd page on.

Two brokers were pinned at 100%. The third still had headroom, which told us this wasn't a uniform cluster-wide misconfiguration; it was a retention and placement problem playing out unevenly across replicas.

The topic config, when we pulled it:

retention.ms=172800000        # 48 hours
retention.bytes=-1            # no byte cap
segment.bytes=536870912       # 512MB segments
compression.type=producer

Forty-eight hours of retention with no byte limit on a topic that ingests K8s logs at scale. That's the kind of config you set once during bootstrap and never revisit until /data screams.

Starting with the obvious — and ruling things out

First instinct: consumer lag. Maybe Logstash fell behind and segments aren't getting cleaned up because consumers haven't committed offsets far enough?

We checked. Lag was ~262. Consumers were keeping up. That dead end mattered — it kept us from burning time scaling consumers as the primary fix. (More on Logstash later; it was a follow-up, not the root cause.)

Next: how big is this topic actually?

du -sh /data/kafka-logs/<topic>-*

Roughly 30G per partition. Forty partitions, RF=2 — the math adds up fast. Even with compression (producer), you're holding two days of high-volume log traffic with no ceiling on total bytes. The disk didn't fill because consumers were slow. It filled because retention policy said "keep everything for 48 hours, however big that gets."

I also briefly entertained a storage hack: attach a third EBS volume and use growpart to extend the existing 850G disk. That doesn't work the way I wanted it to — you can't just bolt on block storage and grow an existing filesystem across unrelated volumes. Ruled out before anyone got too excited about it.

At this point the picture was clear: time-based retention with no byte cap on a firehose topic. Segments age out on the clock, not on disk pressure. Low lag doesn't help if you're still obligated to retain 48 hours of data regardless of volume.

Why two brokers and not three

With RF=2 on 40 partitions across three brokers, replica distribution isn't perfectly even. Leader election and partition assignment meant two brokers ended up holding more of the heavy replicas. The third broker had room — which is almost worse, because it makes the incident look like a broker problem when it's really a topic policy problem showing up asymmetrically.

Running diagnostics locally had its own friction. One broker had bootstrap connection quirks when I tried to run kafka-configs from my laptop — enough to slow me down, not enough to change the diagnosis. I didn't capture the exact error string, but it was the kind of thing where you SSH to the broker and run the command there instead of fighting client config for twenty minutes during an incident.

The fix we could actually do in prod

Manual log deletion on a prod cluster is a last resort. You can orphan consumers, confuse leaders, and create a very exciting afternoon for everyone. We needed Kafka's delete policy to do the work — which meant changing retention so old segments become eligible for cleanup.

The prod-safe path:

1. Tighten retention.ms — we targeted 24h instead of 48h. Halving the time window doesn't instantly free 850G, but it changes which segments are eligible for deletion on the next cleanup cycle.

2. Set retention.bytes — this was the important half. Per-broker byte limits give the delete policy something to act on when time alone isn't enough. With -1, Kafka had no reason to drop data early regardless of disk pressure.

Something like:

kafka-configs --bootstrap-server <broker> \
  --entity-type topics --entity-name <log-topic> \
  --alter --add-config retention.ms=86400000,retention.bytes=<per-broker-cap>

The exact byte cap needs to fit your partition count and RF — you're budgeting across replicas, not pretending one broker owns the whole topic. I don't have the final number we landed on in my notes, but the principle was: set a cap that forces segment deletion before /data hits 100% again, with headroom for normal variance.

3. Pause or scale down producers if needed — before flipping retention on a full disk, you sometimes need to stop the inbound firehose briefly. K8s log shippers don't care about your incident; they'll keep writing. Reducing producer pressure before the config change gives the delete policy room to catch up instead of fighting new segments while old ones are still technically retained.

4. Watch segments actually disappear — this isn't instant. Cleanup runs on a schedule. We monitored /data free space and confirmed segments aging past the new retention window were getting removed. No manual rm -rf in the log directories.

Logstash — real problem, wrong root cause

Only 2 of 7 Logstash instances were consuming this topic. That's worth fixing. Under-provisioned consumption can cause lag, can cause operational blind spots, and is generally sloppy.

But lag was 262. The disk was full because retention said "keep 48 hours, unlimited size." Scaling Logstash would improve throughput and resilience; it would not have emptied an 850G volume holding two days of unrestricted log data. We flagged it as follow-up work, not incident mitigation.

That's a distinction I want to keep sharp: healthy-looking lag masked an unhealthy retention policy. I've seen teams chase consumer scaling during disk incidents before. Sometimes that's right. Here it would've been a distraction.

What I'd do differently

I'd have caught this before /data hit 100%. Disk usage per broker and retention config on high-volume topics belong in the same dashboard. Lag alone is a lie of omission when retention.bytes=-1.

If I couldn't resize the volume during the incident, I'd still open the ticket to make it resizable — storage headroom isn't a substitute for correct retention, but running prod Kafka with no expansion path is its own risk.

Next time I'd also verify Logstash consumer coverage during topic onboarding, not during a disk emergency. Two of seven is a config drift problem waiting to happen.

Takeaways from this one:

Low consumer lag does not mean disk is healthy. Check retention.ms and retention.bytes when /data fills on log-heavy topics.
K8s log shippers feeding Kafka can fill brokers even when every consumer group looks caught up — the firehose doesn't care about your lag graph.
When you can't expand the volume, your only safe lever is retention policy. Set byte caps before you need them.

Why our liveness probe timed out even though /healthCheck never touched Redis

Mridul Tiwari — Sat, 22 Aug 2026 04:42:07 +0000

The alert looked like a dependency outage. Readiness and liveness on our Java service were failing in bursts — context deadline exceeded (Client.Timeout exceeded while awaiting headers) — and around the same window, application logs were full of cache and Redis errors. My first instinct was to trace the health handler and ask whether we'd accidentally started pinging Redis on every probe.

We hadn't. The handler was a static 200 OK. When it actually ran, it logged ~0 ms. That mismatch — probes dying while the health code looked fine — is what sent us down the wrong path for a few hours, and what eventually made the real cause obvious.

What we were running

The service is a Spring Boot app behind an ingress and an AWS target group. Kubernetes runs liveness and readiness as HTTP GETs against the same path: /healthCheck. Period 10 seconds, timeoutSeconds: 1. The load balancer runs its own health check on the same path and port every 10 seconds.

That endpoint is deliberately dumb. It does not call Redis, a database, or anything downstream. Passing the probe only proves the HTTP server can accept a request and return a response. We treat dependency health elsewhere; this path is supposed to be cheap.

Under normal traffic, that design works. During an HPA scale event, it did not.

When things started failing

Failures weren't steady. They clustered when the deployment scaled or pods churned — new replicas coming up, old ones draining, traffic shifting. Fresh pods looked fine. Older ones in the middle of the fleet would fail probes, then sometimes recover after load redistributed. Less often we saw connection refused on pods that were starting or terminating; that part felt like noise until we separated it from the timeout failures.

The timeout failures were the scary ones. Kubernetes marked pods not ready or restarted them. From outside, it looked like the service was unhealthy. From inside, when we grepped logs for the health handler itself, we kept finding fast 200s — but only on the requests that actually reached the handler.

That distinction mattered. We weren't looking at a slow health check. We were looking at health checks that sometimes never got a thread in time.

Chasing Redis (and missing the point)

The timeline overlap with Redis/cache errors made this harder than it should have been. Business API paths were logging codec and connectivity issues. Latency on those endpoints spiked. It was reasonable to wonder if /healthCheck had grown a dependency check we didn't know about, or if Spring's health aggregation had been turned on for something we thought was isolated.

We walked the handler and confirmed: static response, no downstream calls. Optional jar/source review would have been redundant — runtime logs already showed 0 ms when the handler executed. So the cache errors were real, but they weren't in the probe path. They were on API traffic that shared infrastructure with the probe path.

That was our first dead end as a root cause, though not as a contributing factor. Fixing Redis wouldn't fix probe timeouts by making /healthCheck faster — there was nothing left to speed up in the handler itself. But Redis slowness could still make everything worse if it blocked the threads that were supposed to serve probes.

Thread pool saturation

Tomcat serves /healthCheck and every API route from the same worker thread pool. We had the pool capped around 200 threads. Under load, especially during scale events when traffic hadn't yet spread evenly, worker threads filled up with slow API work. Redis trouble on those paths added latency and kept threads busy longer.

Probes don't wait politely. Kubelet hits liveness and readiness every 10 seconds with a 1-second timeout. The target group adds another /healthCheck every 10 seconds per instance. Those requests land in the same queue as everything else. If all 200 threads are tied up waiting on cache calls or slow downstream logic, a probe sits in the accept queue until the client gives up waiting for headers.

Hence the error text: not a connection failure to Redis, not a 500 from the health handler — Client.Timeout exceeded while awaiting headers. The TCP connection might succeed; the response never starts in time.

Once we framed it that way, the intermittent pattern made sense. New pods after scale-up had empty pools and a smaller share of traffic until the service settled. They passed probes immediately. Older pods holding more connections and hotter thread utilization failed first. Pods we terminated showed connection refused because the process was already gone — expected, and a different failure mode from starvation.

We didn't capture exact queue depths or a precise latency number in the notes from that shift. What we had was correlation: HPA events, probe failures, high thread occupancy, and cache errors on API paths — not on the health handler when it ran.

Why "raise the timeout" wasn't the whole story

Raising timeoutSeconds above 1 would give probes more time to wait for a free thread. That's a valid mitigation and probably stops the bleeding on liveness kills. But it treats the symptom. A 3-second probe timeout on a handler that executes in 0 ms is a signal that something else is wrong with capacity or isolation.

We weighed a few changes together:

Increase probe timeout. Low risk, quick to deploy. Buys headroom when the pool is briefly saturated. Doesn't fix API slowness or reduce thread contention.

Separate probe traffic from application traffic. A dedicated connector or a minimal probe path served on a different thread pool (or even a sidecar/admin port) means kubelet and the load balancer aren't competing with /api/... for the same 200 workers. This is more work but addresses the architectural coupling.

Fix the cache codec errors on API paths. Indirect but real. Errors that slow business requests keep threads pinned longer, which increases the odds that a probe waits past 1 second. The health endpoint wasn't broken; the pool was over-subscribed because of problems elsewhere.

Stop triple-stacking the same path. Liveness, readiness, and the target group all hammer /healthCheck on the same port. Each alone is light; combined with API load on one pool, they're another source of contention. Readiness might warrant a slightly richer check; liveness should stay as dumb as possible — but not necessarily on the same threads as heavy traffic.

We didn't treat "make /healthCheck ping Redis so failures are honest" as a fix. That would have made probes fail for the wrong reason and conflated "app process up" with "cache reachable," which is exactly what we'd almost done when we misread the Redis log correlation.

What actually changed

The immediate config change was increasing probe timeoutSeconds so transient saturation during scale events didn't restart pods. In parallel — because the evidence pointed at thread occupancy, not handler logic — we prioritized the cache errors on API routes that were holding threads and traced whether Tomcat's max threads and accept queue were appropriate for peak + probe overhead.

Longer term, the design direction was clear: isolate probe handling from the main servlet traffic, and avoid using one static endpoint for liveness, readiness, and external health checks without explicit headroom in the pool.

After deploy, the pattern we watched for was the same one that fooled us initially: new pods staying green while the fleet scales. That's not proof the problem is gone. It only means those instances haven't hit saturation yet. Validation was watching older replicas through the next HPA event and confirming probes stayed green while API latency and thread utilization stayed within bounds — and that when cache errors appeared, they didn't precede another wave of probe timeouts.

What I'd check first next time

If the health handler is trivial and logs show 0 ms when it runs, but Kubernetes reports probe timeouts, I wouldn't start with dependency connectivity. I'd look at thread pool utilization, probe timeout vs period, and who else hits that path — kubelet twice (liveness + readiness), plus the load balancer — on the same connector as production API traffic.

Probe timeout is not the same as dependency failure. A 1-second timeout on a 0 ms handler is telling you the request didn't get served in time, not that the handler logic failed. connection refused on shutting-down pods is a separate signal from awaiting headers on live ones.

The lesson from this incident isn't "monitor more." It's narrower: a healthy /healthCheck implementation can still fail probes when the HTTP server is saturated, and log lines about Redis on other endpoints can send you on a long detour if you assume every failure mode must flow through the health handler itself.

One Helm template change broke every Argo app — fixing nil `ingressInt` safely

Mridul Tiwari — Sun, 16 Aug 2026 13:30:07 +0000

The Slack thread started the way these things usually do: three people pasting the same Argo CD error within a few minutes of each other. Sync failed. Not one app — several. All on the same shared EKS platform, all pulling from the same application Helm chart we'd been using for ages.

I opened Argo first on the app I knew had deployed recently, then on two others that hadn't changed in weeks. Same failure. That ruled out my first instinct — a bad image tag or a typo in one team's values file. Whatever broke was upstream of individual app config.

A shared chart, a feature merge, and a new template

Our platform pattern is familiar if you've run Kubernetes at any scale: dozens of services, one common chart, per-app and per-env values overlays. Argo CD watches the repo and syncs each Application to its cluster. Most apps define ingress for their primary routing. A subset also define ingressExt for a public-facing ALB. Nobody had needed an internal-only ingress type until recently.

A feature PR had landed that added support for internal ALBs — ingressInt — plus matching templates ingress-int.yaml and ingress-ext.yaml in the shared chart. The author needed internal ingress for one service, so they added an ingressInt block to that app's values file and merged. CI was green. The chart rendered fine in whatever path the PR exercised.

Then Argo tried to sync everyone else.

Narrowing it down

Argo's UI isn't always generous with Helm errors, but the sync logs usually cough up the render failure. The message we kept seeing was the Helm classic:

Error: template: .../ingress-int.yaml:...: executing "..." at <.Values.ingressInt.enabled>: nil pointer evaluating interface {}.enabled

That line tells you almost everything. The template touched .Values.ingressInt.enabled. For the app that got the new values block, ingressInt exists and has an enabled field. For every other consumer of the chart, ingressInt was never defined. In Go-template land, .Values.ingressInt is nil, and dereferencing .enabled on nil is a hard stop — Helm never gets as far as creating or updating resources.

I pulled the diff from the feature merge. Two new template files, and at the top of ingress-int.yaml:

{{- if .Values.ingressInt.enabled -}}

No default in the chart's root values.yaml. No guard checking whether ingressInt exists before reading .enabled. The template runs for every app on every helm template / helm upgrade — the if only skips the body of the template; the condition itself still evaluates .Values.ingressInt.enabled first.

That matched the blast radius. Apps with the new key: fine. Apps with only ingress and maybe ingressExt: broken. Apps that hadn't been touched in months: broken. Argo doesn't isolate chart render failures per consumer when they share a chart version — one bad optional key poisons the whole sync surface.

We briefly wondered whether we should patch values files app by app. There are a lot of them — dev, staging, prod overlays, team forks. Adding ingressInt: { enabled: false } everywhere would work, but it's the kind of churn that hides in a giant PR and still leaves you one missing file away from the next outage. The regression had already been introduced by adding ingressInt to a single app's values while the template assumed every app would have the key. Repeating that pattern at scale felt wrong.

The fix: chart defaults and nil-safe guards

We wanted a chart-only fix — no hunting through dozens of per-env values files, no coordination with every team to add a stub block. Two changes, applied together.

First, add defaults in the chart's values.yaml so ingressInt and ingressExt always exist, even when an app never mentions them:

ingressInt:
  enabled: false
  hosts: []
  annotations: {}
  tls: []

ingressExt:
  enabled: false
  hosts: []
  annotations: {}
  tls: []

Apps that need internal or external ALBs keep overriding these in their own values. Everyone else inherits disabled defaults and never thinks about the keys again.

Second, harden the templates so a missing or partial values merge can't take down render again. The condition in ingress-int.yaml became:

{{- if and .Values.ingressInt .Values.ingressInt.enabled -}}

Same pattern in ingress-ext.yaml for consistency — same class of bug, same class of fix. The and short-circuits: if ingressInt is nil or absent after a bad merge, the template skips cleanly instead of panicking.

We reverted the one-off ingressInt block from the single app's values file. With chart defaults in place, that app could set ingressInt.enabled: true and its hosts/annotations when they actually needed internal ingress, without carrying a special snowflake block that implied the key was app-local rather than chart-wide.

After merging the chart fix, Argo syncs recovered across the board without touching individual Application manifests or env-specific values. The app that originally needed internal ingress still opts in through overrides; the rest of the fleet never knew anything happened except that sync went red and then green again.

Why this works (and what we'd do differently)

Helm merges values layers: chart defaults, then parent charts, then user-supplied -f files and --set. If the chart doesn't define a key, and no values file defines it either, .Values.ingressInt is nil in the template context. Optional features can't assume every consumer opted in — especially on a shared chart where most apps will never use the feature.

Defaults alone would probably have fixed this specific incident, because ingressInt.enabled would resolve to false everywhere. We kept the nil guard anyway. Defaults can be overridden away, subcharts can omit keys, and someone will eventually add a third ingress variant the same way. The guard costs one line and buys immunity to the exact error we saw.

The process miss was visible in the PR itself: the feature was validated on one app that had ingressInt in values, while the template executed for all apps on the chart. A render check that only runs helm template with the happy-path values file wouldn't catch it. Next time, for optional blocks on a shared chart, I'd want CI to render against a minimal values fixture — just enough to deploy a generic app, with no ingressInt — alongside the feature app's overlay. If minimal render passes, you've got both defaults and guards covered, or at least you'll see the nil pointer before merge.

One template change broke every Argo app on the platform. Fixing it took chart-level defaults and a nil-safe condition, not a values-file scavenger hunt across the org. That's the pattern I'd reuse: when you add optional .Values blocks to a shared Helm chart, ship chart defaults and nil-safe template guards — not every consumer will define the key, and Argo will sync all of them on the same chart version whether you planned for that or not.

When TLS 1.3-only broke everything behind Akamai

Mridul Tiwari — Sun, 05 Jul 2026 06:59:46 +0000

The alert wasn't subtle. Requests just stopped hitting our backend nginx box. Internal traffic from our own domains kept flowing, but anything coming in through Akamai-hosted domains — routed via the External LB — went dead. Several frontends all depended on that backend, so they went down together.

We run nginx in front of a shared backend that multiple application frontends talk to. Under normal conditions, Akamai terminates at the edge, traffic crosses the External LB, and nginx forwards to the app tier. Internal paths bypass that edge path entirely, which is why the split behavior was our first real clue: the nginx box wasn't universally broken. Something in the Akamai → External LB path was failing.

Chasing the wrong certificate

We started from the frontend boxes and ran curls against the backend path. The failures looked like they were getting cut off at Akamai — not nginx, not the app. That pointed outward, not inward.

We opened a ticket with Akamai. Their read was an SSL certificate problem. That didn't sit right. We hadn't rotated certs, renewed anything, or touched the cert chain on our side. If nothing on the certificate had changed, why would Akamai suddenly start rejecting handshakes?

We kept digging anyway. The Akamai angle was real — requests really were dying at their edge — but "certificate issue" felt like a symptom label, not the mechanism.

The policy mismatch

While we were going back and forth, someone pulled up the External LB TLS security policy. That was the turn.

The LB had been set to TLS 1.3 only. At Akamai, the property was on their default supported policy — the one that negotiates down through 1.2 → 1.1 → 1.0, not a strict 1.3-only handshake.

TLS 1.3-only on the load balancer means the server side of the handshake insists on 1.3. Akamai's side, configured for broader compatibility, wasn't going to meet it there. The handshake never completed. From the outside it looked like Akamai was cutting requests off — and in a sense it was, because the TLS negotiation failed before anything useful got through. Easy to misread as "SSL cert broken" when the real failure mode is protocol policy mismatch.

We hadn't changed certificates. We had changed (or inherited) a TLS policy that didn't match what Akamai could speak on that property.
What fixed it

We rolled the External LB TLS security policy back to one that supports both TLS 1.2 and TLS 1.3. After that change, traffic from Akamai-hosted domains started reaching nginx again, and the dependent frontends came back.

No nginx reload drama, no cert redeploy — just aligning the LB's minimum/maximum protocol behavior with what the Akamai property actually negotiates.

What I'm keeping in my head

The split between internal (working) and Akamai (broken) saved us from burning time on nginx config. But "Akamai says certificate" plus "we didn't touch certs" should have pushed us to TLS policy sooner.

The operational bit I care about: any time we change TLS policy on the External LB, we need to coordinate with whoever owns the Akamai property config. Edge and origin have to agree on what handshake they're willing to do. A 1.3-only LB in front of an Akamai property still on broad compatibility isn't a subtle drift — it's a hard cutoff.

I'm marking this one complete, but the runbook note is the part that matters for the next person: check LB TLS policy against Akamai's supported cipher/protocol set before you assume the cert is bad.

What is the BEST Programming Language to Start with?

Mridul Tiwari — Sun, 27 Feb 2022 15:41:27 +0000

One of the most popular question that every student asks before they begin their journey of programming is which programming language to choose from OR should i say

WHAT IS THE BEST LANGUAGE TO START WITH ?

A very cliche answer to this curiousity ridden question is "Every Language is Best, its just a matter of your personal choice"
This would have been a good reply to someone who has learn few languages and is on the journey of making one or more, his best. But this isn't a good answer for someone who didn't even started their journey and are looking for something that is worth their time
So, Here is a collection of all information you need to CHOOSE which language to start from.

WHAT ARE THE AVAILABLE OPTIONS?

If you are like me than you must have researched your days off to find out what are the different programming languages one can choose from ?
And you might have come across a few like-

C++
PYTHON
C
JAVA
R
REACT
JAVASCRIPT,etc.

If you have heard the names of any one of those than good you might have some knowledge of what we are going to discuss here
But if you are hearing this for the first time than don't worry we are going to discuss some of them in this article only.

SOME SUBSTITUTES

Now there are two more languages that are not considered programming languages but more of sturcture and styling languages i.e.
HTML AND CSS.
They are the two languages that i recommend you should at least learn for a while if you have never programmed anything because these are very basic languages which can give you a jist of what the other languages might have in store for you and these languages will give you perfect mindset to deal and learn programming languages.

Shortening the List.

Out of the above mentioned 7 programming languages there are 4 or lets say 3 languages most people are confused about that includes

1. C++

2. Python

3. JAVA

4. C

And Now we are going to judge these languages based on some criterias to clear the confusion.

What are the Criterias of Judging?

EASE to Understand
Learning Curve
Interface
User Experience
Any help website of youtube channels
How much time it would take to learn?
What are its real world use?

C

C is one of those language that i think is taught to first years in most of the colleges.And same was happened to me.Being a person who only knew Python by then it was not at all jarring for me to learn this language.
So, the best advice i can give for this language is

"You have to be open Minded and let things sink in first"

-Ease to Understand :* * * * (4/5 stars)

As i mentioned earlier to learn C,One just have to let it sink in you mind for a little bit.By that i mean , Give it some time to let yourself understand its syntaxes and rules ,etc.
In the mean time do some basic questions like Pattern Printing ,searching ,sorting and level up steadily.
-Learning Curve: * * (2/5 stars)

Interface wise i would always recommend use an online compiler for C or a mobile app cause the other IDLE i tried for C for turbo C++ which was terrible but i give it benefit of doubt that it was meant to be used as programming IDLE for developers but for habitual Coding stuff
-Interface: * * * (3/5 stars)

C is a great User Experience being it so easy to understand and learn
-User Experience : * * * *(4/5 stars)

Helping guide for C are very fabulous and there are a lot of it:
Tutorial Point,
Geeks for Geeks,
Free Code Camp,
W3school

-Help Guide: * * * * (4/5 stars)

It would take aprroximately a week or two for you to learn it if you go about only doing it for those weeks.Meaning its not much Time counsuming.
-Time To Learn: * * (2/5 stars)

For the Real world use of C i would highlight what the internet has to say for it:

"Real-World Applications of C. Use of the C programming language is not limited to the development of operating systems and applications. It is also used in GUI development, IDE development, etc."

-Real World Experience: * * * (3/5 stars)

C++

My very first experience with C++ came about a couple of days earlier and i am not kidding once you do C its insanely easy to do C++. There are only syntax changes for the most parts.You will be done with all basic stuff in a day.

Ease of Understanding: * * * (3/5 stars)

As of the Learning Curve its exactly like C so if you did C earlier than this will be a piece of cake but even if you take it as your first language than also it would take you anylonger to understand than if you had take C.

Learning Curve: * * (2/5 stars)

I will deduct a star for interface though cause although i gave C benefit of doubt that Turbo C++ was not meant to be for C.It was catered for C++ and it was not good Experience .I would recommend VS code for sure, here.

Interface: * * (2/5 stars)

If you Use VS code as your text editor than the user experience with this language is very good.Some things are to be nit-picked but still all in all a satisfying user experience.

User Experience: * * * (3/5 stars)

There have been a lot of Helping Media for C++ mainly because most students use it for CP(competitive Programming).So, you will not find lack of resources for C++
some of which include:
W3school
Tutorialpoint
StackOverflow
GFG
freecodecamp

Helping Guide: * * * * (4/5 stars)

As said earlier it won't take any more time than C did, if you are a beginner and this is your first language,But if this comes to not be your first language than it would take even lesser time for you to learn.

Time to Learn: * * (2/5 stars)

Its Real World Use is insanely big.
And i will quote what the internet has to say about it.

"
C++ Application in Real World
GUI Applications.
Operating Systems.
Web Browsers.
Database Management System.
Libraries.
Cloud Computing and Distributed Applications."

Real Word Use: * * * * (4/5 stars)

JAVA

Java is the language that i have been doing for quite some time now. So, I can give my very recent experience.

As for the Ease of Understanding , i will say it looks and feels overwhelming at some times but if you give it time to sink it will become quite easy to grasp once you get over the fact that its imense and so does its code looks like.

Ease of Understanding: * * * (3/5 stars)

Learning Curve is kinda Steep, You might not like but JAVA does have everything for you to figure out yourself but it also gives you very deep understanding of what make something work like it does.There is no hidden library or soemthing everything had to be mentioned by user only.

Learning Curve: * * * (3/5 stars)

I recommend using Intellij IDEA as the IDE for Java, as it has additional features with it and shortcuts for long code structures to make your work load small.

Interface: * * * (3/5 stars)

From A User's Point of view it gives almost all control to the User.So it will do exactly what you tell it to do and will not do some parts on its own.So you have Full Control Over the language.

User Experience: * * * (3/5 stars)

With Java too there is no lack of Helping sites, IT being such vast language you meight never find an error which is not faced by someone else as well.
Some of Helping Guide include:

javapoint
GFG
W3school
Kunal Kushwaha's DSA playlist

Helping Guide: * * * * (4/5 stars)

Java will take some time for you to learn and more of practice is needed in it.Its kinda Steep Learning Curve contribute to its Learning time.

Time to Learn: * * * (3/5 stars)

Java too has a lot of Real world Use and Here also i will quote what the internet has to say for it:

" Applications
1) Desktop GUI Applications.
2) Web Applications.
3) Mobile Applications.
4) Enterprise Applications.
5) Scientific Applications.
6) Web Servers & Applications Servers.
7) Embedded Systems.
8) Server Apps In Financial Industry."

Real Word Use: * * * * (4/5 stars)

PYTHON

Python is the very first language i learned so i can tell from my experience that its fairly easy to Understand So,
**

EASE to Understand: * * * * * (5/5 stars) **

like all languages you have to learn it from very base level but for a beginner or a student as per say it does a lot of work for you as you don't have to tell what each variable's type is.(i.e. you don't have to write int a =10; again and again. you can just write a=10; and the python interpreter will take a as int).So,
- LEARNING CURVE: * (1/5 starts)

Coming to the interface of python its interface depend on the type of IDLE that you use for eg-
If you work on the default IDLE of python provided by its installer setup than its failry OK
But if you use JUPITER NOTEBOOK than its interface is very good with features like line number and inout output bok right below the code space
PYCHARM is also a pyhton IDLE that you can download its specifically made for Python and works fairly better than any other IDLE of Python.
- INTERFACE: * * (2/5 stars)

Python's User Experience also depend on the type of Idle that you use like if you use Pycharm than its pretty good User interface with most of the required windows open on the screen with its own terminal and folder space
But the default IDLE of python is Not user friendly as nothing is mentioned on the IDLE to help the absolute beginners while you can use "help()" function to get help about anything from IDLE.
- USER Experience: * * * * (4/5 Stars)

For Python there very few websites to learn from some of which are
Free Code Camp,
Geeks For Geeks (GFG),
Udacity (free Course)
- Learning Websites: * * (2/5 stars)

Python was fairly easy and fast to learn than any other language. Most of My understanding of python comes from Expreimenting with different projects and using differnet libraries to build one or the other thing so i think
- Time to Learn: * * (2/5 stars)

As for what python is used for I would like to quote something coursera said about python

"Python is commonly used for developing websites and software, task automation, data analysis, and data visualization. Since it's relatively easy to learn, Python has been adopted by many non-programmers such as accountants and scientists, for a variety of everyday tasks, like organizing finances"

But with that we must also know that if a beginner try to start from Python as their first language then you will be able to learn fast but there will be a lot of concepts that will never be understood by you cause in order to make python so easy it has hidden a lot of the basic key features from Storage class to Inheritance (OOPS), Method Overloading etc. which are very useful concepts. So you might learn fast but you will also miss alot of basics which may deter your foundation.

- REAL World USE: * * (2/5 stars)

Conclusion

So, The Conclusion is -
If you wanna learn from Absolute Basic then Go for : HTML and CSS
If you don't have any programming language experience : Python
If you have some experience but wanna know it all : JAVA
If you wanna do some big Projects that have C/C++ : C & C++(both)

All in all Give any language you choose some time and in the mean time do some questions and projects you wanna build etc. Soon you will become so Used to it that it will become like a second nature to you.

And at last i wanna add that this is all my personal experiences and opinions.Don't go around changing your path just cause some Guy on internet said so. This Article and all other materials available on online platforms is meant to Help you on your journey.
And please make sure to keep it that way.

How should I start with programming? "Its all just a dabble"

Mridul Tiwari — Sun, 20 Feb 2022 05:57:10 +0000

If anything I want you to conclude from this Blog, It is this quote right here.

"If he was lost for a moment, he would dive straight back into its honey. ~ Laurence Olivier ".

My very blunt Experience

Of how it feels to begin on the journey of learning programming as not just a curriculum but my career.

BEGINNING

I think it was around April or May of the last Year i.e. 2021. When I was studying for my exams of JEE taking a year drop and all my school friends were in some Colleges pursuing their degrees. I was bored enough and I got a hunch to do something to improve my soft skills. And pretty soon I realized I can't do it alone. I need some sort of a partner to do this. All in All, I and my friends started a small podcast to talk about teenage stuff namely WTeenF.

FIRST ENCOUNTER

And we started building websites or I should say they began building website designs for the podcast and this made me interested in learning what in the world are they doing cause as long as I remember I never learnt anything in HTML that let me make such beautiful websites I was flabbergasted by it. So I researched my day off and found it was CSS behind all that and this led me ways to find out where can I learn CSS from. Then I came across GFG(Geeks for Geeks) it gave me all the written stuff to understand what was programmers using in CSS.
And Yes I said programmers as I didn't consider myself anything back then than just a normal kid, not a programmer for sure.
But I am not that much of learning from text kinda guy so I searched two more days off and for various influencers that are doing God's job of making all that text summarized or let's just say explain it to us such as freecodecamp and w3schools are some of my favourite sites and youtube channel as in the case of former one to learn the codebase of CSS. Their teaching method of learning via programming side by side made me fall in love with the subject at hand and I still love any course they put up there. Obviously, I was excited enough then and kept doing these things and started building different projects some of which are already available for everyone to try and just some beginner-friendly stuff.

Experience with Git

That journey kicked off then and I soon learnt Git. And for those of you who don't know what Git is don't worry I will quote what the internet has to say about it:

"Git is software for tracking changes in any set of files, usually used for coordinating work among programmers collaboratively developing source code during software development."

In layman language, it is software to run open-source projects on your device and push your enhancements or fixes to that open source project, which will be available to everybody.
It is by the way was a totally reversed experience from the previous ones for me. As I couldn't learn Git via any youtube video and Just by chance after searching a lot about it and finding things on my recommended page I found a Course on Git on Udacity and here is its link:
https://www.udacity.com/course/version-control-with-git--ud123
And finally learnt what GitHub is.

What life might throw at you

By this time I had already entered my college first semester and then I found a creator namely "KUNAL KUSHWAHA". This man is really doing some amazing work and I started learning java from him. And for the record, it was not an easy hill for me because my laptop couldn't support any languages Compiler or IDLE nor a text editor other than Sublime Text. So all of my java course was practised on an online compiler namely replit.

CONCLUSION

So this was my journey to the point where I am now and what I can say from all this is, Actually the only thing that has been stopping me from doing anything is the fear of not knowing what it is for me in this in my future. But whenever I dive straight into it I have found myself learning and progressing fairly well...

"With that BEST OF LUCK to all of you people in your journey as well".