DEV Community: Mridul Tiwari

Standing up a greenfield EKS cluster — and every mistake we stepped on

Mridul Tiwari — Sat, 05 Sep 2026 04:25:11 +0000

The Argo CD UI showed SyncFailed on karpenter-node-infra before we'd deployed a single application pod. EC2NodeClass and NodePool didn't exist in the cluster — not because the manifests were wrong, but because the CRDs had never been installed. We'd already burned an afternoon on Terraform provider pins and ARM AMI mismatches. This was supposed to be the easy part: GitOps bootstrap, Karpenter online, move on.

It wasn't.

What we were building

We weren't cloning a legacy self-managed K8s 1.21 ASG stack. This was a greenfield non-prod EKS cluster in an existing VPC — private API only, VPN/office CIDRs on the cluster security group, subnets tagged for cluster shared ownership, internal ELB, and karpenter.sh/discovery.

The compute model had two layers on purpose:

Layer 1 — a bootstrap managed node group: on-demand ARM64, AL2023, t4g.xlarge-class instances, min 1 / max 3, tainted CriticalAddonsOnly=true:NoSchedule. It runs Karpenter controller, Argo CD, EKS addons, and optionally Cluster Autoscaler scoped only to that ASG.

Layer 2 — Karpenter NodePool for application workloads, with a separate EC2NodeClass and IAM role.

We chose Karpenter over Cluster Autoscaler for app scaling, EKS Pod Identity (not IRSA) for Karpenter/LBC/CA auth, Terragrunt for backend/provider DRY, terraform-aws-modules/eks ~20.36, and a dedicated GitOps branch. Target Kubernetes version: 1.36 on create — destroy/recreate, not an upgrade ladder on an empty cluster.

That all sounded clean on paper.

Terraform got mean before the cluster existed

The first terraform plan failure was the AWS provider. We'd drifted to provider 6.x; EKS module 20.x still references launch template fields that 6.x removed. Plan blew up with errors about fields that no longer exist. Fix was blunt: pin hashicorp/aws >= 5.40, < 6.0 alongside the module version and document the upper bound next time we bump either one.

Then architecture assumptions collided with reality. We started with spot on the bootstrap pool; requirement clarified to 100% on-demand. Fine — but we'd already baked spot into early config. More annoying: we paired an ARM64 AMI with x86 instance types. AMI architecture has to match the instance family. Obvious in hindsight, expensive when you're staring at nodes that never join correctly.

The managed node group apply failed when AWS still had desired=0 while we set min_size=1. The EKS module ignores desired_size after create — so on first create you have to set min/desired/max together, or manually bump desired ≥ min once before apply. I didn't know that until apply failed.

Disk was another silent lie. Module defaults said disk_size=50, but instances came up around 20 GiB. The module default doesn't always win against the launch template path we were on. We needed explicit block_device_mappings on the bootstrap launch template — 50 GiB gp3 root — to get what we thought we'd already configured.

IAM role name_prefix exceeded 38 characters. Set a short explicit iam_role_name on the node group instead of letting Terraform generate something too long.

We pinned the AL2023 ARM AMI via data source to cluster_version, not "latest surprise AMI." Addons in order: pod-identity-agent before compute, then coredns, vpc-cni, kube-proxy, ebs-csi. Karpenter got its node role, instance profile, access entry, and controller Pod Identity association in Terraform — which almost caused a duplicate IAM role later when GitOps still pointed at IRSA from a reference cluster. More on that.

One dead end I won't repeat: CONTAINER_RUNTIME=docker on AL2023 / K8s 1.32+. Invalid. Dockershim is gone. containerd via nodeadm is the correct path on AL2023.

GitOps: where reference configs go to die

We copied a reference GitOps branch without scrubbing it. Wrong cluster names, IRSA ARNs, Karpenter version mismatched to our K8s version. I almost created a duplicate Karpenter controller IAM role — Terraform already had Pod Identity wired up, but Helm values still carried IRSA annotations from the old cluster.

Argo CD itself went on the bootstrap nodes with tolerations for CriticalAddonsOnly. Helm install first, then a Git deploy key secret — pods don't have SSH_AUTH_SOCK, so SSH agent forwarding wasn't an option.

Bootstrap order we settled on:

Argo CD on bootstrap nodes
Git deploy key
karpenter-crds app (sync wave -1) — CRDs from chart crds/ dir, ServerSideApply
Karpenter controller (skipCrds: true)
karpenter-node-infra — EC2NodeClass + NodePool manifests

That order exists because of a Helm behavior that's easy to miss: helm template skips the chart crds/ directory. The controller chart never installed NodePool/EC2NodeClass CRDs. Argo kept reporting SyncFailed until we split CRDs into their own Application.

We made it worse before we made it better. Two Argo apps owned the same CRDs, and we had argo-cd self-sync enabled. Sync deadlock. CRD ownership has to be exactly one Argo app; the controller must use skipCrds: true. And the Application for argo-cd itself should not auto-sync — that's another path to deadlock.

valueFiles: ../../helm-values/... failed with parent path escape — Argo blocked it, charts rendered empty. Fix: multi-source with ref: values, or keep values under paths Argo allows inside the chart tree.

Bootstrap node sizing bit us too. t4g.large was too small for Argo + Karpenter + all system addons. We bumped to xlarge and stopped fighting eviction loops on the control plane of our control plane.

The Karpenter vs Cluster Autoscaler confusion

Mid-build someone asked why Cluster Autoscaler wasn't adding nodes for pending app pods. Because app capacity is Karpenter's job, not CA's. We scoped CA to the bootstrap ASG only — optional, and only for that tainted pool. Pending app workloads need a healthy NodePool, EC2NodeClass, and Karpenter controller — not CA scale-out.

That question was a signal we'd documented the two-layer model in Terraform but not clearly enough in runbooks.

Version path we shouldn't have taken

On an empty cluster we walked 1.31 → 1.32 → 1.34 → 1.36 instead of targeting 1.36 upfront and destroy/recreating once. Greenfield means you pick the version before first apply, not ladder upgrades on nothing.

The repo-server red herring

After a bootstrap node recycle, Argo showed ComparisonError and repo-server connection refused. I spent time suspecting chart bugs. Stale comparison state. Once bootstrap pods were healthy again, refresh cleared it — not a chart fix, not a Git problem.

What actually worked

When we stopped fighting copied config and pinned the boring stuff:

Terraform: provider < 6.0, explicit gp3 50 GiB on launch template, min/desired/max aligned on first NG create, Pod Identity for Karpenter/LBC/CA, short IAM role names, AMI arch matched to Graviton instance types.
GitOps: skeleton branch scrubbed for cluster name, discovery tag, Pod Identity vs IRSA, Karpenter version for K8s 1.36. Single CRD owner app with ServerSideApply. Controller with skipCrds: true. No auto-sync on argo-cd self-management.
Runtime: xlarge bootstrap nodes, on-demand only, containerd via nodeadm on AL2023.

End state: private control plane, Pod Identity auth, GitOps-ready Karpenter scaling, system controllers isolated from app workloads on a repeatable greenfield pattern.

What I'd do differently next time

I'd treat the pre-apply checklist as blocking, not advisory:

Confirm on-demand vs spot, ARM vs x86, target K8s version before first terraform apply.
Size bootstrap for Argo + Karpenter + all addons — start at xlarge.
Document the EKS module desired_size ignore behavior and set all three sizing knobs together on create.
Pin AWS provider upper bound whenever we pin EKS module version.
GitOps from a skeleton, never a fork — scrub names, discovery tags, Pod Identity vs IRSA, Karpenter/K8s version alignment.
CRD ownership: one Argo app, controller skips CRDs, argo-cd app doesn't auto-sync.
valueFiles: stay inside allowed paths or use multi-source values refs.

The cluster came up fine in the end. The story isn't "EKS is hard" — it's that greenfield gives you freedom to skip legacy baggage and still step on every sharp edge if you copy someone else's YAML without reading it. We learned more from the SyncFailed CRDs and the provider 6.x plan failure than from the architecture diagram. That's probably how it should be.

When Kafka Hits 100% Disk and the Volume Won't Grow

Mridul Tiwari — Sat, 29 Aug 2026 06:44:43 +0000

The alert didn't come from consumer lag. It came from disk — two of three brokers on our production Kafka cluster reporting /data at 100%, with about 20K free on an 850G EBS volume. That's not "we should look at this tomorrow" territory. That's "something is about to stop accepting writes" territory.

We couldn't expand the volume. Not "we'd prefer not to" — the EBS volume literally wasn't modifiable at that moment. So the usual playbook — bump the disk, watch the graph flatten — was off the table. Whatever we did had to reclaim space from inside Kafka itself, without casually deleting segments on prod.

What we were looking at

This is a three-broker cluster in production. The pain was concentrated on a high-volume log topic: 40 partitions, replication factor 2, fed by Kubernetes log shippers. Three consumer groups were attached. On paper, consumption looked fine — lag was around 262, which is nothing you'd page on.

Two brokers were pinned at 100%. The third still had headroom, which told us this wasn't a uniform cluster-wide misconfiguration; it was a retention and placement problem playing out unevenly across replicas.

The topic config, when we pulled it:

retention.ms=172800000        # 48 hours
retention.bytes=-1            # no byte cap
segment.bytes=536870912       # 512MB segments
compression.type=producer

Forty-eight hours of retention with no byte limit on a topic that ingests K8s logs at scale. That's the kind of config you set once during bootstrap and never revisit until /data screams.

Starting with the obvious — and ruling things out

First instinct: consumer lag. Maybe Logstash fell behind and segments aren't getting cleaned up because consumers haven't committed offsets far enough?

We checked. Lag was ~262. Consumers were keeping up. That dead end mattered — it kept us from burning time scaling consumers as the primary fix. (More on Logstash later; it was a follow-up, not the root cause.)

Next: how big is this topic actually?

du -sh /data/kafka-logs/<topic>-*

Roughly 30G per partition. Forty partitions, RF=2 — the math adds up fast. Even with compression (producer), you're holding two days of high-volume log traffic with no ceiling on total bytes. The disk didn't fill because consumers were slow. It filled because retention policy said "keep everything for 48 hours, however big that gets."

I also briefly entertained a storage hack: attach a third EBS volume and use growpart to extend the existing 850G disk. That doesn't work the way I wanted it to — you can't just bolt on block storage and grow an existing filesystem across unrelated volumes. Ruled out before anyone got too excited about it.

At this point the picture was clear: time-based retention with no byte cap on a firehose topic. Segments age out on the clock, not on disk pressure. Low lag doesn't help if you're still obligated to retain 48 hours of data regardless of volume.

Why two brokers and not three

With RF=2 on 40 partitions across three brokers, replica distribution isn't perfectly even. Leader election and partition assignment meant two brokers ended up holding more of the heavy replicas. The third broker had room — which is almost worse, because it makes the incident look like a broker problem when it's really a topic policy problem showing up asymmetrically.

Running diagnostics locally had its own friction. One broker had bootstrap connection quirks when I tried to run kafka-configs from my laptop — enough to slow me down, not enough to change the diagnosis. I didn't capture the exact error string, but it was the kind of thing where you SSH to the broker and run the command there instead of fighting client config for twenty minutes during an incident.

The fix we could actually do in prod

Manual log deletion on a prod cluster is a last resort. You can orphan consumers, confuse leaders, and create a very exciting afternoon for everyone. We needed Kafka's delete policy to do the work — which meant changing retention so old segments become eligible for cleanup.

The prod-safe path:

1. Tighten retention.ms — we targeted 24h instead of 48h. Halving the time window doesn't instantly free 850G, but it changes which segments are eligible for deletion on the next cleanup cycle.

2. Set retention.bytes — this was the important half. Per-broker byte limits give the delete policy something to act on when time alone isn't enough. With -1, Kafka had no reason to drop data early regardless of disk pressure.

Something like:

kafka-configs --bootstrap-server <broker> \
  --entity-type topics --entity-name <log-topic> \
  --alter --add-config retention.ms=86400000,retention.bytes=<per-broker-cap>

The exact byte cap needs to fit your partition count and RF — you're budgeting across replicas, not pretending one broker owns the whole topic. I don't have the final number we landed on in my notes, but the principle was: set a cap that forces segment deletion before /data hits 100% again, with headroom for normal variance.

3. Pause or scale down producers if needed — before flipping retention on a full disk, you sometimes need to stop the inbound firehose briefly. K8s log shippers don't care about your incident; they'll keep writing. Reducing producer pressure before the config change gives the delete policy room to catch up instead of fighting new segments while old ones are still technically retained.

4. Watch segments actually disappear — this isn't instant. Cleanup runs on a schedule. We monitored /data free space and confirmed segments aging past the new retention window were getting removed. No manual rm -rf in the log directories.

Logstash — real problem, wrong root cause

Only 2 of 7 Logstash instances were consuming this topic. That's worth fixing. Under-provisioned consumption can cause lag, can cause operational blind spots, and is generally sloppy.

But lag was 262. The disk was full because retention said "keep 48 hours, unlimited size." Scaling Logstash would improve throughput and resilience; it would not have emptied an 850G volume holding two days of unrestricted log data. We flagged it as follow-up work, not incident mitigation.

That's a distinction I want to keep sharp: healthy-looking lag masked an unhealthy retention policy. I've seen teams chase consumer scaling during disk incidents before. Sometimes that's right. Here it would've been a distraction.

What I'd do differently

I'd have caught this before /data hit 100%. Disk usage per broker and retention config on high-volume topics belong in the same dashboard. Lag alone is a lie of omission when retention.bytes=-1.

If I couldn't resize the volume during the incident, I'd still open the ticket to make it resizable — storage headroom isn't a substitute for correct retention, but running prod Kafka with no expansion path is its own risk.

Next time I'd also verify Logstash consumer coverage during topic onboarding, not during a disk emergency. Two of seven is a config drift problem waiting to happen.

Takeaways from this one:

Low consumer lag does not mean disk is healthy. Check retention.ms and retention.bytes when /data fills on log-heavy topics.
K8s log shippers feeding Kafka can fill brokers even when every consumer group looks caught up — the firehose doesn't care about your lag graph.
When you can't expand the volume, your only safe lever is retention policy. Set byte caps before you need them.

Why our liveness probe timed out even though /healthCheck never touched Redis

Mridul Tiwari — Sat, 22 Aug 2026 04:42:07 +0000

The alert looked like a dependency outage. Readiness and liveness on our Java service were failing in bursts — context deadline exceeded (Client.Timeout exceeded while awaiting headers) — and around the same window, application logs were full of cache and Redis errors. My first instinct was to trace the health handler and ask whether we'd accidentally started pinging Redis on every probe.

We hadn't. The handler was a static 200 OK. When it actually ran, it logged ~0 ms. That mismatch — probes dying while the health code looked fine — is what sent us down the wrong path for a few hours, and what eventually made the real cause obvious.

What we were running

The service is a Spring Boot app behind an ingress and an AWS target group. Kubernetes runs liveness and readiness as HTTP GETs against the same path: /healthCheck. Period 10 seconds, timeoutSeconds: 1. The load balancer runs its own health check on the same path and port every 10 seconds.

That endpoint is deliberately dumb. It does not call Redis, a database, or anything downstream. Passing the probe only proves the HTTP server can accept a request and return a response. We treat dependency health elsewhere; this path is supposed to be cheap.

Under normal traffic, that design works. During an HPA scale event, it did not.

When things started failing

Failures weren't steady. They clustered when the deployment scaled or pods churned — new replicas coming up, old ones draining, traffic shifting. Fresh pods looked fine. Older ones in the middle of the fleet would fail probes, then sometimes recover after load redistributed. Less often we saw connection refused on pods that were starting or terminating; that part felt like noise until we separated it from the timeout failures.

The timeout failures were the scary ones. Kubernetes marked pods not ready or restarted them. From outside, it looked like the service was unhealthy. From inside, when we grepped logs for the health handler itself, we kept finding fast 200s — but only on the requests that actually reached the handler.

That distinction mattered. We weren't looking at a slow health check. We were looking at health checks that sometimes never got a thread in time.

Chasing Redis (and missing the point)

The timeline overlap with Redis/cache errors made this harder than it should have been. Business API paths were logging codec and connectivity issues. Latency on those endpoints spiked. It was reasonable to wonder if /healthCheck had grown a dependency check we didn't know about, or if Spring's health aggregation had been turned on for something we thought was isolated.

We walked the handler and confirmed: static response, no downstream calls. Optional jar/source review would have been redundant — runtime logs already showed 0 ms when the handler executed. So the cache errors were real, but they weren't in the probe path. They were on API traffic that shared infrastructure with the probe path.

That was our first dead end as a root cause, though not as a contributing factor. Fixing Redis wouldn't fix probe timeouts by making /healthCheck faster — there was nothing left to speed up in the handler itself. But Redis slowness could still make everything worse if it blocked the threads that were supposed to serve probes.

Thread pool saturation

Tomcat serves /healthCheck and every API route from the same worker thread pool. We had the pool capped around 200 threads. Under load, especially during scale events when traffic hadn't yet spread evenly, worker threads filled up with slow API work. Redis trouble on those paths added latency and kept threads busy longer.

Probes don't wait politely. Kubelet hits liveness and readiness every 10 seconds with a 1-second timeout. The target group adds another /healthCheck every 10 seconds per instance. Those requests land in the same queue as everything else. If all 200 threads are tied up waiting on cache calls or slow downstream logic, a probe sits in the accept queue until the client gives up waiting for headers.

Hence the error text: not a connection failure to Redis, not a 500 from the health handler — Client.Timeout exceeded while awaiting headers. The TCP connection might succeed; the response never starts in time.

Once we framed it that way, the intermittent pattern made sense. New pods after scale-up had empty pools and a smaller share of traffic until the service settled. They passed probes immediately. Older pods holding more connections and hotter thread utilization failed first. Pods we terminated showed connection refused because the process was already gone — expected, and a different failure mode from starvation.

We didn't capture exact queue depths or a precise latency number in the notes from that shift. What we had was correlation: HPA events, probe failures, high thread occupancy, and cache errors on API paths — not on the health handler when it ran.

Why "raise the timeout" wasn't the whole story

Raising timeoutSeconds above 1 would give probes more time to wait for a free thread. That's a valid mitigation and probably stops the bleeding on liveness kills. But it treats the symptom. A 3-second probe timeout on a handler that executes in 0 ms is a signal that something else is wrong with capacity or isolation.

We weighed a few changes together:

Increase probe timeout. Low risk, quick to deploy. Buys headroom when the pool is briefly saturated. Doesn't fix API slowness or reduce thread contention.

Separate probe traffic from application traffic. A dedicated connector or a minimal probe path served on a different thread pool (or even a sidecar/admin port) means kubelet and the load balancer aren't competing with /api/... for the same 200 workers. This is more work but addresses the architectural coupling.

Fix the cache codec errors on API paths. Indirect but real. Errors that slow business requests keep threads pinned longer, which increases the odds that a probe waits past 1 second. The health endpoint wasn't broken; the pool was over-subscribed because of problems elsewhere.

Stop triple-stacking the same path. Liveness, readiness, and the target group all hammer /healthCheck on the same port. Each alone is light; combined with API load on one pool, they're another source of contention. Readiness might warrant a slightly richer check; liveness should stay as dumb as possible — but not necessarily on the same threads as heavy traffic.

We didn't treat "make /healthCheck ping Redis so failures are honest" as a fix. That would have made probes fail for the wrong reason and conflated "app process up" with "cache reachable," which is exactly what we'd almost done when we misread the Redis log correlation.

What actually changed

The immediate config change was increasing probe timeoutSeconds so transient saturation during scale events didn't restart pods. In parallel — because the evidence pointed at thread occupancy, not handler logic — we prioritized the cache errors on API routes that were holding threads and traced whether Tomcat's max threads and accept queue were appropriate for peak + probe overhead.

Longer term, the design direction was clear: isolate probe handling from the main servlet traffic, and avoid using one static endpoint for liveness, readiness, and external health checks without explicit headroom in the pool.

After deploy, the pattern we watched for was the same one that fooled us initially: new pods staying green while the fleet scales. That's not proof the problem is gone. It only means those instances haven't hit saturation yet. Validation was watching older replicas through the next HPA event and confirming probes stayed green while API latency and thread utilization stayed within bounds — and that when cache errors appeared, they didn't precede another wave of probe timeouts.

What I'd check first next time

If the health handler is trivial and logs show 0 ms when it runs, but Kubernetes reports probe timeouts, I wouldn't start with dependency connectivity. I'd look at thread pool utilization, probe timeout vs period, and who else hits that path — kubelet twice (liveness + readiness), plus the load balancer — on the same connector as production API traffic.

Probe timeout is not the same as dependency failure. A 1-second timeout on a 0 ms handler is telling you the request didn't get served in time, not that the handler logic failed. connection refused on shutting-down pods is a separate signal from awaiting headers on live ones.

The lesson from this incident isn't "monitor more." It's narrower: a healthy /healthCheck implementation can still fail probes when the HTTP server is saturated, and log lines about Redis on other endpoints can send you on a long detour if you assume every failure mode must flow through the health handler itself.

One Helm template change broke every Argo app — fixing nil `ingressInt` safely

Mridul Tiwari — Sun, 16 Aug 2026 13:30:07 +0000

The Slack thread started the way these things usually do: three people pasting the same Argo CD error within a few minutes of each other. Sync failed. Not one app — several. All on the same shared EKS platform, all pulling from the same application Helm chart we'd been using for ages.

I opened Argo first on the app I knew had deployed recently, then on two others that hadn't changed in weeks. Same failure. That ruled out my first instinct — a bad image tag or a typo in one team's values file. Whatever broke was upstream of individual app config.

A shared chart, a feature merge, and a new template

Our platform pattern is familiar if you've run Kubernetes at any scale: dozens of services, one common chart, per-app and per-env values overlays. Argo CD watches the repo and syncs each Application to its cluster. Most apps define ingress for their primary routing. A subset also define ingressExt for a public-facing ALB. Nobody had needed an internal-only ingress type until recently.

A feature PR had landed that added support for internal ALBs — ingressInt — plus matching templates ingress-int.yaml and ingress-ext.yaml in the shared chart. The author needed internal ingress for one service, so they added an ingressInt block to that app's values file and merged. CI was green. The chart rendered fine in whatever path the PR exercised.

Then Argo tried to sync everyone else.

Narrowing it down

Argo's UI isn't always generous with Helm errors, but the sync logs usually cough up the render failure. The message we kept seeing was the Helm classic:

Error: template: .../ingress-int.yaml:...: executing "..." at <.Values.ingressInt.enabled>: nil pointer evaluating interface {}.enabled

That line tells you almost everything. The template touched .Values.ingressInt.enabled. For the app that got the new values block, ingressInt exists and has an enabled field. For every other consumer of the chart, ingressInt was never defined. In Go-template land, .Values.ingressInt is nil, and dereferencing .enabled on nil is a hard stop — Helm never gets as far as creating or updating resources.

I pulled the diff from the feature merge. Two new template files, and at the top of ingress-int.yaml:

{{- if .Values.ingressInt.enabled -}}

No default in the chart's root values.yaml. No guard checking whether ingressInt exists before reading .enabled. The template runs for every app on every helm template / helm upgrade — the if only skips the body of the template; the condition itself still evaluates .Values.ingressInt.enabled first.

That matched the blast radius. Apps with the new key: fine. Apps with only ingress and maybe ingressExt: broken. Apps that hadn't been touched in months: broken. Argo doesn't isolate chart render failures per consumer when they share a chart version — one bad optional key poisons the whole sync surface.

We briefly wondered whether we should patch values files app by app. There are a lot of them — dev, staging, prod overlays, team forks. Adding ingressInt: { enabled: false } everywhere would work, but it's the kind of churn that hides in a giant PR and still leaves you one missing file away from the next outage. The regression had already been introduced by adding ingressInt to a single app's values while the template assumed every app would have the key. Repeating that pattern at scale felt wrong.

The fix: chart defaults and nil-safe guards

We wanted a chart-only fix — no hunting through dozens of per-env values files, no coordination with every team to add a stub block. Two changes, applied together.

First, add defaults in the chart's values.yaml so ingressInt and ingressExt always exist, even when an app never mentions them:

ingressInt:
  enabled: false
  hosts: []
  annotations: {}
  tls: []

ingressExt:
  enabled: false
  hosts: []
  annotations: {}
  tls: []

Apps that need internal or external ALBs keep overriding these in their own values. Everyone else inherits disabled defaults and never thinks about the keys again.

Second, harden the templates so a missing or partial values merge can't take down render again. The condition in ingress-int.yaml became:

{{- if and .Values.ingressInt .Values.ingressInt.enabled -}}

Same pattern in ingress-ext.yaml for consistency — same class of bug, same class of fix. The and short-circuits: if ingressInt is nil or absent after a bad merge, the template skips cleanly instead of panicking.

We reverted the one-off ingressInt block from the single app's values file. With chart defaults in place, that app could set ingressInt.enabled: true and its hosts/annotations when they actually needed internal ingress, without carrying a special snowflake block that implied the key was app-local rather than chart-wide.

After merging the chart fix, Argo syncs recovered across the board without touching individual Application manifests or env-specific values. The app that originally needed internal ingress still opts in through overrides; the rest of the fleet never knew anything happened except that sync went red and then green again.

Why this works (and what we'd do differently)

Helm merges values layers: chart defaults, then parent charts, then user-supplied -f files and --set. If the chart doesn't define a key, and no values file defines it either, .Values.ingressInt is nil in the template context. Optional features can't assume every consumer opted in — especially on a shared chart where most apps will never use the feature.

Defaults alone would probably have fixed this specific incident, because ingressInt.enabled would resolve to false everywhere. We kept the nil guard anyway. Defaults can be overridden away, subcharts can omit keys, and someone will eventually add a third ingress variant the same way. The guard costs one line and buys immunity to the exact error we saw.

The process miss was visible in the PR itself: the feature was validated on one app that had ingressInt in values, while the template executed for all apps on the chart. A render check that only runs helm template with the happy-path values file wouldn't catch it. Next time, for optional blocks on a shared chart, I'd want CI to render against a minimal values fixture — just enough to deploy a generic app, with no ingressInt — alongside the feature app's overlay. If minimal render passes, you've got both defaults and guards covered, or at least you'll see the nil pointer before merge.

One template change broke every Argo app on the platform. Fixing it took chart-level defaults and a nil-safe condition, not a values-file scavenger hunt across the org. That's the pattern I'd reuse: when you add optional .Values blocks to a shared Helm chart, ship chart defaults and nil-safe template guards — not every consumer will define the key, and Argo will sync all of them on the same chart version whether you planned for that or not.

When TLS 1.3-only broke everything behind Akamai

Mridul Tiwari — Sun, 05 Jul 2026 06:59:46 +0000

The alert wasn't subtle. Requests just stopped hitting our backend nginx box. Internal traffic from our own domains kept flowing, but anything coming in through Akamai-hosted domains — routed via the External LB — went dead. Several frontends all depended on that backend, so they went down together.

We run nginx in front of a shared backend that multiple application frontends talk to. Under normal conditions, Akamai terminates at the edge, traffic crosses the External LB, and nginx forwards to the app tier. Internal paths bypass that edge path entirely, which is why the split behavior was our first real clue: the nginx box wasn't universally broken. Something in the Akamai → External LB path was failing.

Chasing the wrong certificate

We started from the frontend boxes and ran curls against the backend path. The failures looked like they were getting cut off at Akamai — not nginx, not the app. That pointed outward, not inward.

We opened a ticket with Akamai. Their read was an SSL certificate problem. That didn't sit right. We hadn't rotated certs, renewed anything, or touched the cert chain on our side. If nothing on the certificate had changed, why would Akamai suddenly start rejecting handshakes?

We kept digging anyway. The Akamai angle was real — requests really were dying at their edge — but "certificate issue" felt like a symptom label, not the mechanism.

The policy mismatch

While we were going back and forth, someone pulled up the External LB TLS security policy. That was the turn.

The LB had been set to TLS 1.3 only. At Akamai, the property was on their default supported policy — the one that negotiates down through 1.2 → 1.1 → 1.0, not a strict 1.3-only handshake.

TLS 1.3-only on the load balancer means the server side of the handshake insists on 1.3. Akamai's side, configured for broader compatibility, wasn't going to meet it there. The handshake never completed. From the outside it looked like Akamai was cutting requests off — and in a sense it was, because the TLS negotiation failed before anything useful got through. Easy to misread as "SSL cert broken" when the real failure mode is protocol policy mismatch.

We hadn't changed certificates. We had changed (or inherited) a TLS policy that didn't match what Akamai could speak on that property.
What fixed it

We rolled the External LB TLS security policy back to one that supports both TLS 1.2 and TLS 1.3. After that change, traffic from Akamai-hosted domains started reaching nginx again, and the dependent frontends came back.

No nginx reload drama, no cert redeploy — just aligning the LB's minimum/maximum protocol behavior with what the Akamai property actually negotiates.

What I'm keeping in my head

The split between internal (working) and Akamai (broken) saved us from burning time on nginx config. But "Akamai says certificate" plus "we didn't touch certs" should have pushed us to TLS policy sooner.

The operational bit I care about: any time we change TLS policy on the External LB, we need to coordinate with whoever owns the Akamai property config. Edge and origin have to agree on what handshake they're willing to do. A 1.3-only LB in front of an Akamai property still on broad compatibility isn't a subtle drift — it's a hard cutoff.

I'm marking this one complete, but the runbook note is the part that matters for the next person: check LB TLS policy against Akamai's supported cipher/protocol set before you assume the cert is bad.

What is the BEST Programming Language to Start with?

Mridul Tiwari — Sun, 27 Feb 2022 15:41:27 +0000

One of the most popular question that every student asks before they begin their journey of programming is which programming language to choose from OR should i say

WHAT IS THE BEST LANGUAGE TO START WITH ?

A very cliche answer to this curiousity ridden question is "Every Language is Best, its just a matter of your personal choice"
This would have been a good reply to someone who has learn few languages and is on the journey of making one or more, his best. But this isn't a good answer for someone who didn't even started their journey and are looking for something that is worth their time
So, Here is a collection of all information you need to CHOOSE which language to start from.

WHAT ARE THE AVAILABLE OPTIONS?

If you are like me than you must have researched your days off to find out what are the different programming languages one can choose from ?
And you might have come across a few like-

C++
PYTHON
C
JAVA
R
REACT
JAVASCRIPT,etc.

If you have heard the names of any one of those than good you might have some knowledge of what we are going to discuss here
But if you are hearing this for the first time than don't worry we are going to discuss some of them in this article only.

SOME SUBSTITUTES

Now there are two more languages that are not considered programming languages but more of sturcture and styling languages i.e.
HTML AND CSS.
They are the two languages that i recommend you should at least learn for a while if you have never programmed anything because these are very basic languages which can give you a jist of what the other languages might have in store for you and these languages will give you perfect mindset to deal and learn programming languages.

Shortening the List.

Out of the above mentioned 7 programming languages there are 4 or lets say 3 languages most people are confused about that includes

1. C++

2. Python

3. JAVA

4. C

And Now we are going to judge these languages based on some criterias to clear the confusion.

What are the Criterias of Judging?

EASE to Understand
Learning Curve
Interface
User Experience
Any help website of youtube channels
How much time it would take to learn?
What are its real world use?

C

C is one of those language that i think is taught to first years in most of the colleges.And same was happened to me.Being a person who only knew Python by then it was not at all jarring for me to learn this language.
So, the best advice i can give for this language is

"You have to be open Minded and let things sink in first"

-Ease to Understand :* * * * (4/5 stars)

As i mentioned earlier to learn C,One just have to let it sink in you mind for a little bit.By that i mean , Give it some time to let yourself understand its syntaxes and rules ,etc.
In the mean time do some basic questions like Pattern Printing ,searching ,sorting and level up steadily.
-Learning Curve: * * (2/5 stars)

Interface wise i would always recommend use an online compiler for C or a mobile app cause the other IDLE i tried for C for turbo C++ which was terrible but i give it benefit of doubt that it was meant to be used as programming IDLE for developers but for habitual Coding stuff
-Interface: * * * (3/5 stars)

C is a great User Experience being it so easy to understand and learn
-User Experience : * * * *(4/5 stars)

Helping guide for C are very fabulous and there are a lot of it:
Tutorial Point,
Geeks for Geeks,
Free Code Camp,
W3school

-Help Guide: * * * * (4/5 stars)

It would take aprroximately a week or two for you to learn it if you go about only doing it for those weeks.Meaning its not much Time counsuming.
-Time To Learn: * * (2/5 stars)

For the Real world use of C i would highlight what the internet has to say for it:

"Real-World Applications of C. Use of the C programming language is not limited to the development of operating systems and applications. It is also used in GUI development, IDE development, etc."

-Real World Experience: * * * (3/5 stars)

C++

My very first experience with C++ came about a couple of days earlier and i am not kidding once you do C its insanely easy to do C++. There are only syntax changes for the most parts.You will be done with all basic stuff in a day.

Ease of Understanding: * * * (3/5 stars)

As of the Learning Curve its exactly like C so if you did C earlier than this will be a piece of cake but even if you take it as your first language than also it would take you anylonger to understand than if you had take C.

Learning Curve: * * (2/5 stars)

I will deduct a star for interface though cause although i gave C benefit of doubt that Turbo C++ was not meant to be for C.It was catered for C++ and it was not good Experience .I would recommend VS code for sure, here.

Interface: * * (2/5 stars)

If you Use VS code as your text editor than the user experience with this language is very good.Some things are to be nit-picked but still all in all a satisfying user experience.

User Experience: * * * (3/5 stars)

There have been a lot of Helping Media for C++ mainly because most students use it for CP(competitive Programming).So, you will not find lack of resources for C++
some of which include:
W3school
Tutorialpoint
StackOverflow
GFG
freecodecamp

Helping Guide: * * * * (4/5 stars)

As said earlier it won't take any more time than C did, if you are a beginner and this is your first language,But if this comes to not be your first language than it would take even lesser time for you to learn.

Time to Learn: * * (2/5 stars)

Its Real World Use is insanely big.
And i will quote what the internet has to say about it.

"
C++ Application in Real World
GUI Applications.
Operating Systems.
Web Browsers.
Database Management System.
Libraries.
Cloud Computing and Distributed Applications."

Real Word Use: * * * * (4/5 stars)

JAVA

Java is the language that i have been doing for quite some time now. So, I can give my very recent experience.

As for the Ease of Understanding , i will say it looks and feels overwhelming at some times but if you give it time to sink it will become quite easy to grasp once you get over the fact that its imense and so does its code looks like.

Ease of Understanding: * * * (3/5 stars)

Learning Curve is kinda Steep, You might not like but JAVA does have everything for you to figure out yourself but it also gives you very deep understanding of what make something work like it does.There is no hidden library or soemthing everything had to be mentioned by user only.

Learning Curve: * * * (3/5 stars)

I recommend using Intellij IDEA as the IDE for Java, as it has additional features with it and shortcuts for long code structures to make your work load small.

Interface: * * * (3/5 stars)

From A User's Point of view it gives almost all control to the User.So it will do exactly what you tell it to do and will not do some parts on its own.So you have Full Control Over the language.

User Experience: * * * (3/5 stars)

With Java too there is no lack of Helping sites, IT being such vast language you meight never find an error which is not faced by someone else as well.
Some of Helping Guide include:

javapoint
GFG
W3school
Kunal Kushwaha's DSA playlist

Helping Guide: * * * * (4/5 stars)

Java will take some time for you to learn and more of practice is needed in it.Its kinda Steep Learning Curve contribute to its Learning time.

Time to Learn: * * * (3/5 stars)

Java too has a lot of Real world Use and Here also i will quote what the internet has to say for it:

" Applications
1) Desktop GUI Applications.
2) Web Applications.
3) Mobile Applications.
4) Enterprise Applications.
5) Scientific Applications.
6) Web Servers & Applications Servers.
7) Embedded Systems.
8) Server Apps In Financial Industry."

Real Word Use: * * * * (4/5 stars)

PYTHON

Python is the very first language i learned so i can tell from my experience that its fairly easy to Understand So,
**

EASE to Understand: * * * * * (5/5 stars) **

like all languages you have to learn it from very base level but for a beginner or a student as per say it does a lot of work for you as you don't have to tell what each variable's type is.(i.e. you don't have to write int a =10; again and again. you can just write a=10; and the python interpreter will take a as int).So,
- LEARNING CURVE: * (1/5 starts)

Coming to the interface of python its interface depend on the type of IDLE that you use for eg-
If you work on the default IDLE of python provided by its installer setup than its failry OK
But if you use JUPITER NOTEBOOK than its interface is very good with features like line number and inout output bok right below the code space
PYCHARM is also a pyhton IDLE that you can download its specifically made for Python and works fairly better than any other IDLE of Python.
- INTERFACE: * * (2/5 stars)

Python's User Experience also depend on the type of Idle that you use like if you use Pycharm than its pretty good User interface with most of the required windows open on the screen with its own terminal and folder space
But the default IDLE of python is Not user friendly as nothing is mentioned on the IDLE to help the absolute beginners while you can use "help()" function to get help about anything from IDLE.
- USER Experience: * * * * (4/5 Stars)

For Python there very few websites to learn from some of which are
Free Code Camp,
Geeks For Geeks (GFG),
Udacity (free Course)
- Learning Websites: * * (2/5 stars)

Python was fairly easy and fast to learn than any other language. Most of My understanding of python comes from Expreimenting with different projects and using differnet libraries to build one or the other thing so i think
- Time to Learn: * * (2/5 stars)

As for what python is used for I would like to quote something coursera said about python

"Python is commonly used for developing websites and software, task automation, data analysis, and data visualization. Since it's relatively easy to learn, Python has been adopted by many non-programmers such as accountants and scientists, for a variety of everyday tasks, like organizing finances"

But with that we must also know that if a beginner try to start from Python as their first language then you will be able to learn fast but there will be a lot of concepts that will never be understood by you cause in order to make python so easy it has hidden a lot of the basic key features from Storage class to Inheritance (OOPS), Method Overloading etc. which are very useful concepts. So you might learn fast but you will also miss alot of basics which may deter your foundation.

- REAL World USE: * * (2/5 stars)

Conclusion

So, The Conclusion is -
If you wanna learn from Absolute Basic then Go for : HTML and CSS
If you don't have any programming language experience : Python
If you have some experience but wanna know it all : JAVA
If you wanna do some big Projects that have C/C++ : C & C++(both)

All in all Give any language you choose some time and in the mean time do some questions and projects you wanna build etc. Soon you will become so Used to it that it will become like a second nature to you.

And at last i wanna add that this is all my personal experiences and opinions.Don't go around changing your path just cause some Guy on internet said so. This Article and all other materials available on online platforms is meant to Help you on your journey.
And please make sure to keep it that way.

How should I start with programming? "Its all just a dabble"

Mridul Tiwari — Sun, 20 Feb 2022 05:57:10 +0000

If anything I want you to conclude from this Blog, It is this quote right here.

"If he was lost for a moment, he would dive straight back into its honey. ~ Laurence Olivier ".

My very blunt Experience

Of how it feels to begin on the journey of learning programming as not just a curriculum but my career.

BEGINNING

I think it was around April or May of the last Year i.e. 2021. When I was studying for my exams of JEE taking a year drop and all my school friends were in some Colleges pursuing their degrees. I was bored enough and I got a hunch to do something to improve my soft skills. And pretty soon I realized I can't do it alone. I need some sort of a partner to do this. All in All, I and my friends started a small podcast to talk about teenage stuff namely WTeenF.

FIRST ENCOUNTER

And we started building websites or I should say they began building website designs for the podcast and this made me interested in learning what in the world are they doing cause as long as I remember I never learnt anything in HTML that let me make such beautiful websites I was flabbergasted by it. So I researched my day off and found it was CSS behind all that and this led me ways to find out where can I learn CSS from. Then I came across GFG(Geeks for Geeks) it gave me all the written stuff to understand what was programmers using in CSS.
And Yes I said programmers as I didn't consider myself anything back then than just a normal kid, not a programmer for sure.
But I am not that much of learning from text kinda guy so I searched two more days off and for various influencers that are doing God's job of making all that text summarized or let's just say explain it to us such as freecodecamp and w3schools are some of my favourite sites and youtube channel as in the case of former one to learn the codebase of CSS. Their teaching method of learning via programming side by side made me fall in love with the subject at hand and I still love any course they put up there. Obviously, I was excited enough then and kept doing these things and started building different projects some of which are already available for everyone to try and just some beginner-friendly stuff.

Experience with Git

That journey kicked off then and I soon learnt Git. And for those of you who don't know what Git is don't worry I will quote what the internet has to say about it:

"Git is software for tracking changes in any set of files, usually used for coordinating work among programmers collaboratively developing source code during software development."

In layman language, it is software to run open-source projects on your device and push your enhancements or fixes to that open source project, which will be available to everybody.
It is by the way was a totally reversed experience from the previous ones for me. As I couldn't learn Git via any youtube video and Just by chance after searching a lot about it and finding things on my recommended page I found a Course on Git on Udacity and here is its link:
https://www.udacity.com/course/version-control-with-git--ud123
And finally learnt what GitHub is.

What life might throw at you

By this time I had already entered my college first semester and then I found a creator namely "KUNAL KUSHWAHA". This man is really doing some amazing work and I started learning java from him. And for the record, it was not an easy hill for me because my laptop couldn't support any languages Compiler or IDLE nor a text editor other than Sublime Text. So all of my java course was practised on an online compiler namely replit.

CONCLUSION

So this was my journey to the point where I am now and what I can say from all this is, Actually the only thing that has been stopping me from doing anything is the fear of not knowing what it is for me in this in my future. But whenever I dive straight into it I have found myself learning and progressing fairly well...

"With that BEST OF LUCK to all of you people in your journey as well".