DEV Community: Abhimanyu Bajaj

October: Kubernetes memory semantics and Collector edge fixes

Abhimanyu Bajaj — Sun, 01 Nov 2026 10:32:40 +0000

In October I had 10 PRs merged across four projects. The common thread was practical observability work: clearer Kubernetes memory conventions, Helm chart behavior that is easier to reason about, less flaky Collector tests, and maintainership for components that need active review.

Most of my week goes into one question: why does something that looks correctly configured not work in a real cluster? These PRs came from the same place.

TL;DR

Fixed flaky Kubernetes inventory cache-sync tests.

Removed a Java 11+ startup failure in a DataPower monitoring extension.

Let Operator chart users override only the Target Allocator repository.

Added Kubernetes memory utilization conventions.

Clarified Kubernetes working set descriptions.

Added Pod-level memory limit and request conventions.

Joined Flink Metrics Receiver code ownership.

Fixed broad Flink operator metric matching.

Documented Operator chart certificate drift with Argo CD.

Restored Azure Data Explorer Exporter active ownership and beta status.

Kubernetes memory conventions became more precise

Pod memory limits and requests now have desired and current forms

The Kubernetes semantic conventions had Pod-level CPU limit and request metrics, but not the same shape for memory. I added k8s.pod.memory.limit.{desired,current} and k8s.pod.memory.request.{desired,current} so the model can represent values from spec.resources and status.resources.

This shows up in OpenTelemetry semantic conventions, which define standards for consistent telemetry. It closes issue #4133, and there is no release listed for this PR yet.

Once implemented downstream, dashboards and alerts can distinguish desired Pod memory settings from current Pod memory status.

open-telemetry/semantic-conventions PR 4158

Working set descriptions now say what Linux includes

The working set descriptions for k8s.pod.memory.working_set, k8s.node.memory.working_set, and container.memory.working_set were clarified. On Linux, working set excludes inactive file-backed memory, while active file-backed memory remains included.

The generated semantic convention documentation now also says that kubelet uses Pod working set when ranking Pods for eviction, and node working set when determining memory availability for the memory.available eviction signal. There is no release listed for this PR yet.

The next effect is clearer documentation for comparing usage, working set, and eviction behavior.

open-telemetry/semantic-conventions PR 4184

Memory utilization now uses working set as the numerator

I added four Kubernetes memory utilization metrics: Pod and container limit utilization, and Pod and container request utilization. These use memory working set as the numerator, not total memory usage.

The PR explains the reason: total usage includes reclaimable page cache, which can make a workload look close to its limit even when part of that memory is reclaimable. I validated the behavior against a kubelet Summary API using kind with Kubernetes v1.37.0. There is no release listed for this PR yet.

Once implemented downstream, memory utilization can be shown against limits and requests with the numerator semantics documented in the convention and migration guide.

open-telemetry/semantic-conventions PR 4219

Operator Helm chart configuration and docs got clearer

Private registry users can override only the Target Allocator repository

The OpenTelemetry Operator Helm chart did not let users override only manager.targetAllocatorImage.repository while keeping the chart’s default tag. I added explicit defaults for the Target Allocator repository and tag, plus validation that repository and tag are either both set or both empty.

This is for the OpenTelemetry Helm Charts project. It closes issue #1866, where 2 people asked for it, and the chart version was bumped to 0.125.0. There is no release listed for this PR yet.

The practical effect is simpler values for private registry or air-gapped installs that redirect the Target Allocator image. The upgrade guide also documents the configuration precedence change.

open-telemetry/opentelemetry-helm-charts PR 2456

Argo CD users get documentation for auto-generated webhook certs

When admissionWebhooks.autoGenerateCert is used, the Operator chart can generate a new CA on each render. With Argo CD, or other tools that use helm template, Helm cannot use lookup during rendering to reuse the existing Secret, so the webhook caBundle, certificate Secret, and running operator may become inconsistent.

I documented this in the README’s TLS certificate section, including the possible x509: certificate signed by unknown authority failure. I reproduced the behavior with Argo CD v3.5.5, chart 0.126.0, and Kubernetes v1.37.0. This shipped in opentelemetry-operator-0.127.1.

The change is documentation: Argo CD users have a clearer reason to use cert-manager or supply certificate material independently of render-time generation.

open-telemetry/opentelemetry-helm-charts PR 2473

Collector contrib tests, receivers, and ownership moved forward

Kubernetes inventory cache-sync tests are deterministic now

Two internal/k8sinventory/informer tests were flaky because they depended on timing around fake client cache sync. I changed the tests to block the initial Pod List call explicitly, so the failure paths are deterministic without changing production behavior.

This is internal test code in the OpenTelemetry Collector contrib repository. It closes issue #50433, which had 24 comments about failed tests on main. There is no release listed for this PR yet.

The effect is less noise from racey tests when contributors work on Kubernetes inventory code.

open-telemetry/opentelemetry-collector-contrib PR 51657

The Flink Metrics Receiver has another code owner

The Flink Metrics Receiver needed additional maintainers. I added myself as a code owner after reviewing receiver changes, running unit tests, and validating changes against a local Apache Flink 2.3.0 deployment in Docker.

This shows up in CODEOWNERS and generated README metadata in the OpenTelemetry Collector contrib repository. There is no release listed for this PR yet, and it does not change runtime behavior.

The practical change is maintenance coverage for reviews, triage, bug fixes, and receiver upkeep.

open-telemetry/opentelemetry-collector-contrib PR 52019

Flink operator metrics now match exact suffixes

The Flink Metrics Receiver used strings.Contains for operator-level metric names. That could also match longer names like numRecordsInPerSecond when looking for numRecordsIn.

I changed the four affected operator metric cases to exact suffix matching: numRecordsIn, numRecordsOut, numLateRecordsDropped, and currentOutputWatermark. Existing metric names, attributes, and configuration are preserved. There is no release listed for this PR yet.

The result is more precise matching for those receiver metrics.

open-telemetry/opentelemetry-collector-contrib PR 52406

Azure Data Explorer Exporter has active ownership again

The Azure Data Explorer Exporter had been marked unmaintained after its previous active code owner moved to emeritus. I added myself as an active code owner and restored the previous beta stability designation for traces, metrics, and logs.

This is visible in generated README, CODEOWNERS, component labels, status metadata, and the changelog. It closes issue #52282, and there is no release listed for this PR yet.

The practical effect is that the component again has an active maintainer path for review and triage.

open-telemetry/opentelemetry-collector-contrib PR 52500

A Machine Agent monitoring extension starts again on Java 11+

The DataPower extension no longer depends on JDK SAAJ

On Machine Agents running a Java 11+ JRE, the DataPower monitoring extension could fail to instantiate because the JDK no longer ships the SAAJ implementation removed by JEP 320. The extension only used SAAJ to build a fixed SOAP request envelope.

I removed the javax.xml.soap dependency and now build the envelope directly as a string, with XML attribute escaping for domain and status-class values. The PR bumps the extension version to 2.0.2, and there is no release link listed for this PR yet.

The practical change is that Machine Agent users on Java 8 through 21 have a startup path that does not depend on a removed JDK implementation.

Appdynamics/datapower-monitoring-extension PR 13

Still in review

Sources

Drafted with AI assistance from the PRs themselves and reviewed by me.

EKS App Slow, CPU Fine? Scale on Splunk Latency with KEDA

Abhimanyu Bajaj — Tue, 27 Oct 2026 11:20:45 +0000

This lab fixes the autoscaling gap CPU HPA misses: users are waiting, but CPU is barely moving.

You will reproduce the failure on an I/O-bound Python service, then use Splunk Observability, the OpenTelemetry Collector, and KEDA to scale on request latency instead. In my run, the app sat at 1 replica with 3049.2 ms peak latency and 2.8% peak CPU, then scaled from 1 to 5 ready replicas in 48 seconds once KEDA read the Splunk latency metric.

I work a lot with the OpenTelemetry Collector in Kubernetes, and I reproduce problems like this in a throwaway AWS account before I write about them.

Here is how it works, and how to build it yourself.

TL;DR

What you get: an EKS Deployment whose replica count follows request latency, not CPU.

The trick: the app exposes app_request_latency_ms, the Splunk OpenTelemetry Collector sends it to Splunk Observability Cloud, and KEDA feeds that value into an HPA.

The proof: before, replicas stayed at 1; after, they went 1 -> 5 under load and back to 1 after load stopped.

Time and cost: the README estimates about 40 minutes and well under 1 USD; my recorded reproduced-and-fixed run took 31.3 minutes.

Why CPU autoscaling misses this slow EKS service

The app in this lab is intentionally I/O-bound. Each pod has 4 backend slots. Each request holds a slot for 200 ms using sleep, not CPU work. When 60 clients arrive, requests queue inside the service. Users feel latency, but the CPU graph barely moves.

Here is the before state from reproduce.sh:

== Autoscaling objects in the namespace:
No resources found in demo namespace.
NAME          READY   UP-TO-DATE   AVAILABLE   AGE
latency-app   1/1     1            1           7s
...
== Summary: peak latency 3049.2 ms, peak CPU 2.8% of one core, replicas still 1
   Users feel multi-second latency, CPU stays low, nothing scales.

A normal CPU HPA has no useful signal here. The better signal is the one users already feel: request latency.

The trick: let Splunk become the external metric KEDA reads

KEDA makes the scaling decision, but Splunk Observability Cloud is the metric source.

The path is short: the Python app exposes app_request_latency_ms on /metrics, the pod has Prometheus scrape annotations, the Splunk OpenTelemetry Collector sends the metric to Splunk, and KEDA runs a SignalFlow query through the splunk-observability scaler.

This is the key part of the ScaledObject from the lab:

triggers:
  - type: splunk-observability
    metricType: Value
    metadata:
      query: "data('app_request_latency_ms', filter=filter('k8s.cluster.name', '${LAB_NAME}')).max().publish()"
      duration: "40"
      queryAggregator: "max"
      targetValue: "400"
      activationTargetValue: "0"
    authenticationRef:
      name: splunk-observability-auth

fix.sh expands ${LAB_NAME} before applying the manifest. If you paste this into a YAML file yourself, replace it with your cluster name. For your own service, change the metric name, cluster filter, and target value.

Build it yourself in 5 steps

Prerequisites: an AWS account, a Splunk Observability Cloud org, an ingest token, a read-only API token, the matching Splunk realm, and the CLIs used by the scripts: aws, kubectl, helm, jq, curl, and terraform.

Step 1: Create the EKS lab cluster (cluster creation was 6m15s in my run)

This creates a 2-node EKS cluster with self-managed t3.large EC2 nodes in public subnets.

git clone https://github.com/Abhimanyu9988/Abhimanyu9988.github.io.git
cd Abhimanyu9988.github.io/labs/eks-app-slow-cpu-fine-scale-on-splunk-latency-with-keda

export AWS_REGION=eu-central-1
export LAB_NAME=scale-an-eks-app-on-requ-041971
export TF_VAR_lab_name=$LAB_NAME

export SPLUNK_REALM=eu0
export SPLUNK_ACCESS_TOKEN=<splunk-ingest-token>
export SPLUNK_API_TOKEN=<splunk-read-only-api-token>

terraform init && terraform apply

The realm must match your Splunk Observability Cloud org. The ingest token is for the collector. The read-only API token is for KEDA and the proof queries.

Step 2: Reproduce the slow-app, low-CPU state

This deploys the namespace, app, Service, and a load generator with 60 concurrent clients.

Step 3: Install the collector, KEDA, and the Splunk scaler config

The second script installs the Splunk OpenTelemetry Collector chart 0.163.1 and KEDA chart 2.21.0. It also creates the Secret, TriggerAuthentication, and ScaledObject.

bash reproduce.sh
bash fix.sh

fix.sh runs the Helm installs, waits for the collector agent, checks Splunk through the API, applies the KEDA objects, and starts the same load generator again.

Step 4: Check that Splunk has both cluster and app metrics

Before scaling, the lab proves that data arrived in Splunk Observability Cloud.

Installed charts:
NAME                  NAMESPACE   REVISION   STATUS     CHART                         APP VERSION
keda                  keda        1          deployed   keda-2.21.0                   2.21.0
splunk-otel-collector splunk      1          deployed   splunk-otel-collector-0.163.1 0.163.1
...
k8s.cluster.name=scale-an-eks-app-on-requ-041971: 1609 metric time series, distinct metric names: 74
app metric app_request_latency_ms: 1 time series; dimensions of one series:

That matters because KEDA is not scraping the pod directly. It asks Splunk for the metric, then exposes that value to Kubernetes as an external metric.

Step 5: Watch the HPA follow the Splunk latency metric

Once the ScaledObject is ready, KEDA creates an HPA. At idle, the metric is 0/400. Under load, the Splunk latency value crosses the target and the HPA raises the replica count.

The timings below come from sampled script output, with about 10 seconds between checks. The query also uses a 40-second window, so the 48s and 72s figures are approximate, and the metric can stay stale briefly after load stops.

The proof: 3-second latency became a 5-pod scale-out

Stage	Result
Before load test	1 replica, no HPA
Before peak latency	3049.2 ms
Before peak CPU	2.8% of one core
Splunk cluster time series found	1609
Splunk app latency time series found	1
Scale-up under load	1 -> 5 ready replicas in 48s
Scale-down after load stopped	back to 1 in 72s

Here is the scale-up section from the lab output:

T+  7s  replicas desired=1 ready=1  splunk_latency_ms(max)=0
T+ 21s  replicas desired=1 ready=1  splunk_latency_ms(max)=0
T+ 34s  replicas desired=1 ready=1  splunk_latency_ms(max)=0
T+ 48s  replicas desired=5 ready=5  splunk_latency_ms(max)=3008
Scale-up: 1 -> 5 ready replicas in 48s after load started

And after the load generator stops:

T+  4s  replicas desired=5 running=5  splunk_latency_ms(max)=3008
T+ 31s  replicas desired=5 running=5  splunk_latency_ms(max)=3008
T+ 58s  replicas desired=5 running=5  splunk_latency_ms(max)=3008
T+ 72s  replicas desired=1 running=1  splunk_latency_ms(max)=0
Summary: 1 -> 5 replicas in 48s under load; back to 1 in 72s after load stopped (driven by Splunk latency, not CPU).

That is the point of the lab: Kubernetes scaled on the symptom users feel, not on a resource that stayed quiet.

Use it for services where CPU is the wrong proxy

I/O-bound APIs: if your service waits on databases, downstream HTTP calls, or storage, latency may rise long before CPU does.
Queue-like request handlers: if work piles up inside the process, a latency gauge can be a better scaling input than container CPU.
Existing observability signals: if Splunk already has the metric, KEDA can turn it into a Kubernetes scaling signal.

For a real service, I would spend most of the time choosing the query and target. The harder question is: what metric means "add pods now" without scaling on noise?

Clean up (the EKS cluster and nodes cost money while they run)

Run the cleanup script first so Helm releases and namespaces are removed, then destroy the AWS infrastructure.

bash cleanup.sh
terraform destroy

The lab cleanup removed the Helm releases and namespaces, and the final destroy removed 18 Terraform resources.

What I did not test

I did not test scale-to-zero. This lab keeps minReplicaCount: 1.
I did not compare latency scaling against a CPU HPA on the same Deployment.
I did not tune the SignalFlow query for production alert quality; I used the lab metric and a 400 ms target.

Every file in this lab

Everything above runs from these files. Open any of them to copy it, or clone the repo as shown:

Sources

Disclosure: I work at Splunk, part of Cisco. This is my personal blog; the post isn't sponsored or reviewed by my employer.

I built this in a disposable AWS lab. The lab code and raw output are in the repo. Drafted with AI assistance and reviewed by me.

From Broken Ingress to Healthy ALB: An EKS and Splunk Hands-On Lab

Abhimanyu Bajaj — Thu, 22 Oct 2026 11:57:38 +0000

You’ll take an EKS Ingress with an empty ADDRESS and turn it into a working internet-facing ALB, then prove it with curl, healthy targets, controller reconciliation logs, and Splunk metrics. This is the kind of failure I see in real clusters: the Kubernetes YAML can look right while IAM, Pod Identity, subnet tags, and load balancer reconciliation decide whether traffic ever reaches a pod.

In this lab, the before-state is one specific case: the alb Ingress exists before the AWS Load Balancer Controller exists. The lab still creates the IAM role, Pod Identity association, and public subnet tags that the controller needs after it is installed.

In my run, the proof ended with HTTP 200, 2/2 healthy ALB targets, 1 successful reconciliation, 0 reconciler errors, and 1,968 Splunk cluster time series. Here’s how it works, and how to build it yourself.

TL;DR

What you get: EKS 1.32, two t3.large worker nodes, nginx, an alb Ingress, an internet-facing ALB, and Splunk Observability Cloud data.

The trick: the Ingress only becomes an ALB after the AWS Load Balancer Controller is installed with AWS permissions through EKS Pod Identity.

The proof: curl returned HTTP 200, ALB target health was 2/2, the controller logged 1 successful reconciliation and 0 errors, and Splunk returned cluster/controller/app series.

Time and cost: the reproduced-and-fixed run took 28.3 minutes. The lab uses one EKS control plane, two t3.large instances, and one ALB for about an hour, estimated as well under 1 USD.

Why a valid Ingress can still have no ALB

A Kubernetes Ingress is a request. It is not the load balancer by itself.

Terraform creates the VPC, two public subnets tagged kubernetes.io/role/elb=1, the EKS cluster, two EC2 worker nodes, the Pod Identity agent, and the IAM role for the controller. Then reproduce.sh deploys the app and Ingress before any controller is installed.

The Deployment is ready and the Service exists, but nothing is watching ingressClassName: alb yet:

== Ingress status (ADDRESS should be empty: nothing reconciles it)
NAME   CLASS   HOSTS   ADDRESS   PORTS   AGE
web    alb     *                 80      34s
Ingress hostname: ''

== Ingress events
Events:       <none>

== IngressClass objects and controller pods
No resources found
aws-load-balancer-controller deployments: 0

That is the useful gap: no controller, no IngressClass, no ALB, no target group, and no Splunk data for this cluster yet.

The trick: give the `alb` Ingress a controller with AWS permissions

The AWS Load Balancer Controller watches Kubernetes resources and creates AWS load balancer resources from them. In this lab, Terraform binds the controller IAM role to the service account kube-system/aws-load-balancer-controller with EKS Pod Identity.

The Ingress stays small:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/healthcheck-path: /healthz
spec:
  ingressClassName: alb
...

For your service, change the namespace, service name, port, health check path, and whether the ALB should be internet-facing.

Build it yourself in 4 steps

You need Terraform, aws CLI v2, kubectl, helm 3.16 or newer, jq, and curl, plus a Splunk Observability Cloud ingest token and API token. The access token sends data; the API token is used to query for proof.

Step 1: Clone the lab and set the names once

This keeps Terraform, Kubernetes labels, and Splunk queries on the same lab name.

git clone https://github.com/Abhimanyu9988/Abhimanyu9988.github.io.git
cd Abhimanyu9988.github.io/labs/from-broken-ingress-to-healthy-alb-an-eks-and-splunk-hands

export AWS_REGION=eu-central-1
export LAB_NAME=alb-ingress-lab
export TF_VAR_lab_name=$LAB_NAME
export SPLUNK_REALM=us1
export SPLUNK_ACCESS_TOKEN=...
export SPLUNK_API_TOKEN=...

Use your own Splunk realm and tokens from your Splunk Observability Cloud org.

Step 2: Create the EKS foundation (609 seconds in my run)

The repo includes the Terraform. The provider setup supplies the region and permissions boundary, so do not pass those as Terraform variables here.

terraform init && terraform apply

My apply created 23 resources and printed the cluster name, VPC ID, region, and controller role ARN.

Step 3: Reproduce the empty ADDRESS state (48 seconds in my run)

This deploys nginx and the Ingress before the controller is installed.

./reproduce.sh

You should see an empty Ingress hostname, no IngressClass, 0 controller deployments, and 0 ALBs tagged for the cluster.

Step 4: Install the controller and prove the ALB (148 seconds in my run)

The script installs the AWS Load Balancer Controller chart 1.13.0 for app v2.13.0, installs the Splunk OpenTelemetry Collector chart 0.116.0, waits for the ALB, checks targets, curls the ALB, checks logs, and queries Splunk.

./fix.sh

One lab detail: recent Helm versions reject that Splunk chart schema because one deprecated field is a string where the metaschema wants a boolean, so the script uses --skip-schema-validation when Helm supports it.

The proof: HTTP 200, 2 healthy targets, and Splunk series

Check	Result
AWS Load Balancer Controller deployment	1/1 available
Ingress ADDRESS	`k8s-demo-web-70bb130243-28212986.eu-central-1.elb.amazonaws.com`
ALB state	active
ALB target health	2/2 healthy
HTTP status through ALB	200
Controller reconciliations	1 successful, 0 errors
Splunk cluster time series	1,968
Splunk controller pod time series	1,621
Splunk app pod time series	34
Controller Prometheus metric series	12

The target health and curl output prove AWS is sending traffic to pod IPs:

== 4. ALB target health (target-type ip, one target per pod)
10.20.0.106 80  healthy
10.20.0.53  80  healthy
healthy targets: 2/2

== 5. curl through the load balancer: http://k8s-demo-web-70bb130243-28212986.eu-central-1.elb.amazonaws.com/
HTTP status: 200

The controller logged the reconciliation path:

{"level":"info","ts":"2026-10-22T09:51:01Z","logger":"controllers.ingress","msg":"creating targetGroup","stackID":"demo/web","resourceID":"demo/web-web:80"}
{"level":"info","ts":"2026-10-22T09:51:02Z","logger":"controllers.ingress","msg":"created loadBalancer","stackID":"demo/web","resourceID":"LoadBalancer","arn":"arn:aws:elasticloadbalancing:eu-central-1:<account-id>:loadbalancer/ap
{"level":"info","ts":"2026-10-22T09:51:02Z","logger":"controllers.ingress","msg":"successfully deployed model","ingressGroup":"demo/web"}
{"level":"info","ts":"2026-10-22T09:51:03Z","msg":"registering targets","arn":"arn:aws:elasticloadbalancing:eu-central-1:<account-id>:targetgroup/k8s-demo-web-f26d64a1b9/dc37538e55c037db","targets":[{"Id":"10.20.0.106","Availabili
successful reconciliations: 1, reconciler errors: 0

Splunk had data for the same lab name on the first attempt:

attempt 1: cluster=1968 controller-pods=1621 app-pods=34
-- controller Prometheus metrics (best effort):
controller_runtime_reconcile_total series: 12

Proof: curl 200 via k8s-demo-web-70bb130243-28212986.eu-central-1.elb.amazonaws.com, 2/2 targets healthy, controller reconciled, Splunk has cluster/controller/app series.

Use it for real ingress checks

Check reconciliation before changing the app. If ADDRESS is empty, confirm the IngressClass, controller deployment, Pod Identity association, subnet tags, and controller logs.
Keep health checks boring. This lab uses /healthz, a 10-second interval, and a healthy threshold of 2.
Watch Kubernetes and AWS. Pod readiness tells you the app is ready. ALB target health tells you AWS can reach it. Splunk gives you the cluster, controller, and app view together.

Clean up (the ALB costs money while it runs)

Delete Kubernetes-created load balancer resources before destroying Terraform. The cleanup script waits for the controller-managed ALB, target groups, Helm releases, and security groups to go away.

./cleanup.sh
terraform destroy

My cleanup script finished after 37 seconds, and Terraform destroyed 23 resources.

What I didn't test

HTTPS listeners, ACM certificates, or redirects from HTTP to HTTPS.
Private/internal ALBs.
Production sizing, autoscaling, or multi-AZ pod spread beyond this two-node lab.

Every file in this lab

Everything above runs from these files. Open any of them to copy it, or clone the repo as shown:

Sources

AWS Load Balancer Controller documentation
Amazon EKS Pod Identities
AWS Load Balancer Controller v3.6.1 release — background source; this lab pins chart 1.13.0 and app v2.13.0 so Terraform downloads the matching IAM policy.

Disclosure: I work at Splunk, part of Cisco. This is my personal blog; the post isn't sponsored or reviewed by my employer.

I built this in a disposable AWS lab. The lab code and raw output are in the repo. Drafted with AI assistance and reviewed by me.

Trace a Java service on EKS with one annotation, zero code changes

Abhimanyu Bajaj — Wed, 21 Oct 2026 18:10:48 +0000

Getting traces out of a Java service usually means a ticket to the app team; in this build, one EKS deployment annotation injects the OpenTelemetry Java agent with no code change and no image rebuild. I’ll start with a plain Java service that has no tracing, then add the Operator, an Instrumentation resource, and a Collector until spans show up with service.name=orders-java.

The same image stays in place: eclipse-temurin:17.0.13_11-jdk. In my run, the instrumented rollout finished in 85 seconds.

I work a lot with observability agents inside Kubernetes clusters, so I want this kind of change proven from cluster output, not only from a diagram.

Here's how it works, and how to build it yourself.

TL;DR

What you get: an EKS cluster where a plain Java service emits OpenTelemetry traces without source edits or an image rebuild.

The trick: the OpenTelemetry Operator watches one pod-template annotation and injects the Java agent plus OTEL environment variables.

The proof: same image before and after, rollout finished in 85s, Collector received 4 Java client spans with service.name: Str(orders-java).

Time and cost: my reproduced run took 23.6 minutes total including cluster creation; the README says the EKS control plane takes about 10 minutes and the run costs well under 1 USD.

Why a healthy Java pod can still produce no traces

A service can serve traffic and still be invisible to tracing. In the baseline run, /orders returned HTTP 200s, but there was no Java agent, no OTEL environment, no init container, and no Collector.

== Evidence of the gap
-- Tracing-related env vars in the app container (JAVA_TOOL_OPTIONS / OTEL_*):
(none)
-- Init containers on the pod:
(none)
-- Java agent mentions in app logs:
count=0
-- OpenTelemetry CRDs / collector in the cluster:
opentelemetry CRDs installed: 0

That is the platform-team problem: the app is up, but nothing is creating spans and nothing is receiving them.

The trick: mutate the pod template, not the Java code

The OpenTelemetry Operator adds a Kubernetes Instrumentation resource. That resource tells the injected agent where to export traces and which defaults to use.

Then this annotation on the Deployment pod template turns on Java auto-instrumentation:

instrumentation.opentelemetry.io/inject-java: "true"

On rollout, the Operator adds an init container, copies the Java agent into the pod, sets JAVA_TOOL_OPTIONS, sets the OTLP endpoint, and gives the service a name. In this lab, that name is orders-java.

Build it yourself in 5 steps

Step 1: Create the EKS lab cluster (about 10 minutes for the control plane)

This creates a VPC, an EKS cluster, and two self-managed t3.medium worker nodes joined with nodeadm.

git clone https://github.com/Abhimanyu9988/Abhimanyu9988.github.io.git
cd Abhimanyu9988.github.io/labs/trace-a-java-service-on-eks-with-one-annotation-zero-code
export AWS_REGION=eu-central-1 LAB_NAME=otel-java TF_VAR_lab_name=otel-java
terraform init && terraform apply -auto-approve
./reproduce.sh

reproduce.sh waits for nodes, applies k8s/app.yaml, starts the load generator, and prints the before-state evidence.

If your clone only has main.tf, add a small Terraform file with the variable declarations and provider wiring the README calls out. Set permissions_boundary to null if you do not use one.

variable "region" {
  type    = string
  default = "eu-central-1"
}

variable "lab_name" {
  type = string
}

variable "permissions_boundary" {
  type    = string
  default = null
}

provider "aws" {
  region = var.region
}

Step 2: Keep the app Deployment plain on purpose

The sample app lives in k8s/app.yaml. It mounts Java source from a ConfigMap and runs it with the JDK image. For your own service, the annotation still belongs under spec.template.metadata.annotations.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-java
  namespace: demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: orders-java
  template:
    metadata:
      labels:
        app: orders-java
    spec:
      containers:
        - name: app
          image: eclipse-temurin:17.0.13_11-jdk
          command: ["java", "-Xmx256m", "/app/Main.java"]
          ports:
            - containerPort: 8080
          readinessProbe:
            tcpSocket:
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 5
          resources:
            requests:
              cpu: 200m
              memory: 384Mi
            limits:
              memory: 768Mi
          volumeMounts:
            - name: src
              mountPath: /app
      volumes:
        - name: src
          configMap:
            name: orders-java-src

Change the image, command, ports, and probes for your service. Keep the pod-template location for the annotation.

Step 3: Add a Collector that proves spans in logs

The Collector manifest is k8s/collector.yaml. It uses the debug exporter so the proof stays inside Kubernetes logs instead of depending on a tracing backend.

apiVersion: v1
kind: Namespace
metadata:
  name: observability
---
apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: otel
  namespace: observability
spec:
  mode: deployment
  replicas: 1
  config:
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
    processors:
      batch: {}
    exporters:
      debug:
        verbosity: detailed
    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [batch]
          exporters: [debug]

In a real cluster, keep the OTLP receiver and replace the debug exporter with your tracing backend exporter.

Step 4: Point Java auto-instrumentation at the Collector

The Instrumentation resource is k8s/instrumentation.yaml. It points Java agents in the demo namespace at the in-cluster Collector service and turns off metrics and logs for this tracing-only lab.

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: java-auto
  namespace: demo
spec:
  exporter:
    endpoint: http://otel-collector.observability.svc.cluster.local:4318
  propagators:
    - tracecontext
    - baggage
  sampler:
    type: parentbased_always_on
  env:
    - name: OTEL_EXPORTER_OTLP_PROTOCOL
      value: http/protobuf
    - name: OTEL_METRICS_EXPORTER
      value: none
    - name: OTEL_LOGS_EXPORTER
      value: none

Change the endpoint if your Collector runs in a different namespace or you send directly to an external endpoint.

Step 5: Install the Operator and add the one annotation

fix.sh installs the pinned Operator Helm chart 0.71.2, applies k8s/collector.yaml and k8s/instrumentation.yaml, patches the Deployment, waits for rollout, and prints Collector logs.

./fix.sh
kubectl -n observability logs deploy/otel-collector --tail=-1

The patch it applies is the only app-side change: instrumentation.opentelemetry.io/inject-java: "true".

The proof: same image, injected agent, Collector received spans

Stage	What changed	Evidence
Before	App deployed with no tracing	no `JAVA_TOOL_OPTIONS`, no init containers, OpenTelemetry CRDs installed: 0
Operator	OpenTelemetry Operator installed	chart `opentelemetry-operator-0.71.2`, app version `0.110.0`
Collector	OTLP receiver created	service exposed `4317/TCP,4318/TCP` inside the cluster
Annotation	Deployment patched	rollout finished in 85s, image still `eclipse-temurin:17.0.13_11-jdk`
Trace proof	Java agent exported spans	4 spans received, `service.name: Str(orders-java)`

After rollout, the new pod showed the auto-instrumentation init container and Java agent startup line:

== Injected pod: orders-java-578744fdb4-lxj4j
-- init containers:
opentelemetry-auto-instrumentation-java
-- agent image:
ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:1.33.5
-- app log head:
Picked up JAVA_TOOL_OPTIONS:  -javaagent:/otel-auto-instrumentation-java/javaagent.jar
[otel.javaagent 2026-10-19 18:15:24:512 +0000] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 1.33.5
orders-java listening on 8080

The Collector excerpt is cut, but it shows the important path: Java agent to OTLP HTTP to Collector debug output.

2026-10-19T18:15:40.063Z    info    ResourceSpans #0
...
     -> service.name: Str(orders-java)
     -> telemetry.sdk.language: Str(java)
InstrumentationScope io.opentelemetry.java-http-client 1.33.5-alpha
Span #0
    Name           : GET
    Kind           : Client
...

The 4 spans in this run are outbound Java HTTP client spans from the /orders to /inventory call. The build proves the Java auto-instrumentation and Collector path; it did not capture inbound server spans for /orders in this run.

Use it for services you do not want to rebuild

Platform-owned rollout: create the Operator, Collector, and Instrumentation resource once per namespace, then let teams opt in with the pod-template annotation.
Service naming: accept the deployment-derived OTEL_SERVICE_NAME, or set resource.opentelemetry.io/service.name on the pod template if your naming standard is different.
Backend swap: keep the same app annotation and Instrumentation shape, but replace the debug exporter with the exporter your tracing platform needs.

Clean up (the EKS cluster and EC2 nodes cost money while they run)

Remove the Kubernetes objects first, then destroy the AWS resources.

./cleanup.sh
terraform destroy -auto-approve

What I didn't test

I did not send these spans to a production tracing backend; I used the Collector debug exporter only.
I did not capture inbound server spans or parent/child links in this run.
I did not test private subnets, managed node groups, Fargate, IRSA, Node.js, .NET, or Python auto-instrumentation.

Every file in this lab

Everything above runs from these files. Open any of them to copy it, or clone the repo as shown:

Sources

I built this in a disposable AWS lab. The lab code and raw output are in the repo. Drafted with AI assistance and reviewed by me.

Roll back an EKS release in 16 seconds without restarting a pod

Abhimanyu Bajaj — Mon, 19 Oct 2026 17:55:14 +0000

Every team running on Kubernetes knows this moment. A release goes out, something looks off, and the only "undo" button is another deploy: roll back the image, wait for new pods, hope nothing else breaks on the way.

I wanted a rollback that is just a dial. So I built one on EKS. The new version gets 10% of real traffic first, and if it misbehaves, one kubectl command sends everyone back to the old one. In my lab, traffic was fully back on v1 16 seconds later, and not a single pod restarted.

Here's how it works, and how to build it yourself.

TL;DR

What you get: an Application Load Balancer in front of two versions of your app, with a traffic dial you turn from Kubernetes.

The trick: the AWS Load Balancer Controller reads the traffic weights from one Ingress annotation.

The proof: 90/10, then 50/50, then a rollback to 100/0 in 16 seconds, with 0 pod restarts.

Time and cost: my run took 25 minutes end to end in a throwaway AWS account; the lab budgets about 45 minutes and under 1 USD.

Why a normal Kubernetes Service can't give you a 10% canary

A Service spreads requests across whatever pods match its selector. Put 3 v1 pods and 1 v2 pod behind one Service, and v2 gets "about a quarter, depending on luck". That's what my lab showed before the build:

== Before: how is traffic split between v1 and v2? (40 in-cluster requests to Service 'app')
     34 version-v1
      6 version-v2
v1=34 v2=6  (ratio follows the pod count 3:1; to get 10% you would need 9 v1 pods per v2 pod)

So a real 10% canary means running nine v1 pods for every v2 pod. And rolling back means scaling or redeploying. There is no weight you can set to zero.

The trick: one Ingress, two target groups, one dial

The AWS Load Balancer Controller turns a Kubernetes Ingress into an ALB. Give the Ingress a weighted forward action, and the ALB splits traffic between two target groups, one per Service, by weight. The number of pods behind each Service no longer matters.

The whole dial is one annotation:

alb.ingress.kubernetes.io/actions.weighted: >-
  {"type":"forward","forwardConfig":{"targetGroups":[{"serviceName":"app-v1","servicePort":80,"weight":90},{"serviceName":"app-v2","servicePort":80,"weight":10}]}}

Change the numbers, and the controller updates the ALB listener rule. Your Deployments never change. The controller gets its AWS permissions through EKS Pod Identity, so there's no IRSA setup and nothing extra on the node role.

Build it yourself in 4 steps

You need Terraform, the AWS CLI v2, kubectl, helm, jq and curl, plus an AWS account you can experiment in.

Step 1: Create the cluster (about 12 to 15 minutes)

This creates an EKS 1.31 cluster with two t3.medium workers, the Pod Identity agent and the IAM role for the controller.

git clone https://github.com/Abhimanyu9988/Abhimanyu9988.github.io.git
cd Abhimanyu9988.github.io/labs/eks-canary-releases-alb
export AWS_REGION=eu-central-1
export LAB_NAME=canary
export TF_VAR_lab_name="$LAB_NAME"
terraform init && terraform apply

You should see Plan: 23 to add, 0 to change, 0 to destroy. before you confirm.

Step 2: Deploy two versions of the app

bash reproduce.sh

This creates app-v1 (3 pods) and app-v2 (1 pod) in the demo namespace, one Service for each, and shows the "before" split above.

Step 3: Add the load balancer and the dial

bash fix.sh

This installs the AWS Load Balancer Controller (Helm chart 1.11.0, controller v2.11.0) with Pod Identity, applies k8s/ingress.yaml with the 90/10 weights, waits for the ALB, and then walks through 90/10, 50/50 and 100/0, counting 200 requests at each stage.

Step 4: Turn the dial yourself

Send half the traffic to v2:

kubectl -n demo annotate ingress app --overwrite 'alb.ingress.kubernetes.io/actions.weighted={"type":"forward","forwardConfig":{"targetGroups":[{"serviceName":"app-v1","servicePort":80,"weight":50},{"serviceName":"app-v2","servicePort":80,"weight":50}]}}'

Something wrong with v2? Roll back:

kubectl -n demo annotate ingress app --overwrite 'alb.ingress.kubernetes.io/actions.weighted={"type":"forward","forwardConfig":{"targetGroups":[{"serviceName":"app-v1","servicePort":80,"weight":100},{"serviceName":"app-v2","servicePort":80,"weight":0}]}}'

The proof: 90/10, 50/50, and a 16-second rollback

Each stage sent 200 requests through the ALB:

Weights	ALB listener rule	What 200 requests hit
90/10	app-v1=90, app-v2=10	v1=179, v2=21
50/50	app-v1=50, app-v2=50	v1=92, v2=108
100/0 (rollback)	app-v1=100, app-v2=0	v1=200, v2=0

The rollback timing, straight from the run:

ALB rule updated 3s after kubectl annotate; 100% of sampled traffic on v1 after 16s

And the pods? The same ones, still running, never restarted:

pod app-v1-5b8cd854d6-2k7kl Running restarts=0 age=4m13s
pod app-v1-5b8cd854d6-6m5vm Running restarts=0 age=4m13s
pod app-v1-5b8cd854d6-krvvr Running restarts=0 age=4m13s
pod app-v2-649d75bcd8-dd555 Running restarts=0 age=4m13s

That's the whole point: v2 can stay deployed and healthy while the load balancer decides how much traffic it gets.

Use it for your own releases

Give each version its own Service, like app-v1 and app-v2. The weighted action points at Services, not Deployments.
Make the annotation your release step. 10%, 50%, 100% and a rollback to 0% are the same one-line change, so they fit any pipeline.
Use the official IAM policy in production. The lab's controller policy is simplified for a throwaway account; take the official one from the controller release.

Clean up (the ALB costs money while it runs)

Delete the Ingress first, so the controller removes the ALB, target groups and security groups, then destroy everything else. Use the same lab name.

export AWS_REGION=eu-central-1
export LAB_NAME=canary
export TF_VAR_lab_name="$LAB_NAME"
bash cleanup.sh
terraform destroy

What I didn't test

TLS certificates, host-based routing, or several paths on one ALB.
Automatic promotion or rollback driven by metrics.
Long-running connections while the weights change.

Sources

I built this in a disposable AWS lab. The lab code and raw output are in the repo. Drafted with AI assistance and reviewed by me.

Fargate SQS worker stuck at 0 tasks? The IF(running > 0) guard never gets to run

Abhimanyu Bajaj — Sun, 18 Oct 2026 15:11:28 +0000

I rebuilt a Stack Overflow setup in a throwaway account: an ECS Fargate worker with min capacity 0, scaled by target tracking on SQS backlog per task, with the guard IF(running > 0, backlog / running, backlog). I sent 10 messages and the service stayed at desired=0 for the whole 8-minute wait. The backlog series had datapoints, but RunningTaskCount and the metric-math result had none, so the guard never produced a value. Swapping in FILL(running, 0) with one put-scaling-policy call got the service scaling out from 0 and draining the queue. Scale-in back to 0 isn't covered by this run.

This hurts when you are trying to save idle Fargate cost for a worker that only wakes up when SQS has work. The Terraform can look reasonable. The expression even has an IF guard for zero tasks. But when messages arrive, nothing starts, the queue sits full, and the target-tracking alarms stay in INSUFFICIENT_DATA.

Most of my week goes on one question: why does something that looks correctly configured not work in a real cluster? I lead a support engineering team for observability agents that run inside Kubernetes clusters, and before I write about a problem, I reproduce it myself in a throwaway AWS account and keep the raw output.

This one was reproduced and fixed in eu-central-1.

TL;DR

Problem: an ECS Fargate SQS worker with min capacity 0 stayed at desired=0 even with 10 visible messages.

Cause: Container Insights RunningTaskCount had no datapoints at zero tasks, so IF(running > 0, backlog / running, backlog) returned no bpt datapoints.

Fix: use IF(FILL(running, 0) > 0, backlog / FILL(running, 0), backlog) so the expression falls back to raw backlog at zero tasks.

What you'll see

The service starts at zero, as intended:

== Initial state: desired=0 running=0
[
    {
        "min": 0,
        "max": 5
    }
]

After sending 10 SQS messages, it stays there for the whole wait:

t+30s desired=0 running=0 visible_in_queue=10
...
t+480s desired=0 running=0 visible_in_queue=10

The target-tracking alarms never get enough data:

TargetTracking-service/scale-to-zero-sqs-worker-447482-cluster/scale-to-zero-sqs-worker-447482-worker-AlarmHigh-bca35fd1-1f83-47f3-bb4f-7e1460dd8a71    INSUFFICIENT_DATA
TargetTracking-service/scale-to-zero-sqs-worker-447482-cluster/scale-to-zero-sqs-worker-447482-worker-AlarmLow-af7b562c-9e1e-4887-b89d-fae66aa4bd5e INSUFFICIENT_DATA
Unchecked: Initial alarm creation

The important part is the metric math. SQS backlog has data. RunningTaskCount does not. The computed backlog-per-task series, bpt, also has no data:

{"Id":"backlog","datapoints":8,"Values":[10.0,10.0,10.0,10.0,10.0,10.0,10.0,10.0]}
{"Id":"running","datapoints":0,"Values":[]}
{"Id":"bpt","datapoints":0,"Values":[]}
bpt datapoints: 0

Final state from the failed run:

Final: desired=0 running=0 messages_waiting=10 bpt_datapoints=0
Messages are waiting but the 'bpt' series has no datapoints (RunningTaskCount missing), so the alarm never fires and the service stays at 0.

Why it happens

The source question reported this pattern: an ECS Fargate worker consumes SQS, target tracking uses backlog per task, and the service has minimum capacity 0. AWS commonly recommends a backlog-per-task metric for queue workers: visible messages divided by running workers.

That is fine while at least one task is running. The awkward edge is zero.

In my lab, Container Insights did not publish a RunningTaskCount datapoint for this service while it had zero running tasks. CloudWatch metric math then returned no datapoints for this expression:

IF(running > 0, backlog / running, backlog)

So the fallback branch did not save the policy. The problem was not division by zero. The problem was missing data. The guard never had a real running datapoint to evaluate.

The Stack Overflow answers suggested wrapping running with FILL(running, 0). My lab confirms that this fix was enough for scale-out from zero. After replacing the policy with:

IF(FILL(running, 0) > 0, backlog / FILL(running, 0), backlog)

Application Auto Scaling set desired count to 5, Fargate tasks reached RUNNING, and the queue drained.

What is only likely, not proven by this run: adding in-flight SQS messages may help avoid scaling in too early while messages are being processed, and a separate bootstrap policy keyed on DesiredTaskCount may be useful in some designs because Fargate takes time to move from desired to running. I did not test those variants here.

Replicate it yourself, step by step

You need AWS CLI, Terraform, jq, and GNU date. The lab uses eu-central-1 through its provider config.

Get the lab.

git clone https://github.com/Abhimanyu9988/Abhimanyu9988.github.io.git
cd Abhimanyu9988.github.io/labs/scale-to-zero-sqs-workers-on-ecs-fargate

In providers.tf, set permissions_boundary to null for your own account if you do not use an IAM permissions boundary.

Initialize Terraform and apply the lab.

terraform init

Real init output from the run:

Initializing provider plugins...
- Finding hashicorp/aws versions matching "~> 5.80"...
- Installing hashicorp/aws v5.100.0...
- Installed hashicorp/aws v5.100.0 (signed by HashiCorp)

Terraform has been successfully initialized!

Now create the resources:

terraform apply

It creates an SQS queue, a VPC with two public subnets, an ECS cluster with Container Insights enabled, a Fargate service at desired count 0, IAM roles, a log group, an Application Auto Scaling target with min 0 and max 5, and the original target-tracking policy. The worker is the public aws-cli image looping on receive-message; each message takes 20 seconds in this lab.

The plan and apply showed 19 resources added:

Plan: 19 to add, 0 to change, 0 to destroy.
...
Apply complete! Resources: 19 added, 0 changed, 0 destroyed.

Outputs:

cluster_name = "scale-to-zero-sqs-worker-447482-cluster"
policy_name = "scale-to-zero-sqs-worker-447482-backlog-per-task"
queue_name = "scale-to-zero-sqs-worker-447482-jobs"
queue_url = "https://sqs.eu-central-1.amazonaws.com/<account-id>/scale-to-zero-sqs-worker-447482-jobs"
resource_id = "service/scale-to-zero-sqs-worker-447482-cluster/scale-to-zero-sqs-worker-447482-worker"
service_name = "scale-to-zero-sqs-worker-447482-worker"

Set the shell values the scripts read from Terraform outputs.

export AWS_DEFAULT_REGION="eu-central-1"
CLUSTER=$(terraform output -raw cluster_name)
SERVICE=$(terraform output -raw service_name)
QURL=$(terraform output -raw queue_url)
QNAME=$(terraform output -raw queue_name)
RID=$(terraform output -raw resource_id)
POLICY=$(terraform output -raw policy_name)

Check the initial ECS service state and scaling target.

aws ecs describe-services --cluster "$CLUSTER" --services "$SERVICE" --query 'services[0].desiredCount' --output text
aws ecs describe-services --cluster "$CLUSTER" --services "$SERVICE" --query 'services[0].runningCount' --output text
aws application-autoscaling describe-scalable-targets --service-namespace ecs --resource-ids "$RID" --query 'ScalableTargets[].{min:MinCapacity,max:MaxCapacity}' --output json

Output:

== Initial state: desired=0 running=0
[
    {
        "min": 0,
        "max": 5
    }
]

Send 10 messages to SQS.

for i in $(seq 1 10); do
  aws sqs send-message --queue-url "$QURL" --message-body "job-$i" >/dev/null
done

Output:

== Sending 10 messages

Watch desired count, running count, and visible messages. The same reads are repeated every 30 seconds for up to 16 checks.

sleep 30
aws ecs describe-services --cluster "$CLUSTER" --services "$SERVICE" --query 'services[0].desiredCount' --output text
aws ecs describe-services --cluster "$CLUSTER" --services "$SERVICE" --query 'services[0].runningCount' --output text
aws sqs get-queue-attributes --queue-url "$QURL" --attribute-names ApproximateNumberOfMessages --query 'Attributes.ApproximateNumberOfMessages' --output text

Output from the run:

== Waiting up to 8 minutes for the service to leave 0 tasks
t+30s desired=0 running=0 visible_in_queue=10
t+60s desired=0 running=0 visible_in_queue=10
...
t+480s desired=0 running=0 visible_in_queue=10

Ask Application Auto Scaling for the target-tracking alarm names, then ask CloudWatch for their state and reason.

ALARMS=$(aws application-autoscaling describe-scaling-policies --service-namespace ecs --resource-id "$RID" --query 'ScalingPolicies[0].Alarms[].AlarmName' --output text)
aws cloudwatch describe-alarms --alarm-names $ALARMS --query 'MetricAlarms[].[AlarmName,StateValue]' --output text
aws cloudwatch describe-alarms --alarm-names $ALARMS --query 'MetricAlarms[0].StateReason' --output text | cut -c1-300

Output:

== Policy alarms
TargetTracking-service/scale-to-zero-sqs-worker-447482-cluster/scale-to-zero-sqs-worker-447482-worker-AlarmHigh-bca35fd1-1f83-47f3-bb4f-7e1460dd8a71    INSUFFICIENT_DATA
TargetTracking-service/scale-to-zero-sqs-worker-447482-cluster/scale-to-zero-sqs-worker-447482-worker-AlarmLow-af7b562c-9e1e-4887-b89d-fae66aa4bd5e INSUFFICIENT_DATA
Unchecked: Initial alarm creation

Query the metric math directly. This is the same broken expression from the original policy.

START=$(date -u -d '15 minutes ago' +%Y-%m-%dT%H:%M:%SZ)
END=$(date -u +%Y-%m-%dT%H:%M:%SZ)
cat > /tmp/q-broken.json <<EOF
[
 {"Id":"backlog","ReturnData":true,"MetricStat":{"Metric":{"Namespace":"AWS/SQS","MetricName":"ApproximateNumberOfMessagesVisible","Dimensions":[{"Name":"QueueName","Value":"$QNAME"}]},"Period":60,"Stat":"Sum"}},
 {"Id":"running","ReturnData":true,"MetricStat":{"Metric":{"Namespace":"ECS/ContainerInsights","MetricName":"RunningTaskCount","Dimensions":[{"Name":"ClusterName","Value":"$CLUSTER"},{"Name":"ServiceName","Value":"$SERVICE"}]},"Period":60,"Stat":"Average"}},
 {"Id":"bpt","ReturnData":true,"Expression":"IF(running > 0, backlog / running, backlog)"}
]
EOF
METRICS=$(aws cloudwatch get-metric-data --start-time "$START" --end-time "$END" --metric-data-queries file:///tmp/q-broken.json)
echo "$METRICS" | jq -c '.MetricDataResults[] | {Id, datapoints: (.Values|length), Values}'
BPT_N=$(echo "$METRICS" | jq '[.MetricDataResults[] | select(.Id=="bpt") | (.Values|length)] | add // 0')
echo "bpt datapoints: $BPT_N"

Output:

== Metric math as the policy evaluates it (last 15 min, 60s period)
{"Id":"backlog","datapoints":8,"Values":[10.0,10.0,10.0,10.0,10.0,10.0,10.0,10.0]}
{"Id":"running","datapoints":0,"Values":[]}
{"Id":"bpt","datapoints":0,"Values":[]}
bpt datapoints: 0

Check scaling activities. There were none in the reproduced failure.

aws application-autoscaling describe-scaling-activities --service-namespace ecs --resource-id "$RID" --query 'ScalingActivities[].[StatusCode,Description]' --output text | head -5

Output:

== Scaling activities

The failed reproduction ends like this:

Final: desired=0 running=0 messages_waiting=10 bpt_datapoints=0
Messages are waiting but the 'bpt' series has no datapoints (RunningTaskCount missing), so the alarm never fires and the service stays at 0.

Apply the fix: replace the target-tracking policy with the same metric, but wrap running in FILL(running, 0).

First confirm the before state:

aws ecs describe-services --cluster "$CLUSTER" --services "$SERVICE" --query 'services[0].desiredCount' --output text
aws ecs describe-services --cluster "$CLUSTER" --services "$SERVICE" --query 'services[0].runningCount' --output text
aws sqs get-queue-attributes --queue-url "$QURL" --attribute-names ApproximateNumberOfMessages --query 'Attributes.ApproximateNumberOfMessages' --output text

Output:

== Before: desired=0 running=0 visible=10

Create the fixed policy configuration:

EXPR='IF(FILL(running, 0) > 0, backlog / FILL(running, 0), backlog)'
cat > /tmp/policy-fixed.json <<EOF
{
 "TargetValue": 2,
 "ScaleOutCooldown": 60,
 "ScaleInCooldown": 300,
 "CustomizedMetricSpecification": {
  "Metrics": [
   {"Id":"backlog","ReturnData":false,"MetricStat":{"Metric":{"Namespace":"AWS/SQS","MetricName":"ApproximateNumberOfMessagesVisible","Dimensions":[{"Name":"QueueName","Value":"$QNAME"}]},"Stat":"Sum"}},
   {"Id":"running","ReturnData":false,"MetricStat":{"Metric":{"Namespace":"ECS/ContainerInsights","MetricName":"RunningTaskCount","Dimensions":[{"Name":"ClusterName","Value":"$CLUSTER"},{"Name":"ServiceName","Value":"$SERVICE"}]},"Stat":"Average"}},
   {"Id":"bpt","Label":"Backlog per task","ReturnData":true,"Expression":"$EXPR"}
  ]
 }
}
EOF

Output:

== Updating policy with: IF(FILL(running, 0) > 0, backlog / FILL(running, 0), backlog)

Push the fixed policy into Application Auto Scaling:

aws application-autoscaling put-scaling-policy \
  --policy-name "$POLICY" --policy-type TargetTrackingScaling \
  --service-namespace ecs --resource-id "$RID" --scalable-dimension ecs:service:DesiredCount \
  --target-tracking-scaling-policy-configuration file:///tmp/policy-fixed.json \
  --query 'PolicyARN' --output text

Output:

arn:aws:autoscaling:eu-central-1:<account-id>:scalingPolicy:0b5eb0c3-92be-46eb-9e09-d6e225ccfe6b:resource/ecs/service/scale-to-zero-sqs-worker-447482-cluster/scale-to-zero-sqs-worker-447482-worker:policyName/scale-to-zero-sqs-worker-447482-backlog-per-task

Watch the service scale out from zero. The same ECS and SQS reads are repeated every 30 seconds.

sleep 30
aws ecs describe-services --cluster "$CLUSTER" --services "$SERVICE" --query 'services[0].desiredCount' --output text
aws ecs describe-services --cluster "$CLUSTER" --services "$SERVICE" --query 'services[0].runningCount' --output text
aws sqs get-queue-attributes --queue-url "$QURL" --attribute-names ApproximateNumberOfMessages --query 'Attributes.ApproximateNumberOfMessages' --output text
aws sqs get-queue-attributes --queue-url "$QURL" --attribute-names ApproximateNumberOfMessagesNotVisible --query 'Attributes.ApproximateNumberOfMessagesNotVisible' --output text

Output:

== Waiting up to 10 minutes for the service to scale out from 0 and a task to reach RUNNING
t+30s desired=5 running=0 visible=10 in_flight=0
t+60s desired=5 running=5 visible=10 in_flight=0

Check scaling activities again.

aws application-autoscaling describe-scaling-activities --service-namespace ecs --resource-id "$RID" --query 'ScalingActivities[].[StatusCode,Description]' --output text | head -5

Output:

== Scaling activities
Successful  Setting desired count to 5.

Watch the queue drain.

aws sqs get-queue-attributes --queue-url "$QURL" --attribute-names ApproximateNumberOfMessages --query 'Attributes.ApproximateNumberOfMessages' --output text
aws sqs get-queue-attributes --queue-url "$QURL" --attribute-names ApproximateNumberOfMessagesNotVisible --query 'Attributes.ApproximateNumberOfMessagesNotVisible' --output text
aws ecs describe-services --cluster "$CLUSTER" --services "$SERVICE" --query 'services[0].desiredCount' --output text
aws ecs describe-services --cluster "$CLUSTER" --services "$SERVICE" --query 'services[0].runningCount' --output text

Output:

== Watching queue drain (up to 2.5 min, bounded by script time budget)
visible=6 in_flight=0 desired=5 running=5
visible=0 in_flight=4 desired=5 running=5
visible=0 in_flight=0 desired=5 running=5

Clean up the live service state before destroying Terraform resources.

Delete the scaling policy so the autoscaler does not relaunch tasks:

aws application-autoscaling delete-scaling-policy --service-namespace ecs --resource-id "$RID" \
  --scalable-dimension ecs:service:DesiredCount --policy-name "$POLICY"

Set the ECS service desired count back to zero:

aws ecs update-service --cluster "$CLUSTER" --service "$SERVICE" --desired-count 0

List any remaining tasks and stop each one:

for t in $(aws ecs list-tasks --cluster "$CLUSTER" --query 'taskArns[]' --output text); do
  aws ecs stop-task --cluster "$CLUSTER" --task "$t"
done

Wait until no task ARNs are returned. The cleanup script repeats this check up to 18 times, 10 seconds apart:

for i in $(seq 1 18); do
  N=$(aws ecs list-tasks --cluster "$CLUSTER" --query 'length(taskArns)' --output text)
  [ -z "$N" ] || [ "$N" = "0" ] && break
  sleep 10
done

Purge the queue:

aws sqs purge-queue --queue-url "$QURL"

The cleanup script suppressed command output in my run.

Destroy the lab.

terraform destroy

Output from destroy:

Plan: 0 to add, 0 to change, 18 to destroy.
...
Destroy complete! Resources: 18 destroyed.

What I didn't test

I did not test scale-in back to zero. The lab notes say the target-tracking low alarm needs about 15 minutes of low data, and this run was bounded by the script time.

I did not test the in-flight-message expression using ApproximateNumberOfMessagesNotVisible.

I did not test a separate 0-to-1 scale-out policy keyed on DesiredTaskCount.

I did not test ECS task scale-in protection. That matters for a different question: whether a busy worker can be stopped during scale-in. If a task is stopped while processing, SQS messages can be redelivered after the 60 second visibility timeout in this lab.

Cost and time

The run was in eu-central-1 and took 19.1 minutes end to end.

Terraform created 19 resources. After the cleanup step deleted the scaling policy, Terraform destroyed 18 resources.

The timed parts from the raw output were: init 6 seconds, apply 36 seconds, reproduce 572 seconds, fix 162 seconds, cleanup 17 seconds, and the later init 3 seconds. The workload used a few minutes of 0.25 vCPU Fargate tasks. Container Insights metrics cost cents.

The takeaway

If your scale-to-zero ECS worker depends on a metric from running tasks, test the zero-task case directly. For SQS backlog per task, treat missing RunningTaskCount as zero with FILL, otherwise your careful IF guard may never get a datapoint to evaluate.

Sources

Stack Overflow: ECS Fargate SQS autoscaling with min capacity 0

I reproduced this in a disposable AWS lab. The lab code and raw output are in the repo. Drafted with AI assistance and reviewed by me.

DEV Community: Abhimanyu Bajaj

October: Kubernetes memory semantics and Collector edge fixes

Kubernetes memory conventions became more precise

Pod memory limits and requests now have desired and current forms

Working set descriptions now say what Linux includes

Memory utilization now uses working set as the numerator

Operator Helm chart configuration and docs got clearer

Private registry users can override only the Target Allocator repository

Argo CD users get documentation for auto-generated webhook certs

Collector contrib tests, receivers, and ownership moved forward

Kubernetes inventory cache-sync tests are deterministic now

The Flink Metrics Receiver has another code owner

Flink operator metrics now match exact suffixes

Azure Data Explorer Exporter has active ownership again

A Machine Agent monitoring extension starts again on Java 11+

The DataPower extension no longer depends on JDK SAAJ

Still in review

Sources

EKS App Slow, CPU Fine? Scale on Splunk Latency with KEDA

Why CPU autoscaling misses this slow EKS service

The trick: let Splunk become the external metric KEDA reads

Build it yourself in 5 steps

Step 1: Create the EKS lab cluster (cluster creation was 6m15s in my run)

Step 2: Reproduce the slow-app, low-CPU state

Step 3: Install the collector, KEDA, and the Splunk scaler config

Step 4: Check that Splunk has both cluster and app metrics

Step 5: Watch the HPA follow the Splunk latency metric

The proof: 3-second latency became a 5-pod scale-out

Use it for services where CPU is the wrong proxy

Clean up (the EKS cluster and nodes cost money while they run)

What I did not test

Every file in this lab

Sources

Disclosure: I work at Splunk, part of Cisco. This is my personal blog; the post isn't sponsored or reviewed by my employer.

From Broken Ingress to Healthy ALB: An EKS and Splunk Hands-On Lab

Why a valid Ingress can still have no ALB

The trick: give the alb Ingress a controller with AWS permissions

Build it yourself in 4 steps

Step 1: Clone the lab and set the names once

Step 2: Create the EKS foundation (609 seconds in my run)

Step 3: Reproduce the empty ADDRESS state (48 seconds in my run)

Step 4: Install the controller and prove the ALB (148 seconds in my run)

The proof: HTTP 200, 2 healthy targets, and Splunk series

Use it for real ingress checks

Clean up (the ALB costs money while it runs)

What I didn't test

Every file in this lab

Sources

Disclosure: I work at Splunk, part of Cisco. This is my personal blog; the post isn't sponsored or reviewed by my employer.

Trace a Java service on EKS with one annotation, zero code changes

Why a healthy Java pod can still produce no traces

The trick: mutate the pod template, not the Java code

Build it yourself in 5 steps

Step 1: Create the EKS lab cluster (about 10 minutes for the control plane)

Step 2: Keep the app Deployment plain on purpose

Step 3: Add a Collector that proves spans in logs

Step 4: Point Java auto-instrumentation at the Collector

Step 5: Install the Operator and add the one annotation

The proof: same image, injected agent, Collector received spans

Use it for services you do not want to rebuild

Clean up (the EKS cluster and EC2 nodes cost money while they run)

What I didn't test

Every file in this lab

Sources

Roll back an EKS release in 16 seconds without restarting a pod

Why a normal Kubernetes Service can't give you a 10% canary

The trick: one Ingress, two target groups, one dial

Build it yourself in 4 steps

Step 1: Create the cluster (about 12 to 15 minutes)

Step 2: Deploy two versions of the app

Step 3: Add the load balancer and the dial

Step 4: Turn the dial yourself

The proof: 90/10, 50/50, and a 16-second rollback

Use it for your own releases

Clean up (the ALB costs money while it runs)

What I didn't test

Sources

Fargate SQS worker stuck at 0 tasks? The IF(running > 0) guard never gets to run

What you'll see

Why it happens

The trick: give the `alb` Ingress a controller with AWS permissions