DEV Community: Calin Florescu

Debugging an Invisible Scaling Limit on EKS

Calin Florescu — Fri, 20 Mar 2026 11:27:51 +0000

Introduction

If you've ever tried scaling a deployment past 1000 pods on EKS and watched everything just... stop, with no errors, no warnings, and pods that look healthy but never actually receive traffic — this one's for you.

I ran into this exact situation on a client's EKS cluster. The HPA was configured to scale well beyond 1000 replicas, and it did spin up the pods. They started fine, containers were healthy, but something was off: the readiness probes weren't even being evaluated. The pods were stuck in a limbo where they existed but didn't really exist as far as the load balancer was concerned.

The cluster was running Kubernetes 1.33 with the AWS Load Balancer Controller v2.12, using IP-mode target registration behind an Application Load Balancer. That last detail turned out to matter a lot.

Debugging

The fact that exactly 1000 instances were spun up and registered normally — and none above — made me think about some kind of quota or limit being reached on the EKS or AWS level. A quick check confirmed two important AWS quotas: Targets per Target Group per Region and Targets per Application Load Balancer, both with a default value of 1000. Raising these was a necessary first step.

But even after raising the quotas, the pods were not marked as ready, and they were not showing up as new targets. From a container perspective, everything looked fine — processes were starting correctly, no crazy delays with the readiness probes.

The nodes were fine too. Kubelets weren't starved of resources, there weren't too many pods scheduled per node, and IPs were correctly assigned. Nothing pointed to anything being off in that direction.

The next thing I dug into was the internal networking of Kubernetes and how the pods get registered as targets for the target groups. And that's when I found the culprit: the AWS Load Balancer Controller and the way it uses the Endpoint API to manage target registration.

The controller monitors several Kubernetes resources — Services, Ingresses, Pods, Nodes, and critically, Endpoints/EndpointSlices — to determine which backends should be registered as targets in AWS Elastic Load Balancing target groups.

This behaviour depends on the target type configured for the controller. In my case, it was set to IP, which heavily relies on Endpoints/EndpointSlices.

In IP mode, the controller registers individual pod IPs directly into the target group:

The controller's reconciliation loop watches for changes to the Endpoints objects for the relevant Service.
When an EndpointSlice is created or updated, the controller extracts the list of ready pod IP addresses and their associated ports.
It compares this set against the currently registered targets in the AWS target group.
It calls RegisterTargets for any new pod IPs and DeregisterTargets for any stale ones.

After understanding this, the problem was obvious: my controller was configured to use the Endpoint API for IP registration, and according to the Kubernetes docs, each Endpoint object created for a service is capped at 1000 entries.

The Endpoint object for my Service was truncated, so no changes were detected for the newly added pods, and no reconciliation was triggered on the controller side to register the new targets.

But wait — why did the readiness probes look weird?

There was still a nagging question: why did the pods look ready at a glance, even though the readiness probe seemingly never ran?

This is where the pod readiness gate comes in. The AWS LB Controller uses a custom readiness condition (target-health.elbv2.k8s.aws/targetgroupbinding) that works like this:

A pod starts and gets an IP, but that custom condition starts as False.
The controller registers the pod IP in the target group, polls the AWS health check, and only patches the condition to True once AWS reports the target as healthy. Until that happens, the pod isn't truly "Ready" from the EndpointSlice's perspective, meaning kube-proxy won't route traffic to it either.

It's actually a nice mechanism — it prevents dropped connections during rollouts by ensuring the load balancer marks a target as healthy before Kubernetes starts sending traffic to it.

But here's the catch: if the pod never gets registered in the target group in the first place (because the Endpoint object was truncated at 1000), the controller never even starts that health check dance. The readiness gate stays False forever, silently. No error, no event, nothing in the logs saying "hey, I skipped this pod." It just doesn't happen.

That's what made this so tricky to debug. The symptom looked like a readiness probe issue, but the actual cause was three layers deeper.

Solution

After understanding how the AWS LB Controller worked and how it was configured, the solution was straightforward: configure it to use the EndpointSlices API instead of the limited Endpoint one. EndpointSlices don't have the same 1000-entry cap per object — they split endpoints across multiple slices, so the controller can see all of your pod IPs regardless of scale.

Combined with the AWS quota increases mentioned earlier, this got the deployment scaling well beyond the 1000 pod barrier.

Conclusion

I wrote this up for two reasons. The obvious one: if you're hitting this exact wall, I hope this saves you the hours I spent staring at perfectly healthy pods that refused to serve traffic.

The less obvious one is a reminder — mostly to myself — about what happens when we work on top of deep abstraction layers day after day. The Endpoint API silently truncating at 1000 entries, with no warning and no error, buried under controllers, CRDs, and cloud provider integrations — that's the kind of thing you can only debug if you understand what's actually happening beneath the tools you're using. Abstractions are great, they're how we make progress, but when they break, they break quietly. And the only thing that helps at that point is knowing what's underneath.

Streamlining Microservices Management: A Unified Helm Chart Approach

Calin Florescu — Tue, 07 May 2024 05:55:28 +0000

Introduction

Hello, I want to discuss with you today a problem I recently encountered with the microservices architecture and the solution I found. I hope my experience with this matter will help someone in the future or save some time.

This article is relevant if you use a container orchestration tool like Kubernetes or Openshift and manage the releases using Helm. However, it can also be a starting point for an idea in multiple areas.

These concepts are familiar, and I have not invented them; I combined them into a solution that works for the current situation of the project I am working on.

With this in mind, let’s start!

The problem

Microservice architecture is great for scalability, development speed, and making your solution technology agnostic by allowing each team to develop their service in the preferred technology. It also leverages the service's ownership to the development team that created it.

Even if everything sounds great, a problem arises: engineers are managing the Helm Charts that are the bases for the releases that will be installed in the clusters.

Usually, the main focus of the engineers is to make the most efficient code possible, so the focus won’t be on having the best Helm templates or configurations.

It’s also usually impossible in a large ecosystem to have a DevOps engineer dedicated to each team managing a microservice that ensures the best possible configuration and best practices.

This combination of factors is generating a situation where even if 80% of all services are using the same templates (e.g. Deployments, Services, etc.), all of them are written differently, the best practices are not respected, and there might be code duplication or even unused code because of the copy-paste pattern that is happening when a new service is created (this can be avoided using a template repository). Also, if you need to change something on all microservices, you must work in multiple repositories.

As a DevOps, working in this setup is like speaking a new language with each microservice.

The Solution

A centralised helm chart with template definitions that can be added as a dependency in all microservice charts sounds like a great idea. This way, the engineers are left only to configure the templates with the necessary values based on the service specification and decide what Kubernetes resources they need.

Also, when there is a need to update the configuration for a specific resource, e.g. adding affinities, tolerations, etc., a DevOps engineer will need to do the change in a single place and using the versioning feature will be able to propagate the change to all Releases.

To integrate the central charts into the microservice one, we can use the dependency functionality:

dependencies:
- name: unified-templates
  version: 1.0.0
  repository: <repository_where_chart_is_stored>

Implementation

I created a new repository and initialised a Helm chart using the helm create command to implement this.

Regarding the way the templates are imported, I decided to create functions. It is easier to track what objects are in use on a microservice, and you can easily template the object with the local values scope.

## Defining a template
{{ define "deployment" }}
api: v1
kind: Deployment
metadata:
  name: {{ include "unified-helm-template.fullname" . }}
spec:
  {{- with .Values.deployment }}
  replicas: {{ .replicas }}
  {{- end }}
{{ end }}

## Importing a template
{{ include "deployment" . }}

## Configuring the chart
fullnameOverride: <service_release_name>

## Configuring a template
deployment:
 replicas: 2

For the helpers functions to work correctly, since the templating will be done in the service chart, we need to define the fullnameOverride variable to match the release name. Like this, we can use in the unified charts the predefined helpers, which will use the override value to generate proper naming and labels.

To configure the template in the child chart, an engineer needs to define an object matching the name of the functions under which it will configure the required values.

Overriding

Sometimes, you must implement custom functionality, add some logic, or extend the template's functionality quickly. For this situation, I don’t want to restrict the user with the initial implementation, but it would be cool to allow them to override it.

To achieve this, I added a change to how the template is defined. Now, you can merge a second function with the initial one, obtaining the override functionality.

## Template definition with override logic
{{- define "deployment" -}}
    {{- $override := include "deploymentOverride" . | fromYaml -}}
    {{- $base := include "deploymentInstance" . | fromYaml -}}
    {{- if $override -}}
        {{- $merged := mustMergeOverwrite (dict) $base $override -}}
        {{- toYaml $merged -}}
    {{- else -}}
        {{- toYaml $base -}}
    {{- end -}}
{{- end -}}

## Importing the template
{{ include "deployment" .}}

## Override for custom use case
{{ define "deploymentOverride" }}
spec:
 replicas: {{ ternary true false (eq .Values.customVar "true") }}
{{ end }}

Documentation

Good documentation for the templates' configurations is essential for centralising them. To manage this, I used a tool to generate markdown documentation from Helm value files.

# -- Object that configures Deployment instance
deployment:
  # -- Define the minimum number of seconds for which the pod should be running without crushing before being considered healthy
  minReadySeconds:
  # -- Decide if the pods should not be restarted if you have secrets used as env vars and they are updated
  skipSyncSecrets: true
  # -- Decide if the pods should be restarted if you have configs used as env vars and they are updated
  skipSyncConfigs: true
  # -- Update strategy
  strategy:
  # -- Number of replicas that will be created

To automate the process of creating and updating the documentation, I added the execution of this tool as a pre-commit hook so that every time someone contributes to the repo, the documentation will be automatically updated:

repos:
  - repo: <https://github.com/norwoodj/helm-docs>
    rev: v1.2.0
    hooks:
      - id: helm-docs
        args:
          # Make the tool search for charts only under the `unified-templates` directory
          - --chart-search-root=unified-templates

Unit Testing

I needed a way to test the templates I had written before exposing them to the engineers, so I used the helm-unittest plugin to write some unit tests. I know it’s not the perfect way to test functionality, but we are assured that the templating works as desired.

suite: Deployment tests
templates:
  - deployment/deployment.yaml
tests:
  - it: renders a valid deployment instance resource
    values:
    - ./values/deployment.yaml
    asserts:
    - isKind:
        of: Deployment
    - matchSnapshot: {}
  - it: configures the deployment name correctly
    values:
    - ./values/deployment.yaml
    asserts:
    - equal:
        path: metadata.name
        value: service-template

Conclusions

There is a fine line between a guideline and a rule, between adding some generic best practices and forcing someone to adapt to a particular style. A good engineer has to develop solutions that consider all of these things.

The solution described above combines all of them, providing a guideline and some strict rules but also allowing for adaptation when necessary.

What is good for me might be bad for you, and as a DevOps, you have to consider everyone’s well-being, from client to engineer. It’s hard at times, but there lies the beauty.

You can find a template of this solution here.