DEV Community

Fernando Azevedo
Fernando Azevedo

Posted on Originally published at fernando.moretes.com

ADR: EKS Provisioned Control Plane and 40x HPA Concurrency

#aiagents #eks #kubernetes #hpa

On July 28, 2026, AWS announced that EKS Provisioned Control Plane increases HPA sync concurrency to up to 40 times the Kubernetes default — with no configuration changes required from the operator. For those running clusters with hundreds or thousands of HPA objects, this is not a cosmetic adjustment: it is the removal of a structural bottleneck that, until now, made metric-driven autoscaling fundamentally non-deterministic under load.

Context and Forces: The Problem Nobody Was Monitoring Correctly

The Kubernetes Horizontal Pod Autoscaler operates on a control loop. Every cycle — by default every 15 seconds — the controller manager evaluates all HPA objects in the cluster, queries the metrics server or an external metrics adapter, calculates the desired replica count, and issues an update on the Deployment or StatefulSet object. What most teams miss is that this loop is serially bounded by the --horizontal-pod-autoscaler-sync-period parameter combined with evaluation concurrency. In standard upstream Kubernetes, that concurrency is intentionally low — the project assumes a shared control plane where concurrent workloads could saturate the API server.

In the financial environments I operate, it is common to have between 200 and 800 active HPA objects in a single production cluster: quote services, risk engines, onboarding APIs, market event processing workers. Each of those objects needs to be evaluated within the sync window. When load arrives — a volume spike at market open, a news event triggering simultaneous requests — the control plane must process all those objects before any scaling happens. With low concurrency and 400 HPA objects, you may be waiting through multiple 15-second cycles before the critical workload is evaluated. This translates to scaling latency of 45 to 90 seconds in realistic scenarios — enough time for an availability SLO to be violated.

The architectural force here is clear: autoscaler reaction speed is a component of the availability SLO, not just an infrastructure metric. Teams that define SLOs of p99 < 500ms for trading APIs need to include scaling latency in their error budget — and most do not.

The Structural Bottleneck: How HPA Sync Concurrency Works

The upstream kube-controller-manager exposes the flag --concurrent-horizontal-pod-autoscaler-syncs, which controls how many parallel goroutines process HPA objects simultaneously. The historical default value is 5. This means that in a cluster with 500 HPA objects, the controller needs at least 100 serial goroutine iterations to complete a full evaluation cycle. If each evaluation takes ~100ms (including metrics server call, calculation, and write to the API server), a full cycle takes ~10 seconds — already consuming 67% of the default 15-second sync window.

EKS Provisioned Control Plane, per the July 28, 2026 announcement, increases this concurrency to up to 40x the default value. Mathematically, this means the same 500-HPA-object cluster can be evaluated in approximately 2.5 goroutine iterations instead of 100 — reducing cycle time from ~10 seconds to sub-second under normal conditions. The practical effect is that the time between load detection and the start of scaling collapses from multiple cycles to a fraction of the first cycle.

It is important to understand that Provisioned Control Plane is AWS's premium offering where you get a dedicated, non-shared control plane. This is what makes it viable to increase concurrency without impacting other tenants — the underlying API server has reserved capacity to absorb the additional write and read throughput that concurrent evaluation generates. On a standard shared control plane, aggressively increasing this concurrency could generate API server throttling and worsen the situation. Here, the AWS architectural decision is coherent: the benefit is only safe because resource isolation already exists.

Numbers That Matter for the Decision

  • 40x — HPA concurrency increase on Provisioned Control Plane. Versus Kubernetes upstream default (5 goroutines)
  • ~5 — Default HPA goroutines in upstream Kubernetes. Historical value of --concurrent-horizontal-pod-autoscaler-syncs flag
  • 76% — Scale-out time improvement on ECS (AWS benchmark, Jun/2026). From 363s to 86s — signal that AWS is attacking autoscaling latency across the platform
  • 43% — Node scale-out improvement in EKS Auto Mode (Karpenter). Announced Jun/2026 — node layer complementary to the pod layer

Options Considered: How to Solve the HPA Bottleneck at Scale Before This Announcement

Option A: EKS Standard + Manual HPA Tuning

Pros

  • Lower cost — shared control plane without surcharge
  • Full control over sync parameters via custom add-on

Cons

  • Cannot increase --concurrent-horizontal-pod-autoscaler-syncs on EKS Standard — the control plane is managed and does not expose this flag
  • Workarounds like KEDA introduce operational complexity and another control plane to manage
  • Risk of throttling on shared API server under high evaluation concurrency

Verdict: Inadequate for clusters with >200 HPA objects in financial environments with tight SLOs

Option B: KEDA (Kubernetes Event-Driven Autoscaling) as HPA replacement

Pros

  • Native support for external metric sources: Kafka lag, SQS depth, Prometheus, Datadog
  • Scale-to-zero capability — relevant for batch workloads
  • Does not depend on kube-controller-manager sync loop

Cons

  • Introduces an additional operator with its own lifecycle, versioning, and IAM attack surface
  • ScaledObjects are CRD objects — adds API server and etcd overhead
  • Does not solve the problem for workloads already using native HPA with metrics server

Verdict: Valid as complement for external metric sources, not as universal HPA replacement

Option C: Cluster sharding (split by business domain)

Pros

  • Reduces the number of HPA objects per cluster, relieving the concurrency bottleneck
  • Improves blast radius isolation between domains

Cons

  • Multiplication of control plane costs, observability tooling, and operational overhead
  • Network and service mesh complexity between clusters for cross-domain communication
  • Does not scale well — the problem returns as each cluster grows

Verdict: Expensive architectural workaround that resolves the symptom without addressing the root cause

Option D: EKS Provisioned Control Plane (chosen decision)

Pros

  • 40x HPA concurrency with no configuration changes — zero operational debt
  • Dedicated control plane with availability SLA and reserved API server capacity
  • Compatible with KEDA, VPA, Karpenter — not exclusive, it is additive
  • Aligned with AWS platform direction: EKS Auto Mode also runs on Provisioned Control Plane

Cons

  • Additional cost of Provisioned Control Plane versus the standard tier
  • Migration of existing clusters requires planning — not a trivial in-place upgrade

Verdict: Correct decision for financial production clusters with >150 HPA objects and defined availability SLOs

The Decision: Why Provisioned Control Plane Is the Right Choice for Financial Environments

The decision to adopt EKS Provisioned Control Plane is not motivated solely by the HPA concurrency improvement announced in July 2026. It is motivated by an architectural posture shift: the Kubernetes control plane is critical data infrastructure, not a convenience service. In financial environments, the control plane determines the system's reaction time to market events. Treating it as a shared commodity resource is the same as sharing the data bus of a trading system — it works until it does not.

Provisioned Control Plane offers dedicated API server capacity, which means the additional throughput generated by 40x more HPA evaluation goroutines does not compete with other tenants. This is fundamental: increasing concurrency without resource isolation is a recipe for cascading degradation. The Kubernetes API server has configurable rate limiting limits (--max-requests-inflight, --max-mutating-requests-inflight), and on a shared control plane, parallel evaluation of 200 HPA objects can easily saturate these limits, generating 429 Too Many Requests and delaying not just autoscaling, but also deployment operations, health checks, and reconciliation of other controllers.

The decision is also consistent with the AWS platform trajectory. EKS Auto Mode — which includes Karpenter with 43% faster node scale-out and 69% faster consolidation — runs on Provisioned Control Plane. AWS is clearly building its highest-value capabilities on this layer. Teams that have not migrated to it are, effectively, accumulating platform technical debt.

For clusters with fewer than 50 HPA objects and workloads without defined scaling latency SLOs, the standard tier may be sufficient. But that is an exception, not the rule in financial production environments.

HPA Evaluation Pipeline: Standard vs. Provisioned Control Plane (40x)

Comparison of HPA evaluation flow in a cluster with 400 HPA objects. Left: default behavior with 5 serial goroutines. Right: Provisioned Control Plane with up to 200 parallel goroutines. Edges show the critical path that determines scaling latency.

📊 Metrics Layer

  • Metrics Server / Prometheus Adapter (data)
  • CloudWatch Container Insights (data)

🔁 Standard Control Plane (5 goroutines)

  • HPA Controller 5 concurrent syncs ~100 iterations/cycle (compute)
  • API Server (shared tenant) throttle risk (network)
  • Scale Event latency: 45-90s (400 HPAs) (edge)

⚡ Provisioned Control Plane (40x concurrency)

  • HPA Controller ~200 concurrent syncs ~2.5 iterations/cycle (compute)
  • API Server (dedicated capacity) no throttle risk (network)
  • Scale Event latency: <5s (400 HPAs) (edge)

🚀 Node Scaling Layer

  • Karpenter (EKS Auto Mode) 43% faster nodes (compute)
  • EC2 Nodes Provisioned (compute)

Flows

  • ms -> hpa_std: metrics per cycle
  • ms -> hpa_pcp: metrics per cycle
  • cw -> ms: container metrics
  • hpa_std -> api_std: serial writes (throttle risk)
  • hpa_pcp -> api_pcp: parallel writes (dedicated)
  • api_std -> scale_std: replica update
  • api_pcp -> scale_pcp: replica update
  • scale_pcp -> karp: pending pods trigger
  • karp -> ec2: node provisioning

Operational Consequences: What Changes in Practice

The first operational consequence is positive and immediate: existing clusters on Provisioned Control Plane receive the benefit without any action. This is rare in infrastructure improvements and deserves recognition — AWS absorbed the tuning complexity into the managed service.

The second consequence is more subtle and potentially problematic: workloads that were sized based on slow HPA behavior may now scale more aggressively than expected. Imagine a service that had scaleUp.stabilizationWindowSeconds configured at 120 seconds precisely because the team knew the HPA was slow to react and wanted to avoid flapping. With the new concurrency, the HPA reacts in seconds — and the stabilization window may no longer be sufficient to absorb transient metric spikes. The result could be unnecessary scale-up followed by scale-down, generating additional cost and potential instability.

The third consequence affects observability. If you monitor kube_horizontalpodautoscaler_status_current_replicas versus kube_horizontalpodautoscaler_spec_max_replicas as a proxy for scaling pressure, the time series behavior will change. What was previously a gradual upward curve (because the HPA took time to react) may now be a near-instantaneous step function. Capacity planning dashboards that depend on the slope of that curve to predict node needs must be recalibrated.

Finally, there is the interaction with Karpenter/EKS Auto Mode. The faster HPA will generate Pending pods more quickly, which triggers Karpenter to provision nodes. With Karpenter 43% faster (June 2026 announcement), the complete pod-scale → node-provision chain now operates within a much smaller time window. This is excellent for SLOs — but it means the Karpenter NodePool needs well-defined maximum capacity limits to avoid excessive provisioning in runaway scaling scenarios.

Warning: Review Your HPA Policies Before Celebrating: The concurrency improvement is automatic — but your HorizontalPodAutoscaler objects were written assuming slow behavior. Review immediately: (1) scaleUp.stabilizationWindowSeconds — values above 60s may have been set as a workaround for HPA latency; with the new concurrency, they may be overly conservative or, worse, insufficient to prevent flapping on noisy metrics. (2) scaleDown.stabilizationWindowSeconds — the Kubernetes default of 300s is generally correct, but verify it was not reduced to compensate for slow scale-up. (3) maxReplicas limits — with faster scale-up, a poorly sized maxReplicas can be hit before you notice, creating an artificial ceiling that violates SLOs. (4) behavior.scaleUp.policies with type: Pods and absolute values — what was previously a safe scaling rate may now be reached in seconds instead of minutes. Monitor kube_horizontalpodautoscaler_status_condition with reason=ScalingLimited in the first cycle after migration.

Observability: What to Instrument After Migration

Migration to Provisioned Control Plane with 40x HPA concurrency requires a review of the observability model. The set of metrics you need to monitor changes in character, not just in values.

Control plane metrics that become more important: apiserver_request_duration_seconds for patch and update verbs in the autoscaling/v2 group now reflects actual HPA throughput. On a dedicated control plane, you should see latencies consistently below 50ms for these requests. If you observe spikes above 200ms, this is a signal that the metrics server is the bottleneck, not the HPA controller. Instrument metrics_server_api_metric_freshness_seconds to ensure metrics feeding the HPA are fresh enough — with faster HPA, metrics with 30-60 second staleness become a real limiter.

Scaling latency SLO: Explicitly define an SLO for scaling time. A useful composite metric is: time_from_metric_threshold_breach_to_first_new_pod_ready. This requires correlation between HPA events (kube_horizontalpodautoscaler_status_current_replicas increasing), pod creation (kube_pod_created), and readiness (kube_pod_status_ready). With OpenTelemetry, you can create a synthetic span covering this complete pipeline — from breach to pod ready — and expose it as a measurable SLI.

Runaway scaling alerts: Configure an alert in CloudWatch or Datadog for kube_horizontalpodautoscaler_status_current_replicas / kube_horizontalpodautoscaler_spec_max_replicas > 0.85 with a 2-minute window. This detects when a workload is rapidly approaching the replica ceiling — a situation that, with slower HPA, took enough time for manual intervention, but can now happen before any traditional latency alert fires.

Cost observability: Enable AWS Cost Anomaly Detection with a monitor specific to the cluster tag. Faster scale-up means EC2/Fargate cost spikes also happen faster. The cost error budget needs to be recalibrated alongside the availability error budget.

Well-Architected Analysis: EKS Provisioned Control Plane with 40x HPA

  • security: The dedicated control plane reduces the noisy neighbor attack surface at the control plane level. Combined with IRSA (IAM Roles for Service Accounts) for granular metrics server access to CloudWatch, and Network Policies restricting communication between the HPA controller and external metric sources, the security model is stronger than on a shared control plane.
  • reliability: The HPA concurrency increase directly reduces overload recovery time — a critical component of workload-level RTO. In financial environments, this improves availability SLO compliance without code changes. The residual risk is flapping from noisy metrics — mitigated with appropriate stabilizationWindowSeconds and low-latency metrics.
  • performance: The performance benefit is second-order: faster HPA reduces the time the system operates under-provisioned, reducing API latency during spikes. For payment and trading systems, this can mean the difference between meeting or violating p99 latency SLOs during volume peaks.
  • cost: Provisioned Control Plane has additional cost versus the standard tier. This cost must be justified by the value of the SLO it protects. For workloads where one minute of degradation has measurable financial impact (lost transactions, regulatory fines, churn), the ROI is clear. For internal workloads without a formal SLO, the analysis is less straightforward.

Anti-Patterns to Avoid After Migration

  • Assuming faster HPA eliminates the need for pre-warming. HPA reacts to existing load — for predictable events (market open, marketing campaigns), pre-scaling via CronJob or KEDA ScaledJob is still necessary.
  • Removing stabilizationWindowSeconds entirely to maximize reaction speed. CPU and memory metrics have inherent noise — without a stabilization window, you will have constant flapping that degrades user experience and generates unnecessary cost.
  • Not reviewing maxReplicas after migration. With faster HPA, a conservative maxReplicas can be hit within seconds during a spike, creating an invisible artificial ceiling in latency dashboards.
  • Using high-latency metrics (e.g., CloudWatch with 1-minute period) as HPA source for workloads that can now scale in seconds. The HPA controller speed exceeds the metric update frequency — the bottleneck migrates to the metric source.
  • Migrating to Provisioned Control Plane without defining an explicit scaling latency SLO. Without an SLO, you have no way to measure whether the additional investment is generating measurable value.

Architect's Note: In my experience with financial production clusters, the HPA sync concurrency bottleneck was one of the hardest problems to diagnose because its symptoms — elevated latency during spikes — were indistinguishable from application capacity problems. The lesson I learned the hard way is that you need to measure scaling time as a first-class SLI, not as a secondary infrastructure metric. Before migrating to Provisioned Control Plane, do a complete inventory of all HPA objects and classify them by business criticality — you will find that 20% of them have stabilizationWindowSeconds configured as workarounds for slowness that no longer exists. My practical recommendation: instrument kube_horizontalpodautoscaler_status_condition with workload labels in Datadog or CloudWatch before migrating, establish a behavioral baseline, and only then activate Provisioned Control Plane — that way you have a measurable before/after to justify the additional cost to the CFO.

Final Decision: Migrate to Provisioned Control Plane if You Operate Financial Workloads with SLOs

The 40x HPA concurrency increase in EKS Provisioned Control Plane is not a niche feature — it is the correction of a structural Kubernetes limit that affected any cluster with more than ~100 HPA objects. For financial environments, where scaling latency is a direct component of the availability SLO, this change alone justifies migration to Provisioned Control Plane. The decision is reinforced by alignment with AWS platform direction (EKS Auto Mode, faster Karpenter) and the fact that the benefit is delivered without configuration change. The additional cost of Provisioned Control Plane must be evaluated against the cost of SLO violations — in payment, trading, or any system where one minute of degradation has measurable financial impact, the ROI is positive. The immediate action is not to migrate — it is to review all existing HPA policies, instrument scaling latency as an SLI, and then migrate with a measurable baseline. Clusters with fewer than 50 HPA objects and no formal scaling SLOs may remain on the standard tier for now.

Rating: Strongly Recommended for financial-grade

References

Originally published at fernando.moretes.com. By Fernando F. Azevedo — Senior Solutions Architect.

Top comments (0)