Moshe Avdiel

Posted on Jul 9 • Originally published at github.com

Autoscale Guardrails as Live Policy — Cap Max Replicas Before the Bill Explodes (Java SDK)

Friday 00:08 UTC — Black Friday. payments-capture is at 38 replicas and the HPA wants 52. Cloud cost alerts in #payments-finops show EC2 spend already 2.1× the Thanksgiving forecast. Checkout conversion is healthy; the problem is not capacity — it is unbounded scale-out on a service whose maxReplicas: 60 was set last spring when nobody modeled a 4× traffic multiplier.

SRE lead Priya Nair is on the bridge with platform engineer Marco Delgado. Priya does not want to touch minReplicas — that would risk queue buildup on the card-auth handoff. She needs to lower the ceiling so the HPA stops adding pods while traffic stays green:

"Drop max_replicas_ceiling from 60 to 35 now. Keep min at 12. I am not waiting for a Helm PR while we mint expensive pods."

Marco opens values-prod.yaml. The ceiling is not even a first-class HPA field — it is a comment in the runbook and a custom autoscaler sidecar that reads MAX_REPLICA_CEILING = 60 from a Java constant compiled into payments-autoscale-guard three weeks ago.

The FinOps owner asks what everyone is thinking:

"We already compute desired replica count on every metrics tick. Why does spend guardrail require a deploy when the number we need to change is an integer?"

Most Java payment fleets treat autoscale guardrails as infra state: Helm maxReplicas, Terraform ASG caps, and a sidecar constant that only changes after a rolling restart. Kiponos.io collapses ceiling overrides, scale-rate limits, and emergency posture flags into one operational tree — readable on every autoscale evaluation with local get*() calls and adjustable from the dashboard while JVMs keep running.

The problem — max_replicas_ceiling baked into static config

A typical payments autoscale guard clamps HPA output like this:

@Service
public class ReplicaCeilingGuard {
    private static final int MAX_REPLICA_CEILING = 60;
    private static final int MIN_REPLICA_FLOOR = 12;

    public int clampDesired(int hpaDesired, String deployment) {
        return Math.min(Math.max(hpaDesired, MIN_REPLICA_FLOOR), MAX_REPLICA_CEILING);
    }
}

Ceiling policy usually lives elsewhere — scattered and deploy-bound:

# charts/payments-capture/values-prod.yaml — GitOps, not incident knob
autoscaling:
  minReplicas: 12
  maxReplicas: 60
  targetCPUUtilizationPercentage: 65

Or worse — the HPA max and the guard constant disagree:

// Sidecar compiled Monday; Helm still says 60; SRE wants 35 at midnight
private static final int MAX_REPLICA_CEILING = 60;

The autoscale loop runs every 15–30 seconds per deployment. During a spend anomaly you need to:

Lower guardrails/max_replicas_ceiling before the next HPA tick adds eight more c5.2xlarge nodes
Keep guardrails/min_replicas_floor stable so checkout queues do not stall
Flip posture/emergency_scale_freeze to block all scale-out except manual overrides

Doing that through Helm while replicas climb is not FinOps — it is invoice theater with compound interest.

What teams believe vs production reality

Belief	Production reality
"HPA maxReplicas is the ceiling"	Custom guards and FinOps sidecars often enforce a second cap in Java
"Cluster autoscaler protects the budget"	CA adds nodes; HPA adds pods — neither reads your CFO's daily burn target
"We'll scale down after the event"	Scale-down lags; expensive hours already booked
"GitOps maxReplicas is fine for Black Friday"	Peak needs mid-event tuning when traffic shape surprises rehearsal
"One global ceiling works for all payment services"	Capture, fraud, and settlement have different unit economics per replica

The Aha

max_replicas_ceiling is operational config — it changes during invoice spikes, capacity drills, and Black Friday bridges. It belongs in a live tree the autoscale guard already reads with getInt(), not in a static final imported at JVM boot.

What Kiponos.io is for autoscale guardrails

Kiponos.io is a real-time configuration hub with Java and Python SDKs. Kiponos.createForCurrentTeam() connects over WebSocket; the profile tree — for example ['payments']['prod']['autoscale'] — hydrates into in-process memory at service startup.

When Priya sets guardrails/max_replicas_ceiling to 35, a delta patches only that key. The next kiponos.path("guardrails").getInt("max_replicas_ceiling") on the metrics tick is a local memory read — no HTTP to a config API, no poll loop, no extra etcd round-trip for policy.

afterValueChanged logs ceiling flips and can emit autoscale_ceiling_changed metrics to your FinOps dashboard without restarting the sidecar JVM.

No restart. No redeploy. No @RefreshScope bean recycle.

Honest boundary: Kiponos does not replace Terraform for node pools, Kubernetes for HPA CRDs, or your metrics server. It owns operational ceilings Java guards read on every scale evaluation.

Architecture

Helm documents baseline desired state; authoritative incident ceilings live in Kiponos where lowering them takes seconds.

Config tree — guardrails, deployments, posture, and audit

Five folders — guardrails, deployments, posture, rates, audit:

guardrails/
  max_replicas_ceiling: 60
  min_replicas_floor: 12
  default_ceiling: 60
  enforce_ceiling: true
deployments/
  payments-capture/
    max_replicas_ceiling: 60
    min_replicas_floor: 12
    enabled: true
  payments-fraud/
    max_replicas_ceiling: 40
    min_replicas_floor: 8
    enabled: true
  payments-settlement/
    max_replicas_ceiling: 25
    min_replicas_floor: 6
    enabled: true
posture/
  emergency_scale_freeze: false
  freeze_message: "Scale-out frozen — SRE incident"
  allow_manual_override: true
rates/
  max_scale_up_per_tick: 4
  max_scale_down_per_tick: 2
  evaluation_interval_sec: 15
audit/
  last_ceiling_change_by: ""
  last_ceiling_change_at_ms: 0
  emit_clamp_metrics: true

One tree. One profile path: ['payments']['prod']['autoscale']. Staging scale drills share identical key layout — only values differ.

Java integration — per-deployment ceiling guard + scale-freeze posture

import io.kiponos.sdk.Kiponos;
import org.springframework.context.annotation.Configuration;
import org.springframework.stereotype.Service;

@Configuration
public class KiponosAutoscaleConfig {
    @Bean
    public Kiponos kiponos() {
        Kiponos client = Kiponos.createForCurrentTeam();
        // Profile: ['payments']['prod']['autoscale'] via -Dkiponos=... JVM arg
        client.afterValueChanged(change ->
            log.info("Autoscale guard delta: path={} value={}", change.path(), change.newValue())
        );
        return client;
    }
}

@Service
public class ReplicaCeilingGuard {
    private final Kiponos kiponos;

    public ReplicaCeilingGuard(Kiponos kiponos) {
        this.kiponos = kiponos;
    }

    public int clampDesired(String deployment, int hpaDesired) {
        var posture = kiponos.path("posture");
        if (posture.getBool("emergency_scale_freeze", false) && hpaDesired > currentReplicas(deployment)) {
            if (!posture.getBool("allow_manual_override", true)) {
                metrics.inc("autoscale_freeze_blocked", deployment);
                return currentReplicas(deployment);
            }
        }

        var guardrails = kiponos.path("guardrails");
        int floor = resolveMinFloor(deployment, guardrails);
        int ceiling = resolveMaxCeiling(deployment, guardrails);

        int clamped = Math.min(Math.max(hpaDesired, floor), ceiling);
        if (clamped < hpaDesired && guardrails.getBool("enforce_ceiling", true)) {
            if (kiponos.path("audit").getBool("emit_clamp_metrics", true)) {
                metrics.inc("autoscale_ceiling_clamped", deployment);
            }
        }
        return clamped;
    }

    private int resolveMaxCeiling(String deployment, Kiponos.Path guardrails) {
        var dep = kiponos.path("deployments", deployment);
        if (dep.exists() && dep.getBool("enabled", true)) {
            return dep.getInt("max_replicas_ceiling", guardrails.getInt("default_ceiling", 60));
        }
        return guardrails.getInt("max_replicas_ceiling", 60);
    }

    private int resolveMinFloor(String deployment, Kiponos.Path guardrails) {
        var dep = kiponos.path("deployments", deployment);
        if (dep.exists() && dep.getBool("enabled", true)) {
            return dep.getInt("min_replicas_floor", guardrails.getInt("min_replicas_floor", 12));
        }
        return guardrails.getInt("min_replicas_floor", 12);
    }
}

@Service
public class AutoscaleMetricsLoop {
    private final ReplicaCeilingGuard guard;
    private final Kiponos kiponos;

    public void onMetricsTick(String deployment, int hpaDesired) {
        int clamped = guard.clampDesired(deployment, hpaDesired);
        var rates = kiponos.path("rates");
        int maxUp = rates.getInt("max_scale_up_per_tick", 4);
        int current = currentReplicas(deployment);
        int delta = clamped - current;
        if (delta > maxUp) {
            clamped = current + maxUp;
        }
        applyDesiredReplicas(deployment, clamped);
    }
}

Every getInt() and getBool() on the autoscale path is O(1) local cache — microseconds, not cross-region config service RTT.

Replica counts and node types stay in Kubernetes and Terraform — Kiponos owns the ceilings that change when finance rings the alarm.

Real scenarios

Event	Without Kiponos	With Kiponos
Black Friday — SRE lowers max replica ceiling while keeping min stable	Helm PR + rolling restart; 20+ extra pods during merge window	Dashboard: `deployments/payments-capture/max_replicas_ceiling: 35` live
FinOps daily burn exceeds forecast	Manual cordon/drain nodes; risky	Lower global `guardrails/max_replicas_ceiling`; HPA clamps on next tick
Fraud service needs headroom; capture must cap	Two deploy tracks fighting	Per-deployment subtree — fraud ceiling 45, capture 35
Runaway scale-up loop after bad metric	Emergency HPA suspend via kubectl	`posture/emergency_scale_freeze: true` from dashboard
Post-peak restore	Second Helm PR	Reset ceilings and posture in one edit

Performance — hot path on the autoscale evaluation loop

Ceiling read per tick — getInt() is in-memory tree lookup; no HTTP on the scale path
Per-deployment nesting — capture, fraud, settlement each get a folder; no flat key sprawl
Delta updates — changing one deployment ceiling sends one patch, not a full config document
Freeze posture flip — one boolean blocks scale-out cluster-wide; no kubectl script per service
One WebSocket per JVM — background sync; metrics loop never blocks on config API RTT
Complements GitOps baseline — Helm still owns chart defaults; Kiponos owns incident overrides

Compare to alternatives

Approach	Mid-event ceiling lower	Per-deployment caps	Emergency freeze
Helm maxReplicas only	Poor — PR + rollout	Medium — values files per chart	kubectl patch HPA
Custom operator + CRD	Good — but cluster change	Good	CRD apply latency
Redis hash of ceilings	Extra RTT or stale cache	Medium — key sprawl	Separate flag keys
Spring Cloud Config	Network + `@RefreshScope`	Medium	Slow during incident
Spreadsheet + human	N/A	Humans edit; guard unchanged	Bridge chaos
Kiponos live hub	Seconds — dashboard delta	Per-deployment subtree	One posture boolean

When not to use Kiponos

Case	Use instead
Node pool instance types and ASG definitions	Terraform / cluster autoscaler
HPA metric targets and CPU thresholds	Kubernetes manifests — Git-reviewed
IAM roles and service account bindings	Cloud IAM / GitOps
Immutable cloud invoice line items	Billing warehouse — not live config
One-time bootstrap min/max from capacity planning	Helm values at deploy time

Getting started (15 minutes)

Sign up at kiponos.io (TeamPro).
Create profile path ['payments']['prod']['autoscale'].
Add guardrails/max_replicas_ceiling, deployments/payments-capture/max_replicas_ceiling, and wire ReplicaCeilingGuard.clampDesired() in your metrics loop.
./gradlew bootRun — confirm log shows WebSocket handshake.
Lower deployments/payments-capture/max_replicas_ceiling in dashboard; watch next HPA tick clamp without JVM restart.
Drill: flip posture/emergency_scale_freeze during staging; confirm scale-out blocks.

DEV Community