DEV Community: Olivier Bourgeois

Scaling stateful Gemini 4 Argon agent sessions on GKE: Connection draining, startup boost, and session persistence

Olivier Bourgeois — Thu, 22 Oct 2026 16:05:20 +0000

Most Kubernetes architectures are built around a comfortable assumption: HTTP requests arrive, execute stateless logic in under 200 milliseconds, and return a response. Multi-turn Gemini 4 Argon agent workflows invert every part of that profile. A single user request can initiate a 3-minute bidirectional streaming session where the agent plans, executes external tools, ingests multi-megabyte documents into its context window, and streams intermediate reasoning steps back to the client.

When you deploy these long-running, stateful agent orchestrators onto Google Kubernetes Engine (GKE) using default web-service configurations, three production failure modes appear almost immediately:

Startup CPU throttling: Python and Node.js agent runtimes that initialize heavy AI SDKs, OpenTelemetry instrumentation, and local tokenizer caches spike CPU during boot, causing new Pods to fail readiness probes or stall Horizontal Pod Autoscaler (HPA) scale-out events.
Dropped in-flight reasoning loops during rollouts: Default 30-second load balancer timeouts and abrupt SIGTERM shutdowns sever active WebSocket or Server-Sent Events (SSE) streams mid-turn when GKE scales down or rolls out a new image.
Lost conversation and checkpoint state: Storing multi-turn reasoning history purely in Pod memory means any container restart forces the agent to replay expensive Gemini 4 Argon token contexts from scratch.

In this article, I will show you how to architect GKE for stateful Gemini 4 Argon agent runtimes by combining VPA CPU Startup Boost, long-horizon connection draining via GCPBackendPolicy, and external state checkpointing.

The lifecycle of a stateful Gemini 4 Argon agent Pod

An agent runtime Pod traverses three distinct operational phases across its lifecycle:

Admission and cold boot (0s to 15s): GKE's VPA admission webhook injects a temporary CPU Startup Boost so the container initializes AI SDKs, tokenizers, and tracing pipelines without CPU throttling. Once it passes its readinessProbe, GKE uses In-Place Pod Resize (IPPR) to step CPU down to baseline without restarting the container.
Steady-state streaming (up to 10 minutes): The GKE Gateway routes long-lived SSE or WebSocket streams with a 600-second backend timeout and cookie-based session affinity. The worker executes multi-turn tool loops with Gemini 4 Argon, checkpointing turn state to Cloud Memorystore.
Graceful shutdown and draining: On rolling updates or scale-down events, a preStop hook and a 300-second termination grace period allow active reasoning loops to finish and flush final checkpoints before the container exits.

Each GKE primitive addresses a specific stage of this lifecycle:

Operational challenge	Default Kubernetes behavior	GKE production pattern for Gemini 4 Argon agents
Heavy SDK initialization	Container throttled by baseline CFS CPU quota during boot	VPA CPU startup boost (`updateMode: "Off"`) grants temporary CPU headroom and resizes down live via IPPR after readiness
Long streaming sessions	Load balancer closes connection at 30 seconds	`GCPBackendPolicy` sets `timeoutSec: 600` and `drainingTimeoutSec: 300` with cookie-based session affinity
Pod termination during rollout	`SIGTERM` followed by `SIGKILL` after 30 seconds	`terminationGracePeriodSeconds: 300` paired with a graceful drain handler that completes active Gemini 4 Argon turns and checkpoints state

Eliminating cold-start throttling with VPA CPU startup boost

Agent orchestrator images frequently bundle the google-genai SDK, gRPC C-extensions, Pydantic schema validators, and document parsing libraries. If you size your container's baseline CPU request for steady-state I/O waiting (where the Pod spends most of its time awaiting network streams from the Gemini platform), the container only needs 500m CPU at steady state, but requires 1500m CPU during the first 10 seconds of startup.

Using VPA CPU startup boost (available in Public Preview on GKE 1.36.0-gke.4447000 and later in the Rapid Channel), GKE temporarily boosts the container's CPU request at admission and uses Kubernetes In-Place Pod Resize (IPPR) to lower the CPU allocation back to 500m without restarting the container once it passes its readinessProbe.

GKE Autopilot vs. GKE Standard sizing considerations

GKE Autopilot: Autopilot validates CPU-to-memory ratios at admission. Ensure your Pod's memory request supports the boosted CPU value (on the general-purpose Autopilot compute platform, memory must fall between 1 GiB and 8 GiB per vCPU), or assign the workload to a Performance ComputeClass.
GKE Standard: Ensure your worker nodes have sufficient allocatable CPU headroom to accommodate the temporary startup boost factor.

Create a VerticalPodAutoscaler policy named agent-runtime-startup-boost.yaml that triples the baseline 500m CPU allocation to 1500m during boot and unboosts immediately (durationSeconds: 0) upon readiness. Setting updateMode: "Off" applies the startup boost and in-place downscale without enabling continuous VPA actuation, avoiding conflicts with Horizontal Pod Autoscaler (HPA):

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: gemini-agent-startup-boost
  namespace: agent-runtime
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: gemini-session-orchestrator
  updatePolicy:
    updateMode: "Off"
  startupBoost:
    cpu:
      type: Factor
      factor: 3
      durationSeconds: 0

Configuring long-lived streaming and graceful termination

To prevent GKE from severing active Gemini 4 Argon streaming sessions during a rolling deployment, you need to coordinate timeouts across two layers: the Kubernetes Pod lifecycle (terminationGracePeriodSeconds and preStop hooks) and the GKE Gateway load balancer (GCPBackendPolicy).

Save the following manifest as agent-session-deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: gemini-session-orchestrator
  namespace: agent-runtime
spec:
  replicas: 4
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 0
  selector:
    matchLabels:
      app: gemini-session-orchestrator
  template:
    metadata:
      labels:
        app: gemini-session-orchestrator
    spec:
      serviceAccountName: gemini-agent-sa
      terminationGracePeriodSeconds: 300
      containers:
      - name: agent-server
        image: us-central1-docker.pkg.dev/production-ai-platform/agents/gemini-4-session-server:v1.0.0
        ports:
        - containerPort: 8080
        env:
        - name: GOOGLE_CLOUD_PROJECT
          value: "production-ai-platform"
        - name: REDIS_HOST
          value: "10.128.0.45"
        resources:
          requests:
            cpu: "500m"
            memory: "2Gi"
          limits:
            memory: "2Gi"
        readinessProbe:
          httpGet:
            path: /healthz/ready
            port: 8080
          periodSeconds: 5
          failureThreshold: 2
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh", "-c", "sleep 10"]
---
apiVersion: v1
kind: Service
metadata:
  name: gemini-session-svc
  namespace: agent-runtime
spec:
  selector:
    app: gemini-session-orchestrator
  ports:
  - port: 80
    targetPort: 8080
---
apiVersion: networking.gke.io/v1
kind: GCPBackendPolicy
metadata:
  name: gemini-streaming-backend-policy
  namespace: agent-runtime
spec:
  default:
    timeoutSec: 600
    connectionDraining:
      drainingTimeoutSec: 300
    sessionAffinity:
      type: GENERATED_COOKIE
      cookieTtlSec: 3600
  targetRef:
    group: ""
    kind: Service
    name: gemini-session-svc

Notice three critical settings in this configuration:

maxUnavailable: 0: Ensures GKE brings new, startup-boosted Pods into a Ready state before terminating any existing Pod holding active sessions.
preStop sleep hook (sleep 10): Because Kubernetes endpoint removal and Google Cloud load balancer programming happen asynchronously with SIGTERM, sleeping for 10 seconds before the application stops accepting new connections prevents in-flight requests at the load balancer edge from hitting a closing socket.
timeoutSec: 600 and drainingTimeoutSec: 300: Allows streaming SSE or WebSocket responses from Gemini 4 Argon to run for up to 10 minutes and gives draining Pods 5 minutes to finish active turns.

Implementing per-turn state checkpointing and graceful drain

Even with a 300-second drain window, an agent session might span multiple user interactions over an hour, or a node preemption on Spot capacity might give only 30 seconds of warning. By checkpointing the conversation history after every Gemini 4 Argon turn into Redis (such as Memorystore for Redis), any replica in the Deployment can transparently resume the session if the original Pod terminates.

Here is the core session server (session_server.py) showing graceful SIGTERM tracking and per-turn Redis checkpointing with the google-genai SDK:

import asyncio
import json
import os
import signal
from contextlib import asynccontextmanager
from fastapi import FastAPI, HTTPException
from fastapi.responses import StreamingResponse
from pydantic import BaseModel
import redis.asyncio as redis
from google import genai
from google.genai import types

draining = False
active_turns = 0


class TurnRequest(BaseModel):
    user_message: str


def handle_sigterm(*_args):
    global draining
    draining = True


@asynccontextmanager
async def lifespan(_app: FastAPI):
    signal.signal(signal.SIGTERM, handle_sigterm)
    yield
    # Wait for active Gemini 4 Argon turns to finish and checkpoint before exiting
    while active_turns > 0:
        await asyncio.sleep(1)
    await redis_client.aclose()


app = FastAPI(lifespan=lifespan)
genai_client = genai.Client(
    vertexai=True,
    project=os.environ["GOOGLE_CLOUD_PROJECT"],
    location=os.environ.get("VERTEX_LOCATION", "us-central1"),
)
redis_client = redis.Redis(host=os.environ["REDIS_HOST"], port=6379, decode_responses=True)
MODEL_ID = os.environ.get("GEMINI_MODEL", "gemini-4-argon")


@app.get("/healthz/ready")
async def readiness_check():
    if draining:
        raise HTTPException(status_code=503, detail="Pod is draining active sessions")
    return {"status": "ready"}


@app.post("/v1/sessions/{session_id}/stream")
async def stream_agent_turn(session_id: str, request: TurnRequest):
    # Load prior conversation history from Redis checkpoint
    raw_history = await redis_client.get(f"session:{session_id}")
    history = json.loads(raw_history) if raw_history else []
    history.append({"role": "user", "parts": [{"text": request.user_message}]})

    contents = [
        types.Content(role=turn["role"], parts=[types.Part.from_text(text=p["text"]) for p in turn["parts"]])
        for turn in history
    ]

    async def event_generator():
        global active_turns
        active_turns += 1
        assembled_response = []
        try:
            async for chunk in await genai_client.aio.models.generate_content_stream(
                model=MODEL_ID,
                contents=contents,
            ):
                if chunk.text:
                    assembled_response.append(chunk.text)
                    yield f"data: {json.dumps({'text': chunk.text})}\n\n"

            # Checkpoint completed assistant turn back to Redis with a 24-hour TTL
            history.append({"role": "model", "parts": [{"text": "".join(assembled_response)}]})
            await redis_client.set(f"session:{session_id}", json.dumps(history), ex=86400)
        finally:
            active_turns -= 1

    return StreamingResponse(event_generator(), media_type="text/event-stream")

Verifying startup boost and connection draining

Once deployed, confirm that VPA CPU startup boost is accelerating Pod initialization and resizing CPU in place without restarting the container:

# Inspect the startup boost annotation injected on the Pod
kubectl get pods -n agent-runtime \
    -l app=gemini-session-orchestrator \
    -o custom-columns="NAME:.metadata.name,CPU_REQ:.spec.containers[0].resources.requests.cpu,BOOST_ANNOTATION:.metadata.annotations.vpaCpuStartupBoost\/agent-server"

# Watch for the InPlaceResizedByVPA event once the Pod passes readinessProbe
kubectl get events -n agent-runtime --field-selector reason=InPlaceResizedByVPA

Expected event output:

LAST SEEN   TYPE     REASON                 OBJECT                                             MESSAGE
14s         Normal   InPlaceResizedByVPA    pod/gemini-session-orchestrator-6b8f9c7d4-k29lm    Pod was in-place resized by VPA Updater (CPU: 1500m -> 500m)

To verify that rolling updates drain cleanly without dropping streams, trigger a rollout while streaming a multi-step Gemini 4 Argon response and monitor active backend connections in Cloud Monitoring using the metric loadbalancing.googleapis.com/https/backend_request_count alongside kubernetes.io/container/cpu/core_usage_time.

Operational trade-offs to watch

Memory growth on long context windows: While CPU usage drops to near zero while waiting on Gemini platform network streams, assembling 500K-token contexts in Python memory can spike heap allocations. VPA CPU Startup Boost only boosts CPU; size your container memory requests and limits to accommodate the largest concurrent payload your agent processes.
Avoid Spot VMs for long-running streaming gateways: Spot VMs (or spot: true in a GKE ComputeClass) only provide a 30-second preemption notice, which is too short for a 3-minute agent reasoning turn to drain gracefully. Run the lightweight session orchestrator Pods on standard on-demand capacity.
HPA scaling metric selection: Because steady-state agent orchestrators are I/O-bound rather than CPU-bound, scaling HPA solely on CPU utilization will under-scale during concurrent streaming bursts. Pair HPA with custom load balancer concurrency metrics or active session gauges exported to Cloud Monitoring.

Next steps

Running Gemini 4 Argon agents reliably in production requires adapting Kubernetes from short-lived request/response defaults to stateful, long-horizon execution. By combining VPA CPU Startup Boost for fast initialization, GCPBackendPolicy for long streaming timeouts, and per-turn state checkpointing, your agent deployments can scale and roll out new versions without interrupting active users.

To dive deeper into these GKE capabilities, read the VPA CPU Startup Boost guide, review the GKE Gateway GCPBackendPolicy reference, and check the Gemini API documentation on Google Cloud.

Multi-tier model routing on GKE: balancing Gemini 4 Argon reasoning depth with latency and cost

Olivier Bourgeois — Wed, 21 Oct 2026 18:56:20 +0000

Routing every incoming user prompt to a frontier reasoning model is the fastest way to blow through your token budget and degrade p99 tail latency. With the release of Gemini 4 Argon on the Gemini platform, engineering teams have access to deep multi-step reasoning in gemini-4-argon alongside high-throughput, low-latency execution in gemini-3.8-flash and open-weights models running directly on cluster accelerators.

In most production applications, workload complexity follows a steep power law. Roughly 70% of requests involve classification, summarization, schema extraction, or straightforward retrieval that a fast tier handles in a few hundred milliseconds. The remaining 30% require multi-hop synthesis, complex code generation, or ambiguous constraint solving where gemini-4-argon shines. Hardcoding a single model identifier in your application containers forces you to either overpay on simple queries or under-deliver on complex ones.

By building a centralized AI routing layer on Google Kubernetes Engine (GKE) using the GKE Gateway API, platform teams can classify prompt complexity at the ingress edge, route requests across Gemini tiers (gemini-4-argon and gemini-3.8-flash) and self-hosted GKE Inference Gateway pools, and enforce automatic fallback when rate limits occur.

In this article, I will break down the architecture of a multi-tier Gemini 4 Argon router on GKE, configure GKE Standard and Autopilot infrastructure with Workload Identity Federation for GKE, and walk through the routing, health-check, and observability configurations.

Why static model bindings fail in production

When application teams integrate directly with LLM endpoints inside dozens of individual microservices, three operational problems surface quickly:

Unbounded token spend and latency inflation: A microservice calling gemini-4-argon for routine JSON formatting pays frontier token pricing and waits for deep reasoning chains when gemini-3.8-flash would produce an identical schema in a fraction of the time.
Scattered credential and quota management: Managing Gemini platform API quotas, retry backoffs, and regional failover across 30 separate application repositories leads to inconsistent behavior during traffic spikes.
No path to hybrid in-cluster offload: When high-volume, low-complexity tasks could be served at fixed cost by an open-weights model running on GKE GPU or TPU node pools, application code tightly coupled to a single model string cannot shift traffic without code changes and redeployments.

Centralizing model routing behind a GKE Gateway and an intelligent router service decouples client microservices from the underlying model tier.

Multi-tier routing architecture on GKE

In this architecture, internal microservices send standard requests to a regional internal GKE Gateway (gke-l7-rilb). The Gateway forwards traffic to an ai-model-router service that evaluates request metadata or runs a fast complexity check before dispatching the request to one of three model tiers:

Tier 1 (Frontier reasoning): High-complexity tasks, code synthesis, and multi-step planning route to gemini-4-argon on the Gemini platform.
Tier 2 (Low-latency interactive): Routine summarization, classification, and structured extraction route to gemini-3.8-flash on the Gemini platform.
Tier 3 (High-throughput local): High-volume batch transformations and strict data locality workloads route to an in-cluster vLLM pool running Gemma 4 via GKE Inference Gateway.

Here is how the three tiers compare across production operating dimensions:

Routing tier	Target model	Primary workload profile	Fallback path on `429` / timeout
Tier 1: Frontier reasoning	`gemini-4-argon` (Gemini platform)	Multi-step planning, complex code synthesis, ambiguous root-cause analysis	`us` multi-region endpoint, then graceful downgrade to `gemini-3.8-flash`
Tier 2: Low-latency interactive	`gemini-3.8-flash` (Gemini platform)	Real-time chat, structured extraction, classification, single-turn tool calls	`us` multi-region endpoint, then in-cluster open-weights pool
Tier 3: High-throughput local	Open-weights (Gemma 4 via vLLM on GKE)	High-volume entity tagging, PII scrubbing, fixed-cost background jobs	Burst overflow to `gemini-3.8-flash` on the Gemini platform

The sample router later in this article implements the Tier 1 and Tier 2 location failover and the Tier 1 downgrade. The paths into and out of the in-cluster Tier 3 pool depend on your Inference Gateway setup and are left to your own dispatch logic.

Configuring Workload Identity and GKE compute

To call the Gemini platform without distributing static service account keys, bind the ai-model-router Kubernetes ServiceAccount directly to an IAM role using Workload Identity Federation for GKE. This works identically across GKE Standard and GKE Autopilot clusters running GKE 1.32 or later with the GKE Gateway controller enabled (--gateway-api=standard) and an active regional proxy-only subnet (--purpose=REGIONAL_MANAGED_PROXY) in your VPC.

Note on model identifiers and locations: The examples below use gemini-4-argon (Gemini 4 Argon) and gemini-3.8-flash (Gemini 3.8 Flash) configured via environment variables (PRO_MODEL and FLASH_MODEL). Both models are served from the global endpoint and the us and eu multi-region endpoints, not from individual regions such as us-central1, so the router fails over from global to us.

export PROJECT_ID="production-ai-platform"
export PROJECT_NUMBER=$(gcloud projects describe "${PROJECT_ID}" --format="value(projectNumber)")
export LOCATION="us-central1"
export CLUSTER_NAME="ai-gateway-cluster"

# Point kubectl at the cluster
gcloud container clusters get-credentials "${CLUSTER_NAME}" --location="${LOCATION}" --project="${PROJECT_ID}"

# Create the namespace and Kubernetes ServiceAccount
kubectl create namespace ai-routing
kubectl create serviceaccount gemini-router-sa --namespace=ai-routing

# Grant the Kubernetes ServiceAccount permission to invoke Gemini platform models
# (the underlying IAM role string is still roles/aiplatform.user)
gcloud projects add-iam-policy-binding "${PROJECT_ID}" \
    --member="principal://iam.googleapis.com/projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/${PROJECT_ID}.svc.id.goog/subject/ns/ai-routing/sa/gemini-router-sa" \
    --role="roles/aiplatform.user" \
    --condition=None

Provisioning capacity on GKE Autopilot vs. Standard

For the router Deployment itself, standard CPU compute is sufficient. If you also operate Tier 3 (in-cluster open-weights inference via GKE Inference Gateway for high-volume offload), configure your accelerator capacity according to your cluster mode:

GKE Autopilot with ComputeClass: Define a declarative ComputeClass (cloud.google.com/v1) that requests NVIDIA L4 GPUs (gpu.type: nvidia-l4, gpu.count: 2) with prioritized fallback from on-demand capacity to Spot VMs during regional stockouts.
GKE Standard node pools: Create a dedicated GPU node pool (gcloud container node-pools create vllm-l4-pool --cluster="${CLUSTER_NAME}" --location="${LOCATION}" --machine-type=g2-standard-24 --accelerator=type=nvidia-l4,count=2,gpu-driver-version=latest --enable-autoscaling --min-nodes=1 --max-nodes=6).

Implementing the Gemini 4 tier router

The ai-model-router service inspects incoming requests, estimates task complexity (using explicit client headers, token count thresholds, or keyword triggers), and dispatches the call through the google-genai SDK with cross-location failover and cross-tier fallback. It also exposes a /healthz endpoint that both the Pod readinessProbe and the GKE Gateway load balancer health check use to verify backend readiness.

Save the following implementation as router_app.py:

import os
import time
from fastapi import FastAPI, Header, HTTPException
from pydantic import BaseModel
from google import genai
from google.genai import types
from google.genai.errors import APIError

app = FastAPI(title="GKE Gemini 4 Argon Multi-Tier Router")

PROJECT_ID = os.environ["GOOGLE_CLOUD_PROJECT"]
PRIMARY_LOCATION = os.environ.get("PRIMARY_LOCATION", "global")
SECONDARY_LOCATION = os.environ.get("SECONDARY_LOCATION", "us")
PRO_MODEL = os.environ.get("PRO_MODEL", "gemini-4-argon")
FLASH_MODEL = os.environ.get("FLASH_MODEL", "gemini-3.8-flash")

# Project-scoped Gemini platform clients authenticated through Workload Identity Federation for GKE.
# The vertexai=True flag is the SDK switch for project-scoped IAM auth instead of an API key.
primary_client = genai.Client(vertexai=True, project=PROJECT_ID, location=PRIMARY_LOCATION)
secondary_client = genai.Client(vertexai=True, project=PROJECT_ID, location=SECONDARY_LOCATION)

REASONING_KEYWORDS = {"architect", "debug", "prove", "multi-step", "optimize", "trade-off", "root cause"}


class PromptRequest(BaseModel):
    prompt: str
    system_instruction: str | None = None
    require_deep_reasoning: bool = False


def select_model_tier(req: PromptRequest, routing_hint: str | None) -> str:
    """Selects the Gemini 4 Argon (frontier) or Gemini 3.8 Flash tier based on client hints and prompt characteristics."""
    if routing_hint == "pro" or req.require_deep_reasoning:
        return PRO_MODEL
    if routing_hint == "flash":
        return FLASH_MODEL

    prompt_lower = req.prompt.lower()
    word_count = len(req.prompt.split())
    has_reasoning_trigger = any(kw in prompt_lower for kw in REASONING_KEYWORDS)

    if word_count > 600 or has_reasoning_trigger:
        return PRO_MODEL
    return FLASH_MODEL


def build_config(model: str, system_instruction: str | None) -> types.GenerateContentConfig:
    """Gemini 4 Argon ignores sampling parameters and defaults to the MEDIUM thinking level,
    so temperature is only set on the Flash tier."""
    if model == PRO_MODEL:
        return types.GenerateContentConfig(system_instruction=system_instruction)
    return types.GenerateContentConfig(system_instruction=system_instruction, temperature=0.2)


@app.get("/healthz")
def health_check():
    return {"status": "ok"}


@app.post("/v1/generate")
def generate_response(
    req: PromptRequest,
    x_model_tier: str | None = Header(default=None),
):
    target_model = select_model_tier(req, x_model_tier)
    start_time = time.monotonic()

    # Attempt the global endpoint, then the us multi-region endpoint, then downgrade to Flash
    # unless the caller explicitly required deep reasoning
    attempt_chain = [
        (primary_client, PRIMARY_LOCATION, target_model),
        (secondary_client, SECONDARY_LOCATION, target_model),
    ]
    explicit_pro = x_model_tier == "pro" or req.require_deep_reasoning
    if target_model == PRO_MODEL and not explicit_pro:
        attempt_chain.append((primary_client, PRIMARY_LOCATION, FLASH_MODEL))

    for attempt_client, location, candidate_model in attempt_chain:
        try:
            response = attempt_client.models.generate_content(
                model=candidate_model,
                contents=req.prompt,
                config=build_config(candidate_model, req.system_instruction),
            )
            latency_ms = round((time.monotonic() - start_time) * 1000, 2)
            usage = response.usage_metadata
            return {
                "model_used": candidate_model,
                "location": location,
                "fallback_triggered": (location != PRIMARY_LOCATION or candidate_model != target_model),
                "latency_ms": latency_ms,
                "usage": {
                    "prompt_tokens": (usage.prompt_token_count or 0) if usage else 0,
                    "output_tokens": (usage.candidates_token_count or 0) if usage else 0,
                    "thoughts_tokens": (usage.thoughts_token_count or 0) if usage else 0,
                },
                "output": response.text,
            }
        except APIError as exc:
            if exc.code in (429, 500, 503, 504):
                continue
            raise HTTPException(status_code=exc.code or 500, detail=str(exc))

    raise HTTPException(status_code=503, detail="All Gemini endpoints exhausted")

Exposing the router via GKE Gateway API and GCPBackendPolicy

Because gemini-4-argon reasoning chains on large contexts can take 30 to 90 seconds to complete, standard 30-second HTTP load balancer timeouts will prematurely drop connections and cause clients to retry expensive requests.

Configure a GKE Gateway, HTTPRoute, and GCPBackendPolicy (networking.gke.io/v1) to extend the backend timeout to 180 seconds and enable connection draining during rolling updates. Unlike GKE Ingress, GKE Gateway does not infer load balancer health checks from the Pod readinessProbe; it probes / by default, which the router answers with a 404. Add a HealthCheckPolicy (networking.gke.io/v1) that points the load balancer at /healthz:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ai-model-router
  namespace: ai-routing
spec:
  replicas: 3
  selector:
    matchLabels:
      app: ai-model-router
  template:
    metadata:
      labels:
        app: ai-model-router
    spec:
      serviceAccountName: gemini-router-sa
      containers:
      - name: router
        image: us-central1-docker.pkg.dev/production-ai-platform/ai-routing/gemini-4-router:v1.0.0
        ports:
        - containerPort: 8080
        env:
        - name: GOOGLE_CLOUD_PROJECT
          value: "production-ai-platform"
        - name: PRIMARY_LOCATION
          value: "global"
        - name: SECONDARY_LOCATION
          value: "us"
        - name: PRO_MODEL
          value: "gemini-4-argon"
        - name: FLASH_MODEL
          value: "gemini-3.8-flash"
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 10
        resources:
          requests:
            cpu: "500m"
            memory: "512Mi"
---
apiVersion: v1
kind: Service
metadata:
  name: ai-model-router-svc
  namespace: ai-routing
spec:
  selector:
    app: ai-model-router
  ports:
  - port: 80
    targetPort: 8080
---
apiVersion: networking.gke.io/v1
kind: GCPBackendPolicy
metadata:
  name: gemini-router-backend-policy
  namespace: ai-routing
spec:
  default:
    timeoutSec: 180
    connectionDraining:
      drainingTimeoutSec: 60
  targetRef:
    group: ""
    kind: Service
    name: ai-model-router-svc
---
apiVersion: networking.gke.io/v1
kind: HealthCheckPolicy
metadata:
  name: gemini-router-healthcheck
  namespace: ai-routing
spec:
  default:
    checkIntervalSec: 10
    timeoutSec: 5
    healthyThreshold: 1
    unhealthyThreshold: 3
    config:
      type: HTTP
      httpHealthCheck:
        port: 8080
        requestPath: /healthz
  targetRef:
    group: ""
    kind: Service
    name: ai-model-router-svc
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: internal-ai-gateway
  namespace: ai-routing
spec:
  gatewayClassName: gke-l7-rilb
  listeners:
  - name: http
    protocol: HTTP
    port: 80
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: gemini-routing-route
  namespace: ai-routing
spec:
  parentRefs:
  - name: internal-ai-gateway
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /v1/generate
    backendRefs:
    - name: ai-model-router-svc
      port: 80

If you also run an in-cluster vLLM pool managed by GKE Inference Gateway (InferencePool in inference.networking.k8s.io/v1 and InferenceObjective in inference.networking.x-k8s.io for workload prioritization), you can attach an additional HTTPRoute rule on the same internal-ai-gateway (or dispatch internally from router_app.py) to route high-volume batch traffic directly to group: inference.networking.k8s.io, kind: InferencePool.

Verifying the Gateway and routing behavior

After applying the manifests, verify that the GKE Gateway controller has programmed the internal load balancer and attached both backend policies:

# Confirm Gateway programmed status and internal IP address
kubectl get gateway internal-ai-gateway -n ai-routing

# Verify that both policies are attached to the Service (look for Reason: Attached)
kubectl describe gcpbackendpolicy gemini-router-backend-policy -n ai-routing
kubectl describe healthcheckpolicy gemini-router-healthcheck -n ai-routing

Because gke-l7-rilb exposes only an internal IP address, look up the address with kubectl, then send the test requests from a client in the same VPC and region (for example, a debug Pod in the cluster):

export GATEWAY_IP=$(kubectl get gateway internal-ai-gateway -n ai-routing -o jsonpath='{.status.addresses[0].value}')

# 1. Fast classification query (routes automatically to gemini-3.8-flash)
curl -s -X POST "http://${GATEWAY_IP}/v1/generate" \
    -H "Content-Type: application/json" \
    -d '{"prompt": "Extract the HTTP status code from this log line: GET /checkout 502 Bad Gateway"}' | jq .

# 2. Complex architectural query (routes automatically to gemini-4-argon)
curl -s -X POST "http://${GATEWAY_IP}/v1/generate" \
    -H "Content-Type: application/json" \
    -d '{"prompt": "Analyze the root cause and trade-offs of etcd leader election timeouts during cross-zone network partitions."}' | jq .

Expected response showing the selected tier, location, fallback status, and token accounting (including reasoning thoughts_tokens):

{
  "model_used": "gemini-4-argon",
  "location": "global",
  "fallback_triggered": false,
  "latency_ms": 2840.15,
  "usage": {
    "prompt_tokens": 24,
    "output_tokens": 412,
    "thoughts_tokens": 186
  },
  "output": "..."
}

Operational caveats and when not to use a router

Before introducing a model router in front of Gemini 4 Argon, weigh these operational trade-offs:

Classifier accuracy vs. quality regressions: Heuristic or embedding-based routers can misclassify terse prompts that actually require deep reasoning (for example, a short mathematical puzzle). Always expose an explicit override header (X-Model-Tier: pro) so latency-tolerant or high-stakes workflows bypass heuristics completely.
Multi-turn conversation state: Switching model tiers mid-conversation can alter output formatting or tool-calling behavior. For stateful chat sessions, pin the model tier selected on the first turn or evaluate tier upgrades only when a tool execution fails.
When not to use a routing layer: If you operate a single application with predictable prompt complexity and uniform latency targets, calling the Gemini platform directly from the application SDK with built-in retry policies avoids maintaining an extra hop in your cluster.

Next steps

Decoupling your microservices from hardcoded model strings lets you adopt Gemini 4 Argon where its reasoning capabilities matter most while keeping routine queries fast and cost-effective on gemini-3.8-flash or in-cluster accelerators.

To expand this architecture, review the GKE Gateway API documentation, explore the GKE Inference Gateway guide for routing to self-hosted open models, and check the Gemini platform model locations and quota documentation for multi-region availability and throughput configuration.

Automated incident triage on GKE with Gemini 4 Argon and Model Context Protocol: Correlating cluster events and VPA logs

Olivier Bourgeois — Tue, 20 Oct 2026 19:14:39 +0000

When an on-call engineer gets paged at 2 AM for a degraded payment service, the hardest part of incident triage is rarely fixing the YAML manifest. Instead, the bottleneck is correlating telemetry across three different control loops: transient Kubernetes Pod events that expire after an hour, application stack traces in Cloud Logging, and vertical autoscaling actions emitted by the control plane.

Consider a common failure mode on Google Kubernetes Engine (GKE): a latency spike triggers an out-of-memory (OOMKilled) termination while the Vertical Pod Autoscaler (VPA) attempts an in-place resize (InPlaceOrRecreate) that gets capped by node capacity or GKE Autopilot compute ratio constraints. Diagnosing that sequence manually requires stitching together kubectl describe pod, container exit codes, and container.googleapis.com/vpa-controller decision logs.

With the release of Gemini 4 Argon and the open-source Google Kubernetes Engine MCP server, platform teams can deploy an in-cluster incident triage agent that autonomously queries Kubernetes state, inspects KCP_VPA control plane logs, identifies the root cause, and generates a verified remediation patch.

In this article, I will walk through the architecture of a read-only Gemini 4 Argon incident triage agent on GKE, show how to configure least-privilege RBAC and Workload Identity Federation for GKE, and demonstrate how the agent correlates live cluster telemetry through the Model Context Protocol (MCP).

Why incident triage requires multi-hop tool reasoning

Single-turn log summarization falls short during real Kubernetes incidents because the first symptom is almost never the root cause. A CrashLoopBackOff status on a Pod tells you only that the kubelet is delaying restarts; it does not tell you whether the failure stems from an expired secret, a failed readiness probe during JVM initialization, or a memory limit that VPA could not resize in place.

Gemini 4 Argon's long-horizon tool calling allows an agent to execute a structured diagnostic workflow across multiple MCP tools before forming a hypothesis:

Trigger ingest: An incoming alert webhook from Cloud Monitoring delivers the initial incident context, affected namespace, and alert threshold to the in-cluster triage agent.
Kubernetes API inspection: The agent calls list_k8s_events and get_k8s_resource through the GKE MCP server to pull active events, Pod termination reasons, and deployment spec history.
Workload telemetry: The agent queries container stdout/stderr logs via get_k8s_logs to capture application stack traces and unhandled exceptions.
Control plane log correlation: The agent queries query_vpa_decision_logs against Cloud Logging to evaluate KCP_VPA decisions, checking whether autoscaling recommendations were capped or throttled.
Remediation synthesis: The agent compiles a root-cause report and generates a suggested YAML diff delivered to an on-call Slack channel or GitOps pull request for human review.

To keep the cluster safe from unintended mutations during an active outage, the triage agent operates under a strict separation of duties:

Operational phase	Access level	Mechanism	Security boundary
Telemetry discovery	Read-only (`get`, `list`, `watch`)	GKE MCP server + Kubernetes `ClusterRole`	Denies `create`, `patch`, `delete`, and access to `Secret` payloads
Control plane log correlation	Read-only (`roles/logging.viewer`)	Cloud Logging API via Workload Identity	Scoped to `k8s_container`, `k8s_pod`, and `vpa-controller` logs
Remediation proposal	Zero direct cluster write access	Structured Markdown report and YAML diff	Requires human approval or GitOps pipeline validation before apply

Preparing the cluster and control plane logging

For the triage agent to inspect vertical autoscaling decisions alongside standard workload logs, enable VPA decision logging (KCP_VPA, available in Public Preview on GKE 1.36.0-gke.1601000 and later) on your GKE Autopilot or GKE Standard cluster. On GKE Autopilot clusters, Vertical Pod Autoscaling is enabled by default; on GKE Standard clusters, ensure --enable-vertical-pod-autoscaling is also enabled.

When updating --logging on an existing cluster, always include your existing logging components (such as SYSTEM and WORKLOAD) so you do not overwrite active settings:

export PROJECT_ID="production-ai-platform"
export PROJECT_NUMBER=$(gcloud projects describe "${PROJECT_ID}" --format="value(projectNumber)")
export LOCATION="us-central1"
export CLUSTER_NAME="commerce-production-cluster"

# Check currently enabled logging components first
gcloud container clusters describe "${CLUSTER_NAME}" \
    --location="${LOCATION}" \
    --project="${PROJECT_ID}" \
    --format="value(loggingConfig.componentConfig.enableComponents)"

# Enable SYSTEM, WORKLOAD, and KCP_VPA logs
gcloud container clusters update "${CLUSTER_NAME}" \
    --location="${LOCATION}" \
    --project="${PROJECT_ID}" \
    --logging=SYSTEM,WORKLOAD,KCP_VPA

Hardening RBAC and Workload Identity for the triage agent

Never grant an AI agent cluster-admin permissions or read access to Kubernetes Secret objects. Create a dedicated sre-triage namespace and a ClusterRole that grants read-only visibility into workloads, events, HPA/VPA policies, and custom compute classes while explicitly excluding secrets.

Save the following manifest as triage-agent-rbac.yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: sre-triage
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: gemini-triage-sa
  namespace: sre-triage
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: gemini-readonly-triage-role
rules:
- apiGroups: [""]
  resources: ["pods", "pods/log", "events", "services", "nodes", "namespaces", "configmaps"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["events.k8s.io"]
  resources: ["events"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
  resources: ["deployments", "statefulsets", "daemonsets", "replicasets"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["autoscaling"]
  resources: ["horizontalpodautoscalers"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["autoscaling.k8s.io"]
  resources: ["verticalpodautoscalers"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["cloud.google.com"]
  resources: ["computeclasses"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: gemini-readonly-triage-binding
subjects:
- kind: ServiceAccount
  name: gemini-triage-sa
  namespace: sre-triage
roleRef:
  kind: ClusterRole
  name: gemini-readonly-triage-role
  apiGroup: rbac.authorization.k8s.io

Apply the RBAC manifest and bind the gemini-triage-sa ServiceAccount to the Gemini platform, GKE Viewer, and Cloud Logging Viewer IAM roles via Workload Identity Federation:

kubectl apply -f triage-agent-rbac.yaml

for ROLE in roles/aiplatform.user roles/container.viewer roles/logging.viewer; do
  gcloud projects add-iam-policy-binding "${PROJECT_ID}" \
      --member="principal://iam.googleapis.com/projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/${PROJECT_ID}.svc.id.goog/subject/ns/sre-triage/sa/gemini-triage-sa" \
      --role="${ROLE}" \
      --condition=None
done

Implementing the Gemini 4 Argon + MCP incident triage loop

The triage agent connects Gemini 4 Argon (gemini-4-argon) to an in-process gke-mcp session alongside a dedicated tool for querying container.googleapis.com/vpa-controller decision logs in Cloud Logging.

Here is the core implementation (triage_agent.py):

import asyncio
import os
from google import genai
from google.genai import types
from google.cloud import logging as cloud_logging
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client

PROJECT_ID = os.environ["GOOGLE_CLOUD_PROJECT"]
LOCATION = os.environ.get("VERTEX_LOCATION", "us-central1")
MODEL_ID = os.environ.get("GEMINI_MODEL_ID", "gemini-4-argon")

genai_client = genai.Client(vertexai=True, project=PROJECT_ID, location=LOCATION)
logging_client = cloud_logging.Client(project=PROJECT_ID)


def query_vpa_decision_logs(
    cluster_name: str, namespace: str, workload_name: str, limit: int = 10
) -> list[dict]:
    """Queries GKE KCP_VPA control plane logs for a specific workload to inspect resize or eviction decisions.

    Args:
        cluster_name: Name of the GKE cluster.
        namespace: Kubernetes namespace of the target workload.
        workload_name: Name of the target Deployment or workload.
        limit: Maximum number of log entries to return.
    """
    log_filter = (
        'resource.type="k8s_control_plane_component" '
        f'AND resource.labels.cluster_name="{cluster_name}" '
        f'AND logName="projects/{PROJECT_ID}/logs/container.googleapis.com%2Fvpa-controller" '
        f'AND jsonPayload.target.namespace="{namespace}" '
        f'AND jsonPayload.target.name="{workload_name}"'
    )
    entries = logging_client.list_entries(
        filter_=log_filter,
        order_by=cloud_logging.DESCENDING,
        max_results=limit,
    )
    results = []
    for entry in entries:
        payload = entry.payload if isinstance(entry.payload, dict) else {}
        results.append({
            "timestamp": str(entry.timestamp),
            "operation": payload.get("operation"),
            "state": payload.get("state"),
            "reason": payload.get("reason"),
            "confidence": payload.get("confidence"),
        })
    return results


async def investigate_workload_incident(cluster_name: str, namespace: str, workload_name: str) -> str:
    """Connects Gemini to the GKE MCP server and Cloud Logging to diagnose a failing workload."""
    server_params = StdioServerParameters(
        command="gke-mcp",
        args=[],
        env=os.environ.copy(),
    )

    async with stdio_client(server_params) as (read_stream, write_stream):
        async with ClientSession(read_stream, write_stream) as mcp_session:
            await mcp_session.initialize()

            prompt = (
                f"Investigate the degraded Deployment '{workload_name}' in namespace '{namespace}' "
                f"on GKE cluster '{cluster_name}'. "
                "1. List recent Kubernetes events and inspect the Deployment and Pod statuses. "
                "2. Check the container termination exit codes and recent Pod logs. "
                "3. Query VPA decision logs using query_vpa_decision_logs to see if an in-place resize "
                "or eviction occurred or was skipped. "
                "4. Provide a concise root-cause timeline and a git-ready YAML patch."
            )

            config = types.GenerateContentConfig(
                system_instruction=(
                    "You are a senior GKE Site Reliability Engineer. Always verify Pod status, exit codes, "
                    "Kubernetes events, and VPA decision logs before proposing a fix. Never guess when "
                    "telemetry can be queried via your tools."
                ),
                tools=[mcp_session, query_vpa_decision_logs],
                automatic_function_calling=types.AutomaticFunctionCallingConfig(
                    maximum_remote_calls=8
                ),
                temperature=0.1,
            )

            response = await genai_client.aio.models.generate_content(
                model=MODEL_ID,
                contents=prompt,
                config=config,
            )
            return response.text

By passing both mcp_session and query_vpa_decision_logs into tools, the google-genai SDK automatically discovers the GKE MCP server tools (list_k8s_events, get_k8s_resource, get_k8s_logs) and orchestrates the multi-step investigation up to the configured maximum_remote_calls limit. Passing env=os.environ.copy() to StdioServerParameters ensures the gke-mcp subprocess inherits in-cluster Kubernetes service environment variables (KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT) and GOOGLE_CLOUD_PROJECT.

Verifying the diagnosis on an OOMKilled workload

To see what the triage agent captures in practice, suppose the checkout-service Deployment in the payments namespace starts restarting with exit code 137 (OOMKilled) while managed by a VerticalPodAutoscaler in InPlaceOrRecreate mode.

When triggered, Gemini 4 Argon executes four tool calls in sequence and produces a structured diagnostic report:

Root-cause summary for Deployment payments/checkout-service:
1. Pod status & exit code: Pod checkout-service-7c9d8b6f4-x82lp terminated with reason OOMKilled (exit code 137) at 06:14:22 UTC after memory usage reached its 1Gi limit.
2. Kubernetes events: BackOff restarting failed container; InPlaceResizeFailed event reported on node gke-commerce-pool-4a91f.
3. VPA control plane logs (KCP_VPA):
   - Operation: APPLY_RECOMMENDATION_IN_PLACE | State: FAILED | Confidence: HIGH
   - Reason: Recommended memory target (2Gi) exceeded maxAllowed policy bound (1Gi) defined in VerticalPodAutoscaler checkout-service-vpa.
4. Recommended remediation: Raise resourcePolicy.containerPolicies[0].maxAllowed.memory in checkout-service-vpa from 1Gi to 3Gi so VPA can apply high-confidence in-place memory resizes without triggering OOM terminations.

You can independently verify the agent's finding using kubectl and gcloud:

# Verify the VPA resource policy cap on the cluster
kubectl get vpa checkout-service-vpa -n payments -o yaml

# Inspect the corresponding KCP_VPA log entry in Cloud Logging
gcloud logging read "resource.type=\"k8s_control_plane_component\" AND resource.labels.cluster_name=\"${CLUSTER_NAME}\" AND logName=\"projects/${PROJECT_ID}/logs/container.googleapis.com%2Fvpa-controller\" AND jsonPayload.target.name=\"checkout-service\"" \
    --project="${PROJECT_ID}" \
    --limit=3 \
    --format=json

Operational guardrails and trade-offs

When running an AI-assisted triage agent inside your production GKE environment, apply these operational safeguards:

Keep remediation out-of-band: Even with Gemini 4 Argon's strong reasoning accuracy, an autonomous agent should not directly run kubectl apply or mutate resource limits during an active incident. Have the agent open a pull request against your GitOps repository (managed by Config Sync or Argo CD) so human operators can review the diff.
Scrub sensitive payloads from Pod logs: Application containers occasionally log PII or bearer tokens in stack traces during crashes. Configure a log-scrubbing filter before forwarding raw pods/log output to the model API, and ensure your Gemini platform data governance policies disable prompt logging where required.
Bound tool-call iterations during alert storms: If a node outage triggers 50 Pod alerts simultaneously, firing 50 parallel multi-turn Gemini 4 Argon investigations will exhaust both your Kubernetes API server client QPS limits and Gemini platform token quotas. Deduplicate alerts by owner reference (Deployment or StatefulSet) and cap each investigation to a maximum of 8 tool-calling turns.

Next steps

Combining Gemini 4 Argon with the GKE Model Context Protocol server and structured control plane logs (KCP_VPA) turns multi-system incident correlation from a 30-minute manual scavenger hunt into an automated, auditable workflow.

To deploy this pattern on your clusters, explore the open-source Google Kubernetes Engine MCP server repository, read the GKE VPA decision logs documentation, and review Workload Identity Federation for GKE for zero-key IAM authentication.

Autonomous agent loops with Gemini 4 Argon and GKE Agent Sandbox: Running untrusted code safely at scale

Olivier Bourgeois — Mon, 19 Oct 2026 22:20:01 +0000

Autonomous AI systems are only as useful as the actions they can verify. With the release of Gemini 4 Argon, multi-step reasoning and native tool calling allow agents to write data transformation scripts, generate infrastructure migrations, and debug failing tests across long execution horizons. However, a fundamental architectural bottleneck emerges the moment the model outputs a block of Python or Bash: something has to execute that code and return the output so the model can iterate.

Running LLM-generated code inside the same Kubernetes Pod as your orchestrator application is an immediate security liability. A hallucinated command, a prompt-injection payload, or a dependency conflict can corrupt the container filesystem, exfiltrate mounted service account credentials, or starve adjacent workloads of memory. Conversely, spinning up a fresh Kubernetes Job or virtual machine for every turn of a multi-step reasoning loop introduces 5 to 30 seconds of cold-start latency per iteration, stalling the agent workflow.

By pairing a Gemini 4 Argon reasoning loop with GKE Agent Sandbox (Generally Available on GKE 1.32 and later across both GKE Autopilot and GKE Standard), platform teams can isolate every code execution step inside a kernel-hardened gVisor runtime with sub-second allocation from pre-warmed Pod pools.

In this article, I will walk through the architecture of a self-correcting Gemini 4 Argon code execution loop on Google Kubernetes Engine (GKE), show how to configure warm sandbox pools across GKE Autopilot and Standard, and implement a Python orchestrator that claims, executes, and recycles isolated sandboxes.

Architectural tension in multi-turn code execution

When a Gemini 4 Argon agent solves an analytical or engineering task, it rarely stops at a single code snippet. Instead, it operates in an iterative loop:

Task submission: The client sends an analytical query or code generation request to the GKE orchestrator Pod.
Model reasoning and tool dispatch: The orchestrator invokes Gemini 4 Argon with an execute_python tool declaration. The model emits a tool call containing the proposed script.
Warm pool allocation: The orchestrator requests a pre-warmed sandbox from the GKE Sandbox Router, which allocates an initialized gVisor Pod in under 200ms.
Isolated execution: The untrusted script runs inside the sandbox, capturing stdout, stderr, and exit codes without host kernel access.
Recycling and feedback: The used sandbox is returned to the router for teardown, and execution outputs are fed back to Gemini 4 Argon for iterative refinement until the final answer is ready.

Evaluating where to execute that generated code requires balancing isolation boundaries against allocation latency:

Execution pattern	Isolation boundary	Allocation latency	Operational trade-off
In-process `subprocess` in orchestrator Pod	Shared container kernel and filesystem	< 10ms	Exposes Workload Identity tokens, application memory, and internal cluster networking to arbitrary code
Per-invocation Kubernetes `Job`	Standard container (`runc`) or `RuntimeClass`	3s to 20s	Scheduling, image pulling, and container initialization add seconds of delay to every turn of the agent loop
GKE Agent Sandbox with `SandboxWarmPool`	User-space kernel via gVisor + default-deny network	< 200ms (p90)	Requires maintaining a small footprint of pre-warmed idle Pods in exchange for sub-second isolation

GKE Agent Sandbox resolves this tension by combining four open-source custom resources from the kubernetes-sigs/agent-sandbox project with GKE kernel isolation:

Kernel-level isolation via Sandbox and gVisor: Each Sandbox (agents.x-k8s.io/v1alpha1) manages an isolated Pod running inside GKE Sandbox, intercepting application system calls in user space so untrusted code never interacts directly with the host Linux kernel.
Reusable blueprints via SandboxTemplate: A SandboxTemplate (extensions.agents.x-k8s.io/v1alpha1) defines the Pod specification, resource boundaries, and security context once for a class of agent tools.
Pre-warmed capacity via SandboxWarmPool: Rather than waiting for the Kubernetes scheduler and container runtime on each tool call, a SandboxWarmPool (extensions.agents.x-k8s.io/v1alpha1) maintains a standing buffer of initialized Sandbox Pods ready to be claimed in under 200 milliseconds.
Declarative allocation via SandboxClaim and zero-trust posture: When an agent needs to run code, the Sandbox Router creates a SandboxClaim (extensions.agents.x-k8s.io/v1alpha1) to bind a warm Sandbox Pod and deletes the claim on release. Sandbox Pods disable Kubernetes service account token automounting (automountServiceAccountToken: false) and enforce strict network egress policies, keeping your orchestrator's Workload Identity Federation for GKE credentials unreachable from the code execution environment.

Configuring the cluster and sandbox node pools

Before deploying the sandbox templates and the Gemini orchestrator, you need a GKE cluster with gVisor (GKE Sandbox) support and the Agent Sandbox controller installed.

Enabling Agent Sandbox on GKE Autopilot

On GKE Autopilot, GKE Sandbox is supported out of the box and automatically provisions gVisor-enabled compute capacity whenever Pods request runtimeClassName: gvisor. If you are using the managed GKE Agent Sandbox addon preview, pass --enable-agent-sandbox via gcloud beta; otherwise, create a standard Autopilot cluster and install the upstream controller manifests in the next step:

export PROJECT_ID="production-ai-platform"
export LOCATION="us-central1"
export CLUSTER_NAME="agent-execution-cluster"

gcloud beta container clusters create-auto "${CLUSTER_NAME}" \
    --project="${PROJECT_ID}" \
    --location="${LOCATION}" \
    --enable-agent-sandbox

Enabling Agent Sandbox on GKE Standard

On GKE Standard, your cluster must keep at least one regular node pool for system Pods while isolating untrusted execution onto a dedicated node pool configured with --image-type=cos_containerd and --sandbox=type=gvisor. Separating the untrusted execution node pool from the node pool hosting your orchestrator Pods also prevents noisy-neighbor resource contention:

# Enable the Agent Sandbox addon on an existing Standard cluster (beta)
gcloud beta container clusters update "${CLUSTER_NAME}" \
    --project="${PROJECT_ID}" \
    --location="${LOCATION}" \
    --enable-agent-sandbox

# Create a dedicated gVisor node pool for untrusted code execution
gcloud container node-pools create sandbox-execution-pool \
    --cluster="${CLUSTER_NAME}" \
    --project="${PROJECT_ID}" \
    --location="${LOCATION}" \
    --machine-type=e2-standard-8 \
    --image-type=cos_containerd \
    --sandbox=type=gvisor \
    --num-nodes=3 \
    --enable-autoscaling \
    --min-nodes=2 \
    --max-nodes=10

If you are installing the open-source kubernetes-sigs/agent-sandbox controller and extension CRDs (Sandbox, SandboxTemplate, SandboxClaim, and SandboxWarmPool) directly onto any GKE Autopilot or Standard cluster, apply the release manifests:

export AGENT_SANDBOX_VERSION="v0.1.0"

kubectl apply -f "https://github.com/kubernetes-sigs/agent-sandbox/releases/download/${AGENT_SANDBOX_VERSION}/manifest.yaml"
kubectl apply -f "https://github.com/kubernetes-sigs/agent-sandbox/releases/download/${AGENT_SANDBOX_VERSION}/extensions.yaml"

Defining the SandboxTemplate, SandboxWarmPool, and SandboxClaim

Once the cluster and CRDs are ready, you define the execution environment using SandboxTemplate and SandboxWarmPool. When you specify runtimeClassName: gvisor on GKE, the GKE admission controller automatically injects the required sandbox.gke.io/runtime: gvisor node selector and taint tolerations on both Autopilot and Standard clusters.

Create a manifest named analytics-sandbox-pool.yaml in a dedicated agent-sandboxes namespace. This configuration enforces non-root execution, drops all Linux capabilities, disables service account token mounting, and keeps 5 pre-warmed Python runtime Pods ready for tool calls:

apiVersion: v1
kind: Namespace
metadata:
  name: agent-sandboxes
---
apiVersion: extensions.agents.x-k8s.io/v1alpha1
kind: SandboxTemplate
metadata:
  name: python-analytics-template
  namespace: agent-sandboxes
spec:
  podTemplate:
    metadata:
      labels:
        app.kubernetes.io/component: execution-sandbox
        sandbox-tier: untrusted-python
    spec:
      # GKE automatically adds the gVisor nodeSelector and toleration for this RuntimeClass
      runtimeClassName: gvisor
      automountServiceAccountToken: false
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        seccompProfile:
          type: RuntimeDefault
      containers:
      - name: python-runtime
        image: registry.k8s.io/agent-sandbox/python-runtime-sandbox:v0.1.0
        ports:
        - containerPort: 8888
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
        resources:
          requests:
            cpu: "500m"
            memory: "1Gi"
            ephemeral-storage: "2Gi"
          limits:
            cpu: "1000m"
            memory: "1Gi"
            ephemeral-storage: "2Gi"
---
apiVersion: extensions.agents.x-k8s.io/v1alpha1
kind: SandboxWarmPool
metadata:
  name: python-analytics-warmpool
  namespace: agent-sandboxes
spec:
  replicas: 5
  sandboxTemplateRef:
    name: python-analytics-template

Under the hood, whenever the Sandbox Router (or a Kubernetes client) allocates a sandbox from this warm pool, it creates a SandboxClaim resource targeting python-analytics-template:

apiVersion: extensions.agents.x-k8s.io/v1alpha1
kind: SandboxClaim
metadata:
  name: analytics-task-claim-example
  namespace: agent-sandboxes
spec:
  sandboxTemplateRef:
    name: python-analytics-template

To prevent untrusted code from scanning internal cluster services or calling the GKE metadata server, apply a default-deny NetworkPolicy (isolate-untrusted-sandboxes.yaml) in the agent-sandboxes namespace that blocks all egress and only permits ingress on port 8888 from the Sandbox Router:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: isolate-untrusted-sandboxes
  namespace: agent-sandboxes
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/component: execution-sandbox
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: agent-sandbox-system
    - podSelector:
        matchLabels:
          app.kubernetes.io/component: sandbox-router
    ports:
    - protocol: TCP
      port: 8888
  egress: []

Apply the manifests to your cluster:

kubectl apply -f analytics-sandbox-pool.yaml
kubectl apply -f isolate-untrusted-sandboxes.yaml

Building the Gemini orchestrator loop

With the sandbox pool running in agent-sandboxes, you can deploy the orchestrator service in a separate agent-orchestrator namespace. The orchestrator uses Workload Identity Federation for GKE to authenticate with the Gemini platform and calls the Sandbox Router whenever Gemini invokes the code execution tool.

Here is the core implementation (orchestrator.py) using the google-genai SDK and the agentic_sandbox client. In the snippet below, GEMINI_MODEL defaults to gemini-4-argon:

import os
from google import genai
from google.genai import types
from agentic_sandbox import SandboxClient

# Defaults to gemini-4-argon on the Gemini platform:
MODEL_ID = os.environ.get("GEMINI_MODEL", "gemini-4-argon")

# Initialize the Gemini platform client using Workload Identity credentials
genai_client = genai.Client(
    vertexai=True,
    project=os.environ["GOOGLE_CLOUD_PROJECT"],
    location=os.environ.get("GOOGLE_CLOUD_LOCATION", "us-central1"),
)

# Initialize the GKE Agent Sandbox client (communicates via the Sandbox Router)
sandbox_client = SandboxClient(namespace="agent-sandboxes")


def execute_python_in_sandbox(code: str) -> dict[str, str | int]:
    """Executes Python code inside an isolated GKE gVisor sandbox and returns stdout, stderr, and exit code."""
    sandbox = sandbox_client.claim_sandbox(
        template_name="python-analytics-template",
        timeout_seconds=5,
    )
    try:
        result = sandbox.execute(code, timeout_seconds=30)
        return {
            "stdout": result.output or "",
            "stderr": result.error or "",
            "exit_code": result.exit_code,
        }
    finally:
        # Always release the sandbox so the SandboxClaim is deleted and replaced
        sandbox.release()


def run_autonomous_task(user_prompt: str) -> str:
    """Runs a multi-turn Gemini 4 Argon reasoning loop backed by GKE Agent Sandbox."""
    config = types.GenerateContentConfig(
        system_instruction=(
            "You are a quantitative engineering agent. When solving data or systems problems, "
            "write Python scripts and run them using execute_python_in_sandbox. "
            "If execution returns a non-zero exit_code or stderr traceback, inspect the error, "
            "adjust your code, and re-run until the output is verified."
        ),
        tools=[execute_python_in_sandbox],
        automatic_function_calling=types.AutomaticFunctionCallingConfig(
            maximum_remote_calls=10,
        ),
        temperature=0.1,
    )

    chat = genai_client.chats.create(
        model=MODEL_ID,
        config=config,
    )
    response = chat.send_message(user_prompt)
    return response.text or ""

When genai_client.chats.create receives a tool declaration with a typed Python function, the google-genai SDK automatically manages the multi-turn tool-calling loop (bounded by maximum_remote_calls=10), invoking execute_python_in_sandbox each time the model emits a function call and feeding the structured stdout, stderr, and exit_code dictionary back into the conversation context. Because sandbox.release() runs inside a finally block, every execution turn gets a clean, ephemeral container filesystem, preventing state pollution across unrelated tasks.

Verifying sandbox allocation and observability

After deploying the warm pool and orchestrator, verify that the pre-warmed Pods are scheduled under the gVisor runtime and that the warm pool controller is maintaining target capacity:

# Inspect the warm pool status, active sandboxes, and claims
kubectl get sandboxwarmpool,sandbox,sandboxclaim -n agent-sandboxes

# Confirm that sandbox Pods are running with RuntimeClass gvisor
kubectl get pods -n agent-sandboxes \
    -l app.kubernetes.io/component=execution-sandbox \
    -o custom-columns="NAME:.metadata.name,STATUS:.status.phase,RUNTIME:.spec.runtimeClassName,NODE:.spec.nodeName"

Expected output:

NAME                                         STATUS    RUNTIME   NODE
python-analytics-warmpool-6d8f9b7c4-2x9kl    Running   gvisor    gke-agent-execution-sandbox-pool-8a12f-9b2c
python-analytics-warmpool-6d8f9b7c4-7m4pq    Running   gvisor    gke-agent-execution-sandbox-pool-8a12f-9b2c
python-analytics-warmpool-6d8f9b7c4-9v1zr    Running   gvisor    gke-agent-execution-sandbox-pool-8a12f-c41d

To inspect sandbox claim latency and catch container OOM kills triggered by runaway agent scripts in Cloud Logging, run the following query:

resource.type="k8s_pod"
resource.labels.namespace_name="agent-sandboxes"
(reason="OOMKilling" OR reason="SandboxClaimed" OR reason="SandboxReleased")

In Cloud Monitoring, track kubernetes.io/pod/ephemeral_storage/used_bytes and kubernetes.io/container/memory/used_bytes filtered by namespace_name="agent-sandboxes" to rightsize your SandboxTemplate resource limits based on the actual memory footprint of the scripts Gemini 4 Argon generates.

Operational trade-offs and sharp edges

When running GKE Agent Sandbox with Gemini 4 Argon in production, keep these operational boundaries in mind:

Warm pool sizing vs. burst concurrency: If 20 concurrent Gemini 4 Argon requests trigger tool execution simultaneously and your SandboxWarmPool only holds replicas: 5, the remaining 15 requests will wait for cold Pod creation. Size replicas to cover your p95 burst concurrency during peak traffic hours.
Stateful multi-step sessions: In the example above, releasing the sandbox inside execute_python_in_sandbox destroys the Pod after every turn. If your Gemini 4 Argon agent writes files to disk in turn 1 and analyzes them in turn 2, bind the sandbox instance to the chat session lifecycle and invoke sandbox.release() only after the entire conversation completes.
gVisor syscall overhead for heavy I/O: Because gVisor intercepts system calls in user space, compute-bound Python scripts (such as NumPy or Pandas array operations) run near native speed, whereas workloads performing thousands of small file opens or network socket creations incur additional CPU overhead.
Hard execution timeouts: LLM-generated code can inadvertently introduce infinite while loops. Always enforce a strict timeout_seconds parameter on sandbox.execute() so the orchestrator returns a timeout error to Gemini 4 Argon rather than blocking the request worker indefinitely.

Next steps

Pairing Gemini 4 Argon with GKE Agent Sandbox gives your agents the ability to test and self-correct their own code without exposing your cluster or credentials to untrusted execution. By isolating every tool invocation inside a pre-warmed gVisor Pod with zero network egress, you get both kernel-level safety and sub-second iteration latency.

To build this pattern in your own environment, explore the GKE Agent Sandbox conceptual guide, follow the GKE Agent Sandbox installation instructions, and check out the upstream kubernetes-sigs/agent-sandbox repository.

Tunable CrashLoopBackOff in GKE: Accelerating AI/ML recovery and eliminating risky node hacks

Olivier Bourgeois — Mon, 31 Aug 2026 18:21:41 +0000

In Kubernetes, few status messages are as familiar as CrashLoopBackOff. When a container exits unexpectedly, the kubelet steps in to prevent the failing process from overwhelming the host node. To achieve this, it applies an exponential backoff delay before each restart attempt. While this defensive mechanism protects node stability, its rigid default parameters create friction for modern workloads.

The default Kubernetes restart logic starts at a 10-second delay and doubles after each failure (10s, 20s, 40s, 80s, 160s) until reaching a 5-minute (300-second) ceiling. In fast-moving development environments, distributed AI/ML training runs, and architectures with critical sidecars, waiting up to five minutes for a container to retry stalls entire pipelines.

To solve this operational bottleneck, the GKE team launched the General Availability of tunable CrashLoopBackOff. By exposing crashLoopBackOff.maxContainerRestartPeriod through the GKE NodeSystemConfig API and Custom Compute Classes (CCC), platform teams can now securely reduce restart delays down to 1 second.

In this article, I will explain why fixed restart delays impact modern workloads, how GKE enables native tuning without privileged host workarounds, and how to configure and monitor this capability.

The cost of fixed restart backoffs

Kubernetes designed exponential backoff to protect the kubelet and runtime from CPU exhaustion caused by rapid restart loops. However, a maximum backoff delay of 300 seconds introduces severe delays across several workload patterns:

AI/ML training and inference pipelines: Large-scale distributed training jobs synchronize state across hundreds of accelerator nodes hosting GPUs or TPUs. If a single worker encounters a temporary network timeout, initialization hiccup, or dependency race condition, the container enters CrashLoopBackOff. When one Pod delays by 5 minutes, the entire gang-scheduled training job stalls, leaving expensive accelerators idle.
Critical sidecar initialization: Modern microservices frequently depend on sidecars for service mesh routing, mTLS credential renewal, or secret injection. If a sidecar crashes due to transient backend unavailability, the primary application container cannot serve traffic until the sidecar restarts and passes readiness checks.
Fast developer iteration cycles: During active debugging and CI runs, engineers need containers to restart immediately after updating an environment variable or dependency. Waiting through several minutes of backoff adds unnecessary latency to test suites.

The risks of legacy node workarounds

Because upstream Kubernetes historically lacked a supported interface to tune restart delays, platform teams turned to dangerous workarounds.

The most common hack involved running privileged DaemonSets with host filesystem access (hostPID: true, hostPath: /etc/kubernetes). These DaemonSets executed scripts to overwrite kubelet.config.json or modify systemd unit flags directly on the node, forcing kubelet restarts to apply non-standard configurations.

This approach creates significant liabilities:

Security perimeter violations: Granting containers root privileges and host access bypasses Kubernetes security boundaries, exposing worker nodes to container escape risks.
Node stability and auto-repair failures: Custom filesystem edits interfere with GKE node auto-upgrade and auto-repair mechanisms. When GKE reprovisions or updates a node, custom file modifications can lead to bootstrap failures.
Accelerator node instability: Running unsupported background scripts on GPU and TPU nodes risks disrupting specialized accelerator drivers, device plugins, and NUMA-aware scheduling routines.

Tunable CrashLoopBackOff eliminates these workarounds by providing a native, fully managed control plane configuration.

Native tuning through NodeSystemConfig and ComputeClass

GKE allows administrators to configure the maximum restart delay per node pool using the NodeSystemConfig API in GKE Standard, or via ComputeClass custom resources in GKE Autopilot.

The configuration exposes the following parameters:

Configurable range: The maxContainerRestartPeriod must be an integer between 1 second and 300 seconds. Setting it to 1s forces the kubelet to retry failed containers almost immediately, while values like 10s or 30s provide a balanced compromise.
Node pool isolation: The setting applies at the node pool level, allowing you to configure low restart delays for specialized AI/ML pools while keeping default backoff behavior for standard application pools.
Upstream alignment: This capability builds upon upstream Kubernetes enhancement KEP-4603, ensuring compatibility with core Kubernetes architectural standards.

Configuring restart backoffs on your node pools

You can configure tunable CrashLoopBackOff when creating new node pools or updating existing pools.

Configuring GKE Standard node pools

To create a node pool with a custom restart delay, pass the configuration using a system config file with gcloud:

# node-system-config.yaml
kubeletConfig:
  crashLoopBackOff:
    maxContainerRestartPeriod: 5s

Run the following command to apply the configuration:

gcloud container node-pools create accelerator-pool \
    --cluster=production-cluster \
    --location=us-central1-a \
    --system-config-from-file=node-system-config.yaml \
    --machine-type=g2-standard-24 \
    --accelerator=type=nvidia-l4,count=2

To update an existing node pool:

gcloud container node-pools update accelerator-pool \
    --cluster=production-cluster \
    --location=us-central1-a \
    --system-config-from-file=node-system-config.yaml

Configuring GKE Autopilot via ComputeClass

For clusters leveraging GKE Autopilot or Custom Compute Classes, declare the restart delay inside a ComputeClass manifest:

apiVersion: cloud.google.com/v1
kind: ComputeClass
metadata:
  name: fast-recovery-accelerator
spec:
  nodeConfig:
    systemConfig:
      kubeletConfig:
        crashLoopBackOff:
          maxContainerRestartPeriod: 5s

Workloads requesting this compute class automatically land on nodes provisioned with the 5-second maximum restart delay.

Monitoring and operational best practices

Reducing the maximum restart period causes failing containers to restart more frequently. To maintain cluster health, apply these operational practices:

Monitor restart rates: Track the kubernetes.io/container/restart_count metric in Cloud Monitoring. A sudden surge in restarts indicates an unrecoverable crash requiring debugging rather than rapid retries.
Track node resource utilization: Observe kubelet CPU and memory consumption (kubernetes.io/node/cpu/allocatable_utilization). Rapid restarts generate more container runtime and lifecycle events.
Combine with proper health probes: Ensure Pod definitions use realistic startupProbe and livenessProbe timeouts. Probes must allow sufficient initialization time before failing containers.

Next steps

Tunable CrashLoopBackOff removes a major constraint for high-performance workloads on GKE. By replacing risky DaemonSet workarounds with native control plane configuration, platform teams can accelerate AI training recovery, streamline sidecar startup, and protect node stability.

To configure restart periods for your clusters, review the official GKE node system configuration documentation, explore the GKE ComputeClass reference, and read upstream Kubernetes KEP-4603.

Native CORS support on GKE Gateway: Offloading cross-origin policy management to infrastructure

Olivier Bourgeois — Sun, 30 Aug 2026 19:09:03 +0000

Web browsers enforce the Same-Origin Policy by default to protect users from malicious scripts trying to read data across distinct origins. However, modern application architectures almost always require cross-origin communication. Single-page applications, mobile clients, and embedded web components regularly fetch data and stream AI model inferences across separate domains, subdomains, and ports.

To allow these interactions safely, applications must implement Cross-Origin Resource Sharing (CORS). For years, teams running Kubernetes workloads on Ingress-Nginx handled this using annotations like nginx.ingress.kubernetes.io/enable-cors. When migrating to the Kubernetes Gateway API and GKE Gateway, the lack of native CORS support was a frequent operational pain point, making it one of the most requested missing capabilities.

The GKE team addressed this gap with the Preview release of native CORS support for GKE Gateway and Inference Gateway. In this article, I will break down how the new CORS filter works, why moving cross-origin policy management to the load balancer matters, and the operational nuances you need to consider.

Why manage CORS at the ingress layer?

Implementing CORS inside individual backend applications introduces architectural friction across three main areas:

Redundant application logic: Every backend service or framework (Node.js, FastAPI, Spring, or inference engines like vLLM) must include middleware to evaluate incoming headers and generate preflight responses.
Preflight resource consumption: Complex web requests trigger preflight OPTIONS calls. Routing these preflight requests to backend containers wastes application memory, CPU cycles, and network bandwidth on pure protocol negotiation.
Configuration sprawl and drift: When dozens of microservices manage their own CORS policies, subtle discrepancies in allowed headers, exposed headers, or origin validation create security vulnerabilities and broken client integrations.

By shifting CORS processing to GKE Gateway, Google Cloud Load Balancing terminates OPTIONS preflight requests directly at the network edge and injects required response headers (Access-Control-Allow-Origin, Access-Control-Allow-Methods, and Access-Control-Allow-Headers). Your backend applications only receive validated application requests, removing boilerplate code and reducing compute overhead.

Configuring the CORS filter in HTTPRoute

GKE Gateway implements CORS support directly through the open-source Gateway API specification. You define policies declaratively using a CORS filter within the rules section of an HTTPRoute manifest.

Here is an example configuring a CORS policy on an API route:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: api-cors-route
  namespace: production
spec:
  parentRefs:
  - name: external-gateway
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /api
    backendRefs:
    - name: api-service
      port: 8080
    filters:
    - type: CORS
      cors:
        allowOrigins:
        - "https://app.example.com"
        - "https://*.partner-domain.com"
        allowMethods:
        - GET
        - POST
        - PUT
        - DELETE
        allowHeaders:
        - Authorization
        - Content-Type
        - X-Requested-With
        exposeHeaders:
        - X-Request-ID
        allowCredentials: true
        maxAge: 3600

The cors configuration block gives you granular control over the negotiation parameters:

allowOrigins: Specifies the allowed origins via explicit URLs (https://app.example.com), wildcard patterns (https://*.partner-domain.com), or a catch-all wildcard (*).
allowMethods: Lists the permitted HTTP methods. You can also specify * to allow all methods.
allowHeaders: Defines the HTTP request headers that clients can send in their requests.
exposeHeaders: Lists response headers that the browser makes accessible to client scripts beyond simple response headers.
allowCredentials: Sets the Access-Control-Allow-Credentials header to true or false, dictating whether browsers can share responses with requests carrying cookies or authentication headers.
maxAge: Defines the number of seconds the browser can cache the preflight response (defaulting to 5 seconds), significantly cutting down subsequent OPTIONS traffic.

Security considerations with credentials and wildcards

When designing your CORS policies on GKE Gateway, pay careful attention to the interaction between allowOrigins and allowCredentials.

Under standard browser security rules, browsers block responses to credentialed requests if Access-Control-Allow-Origin is set to a literal wildcard (*). However, when you configure wildcard patterns or a wildcard in allowOrigins in GKE Gateway, the controller dynamically matches and reflects the incoming request origin rather than returning a literal asterisk.

Because the browser sees an explicit origin returned alongside allowCredentials: true, it permits the response. If you configure allowOrigins: ["*"] with allowCredentials: true, any arbitrary website can potentially read authenticated user responses. For authenticated APIs, always define explicit domain lists rather than catch-all wildcards.

Supported GatewayClasses and architectural constraints

This Preview release supports single-cluster GKE Gateway deployments across three primary GatewayClasses:

gke-l7-rilb: Regional internal Application Load Balancer.
gke-l7-regional-external-managed: Regional external Application Load Balancer.
gke-l7-global-external-managed: Global external Application Load Balancer.

It also supports AI inference workloads exposed via Inference Gateway, allowing frontend chat interfaces or client SDKs to query served models directly across origins.

Before implementing this in production, keep the following technical limits in mind:

Single-cluster only: Multi-cluster gateways (gke-l7-gmc-*) do not currently support the CORS filter.
Filter conflicts: You cannot combine a CORS filter and a RequestRedirect filter within the same route rule.
URL map regular expression limits: The GKE Gateway controller translates wildcard origin patterns into regular expressions on the underlying Cloud Load Balancing URL maps. For gke-l7-global-external-managed, there is a limit of one regular expression per Gateway listener, and combining wildcard origins with PathPrefix matches is not supported. For regional external and regional internal GatewayClasses, you can use up to five regular expressions per hostname. Note that exact origins and catch-all * entries do not count against these regex quotas.

Next steps

Native CORS support in GKE Gateway eliminates one of the biggest functional gaps for organizations migrating workloads from legacy Ingress controllers to the Kubernetes Gateway API. By managing cross-origin policies declaratively at the routing layer, platform teams can simplify application code and centralize security posture across all services.

To learn more and begin testing CORS in your clusters, check out the official GKE Gateway CORS documentation and the upstream Gateway API CORS User Guide.

Unlocking workload rightsizing visibility on GKE: How VPA decision logs bring observability to autoscaling

Olivier Bourgeois — Sat, 29 Aug 2026 18:57:42 +0000

Automating resource management in Kubernetes has always required a high degree of trust. When you hand over CPU and memory sizing to the Vertical Pod Autoscaler (VPA), you expect it to rightsize containers efficiently without introducing unexpected restarts or performance regressions. Yet for many platform engineers, running VPA in production has felt like operating a black box.

Until recently, inspecting VPA decisions meant relying on standard Kubernetes events or running kubectl describe vpa. These events are transient, often expiring after an hour. If a Pod was evicted unexpectedly during an overnight batch job, or if an in-place resize failed silently due to node capacity limits, diagnosing the root cause the next morning was frustratingly difficult.

To solve this observability gap, the GKE team launched the Public Preview of VerticalPodAutoscaler (VPA) Logs. Available on GKE clusters running version 1.36.0-gke.1601000 or newer, this feature streams structured VPA decision events directly into Cloud Logging.

In this article, I will explain what VPA logs capture, how they demystify autoscaler decisions, and how you can use them to troubleshoot scaling actions and build reliable autonomous workload management.

The missing link in workload autoscaling observability

Vertical autoscaling decisions are inherently complex. The VPA controller continuously evaluates historical CPU and memory utilization, computes recommendations with upper and lower safety bounds, and determines whether an active container needs adjustment.

Without persistent logging, answering essential operational questions was difficult:

Why did VPA decide to evict a specific Pod instead of keeping it running?
Was an applied resource recommendation modified by Autopilot compute ratios or custom resource policies?
Did an in-place resource resize fail, forcing the controller to fall back to recreation?
How much confidence did the recommendation engine have when calculating new targets?

By exporting VPA decision events to Cloud Logging as first-class control plane logs (KCP_VPA), GKE gives platform operators a permanent audit trail. Combined with existing Horizontal Pod Autoscaler (HPA) logging, teams now have complete visibility across horizontal and vertical scaling dimensions.

Understanding the structure of VPA decision logs

VPA logs are emitted by the vpa-controller control plane component and stored under the log destination container.googleapis.com/vpa-controller in Cloud Logging. Each log entry arrives as a structured JSON payload containing detailed metadata about the target workload, the evaluation state, and the calculated resource bounds.

The controller categorizes decision logs across four primary operations:

Update recommendation (UPDATE_RECOMMENDATION): Emitted periodically (once per minute per VPA object). This log details the raw recommendation calculated by the recommender, including lower bound, upper bound, target, uncapped target, and recommendation confidence.
Evict Pod (EVICT_POD): Emitted when the VPA updater decides to evict a Pod to apply new resource requests under Recreate mode (or as a fallback if in-place resize fails).
Apply recommendation on eviction (APPLY_RECOMMENDATION_ON_EVICTION): Emitted when a newly scheduled replacement Pod receives resized resource requests during admission.
Apply recommendation in place (APPLY_RECOMMENDATION_IN_PLACE): Emitted when VPA modifies container resource limits and requests live on a running Pod without a restart under InPlaceOrRecreate mode.

Each log entry includes a state field (SUCCEEDED, SKIPPED, or FAILED) and an explanatory reason string. When an operation succeeds, the reason field clarifies whether applied recommendations diverged from raw recommendations due to policy caps or Autopilot ratio constraints.

Crucially, the payload includes a confidence field:

LOW: The recommender has processed fewer than 10 metric samples.
HIGH: The recommender has processed 10 or more metric samples, indicating a mature usage profile.

Enabling VPA decision logs on GKE

VPA logs can be enabled on both new and existing GKE clusters using the Google Cloud CLI.

Enabling logs on cluster creation

To create a new GKE cluster with VPA decision logs enabled, include KCP_VPA in the --logging flag alongside SYSTEM logs:

gcloud container clusters create CLUSTER_NAME \
    --location=LOCATION \
    --project=PROJECT_ID \
    --logging=SYSTEM,KCP_VPA

Updating an existing cluster

When updating an existing cluster, preserve your currently configured logging components so you do not inadvertently overwrite them. Add KCP_VPA to your existing configuration:

gcloud container clusters update CLUSTER_NAME \
    --location=LOCATION \
    --project=PROJECT_ID \
    --logging=SYSTEM,KCP_VPA

Verifying the logging configuration

You can confirm that KCP_VPA is active by retrieving the enabled logging components for the cluster:

gcloud container clusters describe CLUSTER_NAME \
    --location=LOCATION \
    --flatten=loggingConfig \
    --format='csv[delimiter=",",no-heading](componentConfig.enableComponents)'

The output will list KCP_VPA alongside your other active control plane components.

Practical queries for Logs Explorer

Once enabled, you can search and analyze VPA events directly in Google Cloud Logs Explorer.

To view all decision events for a specific workload within a cluster, use the following filter:

resource.type="k8s_control_plane_component"
resource.labels.cluster_name="CLUSTER_NAME"
logName="projects/PROJECT_ID/logs/container.googleapis.com%2Fvpa-controller"
jsonPayload.target.name="WORKLOAD_NAME"

To find instances where VPA skipped or failed an in-place resize operation, query by operation and state:

logName="projects/PROJECT_ID/logs/container.googleapis.com%2Fvpa-controller"
jsonPayload.operation="APPLY_RECOMMENDATION_IN_PLACE"
jsonPayload.state=("SKIPPED" OR "FAILED")

To audit recommendations generated with low confidence, filter on the confidence attribute:

logName="projects/PROJECT_ID/logs/container.googleapis.com%2Fvpa-controller"
jsonPayload.operation="UPDATE_RECOMMENDATION"
jsonPayload.confidence="LOW"

These queries enable site reliability teams to quickly diagnose scaling anomalies, track resize frequency, and identify workloads that require longer profiling before enforcing automated actuation.

Enabling autonomous, intent-based workload autoscaling

The launch of VPA decision logs represents more than a troubleshooting convenience. In modern cloud-native architectures, platform engineering teams are moving toward intent-based infrastructure where autonomous agents monitor, optimize, and heal application environments.

For an AI agent or automated governance pipeline to safely manage container resources, it requires complete observability into both horizontal and vertical scaling lifecycles. By providing a structured, historical record of why the VPA made every sizing decision, GKE equips platform teams and intelligent systems with the data needed to automate workload rightsizing with confidence.

Next steps

VerticalPodAutoscaler logs bring much-needed clarity to container resource optimization on GKE, turning automated rightsizing into a transparent, auditable process.

To get started with VPA logs on your clusters, review the official GKE vertical Pod autoscaler event logs documentation and learn more about configuring Vertical Pod Autoscaling in GKE.

Accelerating JVM startup on GKE: How VPA CPU startup boost eliminates ongoing resource waste

Olivier Bourgeois — Sat, 29 Aug 2026 05:03:06 +0000

Running Java and other JVM-based workloads in Kubernetes has long presented an infrastructure sizing challenge. During initialization, JVM applications require substantial CPU capacity to perform class loading, bytecode verification, framework dependency injection, and Just-In-Time (JIT) compilation. Once this initialization phase finishes, steady-state CPU consumption frequently drops by 50% to 80%.

To avoid slow startups and ensure new Pods pass readiness probes quickly, platform teams have historically over-provisioned CPU requests permanently. This compromise introduces persistent CPU waste—unutilized allocations that inflate infrastructure spend and degrade workload packing density on worker nodes.

To resolve this trade-off, the GKE team launched the Public Preview of VerticalPodAutoscaler (VPA) CPU Startup Boost. Available on GKE clusters running version 1.36.0-gke.4447000 or newer in the Rapid Channel, this feature provides up to 2x faster startup latency while dynamically reclaiming CPU capacity once the application stabilizes.

In this article, I will explain how CPU startup boost operates under the hood, how it leverages Kubernetes In-Place Pod Resize (IPPR), and how to configure it effectively for your clusters.

The JVM startup and CPU waste dilemma

When configuring CPU requests for a container, Kubernetes uses that value for scheduling decisions and CPU bandwidth enforcement through CFS quotas. If you rightsize a Java container for its steady-state requirement—say, 500m CPU—the application may experience severe CPU throttling during boot, stretching startup times from seconds into minutes.

Slow startups create cascading operational challenges:

Horizontal scaling lag: When traffic spikes occur, Horizontal Pod Autoscaler (HPA) creates new replicas, but those replicas take too long to start serving requests.
Flapping readiness probes: Applications that exceed initial startup budgets can fail health checks, triggering unwanted container restarts.
Resource waste: To avoid throttling, teams routinely allocate 2 to 4 vCPUs permanently. After the bootstrap window, those vCPUs sit idle across GKE Standard and Autopilot clusters.

CPU startup boost solves this by providing temporary CPU headroom during initialization, then reducing the allocation back to baseline without restarting the Pod.

How CPU startup boost works under the hood

The CPU startup boost lifecycle executes across three distinct phases:

Admission phase: When a Pod is created, the GKE VPA mutating admission webhook intercepts the request. The webhook calculates the boosted CPU request based on your policy and injects both the elevated CPU values and a tracking annotation (vpaCpuStartupBoost/<container-name>) before the scheduler places the Pod.
Startup phase: The container starts on a node with the higher CPU allocation, allowing JVM class loading and JIT compilation to run without CFS throttling.
Unboosting phase: Once the Pod satisfies its readiness checks and reaches Ready status, the configured durationSeconds timer begins. When the duration expires, the VPA Updater initiates an in-place resize back to baseline.

Because GKE utilizes Kubernetes In-Place Pod Resize (IPPR), this downscale happens live. The container is never terminated or restarted when the boost ends.

Configuring startup boost for your workloads

You configure CPU startup boost directly inside a standard VerticalPodAutoscaler Custom Resource. You can use VPA exclusively for startup boost or combine it with continuous autoscaling.

Using startup boost without continuous VPA actuation

If you manage steady-state resource requests manually in your Deployment manifest, set updateMode: "Off" in the VPA policy:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: java-app-startup-boost
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: customer-service
  updatePolicy:
    updateMode: "Off"
  startupBoost:
    cpu:
      type: Factor
      factor: 2
      durationSeconds: 10

In this manifest:

type: Factor: Multiplies the baseline CPU request (doubling 1 vCPU to 2 vCPUs during startup). You can also specify type: Quantity with a fixed addition like quantity: "2".
durationSeconds: 10: Keeps boosted CPU active for 10 seconds after the Pod reaches Ready state before reclaiming the resource.

Container-level targeting and continuous autoscaling

For multi-container Pods containing sidecars, you can isolate the boost to the primary application container using containerPolicies:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: java-app-advanced-boost
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: payment-service
  updatePolicy:
    updateMode: InPlaceOrRecreate
  resourcePolicy:
    containerPolicies:
    - containerName: app-server
      mode: Auto
      startupBoost:
        cpu:
          type: Quantity
          quantity: "2"
          durationSeconds: 0
    - containerName: envoy-proxy
      mode: "Off"
      startupBoost:
        cpu:
          type: Factor
          factor: 1

Setting updateMode: InPlaceOrRecreate allows GKE to boost startup resources, return to baseline, and then continue evaluating ongoing usage to rightsize the workload over time.

Key architecture and operational considerations

When adopting CPU startup boost across your clusters, keep the following operational rules in mind:

HPA integration: When pairing startup boost with HPA based on CPU utilization, always define a readinessProbe and set durationSeconds: 0. This configuration ensures the Pod unboosts immediately upon becoming ready, preventing startup CPU spikes from triggering false scale-out events.
GKE Autopilot resource rules: Autopilot validates compute ratios during Pod admission. Make sure your Pod's baseline memory allocation can support the boosted CPU ratio. Because the Scale-Out ComputeClass enforces a fixed 1:4 ratio, standard or performance compute classes are recommended.
Node capacity and autoscaling: On GKE Standard, ensure worker nodes have sufficient allocatable CPU to schedule boosted Pods. If a node lacks capacity, GKE caps the boost to what the node can fit.
Pod restart semantics: Startup boost triggers during initial Pod creation. If a container crashes and restarts within an existing Pod, the boost is not reapplied.

Verifying boost and in-place downscale

You can confirm that startup boost is active by inspecting Pod annotations and cluster events with kubectl:

# Check for the tracking annotation injected at admission
kubectl get pod <pod-name> -o jsonpath='{.metadata.annotations.vpaCpuStartupBoost/*}'

# Observe the in-place downscale event after readiness
kubectl get events --field-selector reason=InPlaceResizedByVPA

The InPlaceResizedByVPA event confirms that the container returned to baseline requests without a restart.

Next steps

VPA CPU Startup Boost eliminates a longstanding trade-off in Kubernetes workload sizing, helping Java microservices start faster while cutting idle resource costs.

To start testing startup boost on your GKE clusters, review the official GKE CPU startup boost documentation and explore Vertical Pod Autoscaling in GKE.

Two-step control plane upgrades in GKE: How minor version rollbacks work under the hood

Olivier Bourgeois — Thu, 27 Aug 2026 17:56:45 +0000

Kubernetes control plane minor version upgrades have historically been an all-or-nothing proposition. In standard Kubernetes clusters, upgrading a control plane from one minor version to the next—such as moving from 1.33 to 1.34—commits changes to storage schemas immediately. If an unexpected regression emerges after upgrading the API server, rolling back to the previous minor version was impossible without restoring etcd snapshots.

To eliminate this operational risk, the GKE team drove upstream contributions in Kubernetes Enhancement Proposal KEP-4330 (Compatibility Versions) and introduced two-step control plane upgrades. Following public preview validation with enterprise customers, this capability is now Generally Available (GA) across all GKE release channels.

In this article, I will explain how two-step upgrades work under the hood, how automated rollouts leverage canary analysis, and how to manage rollback-safe upgrades using the Google Cloud CLI and Terraform.

The problem with minor version upgrades

In Kubernetes, minor version releases introduce storage schema changes, deprecated API removals, and modified controller behaviors. When the kube-apiserver binary starts on a newer minor version, it writes resources using newer internal schemas.

Because earlier binary versions cannot parse data stored in newer schemas, Kubernetes prohibits control plane downgrades across minor versions. If an organization encountered an issue after upgrading, platform operators had to either run the degraded control plane or rebuild the cluster.

Two-step upgrades decouple binary execution from API capability enablement. By separating the upgrade into two distinct phases, GKE provides an observation period (also known as a soak window) during which operators or automated systems can monitor cluster behavior and roll back the control plane to the previous minor version with zero data loss.

Decoupling binary execution from emulated versions

The foundation of two-step upgrades is running a newer control plane binary in an emulated compatibility mode.

When a two-step upgrade begins, GKE advances the control plane through two sequential stages:

Step 1: Binary upgrade (emulated mode): GKE upgrades the control plane binary to the target minor version (e.g., 1.34), but configures the API server to emulate the previous minor version (1.33). In this state, the control plane executes the new binary logic, while API schemas match the older version. APIs removed in 1.34 remain accessible. During this soak period, you can safely roll back to 1.33.
Step 2: Emulated version upgrade (finalization): Once the soak window completes without incident, GKE updates the emulated version to match the binary version. This step permanently enables the new minor version API schemas and feature deprecations. After this point, rollback is no longer possible.

During the soak period, worker node pools cannot be upgraded beyond the emulated version to preserve Kubernetes version skew rules.

Automated orchestration with CPRS and Canary Analysis Service

For clusters configured for auto-upgrades, two-step upgrades are enabled out of the box with zero manual configuration required. The GKE rollout engine, known as Control Plane Rollout Service (CPRS), orchestrates the staged lifecycle natively:

CPRS upgrades the control plane binary while locking the emulated version to the previous minor release.
CPRS initiates an automated 24-hour soak window.
The Canary Analysis Service (CAS) monitors cluster health signals throughout the soak duration, including API latency, error rates, and Pod health metrics.
If CAS detects unexpected regressions, the rollout halts to allow automated or user-driven rollbacks.
Once the cluster passes all CAS evaluations and satisfies the soak timer, CPRS triggers step 2 to finalize the emulated version.

This validation framework has helped GKE control plane upgrades achieve a 99.999% (five nines) rolling 30-day success rate across the global fleet.

Initiating manual two-step upgrades

If your team manages upgrades manually, you can execute two-step upgrades using the Google Cloud CLI or Terraform.

To initiate a two-step upgrade with a custom soak duration using gcloud:

gcloud beta container clusters upgrade my-cluster \
    --location=us-central1 \
    --cluster-version=1.34.1-gke.1829001 \
    --control-plane-soak-duration=48h \
    --master

The --control-plane-soak-duration flag defines the rollback-safe window, supporting values from 6 hours up to 7 days (e.g., 48h or 2d).

For infrastructure-as-code workflows, Terraform includes official support for managing two-step control plane upgrades declaratively by specifying the target version and soak parameters in your GKE cluster resources.

Verifying upgrade status and executing a rollback

While the cluster is soaking in emulated mode, you can inspect the active rollback state using the CLI:

gcloud container clusters describe my-cluster \
    --location=us-central1 \
    --format="yaml(rollbackSafeUpgradeStatus)"

The output includes a rollbackSafeUpgradeStatus block with the target binary version, emulated version, remaining soak time, and previousVersion string.

If your monitoring tools uncover a regression during the soak window, you can roll back the control plane to the previous minor patch:

gcloud container clusters upgrade my-cluster \
    --location=us-central1 \
    --cluster-version=1.33.5-gke.1080000 \
    --master

Because the control plane ran in emulated mode, no new data formats were written to etcd. GKE downgrades the control plane binary back to the specified version without risking data corruption.

Completing the upgrade early

If your validation tests pass and you want to unlock new minor version features immediately without waiting for the soak duration to expire, you can finalize the upgrade manually:

gcloud beta container clusters clusters complete-control-plane-upgrade my-cluster \
    --location=us-central1

Once executed, GKE promotes the emulated version to match the binary version, completing the upgrade.

Operational requirements and limits

When planning two-step control plane upgrades, keep the following operational rules in mind:

Two-step upgrades apply to minor version upgrades to GKE 1.33 and later.
Control plane upgrades must proceed one minor version at a time.
Maintenance windows and exclusions are strictly respected.
Autopilot and regional Standard clusters maintain continuous control plane availability during both phases.

Next steps

Two-step control plane upgrades eliminate the risk of irreversible minor version updates in Kubernetes, providing platform engineers with automated safety and an emergency rollback mechanism.

To learn more about configuring two-step upgrades, read the official GKE cluster upgrade documentation and explore KEP-4330: Compatibility Versions.

Progressive cluster upgrades at scale: A technical guide to GKE rollout sequencing with custom stages

Olivier Bourgeois — Thu, 27 Aug 2026 03:10:21 +0000

Upgrading Kubernetes clusters across a large enterprise fleet is often a balancing act between staying current with security patches and avoiding outages. By default, Google Kubernetes Engine (GKE) rolls out automatic upgrades progressively according to Google Cloud regional timelines. While regional rollout works well for standalone clusters, it does not understand your organization's business topology. If you run staging clusters in us-central1 and critical production clusters in us-east1, a standard regional rollout could upgrade your production environment before your pre-production validation completes.

The General Availability (GA) release of GKE rollout sequencing with custom stages solves this challenge. It provides platform teams with declarative control to sequence cluster upgrades across fleets, environments, and even distinct Google Cloud organizations according to business criticality rather than cloud geography.

How rollout sequencing works

Rollout sequencing builds on GKE fleet management. Fleets serve as logical boundaries for environments such as development, staging, and production. With rollout sequencing, you define an ordered pipeline of upgrade stages managed by a central resource called RolloutSequence.

When GKE publishes a new automatic upgrade target for a release channel, or when you explicitly trigger a target version, the system creates a Rollout object. This rollout progresses through your defined stages sequentially:

Control plane upgrades start in the first stage. Once all control planes in that stage reach the target version, a stage soak timer begins.
Node upgrades run in parallel with control plane upgrades, respecting node pool upgrade strategies such as surge or blue-green.
When both control planes and nodes complete their upgrade and satisfy the configured soak duration, the rollout advances to the next stage in the sequence.

If an individual stage contains clusters that take longer than 30 days to finish upgrading—due to restrictive maintenance windows or exclusions—GKE triggers a forced soak period to avoid stalling the entire multi-stage pipeline indefinitely.

Granular stages with label selectors

Earlier fleet-based rollout sequencing operated strictly at the fleet level, meaning an entire fleet had to upgrade before another fleet could begin. Custom stages introduce the ability to split a single fleet into multiple granular rollout phases using Common Expression Language (CEL) label selectors.

For example, within a production fleet, you can label a subset of clusters as canary targets and upgrade them before the rest of production.

Here is an example YAML manifest defining a three-stage sequence:

- stage:
  fleet-projects:
  - projects/dev-fleet-host
  soak-duration: 3d
- stage:
  fleet-projects:
  - projects/prod-fleet-host
  label-selector: resource.labels.tier=='canary'
  soak-duration: 4d
- stage:
  fleet-projects:
  - projects/prod-fleet-host
  soak-duration: 7d

When structuring custom stages, keep two architectural rules in mind:

Catch-all requirement: If you use a label selector to target a subset of clusters in a fleet, the final stage referencing that fleet must omit the label selector. This acts as a catch-all stage to guarantee that all remaining clusters in the fleet are upgraded.
Conflict resolution: If a cluster matches multiple stages within a sequence, GKE assigns that cluster exclusively to the earliest matching stage.

You can register this configuration using the Google Cloud CLI:

gcloud container fleet rolloutsequences create prod-rollout-sequence \
    --display-name="Production rollout sequence" \
    --stage-config=rollout-sequence.yaml \
    --project=central-management-project

Operational controls and upgrade scoping

Real-world production environments require operational agility when unexpected workload anomalies occur. Rollout sequencing with custom stages provides real-time lifecycle controls over active rollouts without requiring you to dismantle your configuration:

Pause and resume: If an issue arises during validation in a canary stage, you can pause the active rollout. Pausing stops GKE from initiating new cluster upgrades in that stage and subsequent stages while allowing running operations to finish. Once mitigated, you can resume the rollout.
Stage completion: If automated testing confirms stability before a soak timer finishes, platform operators can execute a stage completion action (force-complete-stage). This skips the remaining soak duration and moves the rollout immediately to the next stage.
Rollout cancellation: If a version introduces an unresolvable regression, you can cancel the rollout. Canceling stops the progression of that specific target version across the sequence.

In addition to runtime actions, you can restrict the automatic upgrade scope for a rollout sequence. If your platform policy requires manual control over major and minor Kubernetes version changes while automating security patches, you can configure your sequence to roll out only control plane patch upgrades and node patch upgrades.

However, mandatory upgrades—such as control planes that have not received a patch in 90 days or clusters reaching the end of version support—will continue to execute to preserve cluster stability and security.

Multi-tenant and cross-organization management

For enterprises with distributed topologies, rollout sequences support cross-project and cross-organization fleet memberships. Following continuous delivery best practices, Google recommends creating and maintaining RolloutSequence resources in a dedicated host project.

A single rollout sequence can define up to 15 distinct stages, and fleets can accommodate up to 250 clusters (or up to 2,000 clusters when using lightweight memberships with an approved quota increase). Across all stages in a sequence, you can configure a total soak duration of up to 90 days, with individual stage soak times configured up to 30 days.

By shifting from regional upgrade schedules to business-aligned rollout sequences, platform engineers can automate Kubernetes lifecycle management while maintaining safety, visibility, and control across their entire fleet.

To implement progressive cluster upgrades in your environment, explore the official GKE rollout sequencing documentation to review detailed configuration parameters, CEL filter syntax, and gcloud CLI commands.

Taking control of cluster security: A deep dive into GKE ClusterNetworkPolicy

Olivier Bourgeois — Tue, 25 Aug 2026 20:44:29 +0000

In Kubernetes, network security has historically been a developer-centric responsibility. Standard Kubernetes NetworkPolicy objects are namespace-scoped. While this works beautifully for isolating individual microservices within a single application boundary, it introduces major operational headaches at scale. As clusters grow to accommodate multiple teams, business units, and environments, platform administrators are left without a native mechanism to enforce global security guardrails.

If you want to ensure that every pod in your cluster is blocked from reaching the cloud metadata server, or that certain namespaces are completely isolated from all others, you are forced to rely on complex policy engines, custom admission controllers, or automating the injection of namespace-scoped policies. This approach is fragile and difficult to audit.

The introduction of the GKE ClusterNetworkPolicy (currently in Public Preview) addresses this challenge. It introduces a cluster-scoped resource that allows security and platform teams to establish non-overridable security boundaries across the entire cluster, spanning all namespaces.

Evaluation hierarchy

To understand ClusterNetworkPolicy, you must first understand how GKE evaluates network traffic. Unlike standard namespace-scoped NetworkPolicy resources, which are additive (meaning if multiple policies select a pod, the traffic is allowed if any policy permits it), ClusterNetworkPolicy uses a strict, sequential evaluation pipeline where the first matching rule wins.

Traffic flows through three distinct policy tiers in order:

Admin tier: Policies in this tier are evaluated first. This is where administrators define mandatory guardrails that developers cannot bypass. If a rule in this tier matches a traffic flow with an Accept or Deny verdict, evaluation stops immediately.
NetworkPolicy tier: If traffic is not explicitly allowed or blocked by the Admin tier, it falls through to standard namespace-scoped NetworkPolicy resources configured by DevOps teams and developers.
Baseline tier: If no namespace-scoped policies match the traffic, it falls through to the Baseline tier. Here, administrators can define fallback policies, such as a default-deny posture, which developers can choose to override in their respective namespaces.
Default GKE behavior: If a packet matches no rules in any tier, it falls back to GKE's default behavior, which is an implicit allow.

Within each tier, policies are evaluated based on an explicit numeric priority (0 to 1000, where lower numbers indicate higher precedence). Inside a single policy object, rules are evaluated from top to bottom.

Verdict actions: Accept, deny, pass

Every rule in a ClusterNetworkPolicy must trigger one of three actions:

Deny: Blocks the traffic immediately. This action short-circuits the evaluation pipeline.
Accept: Permits the traffic immediately. This also short-circuits the pipeline.
Pass: Transfers the evaluation to the next tier in the hierarchy.

The Pass action is particularly powerful. It enables platform administrators to target specific traffic flows—for example, web traffic on port 8080—and delegate the final decision to namespace owners. The traffic will bypass any remaining rules in the Admin tier and be evaluated against standard namespace-scoped network policies. If the namespace owners configure a policy to accept the traffic, it is allowed; if they have no policy, the traffic proceeds to the Baseline tier. This strikes a balance between centralized compliance and developer agility.

YAML examples

Let's examine how to write these policies. Consider a scenario where you want to isolate a sensitive namespace from all internal cluster traffic. By placing this policy in the Admin tier, you guarantee that namespace-scoped policies cannot override it.

Here is a manifest for a global deny policy targeting a sensitive namespace:

apiVersion: policy.networking.k8s.io/v1alpha2
kind: ClusterNetworkPolicy
metadata:
  name: cluster-wide-deny-sensitive
spec:
  tier: Admin
  priority: 10
  subject:
    namespaces:
      matchLabels:
        kubernetes.io/metadata.name: sensitive-ns
  ingress:
  - action: Deny
    name: deny-all-ingress
    from:
    - namespaces:
        matchLabels: {}
  egress:
  - action: Deny
    name: deny-all-egress
    to:
    - namespaces:
        matchLabels: {}

Now, consider a different scenario: you want to establish a default-deny posture across the cluster, but you want to allow developers to open up traffic as needed for their applications. For this, you use the Baseline tier.

Here is a baseline default-deny policy:

apiVersion: policy.networking.k8s.io/v1alpha2
kind: ClusterNetworkPolicy
metadata:
  name: default-deny-baseline
spec:
  tier: Baseline
  priority: 100
  subject:
    namespaces: {}
  ingress:
  - action: Deny
    name: baseline-deny-all
    from:
    - namespaces: {}

If a developer in the namespace my-app-ns needs to allow ingress traffic to their frontend service from an external ingress controller, they can simply deploy a standard namespace-scoped NetworkPolicy that permits that traffic. Because the standard policy is evaluated in the NetworkPolicy tier (which runs before the Baseline tier), GKE will permit the traffic without needing platform admin intervention.

Under the hood

The GKE implementation of ClusterNetworkPolicy is built entirely on open-source Cilium, reinforcing a commitment to community-driven foundations. GKE backported this support to Cilium 1.19 for GKE clusters, and the upstream capability will be generally available to the broader open-source ecosystem starting with Cilium 1.20.

To use this feature, your GKE clusters must meet the following requirements:

Run GKE version 1.36.0-gke.4447000 or later.
Use GKE Dataplane V2.

As this feature is in Public Preview, there are a few scale limits to keep in mind. Specifically, a single ClusterNetworkPolicy object can contain a maximum of 100 ingress rules and 100 egress rules. You can monitor traffic flow verdicts and troubleshoot your policies using GKE Dataplane V2 Observability tools.

By decoupling cluster-wide guardrails from namespace-level configurations, ClusterNetworkPolicy gives security teams the control they need without hindering developer speed.

To get started with configuring cluster-wide security policies on your clusters, check out the official GKE ClusterNetworkPolicy documentation for step-by-step guides and YAML examples.

Seamless scaling with VPA In-place Pod Resize on GKE

Olivier Bourgeois — Thu, 04 Jun 2026 18:19:03 +0000

Right-sizing Kubernetes workloads is a common platform engineering challenge. Set your requests too high, and you burn cloud budgets on idle capacity; set your limits too low, and your applications face throttling or dreaded OOMKills.

For years, the Vertical Pod Autoscaler (VPA) has been the standard answer to this problem, automatically adjusting CPU and memory requirements based on actual usage. However, this method of scaling came with a significant catch that prevented widespread adoption for critical workloads: applying new resource parameters required evicting and restarting the pod.

This disruption was often unacceptable for stateful applications, long-running connections, or latency-sensitive services.

Introducing In-place Pod Resize (IPPR) on GKE

In-place Pod Resize (IPPR) changes the game by allowing Kubernetes to modify resource requests and limits on live, running containers directly through the underlying container runtime, without triggering a restart.

By combining the intelligence of VPA with the non-disruptive nature of IPPR, GKE users finally have a viable path to dynamic, seamless, and automated right-sizing.

Note: As of writing, VPA IPPR is in Preview on GKE. While it is a massive step forward, I recommend evaluating it in staging environments before rolling it out to production workloads.

Getting started with IPPR

To use In-place Pod Resize, you need a GKE cluster running version 1.34.0-gke.2201000 or later.

GKE Autopilot: VPA is enabled by default.
GKE Standard: Requires the Vertical Pod Autoscaling feature to be enabled.

1. Enable the feature

If you aren't using Autopilot, ensure your cluster is created or updated with the necessary feature flags:

gcloud container clusters create CLUSTER_NAME \
  --project=PROJECT_ID \
  --location=us-east1 \
  --release-channel=rapid \
  --enable-vertical-pod-autoscaling

2. Define your VPA object

Create a VerticalPodAutoscaler resource targeting your Deployment or StatefulSet. The crucial element here is setting spec.updatePolicy.updateMode to InPlaceOrRecreate.

apiVersion: "autoscaling.k8s.io/v1"
kind: "VerticalPodAutoscaler"
metadata:
  name: "my-vpa"
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: "Deployment"
    name: "my-deployment"
  updatePolicy:
    updateMode: "InPlaceOrRecreate"

3. Watch it scale

Apply the resource to your cluster and monitor your application under load. Instead of watching Pods terminate and recreate, you can watch the resources modify live using kubectl describe.

kubectl describe pod POD_NAME

Look for the AllocatedResources field or check the events section. You will see the requests change in real-time to match the VPA recommendations, while the Restart Count remains exactly the same.

The "Or Recreate" Fallback: Keep in mind that physics still apply. If VPA recommends a resource size that exceeds the remaining capacity of the Node your Pod is currently running on, an in-place resize is impossible. In this scenario, VPA will fall back to evicting and recreating the Pod so it can be scheduled onto a larger or emptier Node.

Ready to dive deeper?

While this introduction covers the basics of IPPR, right-sizing is just one part of a robust scaling strategy. Implementing VPA often goes hand-in-hand with horizontal scaling and cluster autoscaling. Check out the guide to master scaling on GKE: Run full-stack workloads at scale on GKE.