DEV Community: Olivier Bourgeois

Tunable CrashLoopBackOff in GKE: Accelerating AI/ML recovery and eliminating risky node hacks

Olivier Bourgeois — Mon, 31 Aug 2026 18:21:41 +0000

In Kubernetes, few status messages are as familiar as CrashLoopBackOff. When a container exits unexpectedly, the kubelet steps in to prevent the failing process from overwhelming the host node. To achieve this, it applies an exponential backoff delay before each restart attempt. While this defensive mechanism protects node stability, its rigid default parameters create friction for modern workloads.

The default Kubernetes restart logic starts at a 10-second delay and doubles after each failure (10s, 20s, 40s, 80s, 160s) until reaching a 5-minute (300-second) ceiling. In fast-moving development environments, distributed AI/ML training runs, and architectures with critical sidecars, waiting up to five minutes for a container to retry stalls entire pipelines.

To solve this operational bottleneck, the GKE team launched the General Availability of tunable CrashLoopBackOff. By exposing crashLoopBackOff.maxContainerRestartPeriod through the GKE NodeSystemConfig API and Custom Compute Classes (CCC), platform teams can now securely reduce restart delays down to 1 second.

In this article, I will explain why fixed restart delays impact modern workloads, how GKE enables native tuning without privileged host workarounds, and how to configure and monitor this capability.

The cost of fixed restart backoffs

Kubernetes designed exponential backoff to protect the kubelet and runtime from CPU exhaustion caused by rapid restart loops. However, a maximum backoff delay of 300 seconds introduces severe delays across several workload patterns:

AI/ML training and inference pipelines: Large-scale distributed training jobs synchronize state across hundreds of accelerator nodes hosting GPUs or TPUs. If a single worker encounters a temporary network timeout, initialization hiccup, or dependency race condition, the container enters CrashLoopBackOff. When one Pod delays by 5 minutes, the entire gang-scheduled training job stalls, leaving expensive accelerators idle.
Critical sidecar initialization: Modern microservices frequently depend on sidecars for service mesh routing, mTLS credential renewal, or secret injection. If a sidecar crashes due to transient backend unavailability, the primary application container cannot serve traffic until the sidecar restarts and passes readiness checks.
Fast developer iteration cycles: During active debugging and CI runs, engineers need containers to restart immediately after updating an environment variable or dependency. Waiting through several minutes of backoff adds unnecessary latency to test suites.

The risks of legacy node workarounds

Because upstream Kubernetes historically lacked a supported interface to tune restart delays, platform teams turned to dangerous workarounds.

The most common hack involved running privileged DaemonSets with host filesystem access (hostPID: true, hostPath: /etc/kubernetes). These DaemonSets executed scripts to overwrite kubelet.config.json or modify systemd unit flags directly on the node, forcing kubelet restarts to apply non-standard configurations.

This approach creates significant liabilities:

Security perimeter violations: Granting containers root privileges and host access bypasses Kubernetes security boundaries, exposing worker nodes to container escape risks.
Node stability and auto-repair failures: Custom filesystem edits interfere with GKE node auto-upgrade and auto-repair mechanisms. When GKE reprovisions or updates a node, custom file modifications can lead to bootstrap failures.
Accelerator node instability: Running unsupported background scripts on GPU and TPU nodes risks disrupting specialized accelerator drivers, device plugins, and NUMA-aware scheduling routines.

Tunable CrashLoopBackOff eliminates these workarounds by providing a native, fully managed control plane configuration.

Native tuning through NodeSystemConfig and ComputeClass

GKE allows administrators to configure the maximum restart delay per node pool using the NodeSystemConfig API in GKE Standard, or via ComputeClass custom resources in GKE Autopilot.

The configuration exposes the following parameters:

Configurable range: The maxContainerRestartPeriod must be an integer between 1 second and 300 seconds. Setting it to 1s forces the kubelet to retry failed containers almost immediately, while values like 10s or 30s provide a balanced compromise.
Node pool isolation: The setting applies at the node pool level, allowing you to configure low restart delays for specialized AI/ML pools while keeping default backoff behavior for standard application pools.
Upstream alignment: This capability builds upon upstream Kubernetes enhancement KEP-4603, ensuring compatibility with core Kubernetes architectural standards.

Configuring restart backoffs on your node pools

You can configure tunable CrashLoopBackOff when creating new node pools or updating existing pools.

Configuring GKE Standard node pools

To create a node pool with a custom restart delay, pass the configuration using a system config file with gcloud:

# node-system-config.yaml
kubeletConfig:
  crashLoopBackOff:
    maxContainerRestartPeriod: 5s

Run the following command to apply the configuration:

gcloud container node-pools create accelerator-pool \
    --cluster=production-cluster \
    --location=us-central1-a \
    --system-config-from-file=node-system-config.yaml \
    --machine-type=g2-standard-24 \
    --accelerator=type=nvidia-l4,count=2

To update an existing node pool:

gcloud container node-pools update accelerator-pool \
    --cluster=production-cluster \
    --location=us-central1-a \
    --system-config-from-file=node-system-config.yaml

Configuring GKE Autopilot via ComputeClass

For clusters leveraging GKE Autopilot or Custom Compute Classes, declare the restart delay inside a ComputeClass manifest:

apiVersion: cloud.google.com/v1
kind: ComputeClass
metadata:
  name: fast-recovery-accelerator
spec:
  nodeConfig:
    systemConfig:
      kubeletConfig:
        crashLoopBackOff:
          maxContainerRestartPeriod: 5s

Workloads requesting this compute class automatically land on nodes provisioned with the 5-second maximum restart delay.

Monitoring and operational best practices

Reducing the maximum restart period causes failing containers to restart more frequently. To maintain cluster health, apply these operational practices:

Monitor restart rates: Track the kubernetes.io/container/restart_count metric in Cloud Monitoring. A sudden surge in restarts indicates an unrecoverable crash requiring debugging rather than rapid retries.
Track node resource utilization: Observe kubelet CPU and memory consumption (kubernetes.io/node/cpu/allocatable_utilization). Rapid restarts generate more container runtime and lifecycle events.
Combine with proper health probes: Ensure Pod definitions use realistic startupProbe and livenessProbe timeouts. Probes must allow sufficient initialization time before failing containers.

Next steps

Tunable CrashLoopBackOff removes a major constraint for high-performance workloads on GKE. By replacing risky DaemonSet workarounds with native control plane configuration, platform teams can accelerate AI training recovery, streamline sidecar startup, and protect node stability.

To configure restart periods for your clusters, review the official GKE node system configuration documentation, explore the GKE ComputeClass reference, and read upstream Kubernetes KEP-4603.

Native CORS support on GKE Gateway: Offloading cross-origin policy management to infrastructure

Olivier Bourgeois — Sun, 30 Aug 2026 19:09:03 +0000

Web browsers enforce the Same-Origin Policy by default to protect users from malicious scripts trying to read data across distinct origins. However, modern application architectures almost always require cross-origin communication. Single-page applications, mobile clients, and embedded web components regularly fetch data and stream AI model inferences across separate domains, subdomains, and ports.

To allow these interactions safely, applications must implement Cross-Origin Resource Sharing (CORS). For years, teams running Kubernetes workloads on Ingress-Nginx handled this using annotations like nginx.ingress.kubernetes.io/enable-cors. When migrating to the Kubernetes Gateway API and GKE Gateway, the lack of native CORS support was a frequent operational pain point, making it one of the most requested missing capabilities.

The GKE team addressed this gap with the Preview release of native CORS support for GKE Gateway and Inference Gateway. In this article, I will break down how the new CORS filter works, why moving cross-origin policy management to the load balancer matters, and the operational nuances you need to consider.

Why manage CORS at the ingress layer?

Implementing CORS inside individual backend applications introduces architectural friction across three main areas:

Redundant application logic: Every backend service or framework (Node.js, FastAPI, Spring, or inference engines like vLLM) must include middleware to evaluate incoming headers and generate preflight responses.
Preflight resource consumption: Complex web requests trigger preflight OPTIONS calls. Routing these preflight requests to backend containers wastes application memory, CPU cycles, and network bandwidth on pure protocol negotiation.
Configuration sprawl and drift: When dozens of microservices manage their own CORS policies, subtle discrepancies in allowed headers, exposed headers, or origin validation create security vulnerabilities and broken client integrations.

By shifting CORS processing to GKE Gateway, Google Cloud Load Balancing terminates OPTIONS preflight requests directly at the network edge and injects required response headers (Access-Control-Allow-Origin, Access-Control-Allow-Methods, and Access-Control-Allow-Headers). Your backend applications only receive validated application requests, removing boilerplate code and reducing compute overhead.

Configuring the CORS filter in HTTPRoute

GKE Gateway implements CORS support directly through the open-source Gateway API specification. You define policies declaratively using a CORS filter within the rules section of an HTTPRoute manifest.

Here is an example configuring a CORS policy on an API route:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: api-cors-route
  namespace: production
spec:
  parentRefs:
  - name: external-gateway
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /api
    backendRefs:
    - name: api-service
      port: 8080
    filters:
    - type: CORS
      cors:
        allowOrigins:
        - "https://app.example.com"
        - "https://*.partner-domain.com"
        allowMethods:
        - GET
        - POST
        - PUT
        - DELETE
        allowHeaders:
        - Authorization
        - Content-Type
        - X-Requested-With
        exposeHeaders:
        - X-Request-ID
        allowCredentials: true
        maxAge: 3600

The cors configuration block gives you granular control over the negotiation parameters:

allowOrigins: Specifies the allowed origins via explicit URLs (https://app.example.com), wildcard patterns (https://*.partner-domain.com), or a catch-all wildcard (*).
allowMethods: Lists the permitted HTTP methods. You can also specify * to allow all methods.
allowHeaders: Defines the HTTP request headers that clients can send in their requests.
exposeHeaders: Lists response headers that the browser makes accessible to client scripts beyond simple response headers.
allowCredentials: Sets the Access-Control-Allow-Credentials header to true or false, dictating whether browsers can share responses with requests carrying cookies or authentication headers.
maxAge: Defines the number of seconds the browser can cache the preflight response (defaulting to 5 seconds), significantly cutting down subsequent OPTIONS traffic.

Security considerations with credentials and wildcards

When designing your CORS policies on GKE Gateway, pay careful attention to the interaction between allowOrigins and allowCredentials.

Under standard browser security rules, browsers block responses to credentialed requests if Access-Control-Allow-Origin is set to a literal wildcard (*). However, when you configure wildcard patterns or a wildcard in allowOrigins in GKE Gateway, the controller dynamically matches and reflects the incoming request origin rather than returning a literal asterisk.

Because the browser sees an explicit origin returned alongside allowCredentials: true, it permits the response. If you configure allowOrigins: ["*"] with allowCredentials: true, any arbitrary website can potentially read authenticated user responses. For authenticated APIs, always define explicit domain lists rather than catch-all wildcards.

Supported GatewayClasses and architectural constraints

This Preview release supports single-cluster GKE Gateway deployments across three primary GatewayClasses:

gke-l7-rilb: Regional internal Application Load Balancer.
gke-l7-regional-external-managed: Regional external Application Load Balancer.
gke-l7-global-external-managed: Global external Application Load Balancer.

It also supports AI inference workloads exposed via Inference Gateway, allowing frontend chat interfaces or client SDKs to query served models directly across origins.

Before implementing this in production, keep the following technical limits in mind:

Single-cluster only: Multi-cluster gateways (gke-l7-gmc-*) do not currently support the CORS filter.
Filter conflicts: You cannot combine a CORS filter and a RequestRedirect filter within the same route rule.
URL map regular expression limits: The GKE Gateway controller translates wildcard origin patterns into regular expressions on the underlying Cloud Load Balancing URL maps. For gke-l7-global-external-managed, there is a limit of one regular expression per Gateway listener, and combining wildcard origins with PathPrefix matches is not supported. For regional external and regional internal GatewayClasses, you can use up to five regular expressions per hostname. Note that exact origins and catch-all * entries do not count against these regex quotas.

Next steps

Native CORS support in GKE Gateway eliminates one of the biggest functional gaps for organizations migrating workloads from legacy Ingress controllers to the Kubernetes Gateway API. By managing cross-origin policies declaratively at the routing layer, platform teams can simplify application code and centralize security posture across all services.

To learn more and begin testing CORS in your clusters, check out the official GKE Gateway CORS documentation and the upstream Gateway API CORS User Guide.

Unlocking workload rightsizing visibility on GKE: How VPA decision logs bring observability to autoscaling

Olivier Bourgeois — Sat, 29 Aug 2026 18:57:42 +0000

Automating resource management in Kubernetes has always required a high degree of trust. When you hand over CPU and memory sizing to the Vertical Pod Autoscaler (VPA), you expect it to rightsize containers efficiently without introducing unexpected restarts or performance regressions. Yet for many platform engineers, running VPA in production has felt like operating a black box.

Until recently, inspecting VPA decisions meant relying on standard Kubernetes events or running kubectl describe vpa. These events are transient, often expiring after an hour. If a Pod was evicted unexpectedly during an overnight batch job, or if an in-place resize failed silently due to node capacity limits, diagnosing the root cause the next morning was frustratingly difficult.

To solve this observability gap, the GKE team launched the Public Preview of VerticalPodAutoscaler (VPA) Logs. Available on GKE clusters running version 1.36.0-gke.1601000 or newer, this feature streams structured VPA decision events directly into Cloud Logging.

In this article, I will explain what VPA logs capture, how they demystify autoscaler decisions, and how you can use them to troubleshoot scaling actions and build reliable autonomous workload management.

The missing link in workload autoscaling observability

Vertical autoscaling decisions are inherently complex. The VPA controller continuously evaluates historical CPU and memory utilization, computes recommendations with upper and lower safety bounds, and determines whether an active container needs adjustment.

Without persistent logging, answering essential operational questions was difficult:

Why did VPA decide to evict a specific Pod instead of keeping it running?
Was an applied resource recommendation modified by Autopilot compute ratios or custom resource policies?
Did an in-place resource resize fail, forcing the controller to fall back to recreation?
How much confidence did the recommendation engine have when calculating new targets?

By exporting VPA decision events to Cloud Logging as first-class control plane logs (KCP_VPA), GKE gives platform operators a permanent audit trail. Combined with existing Horizontal Pod Autoscaler (HPA) logging, teams now have complete visibility across horizontal and vertical scaling dimensions.

Understanding the structure of VPA decision logs

VPA logs are emitted by the vpa-controller control plane component and stored under the log destination container.googleapis.com/vpa-controller in Cloud Logging. Each log entry arrives as a structured JSON payload containing detailed metadata about the target workload, the evaluation state, and the calculated resource bounds.

The controller categorizes decision logs across four primary operations:

Update recommendation (UPDATE_RECOMMENDATION): Emitted periodically (once per minute per VPA object). This log details the raw recommendation calculated by the recommender, including lower bound, upper bound, target, uncapped target, and recommendation confidence.
Evict Pod (EVICT_POD): Emitted when the VPA updater decides to evict a Pod to apply new resource requests under Recreate mode (or as a fallback if in-place resize fails).
Apply recommendation on eviction (APPLY_RECOMMENDATION_ON_EVICTION): Emitted when a newly scheduled replacement Pod receives resized resource requests during admission.
Apply recommendation in place (APPLY_RECOMMENDATION_IN_PLACE): Emitted when VPA modifies container resource limits and requests live on a running Pod without a restart under InPlaceOrRecreate mode.

Each log entry includes a state field (SUCCEEDED, SKIPPED, or FAILED) and an explanatory reason string. When an operation succeeds, the reason field clarifies whether applied recommendations diverged from raw recommendations due to policy caps or Autopilot ratio constraints.

Crucially, the payload includes a confidence field:

LOW: The recommender has processed fewer than 10 metric samples.
HIGH: The recommender has processed 10 or more metric samples, indicating a mature usage profile.

Enabling VPA decision logs on GKE

VPA logs can be enabled on both new and existing GKE clusters using the Google Cloud CLI.

Enabling logs on cluster creation

To create a new GKE cluster with VPA decision logs enabled, include KCP_VPA in the --logging flag alongside SYSTEM logs:

gcloud container clusters create CLUSTER_NAME \
    --location=LOCATION \
    --project=PROJECT_ID \
    --logging=SYSTEM,KCP_VPA

Updating an existing cluster

When updating an existing cluster, preserve your currently configured logging components so you do not inadvertently overwrite them. Add KCP_VPA to your existing configuration:

gcloud container clusters update CLUSTER_NAME \
    --location=LOCATION \
    --project=PROJECT_ID \
    --logging=SYSTEM,KCP_VPA

Verifying the logging configuration

You can confirm that KCP_VPA is active by retrieving the enabled logging components for the cluster:

gcloud container clusters describe CLUSTER_NAME \
    --location=LOCATION \
    --flatten=loggingConfig \
    --format='csv[delimiter=",",no-heading](componentConfig.enableComponents)'

The output will list KCP_VPA alongside your other active control plane components.

Practical queries for Logs Explorer

Once enabled, you can search and analyze VPA events directly in Google Cloud Logs Explorer.

To view all decision events for a specific workload within a cluster, use the following filter:

resource.type="k8s_control_plane_component"
resource.labels.cluster_name="CLUSTER_NAME"
logName="projects/PROJECT_ID/logs/container.googleapis.com%2Fvpa-controller"
jsonPayload.target.name="WORKLOAD_NAME"

To find instances where VPA skipped or failed an in-place resize operation, query by operation and state:

logName="projects/PROJECT_ID/logs/container.googleapis.com%2Fvpa-controller"
jsonPayload.operation="APPLY_RECOMMENDATION_IN_PLACE"
jsonPayload.state=("SKIPPED" OR "FAILED")

To audit recommendations generated with low confidence, filter on the confidence attribute:

logName="projects/PROJECT_ID/logs/container.googleapis.com%2Fvpa-controller"
jsonPayload.operation="UPDATE_RECOMMENDATION"
jsonPayload.confidence="LOW"

These queries enable site reliability teams to quickly diagnose scaling anomalies, track resize frequency, and identify workloads that require longer profiling before enforcing automated actuation.

Enabling autonomous, intent-based workload autoscaling

The launch of VPA decision logs represents more than a troubleshooting convenience. In modern cloud-native architectures, platform engineering teams are moving toward intent-based infrastructure where autonomous agents monitor, optimize, and heal application environments.

For an AI agent or automated governance pipeline to safely manage container resources, it requires complete observability into both horizontal and vertical scaling lifecycles. By providing a structured, historical record of why the VPA made every sizing decision, GKE equips platform teams and intelligent systems with the data needed to automate workload rightsizing with confidence.

Next steps

VerticalPodAutoscaler logs bring much-needed clarity to container resource optimization on GKE, turning automated rightsizing into a transparent, auditable process.

To get started with VPA logs on your clusters, review the official GKE vertical Pod autoscaler event logs documentation and learn more about configuring Vertical Pod Autoscaling in GKE.

Accelerating JVM startup on GKE: How VPA CPU startup boost eliminates ongoing resource waste

Olivier Bourgeois — Sat, 29 Aug 2026 05:03:06 +0000

Running Java and other JVM-based workloads in Kubernetes has long presented an infrastructure sizing challenge. During initialization, JVM applications require substantial CPU capacity to perform class loading, bytecode verification, framework dependency injection, and Just-In-Time (JIT) compilation. Once this initialization phase finishes, steady-state CPU consumption frequently drops by 50% to 80%.

To avoid slow startups and ensure new Pods pass readiness probes quickly, platform teams have historically over-provisioned CPU requests permanently. This compromise introduces persistent CPU waste—unutilized allocations that inflate infrastructure spend and degrade workload packing density on worker nodes.

To resolve this trade-off, the GKE team launched the Public Preview of VerticalPodAutoscaler (VPA) CPU Startup Boost. Available on GKE clusters running version 1.36.0-gke.4447000 or newer in the Rapid Channel, this feature provides up to 2x faster startup latency while dynamically reclaiming CPU capacity once the application stabilizes.

In this article, I will explain how CPU startup boost operates under the hood, how it leverages Kubernetes In-Place Pod Resize (IPPR), and how to configure it effectively for your clusters.

The JVM startup and CPU waste dilemma

When configuring CPU requests for a container, Kubernetes uses that value for scheduling decisions and CPU bandwidth enforcement through CFS quotas. If you rightsize a Java container for its steady-state requirement—say, 500m CPU—the application may experience severe CPU throttling during boot, stretching startup times from seconds into minutes.

Slow startups create cascading operational challenges:

Horizontal scaling lag: When traffic spikes occur, Horizontal Pod Autoscaler (HPA) creates new replicas, but those replicas take too long to start serving requests.
Flapping readiness probes: Applications that exceed initial startup budgets can fail health checks, triggering unwanted container restarts.
Resource waste: To avoid throttling, teams routinely allocate 2 to 4 vCPUs permanently. After the bootstrap window, those vCPUs sit idle across GKE Standard and Autopilot clusters.

CPU startup boost solves this by providing temporary CPU headroom during initialization, then reducing the allocation back to baseline without restarting the Pod.

How CPU startup boost works under the hood

The CPU startup boost lifecycle executes across three distinct phases:

Admission phase: When a Pod is created, the GKE VPA mutating admission webhook intercepts the request. The webhook calculates the boosted CPU request based on your policy and injects both the elevated CPU values and a tracking annotation (vpaCpuStartupBoost/<container-name>) before the scheduler places the Pod.
Startup phase: The container starts on a node with the higher CPU allocation, allowing JVM class loading and JIT compilation to run without CFS throttling.
Unboosting phase: Once the Pod satisfies its readiness checks and reaches Ready status, the configured durationSeconds timer begins. When the duration expires, the VPA Updater initiates an in-place resize back to baseline.

Because GKE utilizes Kubernetes In-Place Pod Resize (IPPR), this downscale happens live. The container is never terminated or restarted when the boost ends.

Configuring startup boost for your workloads

You configure CPU startup boost directly inside a standard VerticalPodAutoscaler Custom Resource. You can use VPA exclusively for startup boost or combine it with continuous autoscaling.

Using startup boost without continuous VPA actuation

If you manage steady-state resource requests manually in your Deployment manifest, set updateMode: "Off" in the VPA policy:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: java-app-startup-boost
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: customer-service
  updatePolicy:
    updateMode: "Off"
  startupBoost:
    cpu:
      type: Factor
      factor: 2
      durationSeconds: 10

In this manifest:

type: Factor: Multiplies the baseline CPU request (doubling 1 vCPU to 2 vCPUs during startup). You can also specify type: Quantity with a fixed addition like quantity: "2".
durationSeconds: 10: Keeps boosted CPU active for 10 seconds after the Pod reaches Ready state before reclaiming the resource.

Container-level targeting and continuous autoscaling

For multi-container Pods containing sidecars, you can isolate the boost to the primary application container using containerPolicies:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: java-app-advanced-boost
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: payment-service
  updatePolicy:
    updateMode: InPlaceOrRecreate
  resourcePolicy:
    containerPolicies:
    - containerName: app-server
      mode: Auto
      startupBoost:
        cpu:
          type: Quantity
          quantity: "2"
          durationSeconds: 0
    - containerName: envoy-proxy
      mode: "Off"
      startupBoost:
        cpu:
          type: Factor
          factor: 1

Setting updateMode: InPlaceOrRecreate allows GKE to boost startup resources, return to baseline, and then continue evaluating ongoing usage to rightsize the workload over time.

Key architecture and operational considerations

When adopting CPU startup boost across your clusters, keep the following operational rules in mind:

HPA integration: When pairing startup boost with HPA based on CPU utilization, always define a readinessProbe and set durationSeconds: 0. This configuration ensures the Pod unboosts immediately upon becoming ready, preventing startup CPU spikes from triggering false scale-out events.
GKE Autopilot resource rules: Autopilot validates compute ratios during Pod admission. Make sure your Pod's baseline memory allocation can support the boosted CPU ratio. Because the Scale-Out ComputeClass enforces a fixed 1:4 ratio, standard or performance compute classes are recommended.
Node capacity and autoscaling: On GKE Standard, ensure worker nodes have sufficient allocatable CPU to schedule boosted Pods. If a node lacks capacity, GKE caps the boost to what the node can fit.
Pod restart semantics: Startup boost triggers during initial Pod creation. If a container crashes and restarts within an existing Pod, the boost is not reapplied.

Verifying boost and in-place downscale

You can confirm that startup boost is active by inspecting Pod annotations and cluster events with kubectl:

# Check for the tracking annotation injected at admission
kubectl get pod <pod-name> -o jsonpath='{.metadata.annotations.vpaCpuStartupBoost/*}'

# Observe the in-place downscale event after readiness
kubectl get events --field-selector reason=InPlaceResizedByVPA

The InPlaceResizedByVPA event confirms that the container returned to baseline requests without a restart.

Next steps

VPA CPU Startup Boost eliminates a longstanding trade-off in Kubernetes workload sizing, helping Java microservices start faster while cutting idle resource costs.

To start testing startup boost on your GKE clusters, review the official GKE CPU startup boost documentation and explore Vertical Pod Autoscaling in GKE.

Two-step control plane upgrades in GKE: How minor version rollbacks work under the hood

Olivier Bourgeois — Thu, 27 Aug 2026 17:56:45 +0000

Kubernetes control plane minor version upgrades have historically been an all-or-nothing proposition. In standard Kubernetes clusters, upgrading a control plane from one minor version to the next—such as moving from 1.33 to 1.34—commits changes to storage schemas immediately. If an unexpected regression emerges after upgrading the API server, rolling back to the previous minor version was impossible without restoring etcd snapshots.

To eliminate this operational risk, the GKE team drove upstream contributions in Kubernetes Enhancement Proposal KEP-4330 (Compatibility Versions) and introduced two-step control plane upgrades. Following public preview validation with enterprise customers, this capability is now Generally Available (GA) across all GKE release channels.

In this article, I will explain how two-step upgrades work under the hood, how automated rollouts leverage canary analysis, and how to manage rollback-safe upgrades using the Google Cloud CLI and Terraform.

The problem with minor version upgrades

In Kubernetes, minor version releases introduce storage schema changes, deprecated API removals, and modified controller behaviors. When the kube-apiserver binary starts on a newer minor version, it writes resources using newer internal schemas.

Because earlier binary versions cannot parse data stored in newer schemas, Kubernetes prohibits control plane downgrades across minor versions. If an organization encountered an issue after upgrading, platform operators had to either run the degraded control plane or rebuild the cluster.

Two-step upgrades decouple binary execution from API capability enablement. By separating the upgrade into two distinct phases, GKE provides an observation period (also known as a soak window) during which operators or automated systems can monitor cluster behavior and roll back the control plane to the previous minor version with zero data loss.

Decoupling binary execution from emulated versions

The foundation of two-step upgrades is running a newer control plane binary in an emulated compatibility mode.

When a two-step upgrade begins, GKE advances the control plane through two sequential stages:

Step 1: Binary upgrade (emulated mode): GKE upgrades the control plane binary to the target minor version (e.g., 1.34), but configures the API server to emulate the previous minor version (1.33). In this state, the control plane executes the new binary logic, while API schemas match the older version. APIs removed in 1.34 remain accessible. During this soak period, you can safely roll back to 1.33.
Step 2: Emulated version upgrade (finalization): Once the soak window completes without incident, GKE updates the emulated version to match the binary version. This step permanently enables the new minor version API schemas and feature deprecations. After this point, rollback is no longer possible.

During the soak period, worker node pools cannot be upgraded beyond the emulated version to preserve Kubernetes version skew rules.

Automated orchestration with CPRS and Canary Analysis Service

For clusters configured for auto-upgrades, two-step upgrades are enabled out of the box with zero manual configuration required. The GKE rollout engine, known as Control Plane Rollout Service (CPRS), orchestrates the staged lifecycle natively:

CPRS upgrades the control plane binary while locking the emulated version to the previous minor release.
CPRS initiates an automated 24-hour soak window.
The Canary Analysis Service (CAS) monitors cluster health signals throughout the soak duration, including API latency, error rates, and Pod health metrics.
If CAS detects unexpected regressions, the rollout halts to allow automated or user-driven rollbacks.
Once the cluster passes all CAS evaluations and satisfies the soak timer, CPRS triggers step 2 to finalize the emulated version.

This validation framework has helped GKE control plane upgrades achieve a 99.999% (five nines) rolling 30-day success rate across the global fleet.

Initiating manual two-step upgrades

If your team manages upgrades manually, you can execute two-step upgrades using the Google Cloud CLI or Terraform.

To initiate a two-step upgrade with a custom soak duration using gcloud:

gcloud beta container clusters upgrade my-cluster \
    --location=us-central1 \
    --cluster-version=1.34.1-gke.1829001 \
    --control-plane-soak-duration=48h \
    --master

The --control-plane-soak-duration flag defines the rollback-safe window, supporting values from 6 hours up to 7 days (e.g., 48h or 2d).

For infrastructure-as-code workflows, Terraform includes official support for managing two-step control plane upgrades declaratively by specifying the target version and soak parameters in your GKE cluster resources.

Verifying upgrade status and executing a rollback

While the cluster is soaking in emulated mode, you can inspect the active rollback state using the CLI:

gcloud container clusters describe my-cluster \
    --location=us-central1 \
    --format="yaml(rollbackSafeUpgradeStatus)"

The output includes a rollbackSafeUpgradeStatus block with the target binary version, emulated version, remaining soak time, and previousVersion string.

If your monitoring tools uncover a regression during the soak window, you can roll back the control plane to the previous minor patch:

gcloud container clusters upgrade my-cluster \
    --location=us-central1 \
    --cluster-version=1.33.5-gke.1080000 \
    --master

Because the control plane ran in emulated mode, no new data formats were written to etcd. GKE downgrades the control plane binary back to the specified version without risking data corruption.

Completing the upgrade early

If your validation tests pass and you want to unlock new minor version features immediately without waiting for the soak duration to expire, you can finalize the upgrade manually:

gcloud beta container clusters clusters complete-control-plane-upgrade my-cluster \
    --location=us-central1

Once executed, GKE promotes the emulated version to match the binary version, completing the upgrade.

Operational requirements and limits

When planning two-step control plane upgrades, keep the following operational rules in mind:

Two-step upgrades apply to minor version upgrades to GKE 1.33 and later.
Control plane upgrades must proceed one minor version at a time.
Maintenance windows and exclusions are strictly respected.
Autopilot and regional Standard clusters maintain continuous control plane availability during both phases.

Next steps

Two-step control plane upgrades eliminate the risk of irreversible minor version updates in Kubernetes, providing platform engineers with automated safety and an emergency rollback mechanism.

To learn more about configuring two-step upgrades, read the official GKE cluster upgrade documentation and explore KEP-4330: Compatibility Versions.

Progressive cluster upgrades at scale: A technical guide to GKE rollout sequencing with custom stages

Olivier Bourgeois — Thu, 27 Aug 2026 03:10:21 +0000

Upgrading Kubernetes clusters across a large enterprise fleet is often a balancing act between staying current with security patches and avoiding outages. By default, Google Kubernetes Engine (GKE) rolls out automatic upgrades progressively according to Google Cloud regional timelines. While regional rollout works well for standalone clusters, it does not understand your organization's business topology. If you run staging clusters in us-central1 and critical production clusters in us-east1, a standard regional rollout could upgrade your production environment before your pre-production validation completes.

The General Availability (GA) release of GKE rollout sequencing with custom stages solves this challenge. It provides platform teams with declarative control to sequence cluster upgrades across fleets, environments, and even distinct Google Cloud organizations according to business criticality rather than cloud geography.

How rollout sequencing works

Rollout sequencing builds on GKE fleet management. Fleets serve as logical boundaries for environments such as development, staging, and production. With rollout sequencing, you define an ordered pipeline of upgrade stages managed by a central resource called RolloutSequence.

When GKE publishes a new automatic upgrade target for a release channel, or when you explicitly trigger a target version, the system creates a Rollout object. This rollout progresses through your defined stages sequentially:

Control plane upgrades start in the first stage. Once all control planes in that stage reach the target version, a stage soak timer begins.
Node upgrades run in parallel with control plane upgrades, respecting node pool upgrade strategies such as surge or blue-green.
When both control planes and nodes complete their upgrade and satisfy the configured soak duration, the rollout advances to the next stage in the sequence.

If an individual stage contains clusters that take longer than 30 days to finish upgrading—due to restrictive maintenance windows or exclusions—GKE triggers a forced soak period to avoid stalling the entire multi-stage pipeline indefinitely.

Granular stages with label selectors

Earlier fleet-based rollout sequencing operated strictly at the fleet level, meaning an entire fleet had to upgrade before another fleet could begin. Custom stages introduce the ability to split a single fleet into multiple granular rollout phases using Common Expression Language (CEL) label selectors.

For example, within a production fleet, you can label a subset of clusters as canary targets and upgrade them before the rest of production.

Here is an example YAML manifest defining a three-stage sequence:

- stage:
  fleet-projects:
  - projects/dev-fleet-host
  soak-duration: 3d
- stage:
  fleet-projects:
  - projects/prod-fleet-host
  label-selector: resource.labels.tier=='canary'
  soak-duration: 4d
- stage:
  fleet-projects:
  - projects/prod-fleet-host
  soak-duration: 7d

When structuring custom stages, keep two architectural rules in mind:

Catch-all requirement: If you use a label selector to target a subset of clusters in a fleet, the final stage referencing that fleet must omit the label selector. This acts as a catch-all stage to guarantee that all remaining clusters in the fleet are upgraded.
Conflict resolution: If a cluster matches multiple stages within a sequence, GKE assigns that cluster exclusively to the earliest matching stage.

You can register this configuration using the Google Cloud CLI:

gcloud container fleet rolloutsequences create prod-rollout-sequence \
    --display-name="Production rollout sequence" \
    --stage-config=rollout-sequence.yaml \
    --project=central-management-project

Operational controls and upgrade scoping

Real-world production environments require operational agility when unexpected workload anomalies occur. Rollout sequencing with custom stages provides real-time lifecycle controls over active rollouts without requiring you to dismantle your configuration:

Pause and resume: If an issue arises during validation in a canary stage, you can pause the active rollout. Pausing stops GKE from initiating new cluster upgrades in that stage and subsequent stages while allowing running operations to finish. Once mitigated, you can resume the rollout.
Stage completion: If automated testing confirms stability before a soak timer finishes, platform operators can execute a stage completion action (force-complete-stage). This skips the remaining soak duration and moves the rollout immediately to the next stage.
Rollout cancellation: If a version introduces an unresolvable regression, you can cancel the rollout. Canceling stops the progression of that specific target version across the sequence.

In addition to runtime actions, you can restrict the automatic upgrade scope for a rollout sequence. If your platform policy requires manual control over major and minor Kubernetes version changes while automating security patches, you can configure your sequence to roll out only control plane patch upgrades and node patch upgrades.

However, mandatory upgrades—such as control planes that have not received a patch in 90 days or clusters reaching the end of version support—will continue to execute to preserve cluster stability and security.

Multi-tenant and cross-organization management

For enterprises with distributed topologies, rollout sequences support cross-project and cross-organization fleet memberships. Following continuous delivery best practices, Google recommends creating and maintaining RolloutSequence resources in a dedicated host project.

A single rollout sequence can define up to 15 distinct stages, and fleets can accommodate up to 250 clusters (or up to 2,000 clusters when using lightweight memberships with an approved quota increase). Across all stages in a sequence, you can configure a total soak duration of up to 90 days, with individual stage soak times configured up to 30 days.

By shifting from regional upgrade schedules to business-aligned rollout sequences, platform engineers can automate Kubernetes lifecycle management while maintaining safety, visibility, and control across their entire fleet.

To implement progressive cluster upgrades in your environment, explore the official GKE rollout sequencing documentation to review detailed configuration parameters, CEL filter syntax, and gcloud CLI commands.

Taking control of cluster security: A deep dive into GKE ClusterNetworkPolicy

Olivier Bourgeois — Tue, 25 Aug 2026 20:44:29 +0000

In Kubernetes, network security has historically been a developer-centric responsibility. Standard Kubernetes NetworkPolicy objects are namespace-scoped. While this works beautifully for isolating individual microservices within a single application boundary, it introduces major operational headaches at scale. As clusters grow to accommodate multiple teams, business units, and environments, platform administrators are left without a native mechanism to enforce global security guardrails.

If you want to ensure that every pod in your cluster is blocked from reaching the cloud metadata server, or that certain namespaces are completely isolated from all others, you are forced to rely on complex policy engines, custom admission controllers, or automating the injection of namespace-scoped policies. This approach is fragile and difficult to audit.

The introduction of the GKE ClusterNetworkPolicy (currently in Public Preview) addresses this challenge. It introduces a cluster-scoped resource that allows security and platform teams to establish non-overridable security boundaries across the entire cluster, spanning all namespaces.

Evaluation hierarchy

To understand ClusterNetworkPolicy, you must first understand how GKE evaluates network traffic. Unlike standard namespace-scoped NetworkPolicy resources, which are additive (meaning if multiple policies select a pod, the traffic is allowed if any policy permits it), ClusterNetworkPolicy uses a strict, sequential evaluation pipeline where the first matching rule wins.

Traffic flows through three distinct policy tiers in order:

Admin tier: Policies in this tier are evaluated first. This is where administrators define mandatory guardrails that developers cannot bypass. If a rule in this tier matches a traffic flow with an Accept or Deny verdict, evaluation stops immediately.
NetworkPolicy tier: If traffic is not explicitly allowed or blocked by the Admin tier, it falls through to standard namespace-scoped NetworkPolicy resources configured by DevOps teams and developers.
Baseline tier: If no namespace-scoped policies match the traffic, it falls through to the Baseline tier. Here, administrators can define fallback policies, such as a default-deny posture, which developers can choose to override in their respective namespaces.
Default GKE behavior: If a packet matches no rules in any tier, it falls back to GKE's default behavior, which is an implicit allow.

Within each tier, policies are evaluated based on an explicit numeric priority (0 to 1000, where lower numbers indicate higher precedence). Inside a single policy object, rules are evaluated from top to bottom.

Verdict actions: Accept, deny, pass

Every rule in a ClusterNetworkPolicy must trigger one of three actions:

Deny: Blocks the traffic immediately. This action short-circuits the evaluation pipeline.
Accept: Permits the traffic immediately. This also short-circuits the pipeline.
Pass: Transfers the evaluation to the next tier in the hierarchy.

The Pass action is particularly powerful. It enables platform administrators to target specific traffic flows—for example, web traffic on port 8080—and delegate the final decision to namespace owners. The traffic will bypass any remaining rules in the Admin tier and be evaluated against standard namespace-scoped network policies. If the namespace owners configure a policy to accept the traffic, it is allowed; if they have no policy, the traffic proceeds to the Baseline tier. This strikes a balance between centralized compliance and developer agility.

YAML examples

Let's examine how to write these policies. Consider a scenario where you want to isolate a sensitive namespace from all internal cluster traffic. By placing this policy in the Admin tier, you guarantee that namespace-scoped policies cannot override it.

Here is a manifest for a global deny policy targeting a sensitive namespace:

apiVersion: policy.networking.k8s.io/v1alpha2
kind: ClusterNetworkPolicy
metadata:
  name: cluster-wide-deny-sensitive
spec:
  tier: Admin
  priority: 10
  subject:
    namespaces:
      matchLabels:
        kubernetes.io/metadata.name: sensitive-ns
  ingress:
  - action: Deny
    name: deny-all-ingress
    from:
    - namespaces:
        matchLabels: {}
  egress:
  - action: Deny
    name: deny-all-egress
    to:
    - namespaces:
        matchLabels: {}

Now, consider a different scenario: you want to establish a default-deny posture across the cluster, but you want to allow developers to open up traffic as needed for their applications. For this, you use the Baseline tier.

Here is a baseline default-deny policy:

apiVersion: policy.networking.k8s.io/v1alpha2
kind: ClusterNetworkPolicy
metadata:
  name: default-deny-baseline
spec:
  tier: Baseline
  priority: 100
  subject:
    namespaces: {}
  ingress:
  - action: Deny
    name: baseline-deny-all
    from:
    - namespaces: {}

If a developer in the namespace my-app-ns needs to allow ingress traffic to their frontend service from an external ingress controller, they can simply deploy a standard namespace-scoped NetworkPolicy that permits that traffic. Because the standard policy is evaluated in the NetworkPolicy tier (which runs before the Baseline tier), GKE will permit the traffic without needing platform admin intervention.

Under the hood

The GKE implementation of ClusterNetworkPolicy is built entirely on open-source Cilium, reinforcing a commitment to community-driven foundations. GKE backported this support to Cilium 1.19 for GKE clusters, and the upstream capability will be generally available to the broader open-source ecosystem starting with Cilium 1.20.

To use this feature, your GKE clusters must meet the following requirements:

Run GKE version 1.36.0-gke.4447000 or later.
Use GKE Dataplane V2.

As this feature is in Public Preview, there are a few scale limits to keep in mind. Specifically, a single ClusterNetworkPolicy object can contain a maximum of 100 ingress rules and 100 egress rules. You can monitor traffic flow verdicts and troubleshoot your policies using GKE Dataplane V2 Observability tools.

By decoupling cluster-wide guardrails from namespace-level configurations, ClusterNetworkPolicy gives security teams the control they need without hindering developer speed.

To get started with configuring cluster-wide security policies on your clusters, check out the official GKE ClusterNetworkPolicy documentation for step-by-step guides and YAML examples.

Seamless scaling with VPA In-place Pod Resize on GKE

Olivier Bourgeois — Thu, 04 Jun 2026 18:19:03 +0000

Right-sizing Kubernetes workloads is a common platform engineering challenge. Set your requests too high, and you burn cloud budgets on idle capacity; set your limits too low, and your applications face throttling or dreaded OOMKills.

For years, the Vertical Pod Autoscaler (VPA) has been the standard answer to this problem, automatically adjusting CPU and memory requirements based on actual usage. However, this method of scaling came with a significant catch that prevented widespread adoption for critical workloads: applying new resource parameters required evicting and restarting the pod.

This disruption was often unacceptable for stateful applications, long-running connections, or latency-sensitive services.

Introducing In-place Pod Resize (IPPR) on GKE

In-place Pod Resize (IPPR) changes the game by allowing Kubernetes to modify resource requests and limits on live, running containers directly through the underlying container runtime, without triggering a restart.

By combining the intelligence of VPA with the non-disruptive nature of IPPR, GKE users finally have a viable path to dynamic, seamless, and automated right-sizing.

Note: As of writing, VPA IPPR is in Preview on GKE. While it is a massive step forward, I recommend evaluating it in staging environments before rolling it out to production workloads.

Getting started with IPPR

To use In-place Pod Resize, you need a GKE cluster running version 1.34.0-gke.2201000 or later.

GKE Autopilot: VPA is enabled by default.
GKE Standard: Requires the Vertical Pod Autoscaling feature to be enabled.

1. Enable the feature

If you aren't using Autopilot, ensure your cluster is created or updated with the necessary feature flags:

gcloud container clusters create CLUSTER_NAME \
  --project=PROJECT_ID \
  --location=us-east1 \
  --release-channel=rapid \
  --enable-vertical-pod-autoscaling

2. Define your VPA object

Create a VerticalPodAutoscaler resource targeting your Deployment or StatefulSet. The crucial element here is setting spec.updatePolicy.updateMode to InPlaceOrRecreate.

apiVersion: "autoscaling.k8s.io/v1"
kind: "VerticalPodAutoscaler"
metadata:
  name: "my-vpa"
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: "Deployment"
    name: "my-deployment"
  updatePolicy:
    updateMode: "InPlaceOrRecreate"

3. Watch it scale

Apply the resource to your cluster and monitor your application under load. Instead of watching Pods terminate and recreate, you can watch the resources modify live using kubectl describe.

kubectl describe pod POD_NAME

Look for the AllocatedResources field or check the events section. You will see the requests change in real-time to match the VPA recommendations, while the Restart Count remains exactly the same.

The "Or Recreate" Fallback: Keep in mind that physics still apply. If VPA recommends a resource size that exceeds the remaining capacity of the Node your Pod is currently running on, an in-place resize is impossible. In this scenario, VPA will fall back to evicting and recreating the Pod so it can be scheduled onto a larger or emptier Node.

Ready to dive deeper?

While this introduction covers the basics of IPPR, right-sizing is just one part of a robust scaling strategy. Implementing VPA often goes hand-in-hand with horizontal scaling and cluster autoscaling. Check out the guide to master scaling on GKE: Run full-stack workloads at scale on GKE.

Surviving the eviction: How to build interrupt-resilient AI workloads on GKE

Olivier Bourgeois — Tue, 02 Jun 2026 20:02:20 +0000

You did everything right. You containerized your massive model training job, deployed it to Google Kubernetes Engine (GKE), and cleverly routed it to a Spot VM node pool to save up to 90% on compute costs.

Everything is humming along perfectly for 38 hours. Then, a priority on-demand customer needs capacity, Google Cloud reclaims your underlying Spot VM, and your node vanishes.

Whether you are using preemptible Spot VMs to save money, or leveraging the Dynamic Workload Scheduler (DWS) to queue for scarce GPUs, you are building on top of ephemeral compute. The hardware will eventually be taken away. To successfully run critical AI workloads on un-committed capacity, your application architecture must assume failure is a given.

Here is a practical guide to building interruptible workloads on GKE.

1. Trap the warning

When Google Cloud reclaims a Spot VM, it doesn't just pull the power cord immediately. It sends an ACPI signal to the underlying node to begin a power off cycle. Kubernetes intercepts this and translates it into a SIGTERM signal sent directly to your running containers.

You have a grace period (up to 15 seconds for non-system pods) between that SIGTERM and the fatal SIGKILL.

Your application must explicitly listen for this signal. When caught, your code should immediately stop accepting new batches, finish its current loop, flush any in-memory data to disk, and exit with a 0 (success) status.

Here is a simple example on how to catch this signal in Python:

import signal
import sys
import time

def handle_sigterm(signum, frame):
    print("Received SIGTERM. Initiating graceful shutdown...")
    # 1. Stop processing new data
    # 2. Flush memory to persistent storage
    # 3. Save final checkpoint
    print("State saved. Exiting cleanly.")
    sys.exit(0)

# Register the signal handler
signal.signal(signal.SIGTERM, handle_sigterm)

# Your main training loop
print("Starting training loop...")
while True:
    # Train model...
    time.sleep(1)

2. Externalize your checkpoints

If your container dies, everything inside its local filesystem dies with it. To survive an interruption, you must periodically save your progress (model weights, optimizer states, epoch counters, etc.) to an external storage location.

Cloud Storage (GCS) is a common solution for this on Google Cloud.

Save frequently: Decide on a checkpointing interval that balances the cost of lost work against the overhead of writing to storage. Saving every epoch or every few thousand steps is common, but this can vary based on your needs.
Keep it local: Ensure your GCS buckets are in the same region as your GKE cluster (e.g., us-central1) to minimize latency and avoid outbound data transfer fees.
Resume, don't restart: The first thing your container's startup script should do is to check for that GCS bucket. If a checkpoint exists in the bucket, load it and resume from that exact step.

3. Design for Idempotency

"Idempotency" is a fancy way of saying that doing something twice yields the same result as doing it once.

Imagine a batch inference job that reads an image, processes it, and writes the result to a database. If your pod is preempted milliseconds after writing to the database but before it can mark the task as complete, the rescheduled pod will likely process that image again.

If your database blindly inserts new rows, you now have unintentional, duplicate data.

To build an idempotent pipeline:

Use UPSERT (update or insert) operations in your database based on a unique identifier (like an image ID).
Check if a record already exists before spending expensive GPU cycles processing it.

4. Decouple work queues for batch processing

If you are running a massive batch processing or inference job across thousands of files, do not write a monolithic Python script that iterates through a static CSV list. If the node dies at row 5,000, managing the state of where to restart is a nightmare.

Instead, decouple the workload:

Publish the work: Break your dataset down into discrete messages and push them into a message broker like Pub/Sub.
Pull the work: Have your Spot VM worker pods pull messages off the queue one by one or as a small chunk (e.g. 10 at a time).
Acknowledge completion: Only send an "ACK" (acknowledgment) back to Pub/Sub once the result is safely stored.

If a Spot node is preempted mid-inference, the worker dies before sending the ACK. After a brief timeout, Pub/Sub will automatically make that specific message available again. Another surviving worker pod will pick it up seamlessly. No data lost, no manual intervention required.

Key takeaways

Running on ephemeral compute like Spot VMs isn't just an infrastructure choice; it is a design choice. By handling termination signals, checkpointing aggressively to GCS, ensuring idempotent operations, and decoupling your queues, you can unlock massive cost savings and tap into scarce GPU pools without sacrificing reliability.

Strategies for running AI workloads on GKE without committed quota

Olivier Bourgeois — Mon, 01 Jun 2026 18:54:52 +0000

You’ve built your model, your training code is containerized, and you’re ready to scale up on Google Kubernetes Engine (GKE). You go to provision your nvidia-h100-80gb node pool and... QUOTA_EXCEEDED.

It’s one of the most common (and frustrating) roadblocks in modern AI development. High-end accelerators like H100s, A100s, and TPUs are in massive demand, and securing permanent, on-demand quota for them can be difficult. But a lack of on-demand quota doesn't mean you're out of options.

GKE provides two powerful, cost-effective strategies for acquiring these scarce resources when you can't get standard, on-demand instances: Spot VMs and the Dynamic Workload Scheduler (DWS).

Let's break down what they are, when to use each, and how to implement them.

Strategy 1: Spot VMs

Spot VMs are Google Cloud's excess compute capacity sold at a massive discount, up to 90% off the price of standard on-demand VMs. They are perfect for workloads that can be interrupted.

The catch is that Spot VMs have no availability guarantee. Google Cloud can "preempt" (i.e., terminate) them at any time if that capacity is needed for on-demand customers. GKE gets a 30-second warning before the node is terminated. Kubernetes uses this window to gracefully shut down your application (giving non-system pods up to 15 seconds to wrap up) before the node vanishes.

When to use Spot VMs for accelerators

Spot VMs are ideal for workloads that are:

Fault-tolerant and stateless: Your application can handle a node vanishing and having its pods rescheduled elsewhere.
Batch processing: Jobs that can be easily restarted or have checkpointing built-in.
CI/CD pipelines: Running tests or builds that don't need 100% uptime.

How to use Spot VMs in GKE

You can easily add a Spot VM node pool to your GKE Standard cluster. The key is to use Spot VMs for your workers, not your critical system pods.

Create a dedicated Spot VM node pool:
When creating a node pool, simply add the --spot flag and apply a taint so standard pods don't accidentally schedule there.

gcloud container node-pools create spot-gpu-pool \
  --cluster=my-cluster \
  --region=northamerica-northeast2 \
  --machine-type=g2-standard-4 \
  --accelerator=type=nvidia-l4,count=1 \
  --spot \
  --node-taints=cloud.google.com/gke-spot=true:NoSchedule

Add the toleration to your workload's YAML:

# You want to "tolerate" that taint only on the specific workloads you want to run there.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-batch-job
spec:
  template:
    # ... other specs
    spec:
      tolerations:
      - key: "cloud.google.com/gke-spot"
        operator: "Equal"
        value: "true"
        effect: "NoSchedule"

This architecture ensures your critical components stay on reliable on-demand nodes, while your interruptible training jobs run on the preemptible Spot nodes. (Note: If you are using GKE Autopilot, you simply request a Spot class in your pod spec and GKE handles the taints and nodes automatically!)

Strategy 2: Dynamic Workload Scheduler (DWS) with flex-start

What if your job can't be interrupted? Many large-scale training jobs can take days. While they might have checkpointing, restarting from scratch every few hours due to Spot preemptions is inefficient and costly.

This is where Dynamic Workload Scheduler (DWS) comes in.

DWS is a feature designed specifically for acquiring large amounts of scarce resources (like GPUs and TPUs) for batch workloads. It changes the request from "Give me this GPU right now" to "Give me this GPU when it becomes available."

The catch here is that your job doesn't start immediately. It enters a queue and might wait for minutes, hours, or even days for the resources to be provisioned.

There are a few massive upsides:

It's a "get-in-line" system: Instead of you writing a script to retry the gcloud command every 5 minutes, DWS queues your request and provisions the nodes automatically when capacity is found.
No preemptions: Once your DWS nodes are provisioned, they are yours for the entire duration of your job (up to seven days). They are not Spot VMs and will not be preempted.
Cost savings: DWS workloads are also offered at a significant discount (up to 53% for L4 GPUs) compared to on-demand instances.

When to use DWS

DWS is perfect for:

Model training or reinforcement learning (RL): Jobs that need to run uninterrupted for many hours or days.
Batch inference: Running a large inference job on a massive dataset.
Any workload that is not time-sensitive to start, but is sensitive to interruptions.

How to use DWS with flex-start

The flex-start mode in DWS is what enables this "wait-in-queue" behavior. If you are using a GKE Autopilot cluster (or a Standard cluster with Node Auto-provisioning enabled), implementing this is incredibly simple.

You do not need to create complex custom resources; you simply signal your intent via a nodeSelector in your standard Kubernetes Job object.

Request flex-start in your Job:
In your Job.yaml, add the cloud.google.com/gke-flex-start: "true" node selector alongside your accelerator request.

apiVersion: batch/v1
kind: Job
metadata:
  name: my-training-job
spec:
  template:
    spec:
      nodeSelector:
        cloud.google.com/gke-flex-start: "true"
        cloud.google.com/gke-accelerator: nvidia-tesla-a100
      containers:
      - name: my-trainer
        image: "gcr.io/my-project/my-training-image"
        resources:
          limits:
            nvidia.com/gpu: 1
      restartPolicy: Never

When you apply this Job, GKE sees the flex-start selector. It puts the Job's Pods into a Pending state until the DWS queueing system can provision the requested A100 node. Once the node is ready, the Pod is scheduled, your job runs to completion without interruption, and the node is automatically deprovisioned.

Which strategy should you choose?

Here's a simple cheat sheet:

Feature	Spot VMs	DWS with flex-start
Best for	Fault-tolerant, interruptible workloads	Long-running, uninterruptible batch jobs
Primary trade-off	Starts fast, can be preempted at any time	Can wait hours/days to start
Cost savings	Up to 90%	Up to 50%
GKE mode	Standard or Autopilot	Standard or Autopilot
Implementation	--spot flag in a Node Pool	cloud.google.com/gke-flex-start nodeSelector

By mastering both Spot VMs and the Dynamic Workload Scheduler, you can build a resilient and cost-effective AI platform on GKE, even when on-demand accelerator quota seems impossible to find.

Hands-on with Gemma 3 on Google Cloud

Olivier Bourgeois — Fri, 05 Dec 2025 16:31:49 +0000

The landscape of generative AI is shifting. While proprietary APIs are powerful, there is a growing demand for open models—models where the architecture and weights are publicly available. This shift puts control back in the hands of developers, offering transparency, data privacy, and the ability to fine-tune for specific use cases.

To help you navigate this landscape, we are releasing two new hands-on labs featuring Gemma 3, Google’s latest family of lightweight, state-of-the-art open models.

Why Gemma?

Built from the same research and technology as Gemini, Gemma models are designed for responsible AI development. Gemma 3 is particularly exciting because it offers multimodal capabilities (text and image) and fits efficiently on smaller hardware footprints while delivering massive performance.

But running a model on your laptop is very different from running it in production. You need scale, reliability, and hardware acceleration (GPUs). The question is: Where should you deploy?

We have prepared two different paths for you, depending on your infrastructure needs: Cloud Run or Google Kubernetes Engine (GKE).

Path 1: The Serverless Approach (Cloud Run)

Best for: Developers who want an API up and running instantly without managing infrastructure, scaling to zero when not in use.

If your priority is simplicity and cost-efficiency for stateless workloads, Cloud Run is your answer. It abstracts away the server management entirely. With the recent addition of GPU support on Cloud Run, you can now serve modern LLMs without provisioning a cluster.

Start the lab!

Lab: Serving Gemma 3 with vLLM on Cloud Run

Objectives:

Containerize vLLM (a high-throughput serving engine).
Deploy Gemma 3 to Cloud Run.
Leverage GPU acceleration for fast inference.
Expose an OpenAI-compatible API endpoint.

Path 2: The Platform Approach (GKE)

Best for: Engineering teams building complex AI platforms, requiring high throughput, custom orchestration, or integration with a broader microservices ecosystem.

When your application graduates from a prototype to a high-traffic production system, you need the control of Kubernetes. GKE Autopilot gives you that power while still handling the heavy lifting of node management. This path creates a seamless journey from local testing to cloud production.

Start the lab!

Lab: Deploying Open Models on GKE

In this lab, you will learn how to:

Prototype locally using Ollama.
Containerize your setup and transition to GKE Autopilot.
Deploy a scalable inference service using standard Kubernetes manifests.
Manage resources effectively for production workloads.

Which Path Will You Choose?

Whether you are looking for the serverless simplicity of Cloud Run or the robust orchestration of GKE, Google Cloud provides the tools to take Gemma 3 from a concept to a deployed application.

Dive into the labs today and start building:

Share your progress and connect with others on the journey using the hashtag #ProductionReadyAI. Happy learning!

These labs are part of the Open Models module in our official Production-Ready AI with Google Cloud program. Explore the full curriculum for more content that will help you bridge the gap from a promising prototype to a production-grade AI application.

Observability in Action: A Google Cloud Next demo

Olivier Bourgeois — Mon, 05 May 2025 18:17:47 +0000

It was only a few weeks ago that over 32,000 cloud practitioners from all over the world came together in Las Vegas to attend Google Cloud Next 2025. Beyond the keynotes, the workshops, and the multiple jam-packed tracks of talks and sessions, an entire expo hall offered attendees the opportunity to observe or play around with more than 500 live demos. Let’s check out one of these demos!

Overview of the demo

The main goals of the Observability in Action demo were twofold. We wanted to showcase various ways of interacting with metrics and logs. And we wanted to give attendees a little bit of an interactive experience. For the interactive part of the demo, we utilized various oversized physical buttons and pedals that could be used to select answers or confirm inputs.

The flow of the demo was as followed:

We ask the attendee to type in a prompt that they wanted sent to an AI model.
The prompt was sent in the background to three different models: Gemma 3 on Cloud Run, Gemini 2.0 Flash on Vertex AI, and Gemini 2.0 Flash-Lite on Vertex AI. This generated logs and metrics.
The attendee was then given a short quiz about these three models. Each quiz input also generated logs and metrics.
At the end of the quiz, we give the attendee a rundown of their answers, and then flip over to the Google Cloud Console.
In Cloud Monitoring, we showcase the various native metrics that Cloud Run offers, custom metrics implemented using OpenTelemetry, as well as the Cloud Trace functionality.
Finally, we turn to BigQuery to showcase how we can mirror logs to a database for further analysis using Jupyter Notebooks.

Architecture

While the demo frontend runs locally, the backend is deployed as a Cloud Run instance. This instance is then talking to Gemini through the Vertex AI SDK and to Gemma through its own Cloud Run instance. The persistent state of the demo resides in a Firestore database. All Cloud Run logs are mirrored to BigQuery using a simple sink.

Visualizing metrics using Cloud Monitoring

Cloud Monitoring provides visibility into the performance and health of your cloud applications and infrastructure. It collects metrics, events, and metadata from Google Cloud services and other sources, allowing you to visualize this data on dashboards and create alerts for critical issues. This is useful for proactively identifying and resolving problems, optimizing resource utilization, improving uptime, and understanding system behavior, ultimately leading to more reliable and cost-effective applications.

For services like Cloud Run which we’re using for the backend of this demo, Cloud Monitoring automatically collects a wide array of native metrics without any setup needed. This includes data points such as request latency, count, container CPU and memory usage, and instance counts. This out-of-the-box integration means developers get immediate insights into their serverless application's performance and resource consumption, simplifying troubleshooting and optimization efforts.

Cloud Trace is a distributed tracing system within Google Cloud that helps you understand request latency across your application and its services. It tracks how long different parts of your application take to process requests, visualizing the entire request flow. This is particularly valuable for identifying performance bottlenecks in microservices architectures by showing where time is spent during a request's lifecycle.

Here’s a real life example: In this demo we send a prompt to multiple models. We were sure we implemented concurrency correctly (so the calls to the three different models should’ve happened in parallel) yet the latency seems significantly higher than expected. When we dug into the trace of a call, we quickly realized that we were accidentally making those calls sequentially! These traces were made available to us via an OpenTelemetry instrumentation we added to our code.

Interact with your logs with BigQuery

BigQuery is a serverless enterprise data warehouse that enables super-fast SQL queries on large datasets without infrastructure management. It's built for scalable analytics, supports diverse data types, and integrates machine learning, offering a powerful platform for insights from real-time and historical data.

With a simple sink, you can directly stream logs from Cloud Logging into BigQuery, transforming it into a powerful, long-term log analytics platform. This allows you to run complex SQL queries across extensive historical log data, which is invaluable for in-depth security audits, compliance, and identifying subtle operational trends.

Connecting BigQuery to Jupyter Notebooks further enhances log analysis capabilities. This empowers users to leverage Python and data science libraries for advanced data exploration, custom visualizations, and machine learning on log data, facilitating deeper insights and shareable, interactive analysis beyond standard logging tools.

For this demo, we built a Jupyter Notebook that did analysis on the various interactive quiz events, cross-referenced answers with an external Firestore database, and built tables and charts of the resulting data.

Try it out!

Want to try this demo from home? The source code is available on GitHub.

Want to learn more about observability on Google Cloud? Check out these resources: