DEV Community: Google Cloud

Tunable CrashLoopBackOff in GKE: Accelerating AI/ML recovery and eliminating risky node hacks

Olivier Bourgeois — Mon, 31 Aug 2026 18:21:41 +0000

In Kubernetes, few status messages are as familiar as CrashLoopBackOff. When a container exits unexpectedly, the kubelet steps in to prevent the failing process from overwhelming the host node. To achieve this, it applies an exponential backoff delay before each restart attempt. While this defensive mechanism protects node stability, its rigid default parameters create friction for modern workloads.

The default Kubernetes restart logic starts at a 10-second delay and doubles after each failure (10s, 20s, 40s, 80s, 160s) until reaching a 5-minute (300-second) ceiling. In fast-moving development environments, distributed AI/ML training runs, and architectures with critical sidecars, waiting up to five minutes for a container to retry stalls entire pipelines.

To solve this operational bottleneck, the GKE team launched the General Availability of tunable CrashLoopBackOff. By exposing crashLoopBackOff.maxContainerRestartPeriod through the GKE NodeSystemConfig API and Custom Compute Classes (CCC), platform teams can now securely reduce restart delays down to 1 second.

In this article, I will explain why fixed restart delays impact modern workloads, how GKE enables native tuning without privileged host workarounds, and how to configure and monitor this capability.

The cost of fixed restart backoffs

Kubernetes designed exponential backoff to protect the kubelet and runtime from CPU exhaustion caused by rapid restart loops. However, a maximum backoff delay of 300 seconds introduces severe delays across several workload patterns:

AI/ML training and inference pipelines: Large-scale distributed training jobs synchronize state across hundreds of accelerator nodes hosting GPUs or TPUs. If a single worker encounters a temporary network timeout, initialization hiccup, or dependency race condition, the container enters CrashLoopBackOff. When one Pod delays by 5 minutes, the entire gang-scheduled training job stalls, leaving expensive accelerators idle.
Critical sidecar initialization: Modern microservices frequently depend on sidecars for service mesh routing, mTLS credential renewal, or secret injection. If a sidecar crashes due to transient backend unavailability, the primary application container cannot serve traffic until the sidecar restarts and passes readiness checks.
Fast developer iteration cycles: During active debugging and CI runs, engineers need containers to restart immediately after updating an environment variable or dependency. Waiting through several minutes of backoff adds unnecessary latency to test suites.

The risks of legacy node workarounds

Because upstream Kubernetes historically lacked a supported interface to tune restart delays, platform teams turned to dangerous workarounds.

The most common hack involved running privileged DaemonSets with host filesystem access (hostPID: true, hostPath: /etc/kubernetes). These DaemonSets executed scripts to overwrite kubelet.config.json or modify systemd unit flags directly on the node, forcing kubelet restarts to apply non-standard configurations.

This approach creates significant liabilities:

Security perimeter violations: Granting containers root privileges and host access bypasses Kubernetes security boundaries, exposing worker nodes to container escape risks.
Node stability and auto-repair failures: Custom filesystem edits interfere with GKE node auto-upgrade and auto-repair mechanisms. When GKE reprovisions or updates a node, custom file modifications can lead to bootstrap failures.
Accelerator node instability: Running unsupported background scripts on GPU and TPU nodes risks disrupting specialized accelerator drivers, device plugins, and NUMA-aware scheduling routines.

Tunable CrashLoopBackOff eliminates these workarounds by providing a native, fully managed control plane configuration.

Native tuning through NodeSystemConfig and ComputeClass

GKE allows administrators to configure the maximum restart delay per node pool using the NodeSystemConfig API in GKE Standard, or via ComputeClass custom resources in GKE Autopilot.

The configuration exposes the following parameters:

Configurable range: The maxContainerRestartPeriod must be an integer between 1 second and 300 seconds. Setting it to 1s forces the kubelet to retry failed containers almost immediately, while values like 10s or 30s provide a balanced compromise.
Node pool isolation: The setting applies at the node pool level, allowing you to configure low restart delays for specialized AI/ML pools while keeping default backoff behavior for standard application pools.
Upstream alignment: This capability builds upon upstream Kubernetes enhancement KEP-4603, ensuring compatibility with core Kubernetes architectural standards.

Configuring restart backoffs on your node pools

You can configure tunable CrashLoopBackOff when creating new node pools or updating existing pools.

Configuring GKE Standard node pools

To create a node pool with a custom restart delay, pass the configuration using a system config file with gcloud:

# node-system-config.yaml
kubeletConfig:
  crashLoopBackOff:
    maxContainerRestartPeriod: 5s

Run the following command to apply the configuration:

gcloud container node-pools create accelerator-pool \
    --cluster=production-cluster \
    --location=us-central1-a \
    --system-config-from-file=node-system-config.yaml \
    --machine-type=g2-standard-24 \
    --accelerator=type=nvidia-l4,count=2

To update an existing node pool:

gcloud container node-pools update accelerator-pool \
    --cluster=production-cluster \
    --location=us-central1-a \
    --system-config-from-file=node-system-config.yaml

Configuring GKE Autopilot via ComputeClass

For clusters leveraging GKE Autopilot or Custom Compute Classes, declare the restart delay inside a ComputeClass manifest:

apiVersion: cloud.google.com/v1
kind: ComputeClass
metadata:
  name: fast-recovery-accelerator
spec:
  nodeConfig:
    systemConfig:
      kubeletConfig:
        crashLoopBackOff:
          maxContainerRestartPeriod: 5s

Workloads requesting this compute class automatically land on nodes provisioned with the 5-second maximum restart delay.

Monitoring and operational best practices

Reducing the maximum restart period causes failing containers to restart more frequently. To maintain cluster health, apply these operational practices:

Monitor restart rates: Track the kubernetes.io/container/restart_count metric in Cloud Monitoring. A sudden surge in restarts indicates an unrecoverable crash requiring debugging rather than rapid retries.
Track node resource utilization: Observe kubelet CPU and memory consumption (kubernetes.io/node/cpu/allocatable_utilization). Rapid restarts generate more container runtime and lifecycle events.
Combine with proper health probes: Ensure Pod definitions use realistic startupProbe and livenessProbe timeouts. Probes must allow sufficient initialization time before failing containers.

Next steps

Tunable CrashLoopBackOff removes a major constraint for high-performance workloads on GKE. By replacing risky DaemonSet workarounds with native control plane configuration, platform teams can accelerate AI training recovery, streamline sidecar startup, and protect node stability.

To configure restart periods for your clusters, review the official GKE node system configuration documentation, explore the GKE ComputeClass reference, and read upstream Kubernetes KEP-4603.

Native CORS support on GKE Gateway: Offloading cross-origin policy management to infrastructure

Olivier Bourgeois — Sun, 30 Aug 2026 19:09:03 +0000

Web browsers enforce the Same-Origin Policy by default to protect users from malicious scripts trying to read data across distinct origins. However, modern application architectures almost always require cross-origin communication. Single-page applications, mobile clients, and embedded web components regularly fetch data and stream AI model inferences across separate domains, subdomains, and ports.

To allow these interactions safely, applications must implement Cross-Origin Resource Sharing (CORS). For years, teams running Kubernetes workloads on Ingress-Nginx handled this using annotations like nginx.ingress.kubernetes.io/enable-cors. When migrating to the Kubernetes Gateway API and GKE Gateway, the lack of native CORS support was a frequent operational pain point, making it one of the most requested missing capabilities.

The GKE team addressed this gap with the Preview release of native CORS support for GKE Gateway and Inference Gateway. In this article, I will break down how the new CORS filter works, why moving cross-origin policy management to the load balancer matters, and the operational nuances you need to consider.

Why manage CORS at the ingress layer?

Implementing CORS inside individual backend applications introduces architectural friction across three main areas:

Redundant application logic: Every backend service or framework (Node.js, FastAPI, Spring, or inference engines like vLLM) must include middleware to evaluate incoming headers and generate preflight responses.
Preflight resource consumption: Complex web requests trigger preflight OPTIONS calls. Routing these preflight requests to backend containers wastes application memory, CPU cycles, and network bandwidth on pure protocol negotiation.
Configuration sprawl and drift: When dozens of microservices manage their own CORS policies, subtle discrepancies in allowed headers, exposed headers, or origin validation create security vulnerabilities and broken client integrations.

By shifting CORS processing to GKE Gateway, Google Cloud Load Balancing terminates OPTIONS preflight requests directly at the network edge and injects required response headers (Access-Control-Allow-Origin, Access-Control-Allow-Methods, and Access-Control-Allow-Headers). Your backend applications only receive validated application requests, removing boilerplate code and reducing compute overhead.

Configuring the CORS filter in HTTPRoute

GKE Gateway implements CORS support directly through the open-source Gateway API specification. You define policies declaratively using a CORS filter within the rules section of an HTTPRoute manifest.

Here is an example configuring a CORS policy on an API route:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: api-cors-route
  namespace: production
spec:
  parentRefs:
  - name: external-gateway
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /api
    backendRefs:
    - name: api-service
      port: 8080
    filters:
    - type: CORS
      cors:
        allowOrigins:
        - "https://app.example.com"
        - "https://*.partner-domain.com"
        allowMethods:
        - GET
        - POST
        - PUT
        - DELETE
        allowHeaders:
        - Authorization
        - Content-Type
        - X-Requested-With
        exposeHeaders:
        - X-Request-ID
        allowCredentials: true
        maxAge: 3600

The cors configuration block gives you granular control over the negotiation parameters:

allowOrigins: Specifies the allowed origins via explicit URLs (https://app.example.com), wildcard patterns (https://*.partner-domain.com), or a catch-all wildcard (*).
allowMethods: Lists the permitted HTTP methods. You can also specify * to allow all methods.
allowHeaders: Defines the HTTP request headers that clients can send in their requests.
exposeHeaders: Lists response headers that the browser makes accessible to client scripts beyond simple response headers.
allowCredentials: Sets the Access-Control-Allow-Credentials header to true or false, dictating whether browsers can share responses with requests carrying cookies or authentication headers.
maxAge: Defines the number of seconds the browser can cache the preflight response (defaulting to 5 seconds), significantly cutting down subsequent OPTIONS traffic.

Security considerations with credentials and wildcards

When designing your CORS policies on GKE Gateway, pay careful attention to the interaction between allowOrigins and allowCredentials.

Under standard browser security rules, browsers block responses to credentialed requests if Access-Control-Allow-Origin is set to a literal wildcard (*). However, when you configure wildcard patterns or a wildcard in allowOrigins in GKE Gateway, the controller dynamically matches and reflects the incoming request origin rather than returning a literal asterisk.

Because the browser sees an explicit origin returned alongside allowCredentials: true, it permits the response. If you configure allowOrigins: ["*"] with allowCredentials: true, any arbitrary website can potentially read authenticated user responses. For authenticated APIs, always define explicit domain lists rather than catch-all wildcards.

Supported GatewayClasses and architectural constraints

This Preview release supports single-cluster GKE Gateway deployments across three primary GatewayClasses:

gke-l7-rilb: Regional internal Application Load Balancer.
gke-l7-regional-external-managed: Regional external Application Load Balancer.
gke-l7-global-external-managed: Global external Application Load Balancer.

It also supports AI inference workloads exposed via Inference Gateway, allowing frontend chat interfaces or client SDKs to query served models directly across origins.

Before implementing this in production, keep the following technical limits in mind:

Single-cluster only: Multi-cluster gateways (gke-l7-gmc-*) do not currently support the CORS filter.
Filter conflicts: You cannot combine a CORS filter and a RequestRedirect filter within the same route rule.
URL map regular expression limits: The GKE Gateway controller translates wildcard origin patterns into regular expressions on the underlying Cloud Load Balancing URL maps. For gke-l7-global-external-managed, there is a limit of one regular expression per Gateway listener, and combining wildcard origins with PathPrefix matches is not supported. For regional external and regional internal GatewayClasses, you can use up to five regular expressions per hostname. Note that exact origins and catch-all * entries do not count against these regex quotas.

Next steps

Native CORS support in GKE Gateway eliminates one of the biggest functional gaps for organizations migrating workloads from legacy Ingress controllers to the Kubernetes Gateway API. By managing cross-origin policies declaratively at the routing layer, platform teams can simplify application code and centralize security posture across all services.

To learn more and begin testing CORS in your clusters, check out the official GKE Gateway CORS documentation and the upstream Gateway API CORS User Guide.

Unlocking workload rightsizing visibility on GKE: How VPA decision logs bring observability to autoscaling

Olivier Bourgeois — Sat, 29 Aug 2026 18:57:42 +0000

Automating resource management in Kubernetes has always required a high degree of trust. When you hand over CPU and memory sizing to the Vertical Pod Autoscaler (VPA), you expect it to rightsize containers efficiently without introducing unexpected restarts or performance regressions. Yet for many platform engineers, running VPA in production has felt like operating a black box.

Until recently, inspecting VPA decisions meant relying on standard Kubernetes events or running kubectl describe vpa. These events are transient, often expiring after an hour. If a Pod was evicted unexpectedly during an overnight batch job, or if an in-place resize failed silently due to node capacity limits, diagnosing the root cause the next morning was frustratingly difficult.

To solve this observability gap, the GKE team launched the Public Preview of VerticalPodAutoscaler (VPA) Logs. Available on GKE clusters running version 1.36.0-gke.1601000 or newer, this feature streams structured VPA decision events directly into Cloud Logging.

In this article, I will explain what VPA logs capture, how they demystify autoscaler decisions, and how you can use them to troubleshoot scaling actions and build reliable autonomous workload management.

The missing link in workload autoscaling observability

Vertical autoscaling decisions are inherently complex. The VPA controller continuously evaluates historical CPU and memory utilization, computes recommendations with upper and lower safety bounds, and determines whether an active container needs adjustment.

Without persistent logging, answering essential operational questions was difficult:

Why did VPA decide to evict a specific Pod instead of keeping it running?
Was an applied resource recommendation modified by Autopilot compute ratios or custom resource policies?
Did an in-place resource resize fail, forcing the controller to fall back to recreation?
How much confidence did the recommendation engine have when calculating new targets?

By exporting VPA decision events to Cloud Logging as first-class control plane logs (KCP_VPA), GKE gives platform operators a permanent audit trail. Combined with existing Horizontal Pod Autoscaler (HPA) logging, teams now have complete visibility across horizontal and vertical scaling dimensions.

Understanding the structure of VPA decision logs

VPA logs are emitted by the vpa-controller control plane component and stored under the log destination container.googleapis.com/vpa-controller in Cloud Logging. Each log entry arrives as a structured JSON payload containing detailed metadata about the target workload, the evaluation state, and the calculated resource bounds.

The controller categorizes decision logs across four primary operations:

Update recommendation (UPDATE_RECOMMENDATION): Emitted periodically (once per minute per VPA object). This log details the raw recommendation calculated by the recommender, including lower bound, upper bound, target, uncapped target, and recommendation confidence.
Evict Pod (EVICT_POD): Emitted when the VPA updater decides to evict a Pod to apply new resource requests under Recreate mode (or as a fallback if in-place resize fails).
Apply recommendation on eviction (APPLY_RECOMMENDATION_ON_EVICTION): Emitted when a newly scheduled replacement Pod receives resized resource requests during admission.
Apply recommendation in place (APPLY_RECOMMENDATION_IN_PLACE): Emitted when VPA modifies container resource limits and requests live on a running Pod without a restart under InPlaceOrRecreate mode.

Each log entry includes a state field (SUCCEEDED, SKIPPED, or FAILED) and an explanatory reason string. When an operation succeeds, the reason field clarifies whether applied recommendations diverged from raw recommendations due to policy caps or Autopilot ratio constraints.

Crucially, the payload includes a confidence field:

LOW: The recommender has processed fewer than 10 metric samples.
HIGH: The recommender has processed 10 or more metric samples, indicating a mature usage profile.

Enabling VPA decision logs on GKE

VPA logs can be enabled on both new and existing GKE clusters using the Google Cloud CLI.

Enabling logs on cluster creation

To create a new GKE cluster with VPA decision logs enabled, include KCP_VPA in the --logging flag alongside SYSTEM logs:

gcloud container clusters create CLUSTER_NAME \
    --location=LOCATION \
    --project=PROJECT_ID \
    --logging=SYSTEM,KCP_VPA

Updating an existing cluster

When updating an existing cluster, preserve your currently configured logging components so you do not inadvertently overwrite them. Add KCP_VPA to your existing configuration:

gcloud container clusters update CLUSTER_NAME \
    --location=LOCATION \
    --project=PROJECT_ID \
    --logging=SYSTEM,KCP_VPA

Verifying the logging configuration

You can confirm that KCP_VPA is active by retrieving the enabled logging components for the cluster:

gcloud container clusters describe CLUSTER_NAME \
    --location=LOCATION \
    --flatten=loggingConfig \
    --format='csv[delimiter=",",no-heading](componentConfig.enableComponents)'

The output will list KCP_VPA alongside your other active control plane components.

Practical queries for Logs Explorer

Once enabled, you can search and analyze VPA events directly in Google Cloud Logs Explorer.

To view all decision events for a specific workload within a cluster, use the following filter:

resource.type="k8s_control_plane_component"
resource.labels.cluster_name="CLUSTER_NAME"
logName="projects/PROJECT_ID/logs/container.googleapis.com%2Fvpa-controller"
jsonPayload.target.name="WORKLOAD_NAME"

To find instances where VPA skipped or failed an in-place resize operation, query by operation and state:

logName="projects/PROJECT_ID/logs/container.googleapis.com%2Fvpa-controller"
jsonPayload.operation="APPLY_RECOMMENDATION_IN_PLACE"
jsonPayload.state=("SKIPPED" OR "FAILED")

To audit recommendations generated with low confidence, filter on the confidence attribute:

logName="projects/PROJECT_ID/logs/container.googleapis.com%2Fvpa-controller"
jsonPayload.operation="UPDATE_RECOMMENDATION"
jsonPayload.confidence="LOW"

These queries enable site reliability teams to quickly diagnose scaling anomalies, track resize frequency, and identify workloads that require longer profiling before enforcing automated actuation.

Enabling autonomous, intent-based workload autoscaling

The launch of VPA decision logs represents more than a troubleshooting convenience. In modern cloud-native architectures, platform engineering teams are moving toward intent-based infrastructure where autonomous agents monitor, optimize, and heal application environments.

For an AI agent or automated governance pipeline to safely manage container resources, it requires complete observability into both horizontal and vertical scaling lifecycles. By providing a structured, historical record of why the VPA made every sizing decision, GKE equips platform teams and intelligent systems with the data needed to automate workload rightsizing with confidence.

Next steps

VerticalPodAutoscaler logs bring much-needed clarity to container resource optimization on GKE, turning automated rightsizing into a transparent, auditable process.

To get started with VPA logs on your clusters, review the official GKE vertical Pod autoscaler event logs documentation and learn more about configuring Vertical Pod Autoscaling in GKE.

Accelerating JVM startup on GKE: How VPA CPU startup boost eliminates ongoing resource waste

Olivier Bourgeois — Sat, 29 Aug 2026 05:03:06 +0000

Running Java and other JVM-based workloads in Kubernetes has long presented an infrastructure sizing challenge. During initialization, JVM applications require substantial CPU capacity to perform class loading, bytecode verification, framework dependency injection, and Just-In-Time (JIT) compilation. Once this initialization phase finishes, steady-state CPU consumption frequently drops by 50% to 80%.

To avoid slow startups and ensure new Pods pass readiness probes quickly, platform teams have historically over-provisioned CPU requests permanently. This compromise introduces persistent CPU waste—unutilized allocations that inflate infrastructure spend and degrade workload packing density on worker nodes.

To resolve this trade-off, the GKE team launched the Public Preview of VerticalPodAutoscaler (VPA) CPU Startup Boost. Available on GKE clusters running version 1.36.0-gke.4447000 or newer in the Rapid Channel, this feature provides up to 2x faster startup latency while dynamically reclaiming CPU capacity once the application stabilizes.

In this article, I will explain how CPU startup boost operates under the hood, how it leverages Kubernetes In-Place Pod Resize (IPPR), and how to configure it effectively for your clusters.

The JVM startup and CPU waste dilemma

When configuring CPU requests for a container, Kubernetes uses that value for scheduling decisions and CPU bandwidth enforcement through CFS quotas. If you rightsize a Java container for its steady-state requirement—say, 500m CPU—the application may experience severe CPU throttling during boot, stretching startup times from seconds into minutes.

Slow startups create cascading operational challenges:

Horizontal scaling lag: When traffic spikes occur, Horizontal Pod Autoscaler (HPA) creates new replicas, but those replicas take too long to start serving requests.
Flapping readiness probes: Applications that exceed initial startup budgets can fail health checks, triggering unwanted container restarts.
Resource waste: To avoid throttling, teams routinely allocate 2 to 4 vCPUs permanently. After the bootstrap window, those vCPUs sit idle across GKE Standard and Autopilot clusters.

CPU startup boost solves this by providing temporary CPU headroom during initialization, then reducing the allocation back to baseline without restarting the Pod.

How CPU startup boost works under the hood

The CPU startup boost lifecycle executes across three distinct phases:

Admission phase: When a Pod is created, the GKE VPA mutating admission webhook intercepts the request. The webhook calculates the boosted CPU request based on your policy and injects both the elevated CPU values and a tracking annotation (vpaCpuStartupBoost/<container-name>) before the scheduler places the Pod.
Startup phase: The container starts on a node with the higher CPU allocation, allowing JVM class loading and JIT compilation to run without CFS throttling.
Unboosting phase: Once the Pod satisfies its readiness checks and reaches Ready status, the configured durationSeconds timer begins. When the duration expires, the VPA Updater initiates an in-place resize back to baseline.

Because GKE utilizes Kubernetes In-Place Pod Resize (IPPR), this downscale happens live. The container is never terminated or restarted when the boost ends.

Configuring startup boost for your workloads

You configure CPU startup boost directly inside a standard VerticalPodAutoscaler Custom Resource. You can use VPA exclusively for startup boost or combine it with continuous autoscaling.

Using startup boost without continuous VPA actuation

If you manage steady-state resource requests manually in your Deployment manifest, set updateMode: "Off" in the VPA policy:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: java-app-startup-boost
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: customer-service
  updatePolicy:
    updateMode: "Off"
  startupBoost:
    cpu:
      type: Factor
      factor: 2
      durationSeconds: 10

In this manifest:

type: Factor: Multiplies the baseline CPU request (doubling 1 vCPU to 2 vCPUs during startup). You can also specify type: Quantity with a fixed addition like quantity: "2".
durationSeconds: 10: Keeps boosted CPU active for 10 seconds after the Pod reaches Ready state before reclaiming the resource.

Container-level targeting and continuous autoscaling

For multi-container Pods containing sidecars, you can isolate the boost to the primary application container using containerPolicies:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: java-app-advanced-boost
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: payment-service
  updatePolicy:
    updateMode: InPlaceOrRecreate
  resourcePolicy:
    containerPolicies:
    - containerName: app-server
      mode: Auto
      startupBoost:
        cpu:
          type: Quantity
          quantity: "2"
          durationSeconds: 0
    - containerName: envoy-proxy
      mode: "Off"
      startupBoost:
        cpu:
          type: Factor
          factor: 1

Setting updateMode: InPlaceOrRecreate allows GKE to boost startup resources, return to baseline, and then continue evaluating ongoing usage to rightsize the workload over time.

Key architecture and operational considerations

When adopting CPU startup boost across your clusters, keep the following operational rules in mind:

HPA integration: When pairing startup boost with HPA based on CPU utilization, always define a readinessProbe and set durationSeconds: 0. This configuration ensures the Pod unboosts immediately upon becoming ready, preventing startup CPU spikes from triggering false scale-out events.
GKE Autopilot resource rules: Autopilot validates compute ratios during Pod admission. Make sure your Pod's baseline memory allocation can support the boosted CPU ratio. Because the Scale-Out ComputeClass enforces a fixed 1:4 ratio, standard or performance compute classes are recommended.
Node capacity and autoscaling: On GKE Standard, ensure worker nodes have sufficient allocatable CPU to schedule boosted Pods. If a node lacks capacity, GKE caps the boost to what the node can fit.
Pod restart semantics: Startup boost triggers during initial Pod creation. If a container crashes and restarts within an existing Pod, the boost is not reapplied.

Verifying boost and in-place downscale

You can confirm that startup boost is active by inspecting Pod annotations and cluster events with kubectl:

# Check for the tracking annotation injected at admission
kubectl get pod <pod-name> -o jsonpath='{.metadata.annotations.vpaCpuStartupBoost/*}'

# Observe the in-place downscale event after readiness
kubectl get events --field-selector reason=InPlaceResizedByVPA

The InPlaceResizedByVPA event confirms that the container returned to baseline requests without a restart.

Next steps

VPA CPU Startup Boost eliminates a longstanding trade-off in Kubernetes workload sizing, helping Java microservices start faster while cutting idle resource costs.

To start testing startup boost on your GKE clusters, review the official GKE CPU startup boost documentation and explore Vertical Pod Autoscaling in GKE.

Two-step control plane upgrades in GKE: How minor version rollbacks work under the hood

Olivier Bourgeois — Thu, 27 Aug 2026 17:56:45 +0000

Kubernetes control plane minor version upgrades have historically been an all-or-nothing proposition. In standard Kubernetes clusters, upgrading a control plane from one minor version to the next—such as moving from 1.33 to 1.34—commits changes to storage schemas immediately. If an unexpected regression emerges after upgrading the API server, rolling back to the previous minor version was impossible without restoring etcd snapshots.

To eliminate this operational risk, the GKE team drove upstream contributions in Kubernetes Enhancement Proposal KEP-4330 (Compatibility Versions) and introduced two-step control plane upgrades. Following public preview validation with enterprise customers, this capability is now Generally Available (GA) across all GKE release channels.

In this article, I will explain how two-step upgrades work under the hood, how automated rollouts leverage canary analysis, and how to manage rollback-safe upgrades using the Google Cloud CLI and Terraform.

The problem with minor version upgrades

In Kubernetes, minor version releases introduce storage schema changes, deprecated API removals, and modified controller behaviors. When the kube-apiserver binary starts on a newer minor version, it writes resources using newer internal schemas.

Because earlier binary versions cannot parse data stored in newer schemas, Kubernetes prohibits control plane downgrades across minor versions. If an organization encountered an issue after upgrading, platform operators had to either run the degraded control plane or rebuild the cluster.

Two-step upgrades decouple binary execution from API capability enablement. By separating the upgrade into two distinct phases, GKE provides an observation period (also known as a soak window) during which operators or automated systems can monitor cluster behavior and roll back the control plane to the previous minor version with zero data loss.

Decoupling binary execution from emulated versions

The foundation of two-step upgrades is running a newer control plane binary in an emulated compatibility mode.

When a two-step upgrade begins, GKE advances the control plane through two sequential stages:

Step 1: Binary upgrade (emulated mode): GKE upgrades the control plane binary to the target minor version (e.g., 1.34), but configures the API server to emulate the previous minor version (1.33). In this state, the control plane executes the new binary logic, while API schemas match the older version. APIs removed in 1.34 remain accessible. During this soak period, you can safely roll back to 1.33.
Step 2: Emulated version upgrade (finalization): Once the soak window completes without incident, GKE updates the emulated version to match the binary version. This step permanently enables the new minor version API schemas and feature deprecations. After this point, rollback is no longer possible.

During the soak period, worker node pools cannot be upgraded beyond the emulated version to preserve Kubernetes version skew rules.

Automated orchestration with CPRS and Canary Analysis Service

For clusters configured for auto-upgrades, two-step upgrades are enabled out of the box with zero manual configuration required. The GKE rollout engine, known as Control Plane Rollout Service (CPRS), orchestrates the staged lifecycle natively:

CPRS upgrades the control plane binary while locking the emulated version to the previous minor release.
CPRS initiates an automated 24-hour soak window.
The Canary Analysis Service (CAS) monitors cluster health signals throughout the soak duration, including API latency, error rates, and Pod health metrics.
If CAS detects unexpected regressions, the rollout halts to allow automated or user-driven rollbacks.
Once the cluster passes all CAS evaluations and satisfies the soak timer, CPRS triggers step 2 to finalize the emulated version.

This validation framework has helped GKE control plane upgrades achieve a 99.999% (five nines) rolling 30-day success rate across the global fleet.

Initiating manual two-step upgrades

If your team manages upgrades manually, you can execute two-step upgrades using the Google Cloud CLI or Terraform.

To initiate a two-step upgrade with a custom soak duration using gcloud:

gcloud beta container clusters upgrade my-cluster \
    --location=us-central1 \
    --cluster-version=1.34.1-gke.1829001 \
    --control-plane-soak-duration=48h \
    --master

The --control-plane-soak-duration flag defines the rollback-safe window, supporting values from 6 hours up to 7 days (e.g., 48h or 2d).

For infrastructure-as-code workflows, Terraform includes official support for managing two-step control plane upgrades declaratively by specifying the target version and soak parameters in your GKE cluster resources.

Verifying upgrade status and executing a rollback

While the cluster is soaking in emulated mode, you can inspect the active rollback state using the CLI:

gcloud container clusters describe my-cluster \
    --location=us-central1 \
    --format="yaml(rollbackSafeUpgradeStatus)"

The output includes a rollbackSafeUpgradeStatus block with the target binary version, emulated version, remaining soak time, and previousVersion string.

If your monitoring tools uncover a regression during the soak window, you can roll back the control plane to the previous minor patch:

gcloud container clusters upgrade my-cluster \
    --location=us-central1 \
    --cluster-version=1.33.5-gke.1080000 \
    --master

Because the control plane ran in emulated mode, no new data formats were written to etcd. GKE downgrades the control plane binary back to the specified version without risking data corruption.

Completing the upgrade early

If your validation tests pass and you want to unlock new minor version features immediately without waiting for the soak duration to expire, you can finalize the upgrade manually:

gcloud beta container clusters clusters complete-control-plane-upgrade my-cluster \
    --location=us-central1

Once executed, GKE promotes the emulated version to match the binary version, completing the upgrade.

Operational requirements and limits

When planning two-step control plane upgrades, keep the following operational rules in mind:

Two-step upgrades apply to minor version upgrades to GKE 1.33 and later.
Control plane upgrades must proceed one minor version at a time.
Maintenance windows and exclusions are strictly respected.
Autopilot and regional Standard clusters maintain continuous control plane availability during both phases.

Next steps

Two-step control plane upgrades eliminate the risk of irreversible minor version updates in Kubernetes, providing platform engineers with automated safety and an emergency rollback mechanism.

To learn more about configuring two-step upgrades, read the official GKE cluster upgrade documentation and explore KEP-4330: Compatibility Versions.

Progressive cluster upgrades at scale: A technical guide to GKE rollout sequencing with custom stages

Olivier Bourgeois — Thu, 27 Aug 2026 03:10:21 +0000

Upgrading Kubernetes clusters across a large enterprise fleet is often a balancing act between staying current with security patches and avoiding outages. By default, Google Kubernetes Engine (GKE) rolls out automatic upgrades progressively according to Google Cloud regional timelines. While regional rollout works well for standalone clusters, it does not understand your organization's business topology. If you run staging clusters in us-central1 and critical production clusters in us-east1, a standard regional rollout could upgrade your production environment before your pre-production validation completes.

The General Availability (GA) release of GKE rollout sequencing with custom stages solves this challenge. It provides platform teams with declarative control to sequence cluster upgrades across fleets, environments, and even distinct Google Cloud organizations according to business criticality rather than cloud geography.

How rollout sequencing works

Rollout sequencing builds on GKE fleet management. Fleets serve as logical boundaries for environments such as development, staging, and production. With rollout sequencing, you define an ordered pipeline of upgrade stages managed by a central resource called RolloutSequence.

When GKE publishes a new automatic upgrade target for a release channel, or when you explicitly trigger a target version, the system creates a Rollout object. This rollout progresses through your defined stages sequentially:

Control plane upgrades start in the first stage. Once all control planes in that stage reach the target version, a stage soak timer begins.
Node upgrades run in parallel with control plane upgrades, respecting node pool upgrade strategies such as surge or blue-green.
When both control planes and nodes complete their upgrade and satisfy the configured soak duration, the rollout advances to the next stage in the sequence.

If an individual stage contains clusters that take longer than 30 days to finish upgrading—due to restrictive maintenance windows or exclusions—GKE triggers a forced soak period to avoid stalling the entire multi-stage pipeline indefinitely.

Granular stages with label selectors

Earlier fleet-based rollout sequencing operated strictly at the fleet level, meaning an entire fleet had to upgrade before another fleet could begin. Custom stages introduce the ability to split a single fleet into multiple granular rollout phases using Common Expression Language (CEL) label selectors.

For example, within a production fleet, you can label a subset of clusters as canary targets and upgrade them before the rest of production.

Here is an example YAML manifest defining a three-stage sequence:

- stage:
  fleet-projects:
  - projects/dev-fleet-host
  soak-duration: 3d
- stage:
  fleet-projects:
  - projects/prod-fleet-host
  label-selector: resource.labels.tier=='canary'
  soak-duration: 4d
- stage:
  fleet-projects:
  - projects/prod-fleet-host
  soak-duration: 7d

When structuring custom stages, keep two architectural rules in mind:

Catch-all requirement: If you use a label selector to target a subset of clusters in a fleet, the final stage referencing that fleet must omit the label selector. This acts as a catch-all stage to guarantee that all remaining clusters in the fleet are upgraded.
Conflict resolution: If a cluster matches multiple stages within a sequence, GKE assigns that cluster exclusively to the earliest matching stage.

You can register this configuration using the Google Cloud CLI:

gcloud container fleet rolloutsequences create prod-rollout-sequence \
    --display-name="Production rollout sequence" \
    --stage-config=rollout-sequence.yaml \
    --project=central-management-project

Operational controls and upgrade scoping

Real-world production environments require operational agility when unexpected workload anomalies occur. Rollout sequencing with custom stages provides real-time lifecycle controls over active rollouts without requiring you to dismantle your configuration:

Pause and resume: If an issue arises during validation in a canary stage, you can pause the active rollout. Pausing stops GKE from initiating new cluster upgrades in that stage and subsequent stages while allowing running operations to finish. Once mitigated, you can resume the rollout.
Stage completion: If automated testing confirms stability before a soak timer finishes, platform operators can execute a stage completion action (force-complete-stage). This skips the remaining soak duration and moves the rollout immediately to the next stage.
Rollout cancellation: If a version introduces an unresolvable regression, you can cancel the rollout. Canceling stops the progression of that specific target version across the sequence.

In addition to runtime actions, you can restrict the automatic upgrade scope for a rollout sequence. If your platform policy requires manual control over major and minor Kubernetes version changes while automating security patches, you can configure your sequence to roll out only control plane patch upgrades and node patch upgrades.

However, mandatory upgrades—such as control planes that have not received a patch in 90 days or clusters reaching the end of version support—will continue to execute to preserve cluster stability and security.

Multi-tenant and cross-organization management

For enterprises with distributed topologies, rollout sequences support cross-project and cross-organization fleet memberships. Following continuous delivery best practices, Google recommends creating and maintaining RolloutSequence resources in a dedicated host project.

A single rollout sequence can define up to 15 distinct stages, and fleets can accommodate up to 250 clusters (or up to 2,000 clusters when using lightweight memberships with an approved quota increase). Across all stages in a sequence, you can configure a total soak duration of up to 90 days, with individual stage soak times configured up to 30 days.

By shifting from regional upgrade schedules to business-aligned rollout sequences, platform engineers can automate Kubernetes lifecycle management while maintaining safety, visibility, and control across their entire fleet.

To implement progressive cluster upgrades in your environment, explore the official GKE rollout sequencing documentation to review detailed configuration parameters, CEL filter syntax, and gcloud CLI commands.

Taking control of cluster security: A deep dive into GKE ClusterNetworkPolicy

Olivier Bourgeois — Tue, 25 Aug 2026 20:44:29 +0000

In Kubernetes, network security has historically been a developer-centric responsibility. Standard Kubernetes NetworkPolicy objects are namespace-scoped. While this works beautifully for isolating individual microservices within a single application boundary, it introduces major operational headaches at scale. As clusters grow to accommodate multiple teams, business units, and environments, platform administrators are left without a native mechanism to enforce global security guardrails.

If you want to ensure that every pod in your cluster is blocked from reaching the cloud metadata server, or that certain namespaces are completely isolated from all others, you are forced to rely on complex policy engines, custom admission controllers, or automating the injection of namespace-scoped policies. This approach is fragile and difficult to audit.

The introduction of the GKE ClusterNetworkPolicy (currently in Public Preview) addresses this challenge. It introduces a cluster-scoped resource that allows security and platform teams to establish non-overridable security boundaries across the entire cluster, spanning all namespaces.

Evaluation hierarchy

To understand ClusterNetworkPolicy, you must first understand how GKE evaluates network traffic. Unlike standard namespace-scoped NetworkPolicy resources, which are additive (meaning if multiple policies select a pod, the traffic is allowed if any policy permits it), ClusterNetworkPolicy uses a strict, sequential evaluation pipeline where the first matching rule wins.

Traffic flows through three distinct policy tiers in order:

Admin tier: Policies in this tier are evaluated first. This is where administrators define mandatory guardrails that developers cannot bypass. If a rule in this tier matches a traffic flow with an Accept or Deny verdict, evaluation stops immediately.
NetworkPolicy tier: If traffic is not explicitly allowed or blocked by the Admin tier, it falls through to standard namespace-scoped NetworkPolicy resources configured by DevOps teams and developers.
Baseline tier: If no namespace-scoped policies match the traffic, it falls through to the Baseline tier. Here, administrators can define fallback policies, such as a default-deny posture, which developers can choose to override in their respective namespaces.
Default GKE behavior: If a packet matches no rules in any tier, it falls back to GKE's default behavior, which is an implicit allow.

Within each tier, policies are evaluated based on an explicit numeric priority (0 to 1000, where lower numbers indicate higher precedence). Inside a single policy object, rules are evaluated from top to bottom.

Verdict actions: Accept, deny, pass

Every rule in a ClusterNetworkPolicy must trigger one of three actions:

Deny: Blocks the traffic immediately. This action short-circuits the evaluation pipeline.
Accept: Permits the traffic immediately. This also short-circuits the pipeline.
Pass: Transfers the evaluation to the next tier in the hierarchy.

The Pass action is particularly powerful. It enables platform administrators to target specific traffic flows—for example, web traffic on port 8080—and delegate the final decision to namespace owners. The traffic will bypass any remaining rules in the Admin tier and be evaluated against standard namespace-scoped network policies. If the namespace owners configure a policy to accept the traffic, it is allowed; if they have no policy, the traffic proceeds to the Baseline tier. This strikes a balance between centralized compliance and developer agility.

YAML examples

Let's examine how to write these policies. Consider a scenario where you want to isolate a sensitive namespace from all internal cluster traffic. By placing this policy in the Admin tier, you guarantee that namespace-scoped policies cannot override it.

Here is a manifest for a global deny policy targeting a sensitive namespace:

apiVersion: policy.networking.k8s.io/v1alpha2
kind: ClusterNetworkPolicy
metadata:
  name: cluster-wide-deny-sensitive
spec:
  tier: Admin
  priority: 10
  subject:
    namespaces:
      matchLabels:
        kubernetes.io/metadata.name: sensitive-ns
  ingress:
  - action: Deny
    name: deny-all-ingress
    from:
    - namespaces:
        matchLabels: {}
  egress:
  - action: Deny
    name: deny-all-egress
    to:
    - namespaces:
        matchLabels: {}

Now, consider a different scenario: you want to establish a default-deny posture across the cluster, but you want to allow developers to open up traffic as needed for their applications. For this, you use the Baseline tier.

Here is a baseline default-deny policy:

apiVersion: policy.networking.k8s.io/v1alpha2
kind: ClusterNetworkPolicy
metadata:
  name: default-deny-baseline
spec:
  tier: Baseline
  priority: 100
  subject:
    namespaces: {}
  ingress:
  - action: Deny
    name: baseline-deny-all
    from:
    - namespaces: {}

If a developer in the namespace my-app-ns needs to allow ingress traffic to their frontend service from an external ingress controller, they can simply deploy a standard namespace-scoped NetworkPolicy that permits that traffic. Because the standard policy is evaluated in the NetworkPolicy tier (which runs before the Baseline tier), GKE will permit the traffic without needing platform admin intervention.

Under the hood

The GKE implementation of ClusterNetworkPolicy is built entirely on open-source Cilium, reinforcing a commitment to community-driven foundations. GKE backported this support to Cilium 1.19 for GKE clusters, and the upstream capability will be generally available to the broader open-source ecosystem starting with Cilium 1.20.

To use this feature, your GKE clusters must meet the following requirements:

Run GKE version 1.36.0-gke.4447000 or later.
Use GKE Dataplane V2.

As this feature is in Public Preview, there are a few scale limits to keep in mind. Specifically, a single ClusterNetworkPolicy object can contain a maximum of 100 ingress rules and 100 egress rules. You can monitor traffic flow verdicts and troubleshoot your policies using GKE Dataplane V2 Observability tools.

By decoupling cluster-wide guardrails from namespace-level configurations, ClusterNetworkPolicy gives security teams the control they need without hindering developer speed.

To get started with configuring cluster-wide security policies on your clusters, check out the official GKE ClusterNetworkPolicy documentation for step-by-step guides and YAML examples.

How to Count Gemini Tokens Locally

Laurent Picard — Thu, 02 Jul 2026 13:50:42 +0000

✨ Overview

This article explores how Gemini tokenizes data and demonstrates how to count or estimate tokens locally. You’ll learn how to use the local tokenizer to estimate text token counts offline, understand the tokenization math for multimodal inputs (images, audio, video, PDFs), and see how to retrieve precise token usage metadata from API responses for accurate tracking and billing.

ℹ️ The complete source code is available in this notebook (including all setup details and future updates) under the Apache 2.0 license. You can also directly open the notebook in Colab. This article reproduces all the results generated by a click on “Run all”.

⚙️ Setup

🐍 Google Gen AI Python SDK

To call the Gemini API, we'll use the Google Gen AI Python SDK. The Gemini API provides a count_tokens method, and the SDK offers an experimental implementation of a LocalTokenizer class.

Make sure you have a recent version of the google-genai package with its local-tokenizer extra:

%pip install --quiet "google-genai[local-tokenizer]>=2.9.0"

🛠️ Google Cloud Project

To get started using the Gemini API on Agent Platform, you must have an existing Google Cloud project and enable the Agent Platform API.

Learn more about setting up a project and a development environment.

import os

# fmt: off
PROJECT_ID = ""  # @param {type: "string", placeholder: "[your-project-id]", isTemplate: true}
LOCATION = "global" # @param {type: "string", placeholder: "[your-region]", isTemplate: true}
# fmt: on

if not PROJECT_ID:
    PROJECT_ID = os.environ.get("GOOGLE_CLOUD_PROJECT")
    assert PROJECT_ID, "❌ Please set the PROJECT_ID variable"
if not LOCATION:
    LOCATION = os.environ.get("GOOGLE_CLOUD_REGION", "global")

🤖 Gen AI SDK Client

To interact with the Gemini API, we initialize a genai.Client. Since we're using the enterprise-ready Agent Platform backend (formerly Vertex AI), we pass enterprise=True along with our Google Cloud project and location:

from google import genai


def print_configuration(client: genai.Client) -> None:
    service = "Agent Platform" if client.vertexai else "Google AI"
    print(f"ℹ️ Using the {service} API", end="")
    if client._api_client.project:
        print(f' with project "{client._api_client.project[:7]}…"', end="")
        print(f' in location "{client._api_client.location}"')
    elif client._api_client.api_key:
        api_key = client._api_client.api_key
        print(f' with API key "{api_key[:5]}…{api_key[-5:]}"', end="")
        print(f" (in case of error, make sure it was created for {service})")


client = genai.Client(enterprise=True, project=PROJECT_ID, location=LOCATION)

print_configuration(client)

ℹ️ Using the Agent Platform API with project "lpdemo-…" in location "global"

🧠 Gemini Model

We'll use gemini-3.1-flash-lite as our default model for token counting and content generation. This lightweight, fast model is ideal for high-throughput tasks.

MODEL_ID = "gemini-3.1-flash-lite"

🧩 The Basics: Tokens and Tokenizers

Tokens

Large language models (LLMs) don't process our inputs directly, nor do they generate the final text or media we see. Instead, they operate on fundamental units called tokens, ingesting them as inputs and generating them as outputs.

Here's what happens when we send an LLM request:

Our inputs are transformed into tokens. In other words, they are tokenized.
The model generates output tokens, which represent the most likely next tokens based on the overall context.
These output tokens are transformed back into the final content we can use.

You can think of a token as a piece of information, and this tokenization process acts as an information compression codec:

Encoding: Input → Input tokens
Decoding: Output tokens → Output

Tokenization is necessary to compress information to the right level of semantic granularity, allowing the model's attention mechanism to focus and develop an understanding of the provided data.

Tokenizers

Gemini is natively multimodal and accepts text, images, audio, video, and PDFs. These media types can be processed by a set of three tokenizers:

Input	Text Tokenizer	Image Tokenizer	Audio Tokenizer	Comment
Text	✅			The original tokenizer type, when LLMs were only chatbots.
Image		✅		An image is can be worth a thousand ~~words~~ tokens!
Audio	✅		✅	Text tokens are used for timestamps (`MM:SS` or `H:MM:SS`).
Video	✅	✅	[✅]	By default, one frame is sampled per second, along with its corresponding timestamp. Audio is optional for videos.
PDF	✅	✅		PDFs are processed by vision tokenizers. Text tokens are used for OCR and pagination data.

As you can see, up to three tokenizers can be involved, depending on the modality.

💡 Keep in mind that not all underlying tokens are necessarily billed. See the usage_metadata section below for examples of tokens actually billed per modality.

Vocabulary

The complete set of unique tokens that an LLM can ingest or generate makes up its vocabulary. Once an LLM is trained, its vocabulary is fixed and is used for inference.

A vocabulary is essentially a lookup table mapping text sequences to token IDs (which correspond to vector representations in a semantic space). This means tokenizers are simply algorithms that use this vocabulary to encode and decode tokens (i.e., to convert data to and from token IDs).

For example, the Gemini text tokenizers process common words like this:

Text	Tokens	Tokenization	Token IDs
`hello`	1	A single token for most common sequences	`23391`
`passion`	1	`passion`	`208039`
`passionate`	2	`pass • ionate`	`4373 • 84242`
`passionné`	2	`passion • né` (passionate in French)	`208039 • 8504`
`passionately`	2	`passion • ately`	`208039 • 2295`
`passionalmente`	2	`pass • ionalmente` (passionately in Italian)	`4373 • 134916`

💡 As you can see, words with the same root aren't necessarily split the same way. Text tokenizers have no concept of syllables, prefixes, or suffixes. They don't think like linguists or grammarians; they think like statisticians and look for statistically optimal combinations.

🌐 Baseline: API Token Counting

The Gemini API lets you count tokens for any multimodal input by sending a count_tokens request. While you need to be authenticated to use it, this method is free of charge, so you can audit your prompts before committing to a paid request. Likewise, the compute_tokens method lets you retrieve the list of corresponding tokens and token IDs.

Let's reproduce the previous table:

from collections.abc import Iterator

import IPython.display

from google.genai.types import (
    ComputeTokensResponse,
    ComputeTokensResult,
    CountTokensResponse,
    CountTokensResult,
)

RowData = tuple[str, str, str, str]


def display_token_info_from_api(model: str, texts: list[str]) -> None:
    def yield_data() -> Iterator[RowData]:
        for text in texts:
            count_result = client.models.count_tokens(model=model, contents=text)
            compute_result = client.models.compute_tokens(model=model, contents=text)
            yield get_text_token_info(text, count_result, compute_result)

    display_token_info(yield_data())


def display_token_info(yield_data: Iterator[RowData]) -> None:
    def yield_row() -> Iterator[RowData]:
        yield "Text", "Tokens", "Tokenization", "Token IDs"
        yield "-", ":-:", "-", "-"
        yield from yield_data

    markdown = "\n".join("| " + " | ".join(row) + " |" for row in yield_row())
    IPython.display.display(IPython.display.Markdown(markdown))


def get_text_token_info(
    text: str,
    count_tokens_res: CountTokensResponse | CountTokensResult,
    compute_tokens_res: ComputeTokensResponse | ComputeTokensResult,
) -> RowData:
    def inline_code(s: str) -> str:
        return f"`{s}`"

    total_tokens = count_tokens_res.total_tokens
    tokens_info = compute_tokens_res.tokens_info
    assert tokens_info is not None and len(tokens_info) == 1
    info = tokens_info[0]
    assert info.tokens is not None and info.token_ids is not None
    tokenization = " • ".join(t.decode("utf-8", errors="replace") for t in info.tokens)
    token_ids = " • ".join(str(token_id) for token_id in info.token_ids)

    return (
        inline_code(text),
        str(total_tokens),
        inline_code(tokenization),
        inline_code(token_ids),
    )


TEXTS = [
    "hello",
    "passion",
    "passionate",
    "passionné",
    "passionately",
    "passionalmente",
]
display_token_info_from_api(MODEL_ID, TEXTS)

Text	Tokens	Tokenization	Token IDs
`hello`	1	`hello`	`23391`
`passion`	1	`passion`	`208039`
`passionate`	2	`pass • ionate`	`4373 • 84242`
`passionné`	2	`passion • né`	`208039 • 8504`
`passionately`	2	`passion • ately`	`208039 • 2295`
`passionalmente`	2	`pass • ionalmente`	`4373 • 134916`

🚀 Why Count Tokens Locally?

Here are a few use cases where counting (or just estimating) tokens locally is useful:

Offline & Speed: You can count tokens completely offline. Plus, even when you're online, doing it locally means you don't have to wait for a network round-trip to the Gemini API just to check your prompt size.
Quotas: While the count_tokens method is free, counting locally saves bandwidth and prevents you from hitting API rate limits, especially during high-volume token counting.
Latency: You can estimate how much time is needed to process your text input before you start receiving a response (for a given model, the time-to-first-token latency is roughly proportional to the number of input tokens).
Cost Control: You can estimate and budget your API costs before committing to a paid request.
Routing: Knowing which token-count bucket your input falls into lets you route requests to different models based on speed, cost, or context size.
Privacy: You can audit the token count of sensitive data without sending it over your network.

🔤 Using the Local Text Tokenizer

Create a local tokenizer for the specific Gemini model you're using:

from google.genai.local_tokenizer import LocalTokenizer

tokenizer = LocalTokenizer(model_name=MODEL_ID)

💡 Remarks

Creating a tokenizer takes a few seconds, during which the configuration and vocabulary are loaded into memory.

On the first call, the tokenizer data is downloaded and stored in a local cache. This step requires an internet connection and about 30MB of storage.

If you want to build a fully offline solution, you can check out the SDK source code and persist the tokenizer assets (e.g., by configuring a persistent cache directory or building a container image).

Checking the internal tokenizer name confirms that the Gemma open-weight models share the same text tokenizer as the Gemini 3 family:

print(f'Text tokenizer name for "{MODEL_ID}": "{tokenizer._tokenizer_name}"')

Text tokenizer name for "gemini-3.1-flash-lite": "gemma4"

Call the count_tokens() method on a small text input:

contents = "Hello World!"
result = tokenizer.count_tokens(contents)

print(f"{result.total_tokens=}")

result.total_tokens=3

Now, let's reproduce the previous API tokenization tests with our local tokenizer:

def display_token_info_from_local_tokenizer(
    tokenizer: LocalTokenizer, texts: list[str]
) -> None:
    def yield_data() -> Iterator[RowData]:
        for text in texts:
            count_result = tokenizer.count_tokens(contents=text)
            compute_result = tokenizer.compute_tokens(contents=text)
            yield get_text_token_info(text, count_result, compute_result)

    display_token_info(yield_data())


display_token_info_from_local_tokenizer(tokenizer, TEXTS)

Text	Tokens	Tokenization	Token IDs
`hello`	1	`hello`	`23391`
`passion`	1	`passion`	`208039`
`passionate`	2	`pass • ionate`	`4373 • 84242`
`passionné`	2	`passion • né`	`208039 • 8504`
`passionately`	2	`passion • ately`	`208039 • 2295`
`passionalmente`	2	`pass • ionalmente`	`4373 • 134916`

💡 As expected, we get exactly the same results, but with 100% local execution this time.

Finally, let's download a longer text, like Hamlet:

import requests


def get_text_from_url(content_url: str, force_encoding: str = "") -> str:
    response = requests.get(content_url, timeout=10)
    response.raise_for_status()
    if force_encoding:  # Use for HTTP headers with unknown/incorrect charset
        response.encoding = force_encoding
    return response.text


TEXT_URL = "https://storage.googleapis.com/dataflow-samples/shakespeare/hamlet.txt"
contents = get_text_from_url(TEXT_URL)

print(contents[:256] + "[…]")

    HAMLET


    DRAMATIS PERSONAE


CLAUDIUS    king of Denmark. (KING CLAUDIUS:)

HAMLET  son to the late, and nephew to the present king.

POLONIUS    lord chamberlain. (LORD POLONIUS:)

HORATIO friend to Hamlet.

LAERTES son to Polonius.

LUCIANUS    nephew to the kin[…]

How many tokens do we need to encode Hamlet?

result = tokenizer.count_tokens(contents)

print(f"{result.total_tokens=:,}")

result.total_tokens=54,660

💡 Hamlet gets broken down locally into 50k+ tokens in a fraction of a second. If you tokenize War and Peace, you'll get 850k+ tokens.

🕵️‍♂️ Accounting for "Hidden" Tokens

When you send a request to Gemini, the total input token count isn't always just the sum of your input data.

To keep things simple, we tested text token counts with default parameters. The count_tokens and compute_tokens methods both have a config parameter. Depending on your request configuration, your inputs and outputs may include additional tokens.

Keep an eye out for these hidden additions:

System Instructions: Any system prompt you set will add to the total token count.
Thinking: If thinking is enabled, an internal chain of thought can generate additional thinking tokens.
Tools and Functions: If you provide a list of tools (like Python execution or custom functions), their declarations, calls, and responses are part of your prompt payload.
Response Schema: Enforcing structured outputs (like JSON) requires the model to process the schema definition you provide, which consumes input tokens.
Chat History: In multi-turn conversations, the entire chat history is sent back to the model with every new message, meaning your input token count grows with each turn.

🧮 Multimodal Token Math

Multimodal inputs (images, audio, video, and documents) aren't tokenized like text. They usually have specific calculation rules based on the model (and its underlying tokenizers), the media type, and the request configuration.

For multimodal inputs, refer to the documentation for details on how token counts are calculated for different media types:

There are generally multiple tokenization options, even for a single modality. You can use the count_tokens method and the calculation rules to estimate the token count of your own payloads. To get a clearer picture, let's look at actual requests and see how token counts are broken down by modality…

🎯 Tracking Actual Token Usage

While estimating token counts is super useful, you should always rely on the usage_metadata returned in the API response when you need to track your actual usage down to the exact token. It's the single source of truth for billing.

Here's the gist of how usage_metadata lets you get the token counts by modality:

class GenerateContentResponse:
    # …
    usage_metadata: Optional[GenerateContentResponseUsageMetadata]
    # …


class GenerateContentResponseUsageMetadata:
    # …
    prompt_token_count: Optional[int]
    prompt_tokens_details: Optional[list[ModalityTokenCount]]
    # …


class ModalityTokenCount:
    modality: Optional[MediaModality]
    token_count: Optional[int]


class MediaModality(StrEnum):
    MODALITY_UNSPECIFIED = "MODALITY_UNSPECIFIED"
    TEXT = "TEXT"
    IMAGE = "IMAGE"
    VIDEO = "VIDEO"
    AUDIO = "AUDIO"
    DOCUMENT = "DOCUMENT"

🐍 Let's define a few helpers:

from google.genai.types import (
    FileData,
    GenerateContentResponse,
    MediaModality,
    Part,
    PartMediaResolution,
    PartMediaResolutionLevel,
    VideoMetadata,
)

TokensPerModality = dict[MediaModality, int]


def display_tokens_per_modality(response: GenerateContentResponse) -> None:
    usage_metadata = response.usage_metadata
    if not usage_metadata:
        print("⚠️ No usage metadata found in the response.")
        return
    prompt_tokens_details = usage_metadata.prompt_tokens_details or []
    tokens_per_modality = get_empty_tokens_per_modality()

    for tokens_details in prompt_tokens_details:
        modality = tokens_details.modality
        if modality and modality in tokens_per_modality:
            tokens_per_modality[modality] += tokens_details.token_count or 0

    prompt_token_count = usage_metadata.prompt_token_count or 0
    display_token_table(tokens_per_modality, prompt_token_count)


def get_empty_tokens_per_modality() -> TokensPerModality:
    return {
        modality: 0
        for modality in MediaModality
        if modality != MediaModality.MODALITY_UNSPECIFIED
    }


def display_token_table(
    tokens_per_modality: TokensPerModality,
    total_tokens: int,
) -> None:
    def yield_row() -> Iterator[list[str]]:
        yield [mod.value for mod in tokens_per_modality.keys()] + ["Total"]
        yield [":-:" for _ in range(len(tokens_per_modality) + 1)]
        yield [f"{t:,d}" for t in tokens_per_modality.values()] + [f"{total_tokens:,d}"]

    markdown = "\n".join("| " + " | ".join(row) + " |" for row in yield_row())
    IPython.display.display(IPython.display.Markdown(markdown))

Let's check a few examples…

🖼️ Image Tokenization

Image token counts depend on the image itself and the configured media resolution:

class PartMediaResolutionLevel(StrEnum):
    MEDIA_RESOLUTION_UNSPECIFIED = "MEDIA_RESOLUTION_UNSPECIFIED"
    MEDIA_RESOLUTION_LOW = "MEDIA_RESOLUTION_LOW"
    MEDIA_RESOLUTION_MEDIUM = "MEDIA_RESOLUTION_MEDIUM"
    MEDIA_RESOLUTION_HIGH = "MEDIA_RESOLUTION_HIGH"
    MEDIA_RESOLUTION_ULTRA_HIGH = "MEDIA_RESOLUTION_ULTRA_HIGH"

For a given media resolution level, the Gemini 3 tokenizers will use these maximum token budgets per image:

`media_resolution`	Tokens
`MEDIA_RESOLUTION_LOW`	280
`MEDIA_RESOLUTION_MEDIUM`	560
`MEDIA_RESOLUTION_HIGH` (default)	1,120
`MEDIA_RESOLUTION_ULTRA_HIGH`	2,240

🐍 Check how this cat image is tokenized by default:

def display_tokens_for_image(
    image_uri: str,
    media_resolution_level: PartMediaResolutionLevel | None = None,
) -> None:
    print(f"🧪 {media_resolution_level=}")
    contents = Part.from_uri(
        file_uri=image_uri,
        mime_type="image/*",
        media_resolution=(
            PartMediaResolution(level=media_resolution_level)
            if media_resolution_level
            else None
        ),
    )
    response = client.models.generate_content(model=MODEL_ID, contents=contents)
    display_tokens_per_modality(response)


IMAGE_URI = "https://storage.googleapis.com/cloud-samples-data/generative-ai/image/chair-cat.png"
display_tokens_for_image(IMAGE_URI)

🧪 media_resolution_level=None

TEXT	IMAGE	VIDEO	AUDIO	DOCUMENT	Total
0	1,080	0	0	0	1,080

💡 This image is tokenized into only 1,080 tokens (instead of the maximum 1,120), saving us 40 tokens! It's a nice touch that helps keep costs down rather than defaulting to the upper limit.

🐍 For less detailed images, you can reduce token counts by a factor of 2 or 4 using the medium or low levels:

display_tokens_for_image(IMAGE_URI, PartMediaResolutionLevel.MEDIA_RESOLUTION_LOW)

🧪 media_resolution_level=<PartMediaResolutionLevel.MEDIA_RESOLUTION_LOW: 'MEDIA_RESOLUTION_LOW'>

TEXT	IMAGE	VIDEO	AUDIO	DOCUMENT	Total
0	264	0	0	0	264

💡 At the other end of the media resolution range, the ultra-high level is great for detailed images (like a photo of a circuit board with many components), ensuring maximum visual understanding. An image at this level uses between 2,000 and 2,240 tokens.

🔊 Audio Tokenization

Audio tokenization currently uses 25 tokens per second to represent the audio stream semantically.

🐍 Here is the tokenization for a 3.049-second audio file:

def display_tokens_for_audio(audio_uri: str) -> None:
    contents = Part.from_uri(file_uri=audio_uri, mime_type="audio/*")
    response = client.models.generate_content(model=MODEL_ID, contents=contents)
    display_tokens_per_modality(response)


AUDIO_URI = "https://storage.googleapis.com/cloud-samples-data/generative-ai/audio/hello_gemini_are_you_there.wav"
display_tokens_for_audio(AUDIO_URI)

TEXT	IMAGE	VIDEO	AUDIO	DOCUMENT	Total
0	0	0	77	0	77

💡 ceil(3.049 s × 25 tok/s) = ceil(76.225 tok) = 77 tok

🐍 A longer, 30.772-second audio file requires 10 times as many tokens, as expected:

AUDIO_URI = "https://storage.googleapis.com/cloud-samples-data/generative-ai/audio/sailor_audio.mp3"
display_tokens_for_audio(AUDIO_URI)

TEXT	IMAGE	VIDEO	AUDIO	DOCUMENT	Total
0	0	0	770	0	770

💡 ceil(30.772 s × 25 tok/s) = ceil(769.3 tok) = 770 tok

🎬 Video Tokenization

For videos:

The audio tokenizer is the same as for standalone audio (25 tokens per second).
Video frames are sampled (1 FPS by default) and tokenized based on the media resolution.

For a given media resolution level, the Gemini 3 tokenizers will use these maximum token budgets per sampled frame:

`media_resolution`	Max. tokens
`MEDIA_RESOLUTION_LOW`/`MEDIA_RESOLUTION_MEDIUM` (default)	70
`MEDIA_RESOLUTION_HIGH`	280

🐍 Here's the tokenization for a 59-second video:

def display_tokens_for_video(
    video_uri: str,
    fps: float | None = None,
    media_resolution_level: PartMediaResolutionLevel | None = None,
) -> None:
    print(f"🧪 {fps=}, {media_resolution_level=}")
    contents = Part(
        file_data=FileData(file_uri=video_uri, mime_type="video/*"),
        video_metadata=VideoMetadata(fps=fps) if fps is not None else None,
        media_resolution=(
            PartMediaResolution(level=media_resolution_level)
            if media_resolution_level
            else None
        ),
    )
    response = client.models.generate_content(model=MODEL_ID, contents=contents)
    display_tokens_per_modality(response)


VIDEO_URI = "https://www.youtube.com/watch?v=0pJn3g8dfwk"
display_tokens_for_video(VIDEO_URI)

🧪 fps=None, media_resolution_level=None

TEXT	IMAGE	VIDEO	AUDIO	DOCUMENT	Total
0	0	3,894	1,475	0	5,369

💡 Details

Video: ceil(59 s × 1 frame/s × 66 tok/frame) = ceil(3894 tok) = 3894 tok

Audio: ceil(59 s × 25 tok/s) = ceil(1475 tok) = 1475 tok

🐍 Doubling the sampling rate requires twice as many video tokens:

display_tokens_for_video(VIDEO_URI, fps=2)

🧪 fps=2, media_resolution_level=None

TEXT	IMAGE	VIDEO	AUDIO	DOCUMENT	Total
0	0	7,788	1,475	0	9,263

💡 Details

Video: ceil(59 s × 2 frame/s × 66 tok/frame) = ceil(7788 tok) = 7788 tok

Audio: ceil(59 s × 25 tok/s) = ceil(1475 tok) = 1475 tok

🐍 If you switch from low/medium to high media resolution, sampled frames are tokenized in greater detail, requiring four times as many video tokens:

VIDEO_URI = "https://www.youtube.com/watch?v=0pJn3g8dfwk"
display_tokens_for_video(
    VIDEO_URI,
    media_resolution_level=PartMediaResolutionLevel.MEDIA_RESOLUTION_HIGH,
)

🧪 fps=None, media_resolution_level=<PartMediaResolutionLevel.MEDIA_RESOLUTION_HIGH: 'MEDIA_RESOLUTION_HIGH'>

TEXT	IMAGE	VIDEO	AUDIO	DOCUMENT	Total
0	0	15,576	1,475	0	17,051

💡 Details

Video: ceil(59 s × 1 frame/s × 264 tok/frame) = ceil(15576 tok) = 15576 tok

Audio: ceil(59 s × 25 tok/s) = ceil(1475 tok) = 1475 tok

📄 Document Tokenization

For a given media resolution level, the Gemini 3 tokenizers will use these maximum token budgets per PDF page:

`media_resolution`	Tokens
`MEDIA_RESOLUTION_LOW`	280
`MEDIA_RESOLUTION_MEDIUM` (default)	560
`MEDIA_RESOLUTION_HIGH`	1,120

🐍 Here's the tokenization for a one-page PDF at different media resolutions:

def display_tokens_for_document(
    document_uri: str,
    media_resolution_level: PartMediaResolutionLevel | None = None,
) -> None:
    print(f"🧪 {media_resolution_level=}")
    contents = Part.from_uri(
        file_uri=document_uri,
        mime_type="application/pdf",
        media_resolution=(
            PartMediaResolution(level=media_resolution_level)
            if media_resolution_level
            else None
        ),
    )
    response = client.models.generate_content(model=MODEL_ID, contents=contents)
    display_tokens_per_modality(response)


DOCUMENT_URI = (
    "https://storage.googleapis.com/cloud-samples-data/generative-ai/pdf/invoice.pdf"
)
media_resolution_levels = [
    PartMediaResolutionLevel.MEDIA_RESOLUTION_LOW,
    PartMediaResolutionLevel.MEDIA_RESOLUTION_MEDIUM,
    PartMediaResolutionLevel.MEDIA_RESOLUTION_HIGH,
]
for media_resolution_level in media_resolution_levels:
    display_tokens_for_document(DOCUMENT_URI, media_resolution_level)

🧪 media_resolution_level=<PartMediaResolutionLevel.MEDIA_RESOLUTION_LOW: 'MEDIA_RESOLUTION_LOW'>

TEXT	IMAGE	VIDEO	AUDIO	DOCUMENT	Total
0	266	0	0	0	266

🧪 media_resolution_level=<PartMediaResolutionLevel.MEDIA_RESOLUTION_MEDIUM: 'MEDIA_RESOLUTION_MEDIUM'>

TEXT	IMAGE	VIDEO	AUDIO	DOCUMENT	Total
0	532	0	0	0	532

🧪 media_resolution_level=<PartMediaResolutionLevel.MEDIA_RESOLUTION_HIGH: 'MEDIA_RESOLUTION_HIGH'>

TEXT	IMAGE	VIDEO	AUDIO	DOCUMENT	Total
0	1,092	0	0	0	1,092

💡 Remarks

Low: 266 tok/pg

Medium: 532 tok/pg

High: 1092 tok/pg

🐍 Here's another test for a 15-page PDF:

DOCUMENT_URI = "https://storage.googleapis.com/cloud-samples-data/generative-ai/pdf/1706.03762v7.pdf"
for media_resolution_level in media_resolution_levels:
    display_tokens_for_document(DOCUMENT_URI, media_resolution_level)

🧪 media_resolution_level=<PartMediaResolutionLevel.MEDIA_RESOLUTION_LOW: 'MEDIA_RESOLUTION_LOW'>

TEXT	IMAGE	VIDEO	AUDIO	DOCUMENT	Total
0	3,990	0	0	0	3,990

🧪 media_resolution_level=<PartMediaResolutionLevel.MEDIA_RESOLUTION_MEDIUM: 'MEDIA_RESOLUTION_MEDIUM'>

TEXT	IMAGE	VIDEO	AUDIO	DOCUMENT	Total
0	7,800	0	0	0	7,800

🧪 media_resolution_level=<PartMediaResolutionLevel.MEDIA_RESOLUTION_HIGH: 'MEDIA_RESOLUTION_HIGH'>

TEXT	IMAGE	VIDEO	AUDIO	DOCUMENT	Total
0	16,530	0	0	0	16,530

💡 Remarks

Low: 3990 tok / 15 pg = 266 tok/pg

Medium: 7800 tok / 15 pg = 520 tok/pg

High: 16530 tok / 15 pg = 1102 tok/pg

🎉 Conclusion

You've now mastered token counting both locally and via the Gemini API!

With the LocalTokenizer, you can estimate text token counts completely offline, saving bandwidth and avoiding rate limits. You've also seen how Gemini's multimodal tokenizers handle images, audio, video, and PDFs, and how to extract precise token usage from usage_metadata for accurate tracking and billing.

➕ More!

Try it yourself: Use the companion notebook (or run the notebook on Colab) to reproduce all results in this article.
Get inspired: Explore typical use cases in the Agent Platform Prompt Gallery.
Stay updated: Follow the Agent Platform Release Notes.
Follow me: Connect with me (@PicardParis) on LinkedIn or Twitter / X for more cloud, applied AI, and Python explorations…

Real-time IP capacity in Google Cloud subnets

Zach S. — Wed, 17 Jun 2026 15:46:58 +0000

When managing Shared VPCs, most teams allocate dedicated IP subnets for each service project to keep firewall rules simple, but this isolation often leads to poor IP utilization — it is not uncommon to see subnet IP utilization hovering in the low teens. On the other hand, using large shared subnets requires coordinating workload deployments to ensure there is enough internal IP address space for everyone. To optimize these shared networks, you need real-time visibility. The WITH_UTILIZATION query parameter on the Method: subnetworks.list | Compute Engine API solves this by returning the exact count of allocated and free IP addresses for each subnet IP range.

This capability is designed for query-time decisions. For example, if you need to deploy a GCE workload requiring 100 instances, you can search for a subnet with enough capacity. This query-time data comes directly from Google Cloud's internal IP allocator and includes both primary and secondary CIDR ranges.

Automating the search with gcloud and jq

To automate capacity checks before you deploy, you can script this check. The script below uses gcloud compute networks subnets list | Google Cloud SDK to grab the utilization data as JSON, and then uses jq to parse, filter, and sort the subnets based on your required capacity:

#!/bin/bash

# --- Configuration (Replace with your details) ---
PROJECT="<YOUR_PROJECT_ID>"
NETWORK_NAME="<YOUR_VPC_NETWORK_NAME>"
REGION="<YOUR_REGION>"
REQUIRED_IP_CAPACITY=100

echo "Searching $NETWORK_NAME in $REGION for subnets with >= $REQUIRED_IP_CAPACITY free IPs..."
echo "------------------------------------------------------------------------"

# Fetch subnets with utilization data, output as JSON, and pipe to jq
gcloud compute networks subnets list \
    --project="$PROJECT" \
    --network="$NETWORK_NAME" \
    --regions="$REGION" \
    --view=WITH_UTILIZATION \
    --format=json | \
jq -r --argjson min_ips "$REQUIRED_IP_CAPACITY" '
  [ 
    .[] | {
      name: .name,
      cidr: .ipCidrRange,
      # Safely extract totalFreeIp: if it is null, substitute "0" before converting to a number
      free_ips: (.utilizationDetails.ipv4Utilizations[0].totalFreeIp // "0" | tonumber)
    } 
    # Keep only the subnets that meet the minimum requirement
    | select(.free_ips >= $min_ips)
  ] 

  # Sort ascending by free_ips, then reverse to get descending order
  | sort_by(.free_ips) 
  | reverse 

  # Format the final output into clean, readable strings
  | .[] 
  | "Subnet: \(.name) | CIDR: \(.cidr) | Free IPs: \(.free_ips)"
'

Let's list the configured subnets in our target region first:

~  gcloud compute networks subnets list \
    --project="my-gcp-project" \
    --format='table(name, region, network, ipCidrRange)'

NAME       REGION      NETWORK  RANGE
subnet-a0  us-south1  vpc-a    10.0.0.0/28
subnet-a1  us-south1  vpc-a    10.0.1.0/28
subnet-a2  us-south1  vpc-a    10.0.2.0/28
subnet-a3  us-south1  vpc-a    10.0.3.0/28
subnet-a4  us-south1  vpc-a    10.0.4.0/28
subnet-a5  us-south1  vpc-a    10.0.5.0/24
subnet-a6  us-south1  vpc-a    10.0.6.0/25
subnet-a7  us-south1  vpc-a
subnet-a8  us-south1  vpc-a
subnet-a9  us-south1  vpc-a

Running the script returns only the subnets that can safely host our 100-instance workload:

~  bash gce-subnet-utilization.sh

Searching vpc-a in us-south1 for subnets with >= 100 free IPv4 addresses...
------------------------------------------------------------------------
Subnet: subnet-a5 | CIDR: 10.0.5.0/24 | Free IPs: 252
Subnet: subnet-a6 | CIDR: 10.0.6.0/25 | Free IPs: 124

Under the hood: Reading the utilization payload

When you request a subnet list with the utilization view, the API returns a utilizationDetails object. For a standard subnet with only a primary IPv4 address configured, the JSON payload looks like this:

"utilizationDetails": {
  "ipv4Utilizations": [
    {
      "totalAllocatedIp": "4",
      "totalFreeIp": "252"
    }
  ]
}

Notice that the totalAllocatedIp is 4. In any primary IPv4 range, Google Cloud reserves four IP addresses for default routing and metadata, as detailed in the Subnets | Virtual Private Cloud - Google Cloud Documentation.

If you have secondary ranges configured (often used for GKE Pods), the API includes utilization metrics for each secondary range, identified by rangeName:

"utilizationDetails": {
  "ipv4Utilizations": [
    {
      "totalAllocatedIp": "4",
      "totalFreeIp": "124"
    },
    {
      "rangeName": "a6-secondary",
      "totalAllocatedIp": "0",
      "totalFreeIp": "4096"
    }
  ]
}

The API also breaks down IPv6 utilization if you are running dual-stack subnets. It tracks external instance IPs, load balancer endpoints, and internal IPv6 allocations separately:

"utilizationDetails": {
  "externalIpv6InstanceUtilization": {
    "totalAllocatedIp": { "high": "0", "low": "0" },
    "totalFreeIp": { "high": "0", "low": "9223372036854775808" }
  },
  "externalIpv6LbUtilization": {
    "totalAllocatedIp": { "high": "0", "low": "0" },
    "totalFreeIp": { "high": "0", "low": "9223372036854775808" }
  },
  "internalIpv6Utilization": {
    "totalAllocatedIp": { "high": "0", "low": "8589934592" },
    "totalFreeIp": { "high": "0", "low": "18446744065119617024" }
  }
}

A few quick constraints

API Support: The WITH_UTILIZATION parameter works with both Method: subnetworks.get | Compute Engine and Method: subnetworks.list | Compute Engine.
gcloud Support: You can pass the parameter in gcloud using --view=WITH_UTILIZATION, as documented in gcloud compute networks subnets list | Google Cloud SDK.
JSON Strings: The API returns allocated and free counts as strings in the JSON payload. Make sure to cast them (like using tonumber in jq) before running any mathematical comparisons.

Next steps

Next time you are building a deployment pipeline, try integrating the WITH_UTILIZATION view. It is a simple way to programmatically ensure you have enough network headroom before kicking off a deployment.

Seamless scaling with VPA In-place Pod Resize on GKE

Olivier Bourgeois — Thu, 04 Jun 2026 18:19:03 +0000

Right-sizing Kubernetes workloads is a common platform engineering challenge. Set your requests too high, and you burn cloud budgets on idle capacity; set your limits too low, and your applications face throttling or dreaded OOMKills.

For years, the Vertical Pod Autoscaler (VPA) has been the standard answer to this problem, automatically adjusting CPU and memory requirements based on actual usage. However, this method of scaling came with a significant catch that prevented widespread adoption for critical workloads: applying new resource parameters required evicting and restarting the pod.

This disruption was often unacceptable for stateful applications, long-running connections, or latency-sensitive services.

Introducing In-place Pod Resize (IPPR) on GKE

In-place Pod Resize (IPPR) changes the game by allowing Kubernetes to modify resource requests and limits on live, running containers directly through the underlying container runtime, without triggering a restart.

By combining the intelligence of VPA with the non-disruptive nature of IPPR, GKE users finally have a viable path to dynamic, seamless, and automated right-sizing.

Note: As of writing, VPA IPPR is in Preview on GKE. While it is a massive step forward, I recommend evaluating it in staging environments before rolling it out to production workloads.

Getting started with IPPR

To use In-place Pod Resize, you need a GKE cluster running version 1.34.0-gke.2201000 or later.

GKE Autopilot: VPA is enabled by default.
GKE Standard: Requires the Vertical Pod Autoscaling feature to be enabled.

1. Enable the feature

If you aren't using Autopilot, ensure your cluster is created or updated with the necessary feature flags:

gcloud container clusters create CLUSTER_NAME \
  --project=PROJECT_ID \
  --location=us-east1 \
  --release-channel=rapid \
  --enable-vertical-pod-autoscaling

2. Define your VPA object

Create a VerticalPodAutoscaler resource targeting your Deployment or StatefulSet. The crucial element here is setting spec.updatePolicy.updateMode to InPlaceOrRecreate.

apiVersion: "autoscaling.k8s.io/v1"
kind: "VerticalPodAutoscaler"
metadata:
  name: "my-vpa"
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: "Deployment"
    name: "my-deployment"
  updatePolicy:
    updateMode: "InPlaceOrRecreate"

3. Watch it scale

Apply the resource to your cluster and monitor your application under load. Instead of watching Pods terminate and recreate, you can watch the resources modify live using kubectl describe.

kubectl describe pod POD_NAME

Look for the AllocatedResources field or check the events section. You will see the requests change in real-time to match the VPA recommendations, while the Restart Count remains exactly the same.

The "Or Recreate" Fallback: Keep in mind that physics still apply. If VPA recommends a resource size that exceeds the remaining capacity of the Node your Pod is currently running on, an in-place resize is impossible. In this scenario, VPA will fall back to evicting and recreating the Pod so it can be scheduled onto a larger or emptier Node.

Ready to dive deeper?

While this introduction covers the basics of IPPR, right-sizing is just one part of a robust scaling strategy. Implementing VPA often goes hand-in-hand with horizontal scaling and cluster autoscaling. Check out the guide to master scaling on GKE: Run full-stack workloads at scale on GKE.

Surviving the eviction: How to build interrupt-resilient AI workloads on GKE

Olivier Bourgeois — Tue, 02 Jun 2026 20:02:20 +0000

You did everything right. You containerized your massive model training job, deployed it to Google Kubernetes Engine (GKE), and cleverly routed it to a Spot VM node pool to save up to 90% on compute costs.

Everything is humming along perfectly for 38 hours. Then, a priority on-demand customer needs capacity, Google Cloud reclaims your underlying Spot VM, and your node vanishes.

Whether you are using preemptible Spot VMs to save money, or leveraging the Dynamic Workload Scheduler (DWS) to queue for scarce GPUs, you are building on top of ephemeral compute. The hardware will eventually be taken away. To successfully run critical AI workloads on un-committed capacity, your application architecture must assume failure is a given.

Here is a practical guide to building interruptible workloads on GKE.

1. Trap the warning

When Google Cloud reclaims a Spot VM, it doesn't just pull the power cord immediately. It sends an ACPI signal to the underlying node to begin a power off cycle. Kubernetes intercepts this and translates it into a SIGTERM signal sent directly to your running containers.

You have a grace period (up to 15 seconds for non-system pods) between that SIGTERM and the fatal SIGKILL.

Your application must explicitly listen for this signal. When caught, your code should immediately stop accepting new batches, finish its current loop, flush any in-memory data to disk, and exit with a 0 (success) status.

Here is a simple example on how to catch this signal in Python:

import signal
import sys
import time

def handle_sigterm(signum, frame):
    print("Received SIGTERM. Initiating graceful shutdown...")
    # 1. Stop processing new data
    # 2. Flush memory to persistent storage
    # 3. Save final checkpoint
    print("State saved. Exiting cleanly.")
    sys.exit(0)

# Register the signal handler
signal.signal(signal.SIGTERM, handle_sigterm)

# Your main training loop
print("Starting training loop...")
while True:
    # Train model...
    time.sleep(1)

2. Externalize your checkpoints

If your container dies, everything inside its local filesystem dies with it. To survive an interruption, you must periodically save your progress (model weights, optimizer states, epoch counters, etc.) to an external storage location.

Cloud Storage (GCS) is a common solution for this on Google Cloud.

Save frequently: Decide on a checkpointing interval that balances the cost of lost work against the overhead of writing to storage. Saving every epoch or every few thousand steps is common, but this can vary based on your needs.
Keep it local: Ensure your GCS buckets are in the same region as your GKE cluster (e.g., us-central1) to minimize latency and avoid outbound data transfer fees.
Resume, don't restart: The first thing your container's startup script should do is to check for that GCS bucket. If a checkpoint exists in the bucket, load it and resume from that exact step.

3. Design for Idempotency

"Idempotency" is a fancy way of saying that doing something twice yields the same result as doing it once.

Imagine a batch inference job that reads an image, processes it, and writes the result to a database. If your pod is preempted milliseconds after writing to the database but before it can mark the task as complete, the rescheduled pod will likely process that image again.

If your database blindly inserts new rows, you now have unintentional, duplicate data.

To build an idempotent pipeline:

Use UPSERT (update or insert) operations in your database based on a unique identifier (like an image ID).
Check if a record already exists before spending expensive GPU cycles processing it.

4. Decouple work queues for batch processing

If you are running a massive batch processing or inference job across thousands of files, do not write a monolithic Python script that iterates through a static CSV list. If the node dies at row 5,000, managing the state of where to restart is a nightmare.

Instead, decouple the workload:

Publish the work: Break your dataset down into discrete messages and push them into a message broker like Pub/Sub.
Pull the work: Have your Spot VM worker pods pull messages off the queue one by one or as a small chunk (e.g. 10 at a time).
Acknowledge completion: Only send an "ACK" (acknowledgment) back to Pub/Sub once the result is safely stored.

If a Spot node is preempted mid-inference, the worker dies before sending the ACK. After a brief timeout, Pub/Sub will automatically make that specific message available again. Another surviving worker pod will pick it up seamlessly. No data lost, no manual intervention required.

Key takeaways

Running on ephemeral compute like Spot VMs isn't just an infrastructure choice; it is a design choice. By handling termination signals, checkpointing aggressively to GCS, ensuring idempotent operations, and decoupling your queues, you can unlock massive cost savings and tap into scarce GPU pools without sacrificing reliability.

Strategies for running AI workloads on GKE without committed quota

Olivier Bourgeois — Mon, 01 Jun 2026 18:54:52 +0000

You’ve built your model, your training code is containerized, and you’re ready to scale up on Google Kubernetes Engine (GKE). You go to provision your nvidia-h100-80gb node pool and... QUOTA_EXCEEDED.

It’s one of the most common (and frustrating) roadblocks in modern AI development. High-end accelerators like H100s, A100s, and TPUs are in massive demand, and securing permanent, on-demand quota for them can be difficult. But a lack of on-demand quota doesn't mean you're out of options.

GKE provides two powerful, cost-effective strategies for acquiring these scarce resources when you can't get standard, on-demand instances: Spot VMs and the Dynamic Workload Scheduler (DWS).

Let's break down what they are, when to use each, and how to implement them.

Strategy 1: Spot VMs

Spot VMs are Google Cloud's excess compute capacity sold at a massive discount, up to 90% off the price of standard on-demand VMs. They are perfect for workloads that can be interrupted.

The catch is that Spot VMs have no availability guarantee. Google Cloud can "preempt" (i.e., terminate) them at any time if that capacity is needed for on-demand customers. GKE gets a 30-second warning before the node is terminated. Kubernetes uses this window to gracefully shut down your application (giving non-system pods up to 15 seconds to wrap up) before the node vanishes.

When to use Spot VMs for accelerators

Spot VMs are ideal for workloads that are:

Fault-tolerant and stateless: Your application can handle a node vanishing and having its pods rescheduled elsewhere.
Batch processing: Jobs that can be easily restarted or have checkpointing built-in.
CI/CD pipelines: Running tests or builds that don't need 100% uptime.

How to use Spot VMs in GKE

You can easily add a Spot VM node pool to your GKE Standard cluster. The key is to use Spot VMs for your workers, not your critical system pods.

Create a dedicated Spot VM node pool:
When creating a node pool, simply add the --spot flag and apply a taint so standard pods don't accidentally schedule there.

gcloud container node-pools create spot-gpu-pool \
  --cluster=my-cluster \
  --region=northamerica-northeast2 \
  --machine-type=g2-standard-4 \
  --accelerator=type=nvidia-l4,count=1 \
  --spot \
  --node-taints=cloud.google.com/gke-spot=true:NoSchedule

Add the toleration to your workload's YAML:

# You want to "tolerate" that taint only on the specific workloads you want to run there.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-batch-job
spec:
  template:
    # ... other specs
    spec:
      tolerations:
      - key: "cloud.google.com/gke-spot"
        operator: "Equal"
        value: "true"
        effect: "NoSchedule"

This architecture ensures your critical components stay on reliable on-demand nodes, while your interruptible training jobs run on the preemptible Spot nodes. (Note: If you are using GKE Autopilot, you simply request a Spot class in your pod spec and GKE handles the taints and nodes automatically!)

Strategy 2: Dynamic Workload Scheduler (DWS) with flex-start

What if your job can't be interrupted? Many large-scale training jobs can take days. While they might have checkpointing, restarting from scratch every few hours due to Spot preemptions is inefficient and costly.

This is where Dynamic Workload Scheduler (DWS) comes in.

DWS is a feature designed specifically for acquiring large amounts of scarce resources (like GPUs and TPUs) for batch workloads. It changes the request from "Give me this GPU right now" to "Give me this GPU when it becomes available."

The catch here is that your job doesn't start immediately. It enters a queue and might wait for minutes, hours, or even days for the resources to be provisioned.

There are a few massive upsides:

It's a "get-in-line" system: Instead of you writing a script to retry the gcloud command every 5 minutes, DWS queues your request and provisions the nodes automatically when capacity is found.
No preemptions: Once your DWS nodes are provisioned, they are yours for the entire duration of your job (up to seven days). They are not Spot VMs and will not be preempted.
Cost savings: DWS workloads are also offered at a significant discount (up to 53% for L4 GPUs) compared to on-demand instances.

When to use DWS

DWS is perfect for:

Model training or reinforcement learning (RL): Jobs that need to run uninterrupted for many hours or days.
Batch inference: Running a large inference job on a massive dataset.
Any workload that is not time-sensitive to start, but is sensitive to interruptions.

How to use DWS with flex-start

The flex-start mode in DWS is what enables this "wait-in-queue" behavior. If you are using a GKE Autopilot cluster (or a Standard cluster with Node Auto-provisioning enabled), implementing this is incredibly simple.

You do not need to create complex custom resources; you simply signal your intent via a nodeSelector in your standard Kubernetes Job object.

Request flex-start in your Job:
In your Job.yaml, add the cloud.google.com/gke-flex-start: "true" node selector alongside your accelerator request.

apiVersion: batch/v1
kind: Job
metadata:
  name: my-training-job
spec:
  template:
    spec:
      nodeSelector:
        cloud.google.com/gke-flex-start: "true"
        cloud.google.com/gke-accelerator: nvidia-tesla-a100
      containers:
      - name: my-trainer
        image: "gcr.io/my-project/my-training-image"
        resources:
          limits:
            nvidia.com/gpu: 1
      restartPolicy: Never

When you apply this Job, GKE sees the flex-start selector. It puts the Job's Pods into a Pending state until the DWS queueing system can provision the requested A100 node. Once the node is ready, the Pod is scheduled, your job runs to completion without interruption, and the node is automatically deprovisioned.

Which strategy should you choose?

Here's a simple cheat sheet:

Feature	Spot VMs	DWS with flex-start
Best for	Fault-tolerant, interruptible workloads	Long-running, uninterruptible batch jobs
Primary trade-off	Starts fast, can be preempted at any time	Can wait hours/days to start
Cost savings	Up to 90%	Up to 50%
GKE mode	Standard or Autopilot	Standard or Autopilot
Implementation	--spot flag in a Node Pool	cloud.google.com/gke-flex-start nodeSelector

By mastering both Spot VMs and the Dynamic Workload Scheduler, you can build a resilient and cost-effective AI platform on GKE, even when on-demand accelerator quota seems impossible to find.