🔴 Ongoing · Reports active since April 2026 · No official GCP acknowledgement
Engineers running workloads on Google Cloud Platform's us-central1 region are encountering a growing problem: GCP cannot provision new n2 VMs on demand. The autoscaler requests the machines. GCP says no. Your pods stay Pending. And the GCP status dashboard sits on green the entire time.
Google officially defines this error as occurring when "the zone does not have enough resources available to fulfill the request" — distinguishing it from quota limits, which are a separate class of problem. The error, when you eventually find it buried in GKE cluster-level logs, looks like this:
Node scale up in zones us-central1-f associated with this pod failed:
GCE out of resources. Pod is at risk of not being scheduled.
The issue has been documented across Google's own developer forums since April 2026, with fresh reports appearing through August. It has affected n1, n2, and e2 instance types across all availability zones in us-central1 — zones a, b, c, and f — making the standard "try a different zone" advice ineffective when the shortage is regional.
What Engineers Are Reporting on Google's Own Forums
The reports started quietly in spring and have grown louder through summer. These are direct quotes from Google Developer Community discussion threads:
"For many years I've been deploying Compute Engine VMs to us-central-1 region but for the past couple days I've constantly received error ZONE_RESOURCE_POOL_EXHAUSTED."
— Kevin_Koehler, April 13, 2026 · Google Developer Community"Assumed it was a temporary resource shortage but it's been happening consistently for the past week, which has been very frustrating."
— nju, May 18, 2026 · Google Developer Community"I tried zones a, b, c, and f. Same issue. I tried e2, n1, n2. Same issue... I guess GCP doesn't want my money anymore 🤷"
— standev, August 13, 2026 · Google Developer Community⚠️ No Official Response. As of the time of writing, no Google employee has responded to these forum threads, and no incident has been posted to the GCP Status Dashboard for this issue. The gap between what users experience and what Google publicly acknowledges is the defining characteristic of this problem.
The Failure Mode That Looks Like Slowness
What makes this particularly dangerous for production systems is how the failure presents itself. When GCP cannot provision a requested machine type, it does not return an error to your application. The GKE cluster autoscaler keeps retrying silently. Your worker pods stay in Pending state. Your job scheduler marks tasks as still running. Your monitoring dashboards show activity. Nothing appears broken.
The only place the truth surfaces is in GKE cluster-level logs — not in your application logs, not in your job scheduler UI. Most engineers only look there when something is visibly broken, which by then is hours later.
In distributed batch workloads on GKE, a common pattern is a lightweight driver pod that coordinates the job and schedules work across executor (worker) pods that do the actual computation. When this capacity issue strikes, the driver pod starts successfully — it's small and fits on an existing node. The executor pods, which need the new n2 VMs the autoscaler is trying to provision, never arrive. From the outside, the job appears to be running. Internally, it's doing nothing.
| Time | What Happens |
|---|---|
| T + 0 min | Driver pod created, job shows as RUNNING. Everything looks normal. |
| T + ongoing | Autoscaler requests new n2 nodes — GCP cannot fulfil. Worker pods remain Pending. No error surfaced to the app layer. |
| T + hours later | GCP eventually finds capacity — workers schedule, job completes. Having waited hours to do minutes of work. |
"The driver node got created around 19:30 UTC but the executors were allocated only around 00:30 UTC — 5 hours later. This is the case with all the tasks."
— Platform engineer's documented incident report
Google's own GKE documentation acknowledges this scenario, stating the resolution is to "try a different zone, or try again later." When the shortage is regional and persistent — spanning all zones for hours — neither of those options helps in the moment.
Which Machine Types Are Affected
Based on community forum reports, the issue is not limited to a single machine family:
| Machine Family | Types Reported | Zones (us-central1) | Community Status |
|---|---|---|---|
| n2 | n2-standard, n2-highmem (all sizes) | a, b, c, f | ❌ Most frequently reported |
| n1 | n1-standard, n1-highmem | a, b, c, f | ❌ Also reported |
| e2 | e2-medium, e2-standard | Some zones | ⚠️ Intermittent reports |
| n2d | n2d-highmem, n2d-standard | a, b, c, f | ⚠️ Some teams found better availability |
| c4 | c4-highmem, c4-standard | — | ✅ No shortage reports found (Aug 2026) |
Three Actions You Can Take Today
1. Stop autoscaling from zero on SLA-critical node pools
When your minimum node count is zero, a capacity crunch becomes an indefinite stall — the autoscaler cannot get new nodes, and no existing nodes are available either. Raising the minimum floor means your workloads always have somewhere to land.
# Before — zero floor, entirely dependent on autoscaler from scratch
min_node_count = 0
max_node_count = 10
# After — warm baseline; autoscaler supplements from here
min_node_count = 4
max_node_count = 10
Always commit this change through your IaC tooling (Terraform / Terragrunt). A manual GCP Console change is not reflected in state and will be silently overwritten on the next apply.
2. Try a different machine type
Google's official troubleshooting documentation explicitly lists "selecting a different machine series or type" as a recommended workaround for ZONE_RESOURCE_POOL_EXHAUSTED. Some teams have had better provisioning success with n2d compared to n2 during this period.
gcloud beta container node-pools update YOUR-NODEPOOL-NAME \
--project YOUR-PROJECT-ID \
--cluster YOUR-CLUSTER-NAME \
--region us-central1 \
--machine-type n2d-highmem-16
Plan for approximately 30 minutes of node pool downtime. Schedule this during a quiet window and validate the new machine type on a non-production cluster first.
3. Alert on pod pending time — not just job failure
This failure mode presents as slowness, not an error. Your job scheduler will not raise an alert — it sees the job as running. Set a Kubernetes-level alert on pod pending duration.
# List all Pending pods across namespaces
kubectl get pods -A --field-selector=status.phase=Pending
# Inspect a specific pod — look in Events for the root cause
kubectl describe pod <pod-name> -n <namespace>
# GCE out of resources → capacity problem (this article)
# Insufficient cpu/memory → resource request too large for available nodes
The Long-Term Answer: Consider Migrating to C4
The C4 machine series — powered by Intel Emerald Rapids processors and Google's Titanium offload chips — is Google's current-generation compute offering, and no documented community reports of ZONE_RESOURCE_POOL_EXHAUSTED for C4 have been found as of August 2026.
| Spec | n2-highmem | c4-highmem |
|---|---|---|
| Processor | Intel Cascade Lake / Ice Lake | Intel Emerald Rapids (5th Gen Xeon) |
| Max vCPUs | 128 | 192 |
| Max memory | 864 GB | 1.5 TB DDR5 |
| Network bandwidth | Up to 100 Gbps | Up to 200 Gbps |
| Price-performance | Baseline | Up to 25% better vs C3 |
💡 Migration approach: The safest path is to create a new C4 node pool alongside your existing n2 pool, validate your workloads on the new pool, then drain and delete the old one. This avoids any single-pool downtime and lets you roll back cleanly if something doesn't behave as expected.
Before migrating, validate on a pre-production cluster first. C4 uses Hyperdisk as its storage backend by default rather than standard persistent disks — confirm compatibility before rolling to production.
Key Takeaways
1. The GCP status page is not your monitoring.
ZONE_RESOURCE_POOL_EXHAUSTED does not appear as a GCP incident. Monitor your cluster directly — pod pending duration is a more reliable early-warning signal than any external status page.
2. Minimum node count zero is a risk for production batch.
Autoscaling from zero is cost-efficient under normal conditions. Under capacity constraints it means indefinite waits. For jobs with SLAs, maintain a warm baseline.
3. Multi-zone helps — regional shortages need a different response.
Zone diversity protects against single-zone hardware failures. When the shortage is regional, the only effective responses are machine type diversity or capacity reservations made in advance.
4. Know your machine type fallback before you need it.
Google explicitly recommends switching machine series as a workaround. Have a tested alternative machine type ready in your runbook — the wrong time to figure it out is during a production incident at 3 AM.
The frustrating reality of ZONE_RESOURCE_POOL_EXHAUSTED is that it sits in a grey zone: widespread enough to affect multiple teams across months, specific enough that Google doesn't surface it as a regional incident. The community has been documenting it since April. The status page has stayed green throughout.
If you're running SLA-critical workloads in us-central1 on n2 — especially with a minimum node count of zero — the fixes above are worth reviewing this week, not after the next unexplained multi-hour delay.
Sources
- Sudden VM Unavailability in us-central-1 (Apr–Aug 2026) — Google Developer Community
- Shortage of Compute Engine Resource in us-central-1 — Google Developer Forums
- Troubleshooting Resource Availability Errors — Google Cloud Docs
- C4 and N4 Machine Series Deep Dive — Google Cloud Blog
- Troubleshoot GKE Standard Node Pools — Google Cloud Docs
If you’re dealing with something like this at work — or you have a war story of your own — I’m on LinkedIn and I actually respond.
I publish one production incident breakdown every week. Subscribe free if you want it directly in your inbox — no tutorials, no fluff:
Let’s Stay in Touch 🪂 Connect with me on: LinkedIn, Instagram, Medium
Top comments (0)