DEV Community

Cover image for Your Stopped GCP VM Is Not Guaranteed to Start Again
Eduardo Pittol
Eduardo Pittol

Posted on

Your Stopped GCP VM Is Not Guaranteed to Start Again

#gcp #googlecloud #devops #bash

The setup: an ephemeral staging box. Nobody reviews releases at 2 AM, so a scheduler auto-stops it every evening after business hours — you don't pay for a machine that's asleep. It's a deliberately disposable environment, which is exactly why the next part stung.

One morning you press start and get this:

The zone us-central1-f does not have enough resources available to fulfill the request.

Under the hood that's ZONE_RESOURCE_POOL_EXHAUSTED. Not quota. Not billing. Not IAM. The instance is right there in the console, stopped, exactly as you left it — and it won't turn on. Google Cloud has simply run out of your machine type in that zone. It's called a stockout, and "just start it in another zone" turns out to be impossible. This post is the recovery script I keep on hand, the one concept that makes it work, and what I learned about how GCP hands out capacity.

Why a stopped VM can even fail to start

It ran fine yesterday. Why can't it start today?

Because capacity errors apply only to new requests, and a stopped VM has already handed its compute back. GCP tracks capacity per machine type, per zone — there is no single generic pool of servers. When your VM stopped, that capacity returned to the e2-medium pool in that zone. Pressing start is a brand-new request against whatever's free right now. If the zone filled up with other people's e2-medium instances overnight, your start loses the race. Stopping to save money quietly means giving up your seat and hoping one's free when you come back.

The fix in one idea: make the disk portable

A stopped VM's boot disk is a zonal resource. It lives in exactly one zone and it cannot move. When you start the VM, GCP has to find capacity for that machine type in that specific zone, because that's where the disk is nailed down. If the zone is stocked out for your shape, you're holding a disk you can't boot and can't relocate. Retrying start just hammers the one zone that already said no.

The escape hatch: a zonal disk can't cross zones, but an image can. So you snapshot the stuck disk into an image, delete the instance, and recreate it from that image in whichever zone (and machine type) actually has room.

A zonal boot disk is pinned to one zone; snapshotting it into a global recovery image lets the VM be recreated in any zone that has capacity.

The script

Set the four variables at the top and run it.

#!/usr/bin/env bash
set -euo pipefail

PROJECT="my-project"
INSTANCE="stg-my-app"
IMAGE="${INSTANCE}-recovery-image"

# Standard-tier fallbacks, cheapest first. All x86 + pd-balanced compatible,
# so no Arm families (T2A/C4A/N4A) and no Hyperdisk-only ones (C4/C3D) —
# the recovery image simply can't boot on those.
MACHINE_TYPES=(e2-medium e2-standard-2 n2d-standard-2 t2d-standard-2 n1-standard-2)

# 1. Find where the stopped instance currently lives.
ZONE=$(gcloud compute instances list --project="$PROJECT" \
  --filter="name=$INSTANCE" --format="value(zone.basename())")
[[ -n "$ZONE" ]] || { echo "Couldn't find $INSTANCE." >&2; exit 1; }

# 2. Discover every UP zone in the instance's region — no hardcoded zone
#    list to maintain as GCP adds or retires zones.
REGION="${ZONE%-*}"                       # us-central1-f -> us-central1
ZONES=($(gcloud compute zones list --project="$PROJECT" \
  --filter="region:$REGION AND status=UP" --format="value(name)"))
[[ ${#ZONES[@]} -gt 0 ]] || { echo "No UP zones in $REGION." >&2; exit 1; }

# 3. Read the config we need to recreate the instance faithfully.
field() { gcloud compute instances describe "$INSTANCE" \
  --project="$PROJECT" --zone="$ZONE" --format="value($1)"; }
NETWORK=$(field "networkInterfaces[0].network.basename()")
SUBNET=$(field "networkInterfaces[0].subnetwork.basename()")
DISK_SIZE=$(field "disks[0].diskSizeGb")

# 4. Freeze the boot disk into a GLOBAL image. This is the key move:
#    a zonal disk can't change zones, but an image can.
gcloud compute images create "$IMAGE" --project="$PROJECT" \
  --source-disk="$INSTANCE" --source-disk-zone="$ZONE"

# 5. Delete the stuck instance so its name is free to reuse.
gcloud compute instances delete "$INSTANCE" \
  --project="$PROJECT" --zone="$ZONE" --quiet

# 6. Cheapest type first: hold each machine type and sweep every zone before
#    escalating to a costlier one. Lands on the cheapest shape that has
#    capacity anywhere, and only pays more when a type is stocked out
#    region-wide. (MACHINE_TYPES is already ordered cheapest -> priciest.)
for mtype in "${MACHINE_TYPES[@]}"; do
  for zone in "${ZONES[@]}"; do
    echo "Trying $mtype in $zone..."
    if gcloud compute instances create "$INSTANCE" \
      --project="$PROJECT" --zone="$zone" --machine-type="$mtype" \
      --image="$IMAGE" --network="$NETWORK" --subnet="$SUBNET" \
      --boot-disk-size="${DISK_SIZE}GB"; then
      echo "Back up in $zone as $mtype."
      gcloud compute images delete "$IMAGE" --project="$PROJECT" --quiet
      exit 0
    fi
  done
done

echo "No capacity anywhere. Recovery image '$IMAGE' kept — recreate by hand." >&2
exit 1
Enter fullscreen mode Exit fullscreen mode

Step 6 is the heart of it: a nested loop that holds the cheapest machine type and sweeps every zone before moving up to a costlier type — so you land on the cheapest shape that has capacity anywhere, and only pay more when the cheap one is stocked out region-wide. The zone list isn't hardcoded — it's pulled from the instance's own region at runtime, so the script keeps working as Google adds or retires zones.

A grid of machine types (rows, cheapest on top) by zones (columns); the cheapest type is out of capacity in every zone, and the run succeeds on the next type up in zone b.

Cheapest type first: sweep it across every zone before paying for a bigger one, and stop at the first hit.

The gotchas that cost me retries

The machine-type list isn't arbitrary. Two constraints are baked in, both learned the hard way:

  • No Arm. The image is built from an x86 box. T2A, C4A, and N4A are Arm — the image won't boot on them. That's not a capacity failure, it's a hard incompatibility, and each one wastes a retry.
  • No Hyperdisk-only families. C4/C4D/C3D want Hyperdisk boot disks, not the pd-balanced this instance uses. Same story: they fail for the wrong reason.

And the rule I'd tattoo on my hand: the recovery image is only deleted after a new instance is confirmed up. You are deliberately deleting your only instance in step 5. If the script also deleted the image on failure, a total stockout would leave you with nothing. On failure it keeps the image — that's your recovery point.

How GCP hands out capacity (so you can bias the odds)

The fallback list looks cheapest-first, but it's really availability-first — it climbs GCP's capacity gradient:

  • You can't check first. There's no API that tells you whether a type has free capacity in a zone. On-demand capacity is opaque; the only signal is attempting the create and catching the stockout. Every strategy is fundamentally attempt-and-fall-back. (I did try gcloud alpha compute advice capacity to get ahead of it — it's gated behind an Alpha allowlist my account isn't on, so: 403.)
  • Commodity beats cutting-edge. The newest series (C3, C4, GPUs) are the scarcest; E2 and N2 are the safe fallbacks. A ladder like C3 → N2 → E2 usually finds room.
  • Smaller beats bigger. Large shapes need contiguous capacity on a single host, so they stock out more. A smaller shape in the same series is likelier to fit.
  • Zone beats type. A stockout is per type and per zone, so sweeping zones is often a bigger lever than swapping types. The loop leans on this: it holds the cheapest type and tries every zone before escalating — diversifying zones first, and paying more only when a type is out region-wide.
  • Spot and on-demand are separate pools. Spot has its own capacity, but it can be preempted — no good when you need the box reliably up for a test session.

Lock down who can run it

"Let a teammate restart the staging box" sounds harmless, but this script deletes an instance. You do not want to hand out compute.admin for that.

I scoped a custom role to exactly the verbs the script needs — create/delete instance, create/delete image, read the config, and (for the dynamic zone lookup) compute.zones.list — and pinned it with an IAM condition so it only applies to this one instance, its disk, and its recovery image. The blast radius of the restart button should be one VM.

Why this happens — and why it's not just you

It's tempting to assume you misconfigured something. You didn't. A zone is a finite pile of physical machines, and sometimes the shape you want isn't in the pile right now. Google's own docs are blunt about it: resource errors are unrelated to your quota and apply only to the exact resource you asked for, at the moment you asked — try a different zone, a different machine type, or again later.

And it's a well-worn rake. Google's gce-discussion group has threads going back years — one stockout stretched close to 24 hours across multiple zones, with a user flatly saying it "affects us to the point we cant use GCP."

If uptime genuinely matters, this script is the wrong tool, and it's worth knowing what the right ones are:

  • Reservations hold capacity for a specific machine type in a specific zone. You pay whether you use it or not, but the capacity is yours.
  • Committed use discounts (CUDs) are easy to confuse with reservations, but they're a billing discount across a series in a region — not a capacity guarantee.
  • MIG instance flexibility is GCP's native "try types until one is available" primitive, with a ranked list of machine types. It's built for stateless, interchangeable VMs, though — wrapping a single stateful staging box in a managed instance group is far more machinery than the problem deserves. The Beyond "Try Again Later" writeup is a good map of these options.

The cheapest fix is prevention

The highest-leverage, lowest-effort move isn't the script at all: don't run staging on scarce hardware. If the box is on C3/C4/GPU, that's very likely why it won't start. Staging rarely needs the newest silicon — move it to E2 or N2 and the stockouts mostly disappear. The script is the seatbelt for the day it happens anyway.

Takeaways

  • A stopped VM is not a reservation. Stopping it releases capacity back to the pool; starting it is a fresh request that can fail.
  • Capacity is per type, per zone, and opaque. There's no API to check it in advance — you attempt and fall back.
  • Zonal disks pin you to a zone. Crossing zones means routing through a global artifact — an image or a snapshot.
  • Commodity types stock out far less. The cheapest fix is not being on scarce hardware in the first place.
  • When a script deletes your only copy, protect the recovery point above everything else.

The cloud markets itself as infinite. It isn't — it's a very large, very finite pile of other people's computers, and once in a while the pile you want is empty. Plan for the empty pile.

Sources: GCP resource-availability troubleshooting · gce-discussion stockout thread · Beyond "Try Again Later" (Google Cloud Community)

Top comments (0)