DEV Community: Eduardo Pittol

Your Stopped GCP VM Is Not Guaranteed to Start Again

Eduardo Pittol — Wed, 22 Jul 2026 19:46:37 +0000

The setup: an ephemeral staging box. Nobody reviews releases at 2 AM, so a scheduler auto-stops it every evening after business hours — you don't pay for a machine that's asleep. It's a deliberately disposable environment, which is exactly why the next part stung.

One morning you press start and get this:

The zone us-central1-f does not have enough resources available to fulfill the request.

Under the hood that's ZONE_RESOURCE_POOL_EXHAUSTED. Not quota. Not billing. Not IAM. The instance is right there in the console, stopped, exactly as you left it — and it won't turn on. Google Cloud has simply run out of your machine type in that zone. It's called a stockout, and "just start it in another zone" turns out to be impossible. This post is the recovery script I keep on hand, the one concept that makes it work, and what I learned about how GCP hands out capacity.

Why a stopped VM can even fail to start

It ran fine yesterday. Why can't it start today?

Because capacity errors apply only to new requests, and a stopped VM has already handed its compute back. GCP tracks capacity per machine type, per zone — there is no single generic pool of servers. When your VM stopped, that capacity returned to the e2-medium pool in that zone. Pressing start is a brand-new request against whatever's free right now. If the zone filled up with other people's e2-medium instances overnight, your start loses the race. Stopping to save money quietly means giving up your seat and hoping one's free when you come back.

The fix in one idea: make the disk portable

A stopped VM's boot disk is a zonal resource. It lives in exactly one zone and it cannot move. When you start the VM, GCP has to find capacity for that machine type in that specific zone, because that's where the disk is nailed down. If the zone is stocked out for your shape, you're holding a disk you can't boot and can't relocate. Retrying start just hammers the one zone that already said no.

The escape hatch: a zonal disk can't cross zones, but an image can. So you snapshot the stuck disk into an image, delete the instance, and recreate it from that image in whichever zone (and machine type) actually has room.

The script

Set the four variables at the top and run it.

#!/usr/bin/env bash
set -euo pipefail

PROJECT="my-project"
INSTANCE="stg-my-app"
IMAGE="${INSTANCE}-recovery-image"

# Standard-tier fallbacks, cheapest first. All x86 + pd-balanced compatible,
# so no Arm families (T2A/C4A/N4A) and no Hyperdisk-only ones (C4/C3D) —
# the recovery image simply can't boot on those.
MACHINE_TYPES=(e2-medium e2-standard-2 n2d-standard-2 t2d-standard-2 n1-standard-2)

# 1. Find where the stopped instance currently lives.
ZONE=$(gcloud compute instances list --project="$PROJECT" \
  --filter="name=$INSTANCE" --format="value(zone.basename())")
[[ -n "$ZONE" ]] || { echo "Couldn't find $INSTANCE." >&2; exit 1; }

# 2. Discover every UP zone in the instance's region — no hardcoded zone
#    list to maintain as GCP adds or retires zones.
REGION="${ZONE%-*}"                       # us-central1-f -> us-central1
ZONES=($(gcloud compute zones list --project="$PROJECT" \
  --filter="region:$REGION AND status=UP" --format="value(name)"))
[[ ${#ZONES[@]} -gt 0 ]] || { echo "No UP zones in $REGION." >&2; exit 1; }

# 3. Read the config we need to recreate the instance faithfully.
field() { gcloud compute instances describe "$INSTANCE" \
  --project="$PROJECT" --zone="$ZONE" --format="value($1)"; }
NETWORK=$(field "networkInterfaces[0].network.basename()")
SUBNET=$(field "networkInterfaces[0].subnetwork.basename()")
DISK_SIZE=$(field "disks[0].diskSizeGb")

# 4. Freeze the boot disk into a GLOBAL image. This is the key move:
#    a zonal disk can't change zones, but an image can.
gcloud compute images create "$IMAGE" --project="$PROJECT" \
  --source-disk="$INSTANCE" --source-disk-zone="$ZONE"

# 5. Delete the stuck instance so its name is free to reuse.
gcloud compute instances delete "$INSTANCE" \
  --project="$PROJECT" --zone="$ZONE" --quiet

# 6. Cheapest type first: hold each machine type and sweep every zone before
#    escalating to a costlier one. Lands on the cheapest shape that has
#    capacity anywhere, and only pays more when a type is stocked out
#    region-wide. (MACHINE_TYPES is already ordered cheapest -> priciest.)
for mtype in "${MACHINE_TYPES[@]}"; do
  for zone in "${ZONES[@]}"; do
    echo "Trying $mtype in $zone..."
    if gcloud compute instances create "$INSTANCE" \
      --project="$PROJECT" --zone="$zone" --machine-type="$mtype" \
      --image="$IMAGE" --network="$NETWORK" --subnet="$SUBNET" \
      --boot-disk-size="${DISK_SIZE}GB"; then
      echo "Back up in $zone as $mtype."
      gcloud compute images delete "$IMAGE" --project="$PROJECT" --quiet
      exit 0
    fi
  done
done

echo "No capacity anywhere. Recovery image '$IMAGE' kept — recreate by hand." >&2
exit 1

Step 6 is the heart of it: a nested loop that holds the cheapest machine type and sweeps every zone before moving up to a costlier type — so you land on the cheapest shape that has capacity anywhere, and only pay more when the cheap one is stocked out region-wide. The zone list isn't hardcoded — it's pulled from the instance's own region at runtime, so the script keeps working as Google adds or retires zones.

Cheapest type first: sweep it across every zone before paying for a bigger one, and stop at the first hit.

The gotchas that cost me retries

The machine-type list isn't arbitrary. Two constraints are baked in, both learned the hard way:

No Arm. The image is built from an x86 box. T2A, C4A, and N4A are Arm — the image won't boot on them. That's not a capacity failure, it's a hard incompatibility, and each one wastes a retry.
No Hyperdisk-only families. C4/C4D/C3D want Hyperdisk boot disks, not the pd-balanced this instance uses. Same story: they fail for the wrong reason.

And the rule I'd tattoo on my hand: the recovery image is only deleted after a new instance is confirmed up. You are deliberately deleting your only instance in step 5. If the script also deleted the image on failure, a total stockout would leave you with nothing. On failure it keeps the image — that's your recovery point.

How GCP hands out capacity (so you can bias the odds)

The fallback list looks cheapest-first, but it's really availability-first — it climbs GCP's capacity gradient:

You can't check first. There's no API that tells you whether a type has free capacity in a zone. On-demand capacity is opaque; the only signal is attempting the create and catching the stockout. Every strategy is fundamentally attempt-and-fall-back. (I did try gcloud alpha compute advice capacity to get ahead of it — it's gated behind an Alpha allowlist my account isn't on, so: 403.)
Commodity beats cutting-edge. The newest series (C3, C4, GPUs) are the scarcest; E2 and N2 are the safe fallbacks. A ladder like C3 → N2 → E2 usually finds room.
Smaller beats bigger. Large shapes need contiguous capacity on a single host, so they stock out more. A smaller shape in the same series is likelier to fit.
Zone beats type. A stockout is per type and per zone, so sweeping zones is often a bigger lever than swapping types. The loop leans on this: it holds the cheapest type and tries every zone before escalating — diversifying zones first, and paying more only when a type is out region-wide.
Spot and on-demand are separate pools. Spot has its own capacity, but it can be preempted — no good when you need the box reliably up for a test session.

Lock down who can run it

"Let a teammate restart the staging box" sounds harmless, but this script deletes an instance. You do not want to hand out compute.admin for that.

I scoped a custom role to exactly the verbs the script needs — create/delete instance, create/delete image, read the config, and (for the dynamic zone lookup) compute.zones.list — and pinned it with an IAM condition so it only applies to this one instance, its disk, and its recovery image. The blast radius of the restart button should be one VM.

Why this happens — and why it's not just you

It's tempting to assume you misconfigured something. You didn't. A zone is a finite pile of physical machines, and sometimes the shape you want isn't in the pile right now. Google's own docs are blunt about it: resource errors are unrelated to your quota and apply only to the exact resource you asked for, at the moment you asked — try a different zone, a different machine type, or again later.

And it's a well-worn rake. Google's gce-discussion group has threads going back years — one stockout stretched close to 24 hours across multiple zones, with a user flatly saying it "affects us to the point we cant use GCP."

If uptime genuinely matters, this script is the wrong tool, and it's worth knowing what the right ones are:

Reservations hold capacity for a specific machine type in a specific zone. You pay whether you use it or not, but the capacity is yours.
Committed use discounts (CUDs) are easy to confuse with reservations, but they're a billing discount across a series in a region — not a capacity guarantee.
MIG instance flexibility is GCP's native "try types until one is available" primitive, with a ranked list of machine types. It's built for stateless, interchangeable VMs, though — wrapping a single stateful staging box in a managed instance group is far more machinery than the problem deserves. The Beyond "Try Again Later" writeup is a good map of these options.

The cheapest fix is prevention

The highest-leverage, lowest-effort move isn't the script at all: don't run staging on scarce hardware. If the box is on C3/C4/GPU, that's very likely why it won't start. Staging rarely needs the newest silicon — move it to E2 or N2 and the stockouts mostly disappear. The script is the seatbelt for the day it happens anyway.

Takeaways

A stopped VM is not a reservation. Stopping it releases capacity back to the pool; starting it is a fresh request that can fail.
Capacity is per type, per zone, and opaque. There's no API to check it in advance — you attempt and fall back.
Zonal disks pin you to a zone. Crossing zones means routing through a global artifact — an image or a snapshot.
Commodity types stock out far less. The cheapest fix is not being on scarce hardware in the first place.
When a script deletes your only copy, protect the recovery point above everything else.

The cloud markets itself as infinite. It isn't — it's a very large, very finite pile of other people's computers, and once in a while the pile you want is empty. Plan for the empty pile.

Sources: GCP resource-availability troubleshooting · gce-discussion stockout thread · Beyond "Try Again Later" (Google Cloud Community)

I had an application taking 10 seconds to load. Turned out file ops functions on aren't real filesystem calls due S3 stream wrapper. This post brings the results of a POC that I built to demonstrate the impact of these invisible of these network calls.

Eduardo Pittol — Thu, 16 Jul 2026 18:26:55 +0000

Eduardo Pittol

Jul 16

Your file_exists() Is Secretly a Network Call

#php #performance #s3 #aws

6 min read

Your file_exists() Is Secretly a Network Call

Eduardo Pittol — Thu, 16 Jul 2026 18:14:10 +0000

The one-line horror

if ( file_exists( $path ) ) {
    // ... use the cached file
}

Nothing about that line looks dangerous. It's the kind of guard clause you've written a thousand times. But suppose $path is:

s3://my-bucket/cache/style-4f2a.css

Now that single file_exists() is not a syscall. It's a HeadObject request to Amazon S3 — a network round trip that leaves your server, waits for S3, and comes back. Against distant S3, that round trip costs tens of milliseconds.

Here's the twist that makes it dangerous: the AWS SDK caches that result in memory, so a second check of the same path is instant. The cost hides behind the cache. But every web request is a fresh PHP process with an empty cache, so the first touch of every path pays full price — every request. And writes never get even that reprieve: nothing caches a PutObject, so every file_put_contents() over S3 is a full round trip, every time. Reads can ride the cache; writes always cross the wire.

Local stat() costs microseconds. The code doesn't change — only the string in $path does. PHP stream wrappers make remote storage look like a local disk, and code written for local-disk economics keeps compiling, keeps passing tests, and silently falls off a performance cliff the moment the path points at a bucket.

Why an innocent function makes a network call

A stream wrapper is a class registered against a URL scheme:

stream_wrapper_register( 's3', S3\StreamWrapper::class );

Once s3:// is registered, PHP routes every filesystem call for that scheme through the wrapper: fopen(), fread(), file_get_contents(), stat(), is_dir(), unlink(). The whole point is that your code doesn't have to know or care whether it's talking to a disk or a bucket — the API surface is identical.

That's exactly the trap. Each operation quietly maps to an S3 API request:

PHP call	S3 request
`file_exists()` / `stat()`	`HeadObject`
`is_dir()`	`ListObjects`
`unlink()`	`DeleteObject`
`file_put_contents()`	`PutObject`

If you've ever hunted down an N+1 query in an ORM, you already understand the failure mode. N+1 is a loop that fires one database round trip per item instead of batching them. This is the same shape — one network round trip per filesystem call — except the round trip hides behind a function whose name says "filesystem." Nobody profiles file_exists(). Static analysis won't flag it. And in local development, where the wrapper points at a fast disk or a nearby S3-compatible service, it's instant. The cost only shows up as latency against real, distant S3 in production — the worst possible place to discover it.

The numbers below come from a controlled rig: a local MinIO standing in for S3, with Toxiproxy injecting a fixed round-trip time (RTT) of 0, 10, 20, or 40 ms — roughly the spread from same-AZ to cross-region.

The benchmark: cost scales with the network

Median latency per call, S3 backend, as RTT climbs (local disk stays ≤ 0.01 ms for all operations at every RTT):

S3 operation	RTT 0 ms	10 ms	20 ms	40 ms
`file_exists`	0.002 ms	0.022 ms	0.025 ms	0.026 ms
`stat`	0.002 ms	0.022 ms	0.029 ms	0.032 ms
`file_put_contents`	1.460 ms	17.961 ms	29.626 ms	49.551 ms

Two things jump out. Reads barely move with latency. Checking the same path in a loop, file_exists() stays around 0.02 ms no matter the RTT — the SDK's in-memory stat cache absorbs the repeats, so they never cross the wire twice. Writes track RTT almost linearly, because nothing caches a PutObject.

Now scale to a page. A single real page in the incident below fired 88 filesystem ops, so reconstruct the page cost as median × 88:

S3 op × 88	RTT 0 ms	10 ms	20 ms	40 ms
`file_exists`	0.14 ms	1.95 ms	2.19 ms	2.25 ms
`stat`	0.17 ms	1.94 ms	2.55 ms	2.81 ms
`file_put_contents`	128.5 ms	1,580.6 ms	2,607.1 ms	4,360.5 ms

At 40 ms RTT — a cross-region hop — 88 writes cost ~4.4 seconds. That's the cliff, and every extra millisecond between you and S3 makes it steeper.

But look at the read rows: 2 ms for 88 checks. That looks harmless — and it's the most misleading number in the table. It's low only because the benchmark checks the same paths repeatedly inside one long-lived process, so the SDK's in-memory stat cache absorbs the repeats. Nothing crosses the wire twice.

PHP in production doesn't work that way. Every web request is served by a fresh process with an empty cache, and that cache is gone the moment the request ends — the LruArrayCache lives in process memory, not in Redis or on disk. So the first touch of each distinct path is always a full round trip, and nothing carries over to the next request. If a page checks 73 distinct files, that's 73 cold round trips — every request. Which is exactly what happened next.

The war story: a 10-second homepage

This isn't hypothetical. A high-traffic WordPress site, with its media library backed by S3, had a homepage that took 10 seconds to return.

New Relic told the story immediately. GET / returned HTTP 200 in 10.07 s — with an empty database-queries tab. No slow SQL. Roughly 90% of the time was synchronous S3 traffic, all inside a single page view:

Segment	Calls	Time
`s3.amazonaws.com`	73	3,281 ms
Guzzle `CurlMultiHandler::tick`	59	2,623 ms
Stream wrapper closure	88	1,702 ms
`AwsClient::execute`	124	1,582 ms
Guzzle `CurlMultiHandler::execute`	78	873 ms

73 real S3 calls to render one homepage. The theme's CSS layer was the culprit: for every style handle, on every breakpoint, on every request, it called file_exists() on the generated stylesheet to decide whether to regenerate — dozens of HeadObjects per page — even though it already held a persisted flag saying the cache was valid. Those were 73 distinct paths, so the in-process cache never helped; and because the SDK's stat cache was request-scoped — an in-memory cache that dies with the PHP process — nothing survived to the next request either. Every check was a cold round trip.

The fix was a tour of where a cache can live — each layer trading something different:

Persistent stat cache. Replace the request-scoped LruArrayCache with an adapter over the WordPress object cache (Redis). It implements the SDK's CacheInterface and is injected when the wrapper is registered — no plugin patching required. Now a stat survives across requests. The cost: a Redis hop instead of memory, still far cheaper than S3.
Local-disk-first. For generated CSS, write to local disk, mirror to S3 asynchronously after the write, and restore from S3 only if the local copy goes missing. Reads become genuine microsecond stat()s again, and — crucially — the writes stop hitting S3 on the request path. The cost: you trade strong read-after-write consistency across nodes for latency, a deliberate, documented trade-off.

The page dropped from ~10 seconds to well under a second. Same features, same S3 bucket — the only thing that changed was refusing to let a network call keep masquerading as a syscall.

How to spot this in your own code

Audit filesystem calls near wrapped paths. Grep for file_exists, stat, is_dir, unlink, and file_put_contents, and ask, for each, whether the path could ever be s3:// (or gs://, or any wrapper). Pay special attention to anything inside a loop and to distinct paths — those never hit the cache.
Read APM segment breakdowns, not just the total. A slow request with an empty SQL tab is the tell. Look for time in the storage SDK and the HTTP handler.
Distrust request-scoped caches. A cache that isn't shared across processes does nothing for a per-request PHP model. Confirm where your cache actually lives.
Remember that a cache can't save a write. Reads can be cached; every PutObject is a real round trip. Batch writes, defer them off the request path, or keep them local and mirror asynchronously.

And the principle underneath all of it: when an abstraction changes the cost model by orders of magnitude, it has to leak that cost somewhere. An abstraction that hides a 50 ms network call behind a microsecond-shaped function isn't a convenience — it's a latency bug waiting for production traffic. Put the cost back where you can see it: a persistent cache, a batched call, or a local-first layer. Don't let file_exists() keep wearing a syscall's clothes.

All the measurements in this post come from a small, self-contained benchmark rig — MinIO standing in for S3, Toxiproxy injecting the round-trip latency, and the PHP scripts that produced every table above. It's on GitHub: edpittol/s3-stream-multiple-operations. Clone it and reproduce the numbers.

Stop your team from rebuilding the same AI skills: a shared catalog that maintains itself

Eduardo Pittol — Sun, 14 Jun 2026 18:39:06 +0000

If your team has started using skills for AI coding agents, you've probably hit this already: someone uses an existing skill or builds one to run the test suite a certain way. Two weeks later, someone else does it their own way for the same job — different steps, different conventions, a different result — because they had no idea the first one existed. Now the agent does the "same" task one way on your machine and another way on theirs.

That's the part that actually hurts. It isn't just the wasted hour rebuilding something that already existed (though that adds up). It's that the team quietly drifts into a dozen slightly different ways of solving the same problem — inconsistent results, no shared standard, and no single place to fix anything when it breaks. Multiply it across a few people and a few months, and "skills" stops being leverage and starts being entropy.

I ran into exactly this. The fix wasn't more discipline — it was a shared catalog that's so easy to contribute to that nobody has an excuse not to — and the whole team reaches for one skill instead of each rolling their own. The trick that made it stick: a skill whose only job is to add other skills to the catalog. This post is how that works, and how to build your own.

The idea: one catalog, one source of truth

The catalog is a single repo. Skills live under skills/<category>/, where the categories are deliberately boring:

engineering — development, testing, code review, automation
productivity — organization, writing, planning, day-to-day workflows
misc — everything else

At the root there's a master table — one row per skill, with its category and a link back to the source it came from. That table is the single source of truth for what the team has. New teammate? Point them at the table. Wondering if a skill already exists? Check the table before you build.

And here's the payoff that's easy to miss: when everyone reaches for the same skill, the catalog becomes a feedback loop. Someone hits a rough edge — a flaky step, a missing case, a prompt the agent keeps misreading — and they fix it in that one canonical skill. The fix lands for the whole team at once. Troubleshooting stops being something each person quietly redoes in private and starts accumulating: a shared skill gets sharper every time anyone uses it. Twelve private copies just stay broken in twelve different ways.

It's a nice idea. It also rots the instant adding a skill becomes a chore.

The catch: cataloging by hand is exactly the friction that kills it

Look at what "just add it to the catalog" actually means by hand:

Pick the right category and copy the folder into skills/<category>/.
Write an entry in that category's README — in the right alphabetical spot.
Resolve a permalink back to the source: the upstream repo URL, the org/repo, and the exact commit hash.
Regenerate the master table at the root, re-sorted alphabetically, without breaking the existing rows.

Nobody wants to do that after every skill. So they don't — and the catalog drifts out of date until it's useless. The friction is the failure mode.

The fix: a skill that catalogs skills

So I made the bookkeeping itself a skill: catalog-a-skill. You point it at a skill folder and it does all four steps. Here's the abridged SKILL.md:

---
name: catalog-a-skill
description: "Catalogs an existing skill into this repo's category structure. Use when a skill folder with a valid SKILL.md already exists and needs to be added to the catalog."
---

# Catalog a Skill

## Workflow

### 1. Read the skill
- Read `<skill-dir>/SKILL.md` and extract `name` and `description`.
- Gate check 1 — missing frontmatter.
- Gate check 2 — folder name ≠ frontmatter `name`.

### 2. Choose the category
Pick engineering / productivity / misc from the description.

### 3. Position the skill
- Copy to `skills/<category>/<skill-name>/`.
- Resolve the upstream reference from the SOURCE repo:
  commit hash (`git rev-parse HEAD`), remote URL, and `org/repo`.

### 4. Update the category README
Insert a section at its alphabetical position.

### 5. Regenerate the master table
One row per skill, sorted by name, with a permalink to the source commit.

### 6. Verify
Run `npx skills list` and confirm the skill appears.

The whole point: the contributor does the fun part (writing a useful skill) and hands the tedious, rule-bound part to the agent. That's the kind of work agents are genuinely good at — deterministic, fiddly, easy to get subtly wrong by hand.

The part that makes it trustworthy: it refuses to do the wrong thing

A naive version would happily corrupt the catalog. The real value is in the gates — the cases where it stops instead of cataloging:

Missing frontmatter. If name or description is absent, it copies the skill into misc but does not touch the README or the master table. Then it tells you exactly which field is missing. A broken skill never silently pollutes the source of truth.
Name divergence. If the folder is named foo but the frontmatter says name: bar, that's a red flag — it copies under the original folder name, refuses to catalog, and reports the mismatch.

That "refuse and report" behavior is what turns a convenient script into something a team can trust to run on its catalog.

How to build your own

You don't need my repo — you need the pattern. To replicate it:

Make the catalog repo. skills/<category>/ directories, a root README with a master table, one README per category. Keep the categories few and obvious.
Decide your source of truth. I use the root table; the rule is "if it's not in the table, it doesn't exist." Pick yours and make it explicit.
Write the meta-skill. Encode the four steps above as a SKILL.md. The non-obvious work is the bookkeeping rules: alphabetical insertion, permalink resolution from the source repo's git remote and commit, and consistent table regeneration.
Add the gates. Decide what "invalid" means for you (missing fields, name mismatch, wrong structure) and make the skill refuse and report instead of pushing bad entries.
Lower the barrier to zero. The whole bet is that a catalog survives only when contributing costs nothing. The meta-skill is what buys that.

That's it. The catalog stays current because keeping it current is now a single command, not a chore — and the team stops rebuilding skills it already has.

The full working version, gate cases and all, is here: github.com/edpittol/skills. Take it apart, adapt the categories and rules to your team, and let your agent keep the books.