Surviving N4 stockouts in GKE

Vincent — Sat, 13 Jun 2026 15:55:44 +0000

The problem I ran into

I was bringing up the startup infrastructure for a solution standardized on the N4 machine series. N4 is a great general-purpose default for us, so the design assumed N4 capacity would be there when we asked for it.

It wasn't. In several of our target regions I simply could not get N4 nodes when I needed them. New nodes wouldn't come up during scale-up events, and worse, I couldn't even create the initial infrastructure in some zones. The cluster autoscaler kept logging the now-familiar Compute Engine signal:

ZONE_RESOURCE_POOL_EXHAUSTED
The zone 'projects/PROJECT_ID/zones/ZONE' does not have enough resources
available to fulfill the request. Try a different zone, or try again later.

This is not a quota problem and not a config problem. It's a capacity problem: GCP didn't have the N4 hardware free in that zone at that moment. A few things made this clear to me as I dug in.

Google's own resource-availability troubleshooting page is explicit that these errors happen when you request resources in a zone that can't currently accommodate them, that they're unrelated to your quota, and that they only apply to new resource requests — existing VMs keep running fine. Their recommended mitigations are to retry in another zone, retry in another region, retry later, or change the hardware configuration you're asking for. (Troubleshooting resource availability errors)

The GKE ComputeClass troubleshooting docs say the same thing from the cluster's point of view: when GKE can't provision nodes for a priority rule — for example because of ZONE_RESOURCE_POOL_EXHAUSTED or QUOTA_EXCEEDED from Compute Engine — the autoscaler immediately moves on to the next rule, with no waiting period (the exception being TPUs and Flex Start). (Troubleshoot custom ComputeClass issues)

And I'm clearly not alone. The lack of capacity for specific machine families in specific zones is a long-running, well-documented reality on GCP. You can find it discussed in places like the GCE discussion group thread on resources not being available to fulfill the request, the ZONE_RESOURCE_POOL_EXHAUSTED thread, and the Google developer-forum report of not being able to provision anything on GKE across multiple European zones. Even Google's custom compute class launch blog quotes Delivery Hero saying the thing I wanted to be able to say: that with fallback rules they stopped having to worry about instance availability at all.

So I stopped trying to force N4 and instead made my infrastructure resilient to N4 being temporarily unavailable.

The fix: a dedicated cluster with a cluster-wide default ComputeClass

My solution doesn't bolt a ComputeClass onto an existing shared cluster, and it deliberately does not attach the class to individual workloads with per-Pod selectors. Instead, I provision a specific, dedicated GKE cluster for the whole solution, and I make a single custom ComputeClass the cluster-level default. Every Pod that lands on that cluster inherits the N4→C4 behavior automatically, whether or not it asks for it. The workload manifests stay completely clean — no nodeSelector, no per-namespace labels, nothing for app teams to remember.

A ComputeClass is a Kubernetes Custom Resource (CRD, apiVersion: cloud.google.com/v1, kind: ComputeClass) that lets me declare an ordered list of node configurations — "priorities" — that GKE walks through when it needs to scale up. (About custom ComputeClasses)

Two features make this work:

Fallback compute priorities. I list N4 first and C4 second. If GKE can't get N4, it doesn't leave my Pods Pending — it falls straight through to C4. C4 is the natural sibling to reach for: it's a current-generation Intel general-purpose series with a comparable shape to N4, so my workloads behave consistently on it.
Active migration back to the preferred option. With activeMigration.optimizeRulePriority: true, once N4 capacity reappears in my location (capacity frees up, or my quota goes up), GKE creates a new N4 node, then cordons and drains the C4 node it had been using as a fallback. The fleet self-heals back to N4 without me touching anything. This N4→C4 example is, in fact, the exact scenario Google documents for active migration. (About custom ComputeClasses)

The reason the dedicated cluster + cluster-level default combination matters — and isn't just a stylistic choice — is covered in the next section. The short version: it's the only configuration where my N4 fail-back actually fires for the whole solution.

The default ComputeClass I deployed

To make a custom ComputeClass the cluster-wide default, you give it the reserved name default. GKE then uses its priority rules as the autoscaling rules for any Pod that doesn't select a different class. (About GKE ComputeClasses)

apiVersion: cloud.google.com/v1
kind: ComputeClass
metadata:
  # The reserved name "default" makes this the cluster-wide default class.
  name: default
spec:
  priorities:
    # 1) Preferred: N4 on-demand
    - machineFamily: n4
      spot: false
    # 2) Fallback: C4 on-demand when N4 is exhausted
    - machineFamily: c4
      spot: false

  # When N4 capacity returns, replace C4 nodes with N4 nodes over time.
  # This is what makes the C4 fallback temporary by design.
  activeMigration:
    optimizeRulePriority: true

  # Let GKE create the node pools for me.
  nodePoolAutoCreation:
    enabled: true

  # Be explicit: if NEITHER N4 nor C4 can be provisioned, keep Pods
  # Pending rather than silently scaling up some default machine type.
  whenUnsatisfiable: DoNotScaleUp

I set whenUnsatisfiable explicitly. The default behavior changed at GKE 1.33: in older versions an unsatisfiable rule meant "scale up anyway" with the cluster's default machine type, which for me would have meant surprise E2 nodes instead of a clear Pending signal. Setting it explicitly means an upgrade won't silently change how the cluster behaves. (About custom ComputeClasses) I use DoNotScaleUp where I'd rather alert on Pending Pods; if you'd rather keep running on anything, use ScaleUpAnyway.

Because this is the cluster default, workloads need no changes at all — there are no nodeSelector blocks and no namespace labels to apply. One caution that informed the dedicated-cluster decision: GKE recommends you use either ComputeClasses or individual node selectors, but not both, since mixing them causes unexpected scheduling. A dedicated cluster where the default class governs everything avoids that conflict entirely. (About GKE ComputeClasses)

The default ComputeClass challenge in versions

This is the part that drove the whole design, and where I lost the most time, so I'm being emphatic about it.

Custom ComputeClasses themselves are old enough now (supported since GKE 1.30.3-gke.1451000) that it's easy to assume "defaults" came with them. They didn't. Setting a ComputeClass as a default is a much newer capability, and the two kinds of default have different version floors — and, critically, different behavior. (Apply ComputeClasses to Pods by default)

Cluster-level default (the class named default): requires GKE 1.33.1-gke.1744000 or later.
Namespace-level default (a cloud.google.com/default-compute-class label on the namespace), applied to only non-DaemonSet Pods: requires GKE 1.33.1-gke.1788000 or later.

The behavioral difference is the trap, and it's the reason a dedicated cluster with a cluster-level default is the right shape for my solution rather than a namespace label on a shared cluster:

With a cluster-level default, activeMigration.optimizeRulePriority does fire — if lower-priority C4 nodes exist and GKE can now create N4 nodes, it migrates. That's exactly my N4 fail-back.
With a namespace-level default, active migration is not triggered at all, because GKE only injects the ComputeClass selector into newly created Pods. Existing Pods are never re-created with the selector, so they never migrate back to N4. (About GKE ComputeClasses)

In other words, if I'd taken the easy path of labeling a namespace on an existing cluster, my Pods would have fallen back to C4 during a stockout and stayed on C4 forever, even after N4 came back. The fail-back — the whole point — would silently not work. The cluster-level default is the only configuration that gives me both the automatic inheritance and the migration back to N4, and that pushed me to stand up a cluster dedicated to this solution where naming a class default is safe and intentional.

Stacking the requirements, the binding version floor for my solution is the combination of:

cluster-level default ComputeClass → 1.33.1-gke.1744000, and
ComputeClass-driven node-pool auto-creation without cluster-level node auto-provisioning → 1.33.3-gke.1136000 on the Rapid release channel.

So I target ≥ 1.33.3-gke.1136000 on Rapid, which clears both. Below the default-CC floor entirely (anything before 1.33.1-gke.1744000), you simply cannot set a cluster-wide default — you'd be forced back into per-workload selectors or namespace labels, and the latter breaks the fail-back as described above.

Version requirements I treat as mandatory

Capability I depend on	Minimum GKE version
Custom ComputeClasses at all (the base feature)	1.30.3-gke.1451000
Cluster-level default ComputeClass (class named `default`)	1.33.1-gke.1744000
Namespace-level default for non-DaemonSet Pods (I deliberately avoid this)	1.33.1-gke.1788000
Node-pool auto-creation from the ComputeClass without cluster-level NAP	1.33.3-gke.1136000, Rapid channel
Exact custom machine types in `priorities` + Kubernetes labels set by the class	1.33.2-gke.1111000
`nodePoolConfig.imageType` in a ComputeClass	1.32.4-gke.1198000

(Sources: default ComputeClasses, node-attributes ComputeClass how-to, node pool auto-creation, GKE release notes.)

One more trap on top of the version floors: nodePoolAutoCreation.enabled: true is not enough on its own on older clusters. Before 1.33.3-gke.1136000, you also have to enable cluster-level node auto-provisioning (NAP), or GKE won't create the node pools for your class and Pods just sit Pending — which looks exactly like the stockout you were trying to solve. At/after that version on Rapid you can let the ComputeClass drive auto-creation directly. (About custom ComputeClasses)

On the tooling side, the versions I standardized on:

Terraform CLI ≥ 1.5 (I run 1.15.x).
hashicorp/google provider 7.x — I pin ~> 7.36. (The 7.0 line has been GA since 2025; 7.36 was current when I wrote this.)
hashicorp/kubernetes provider ≥ 2.30 to apply the ComputeClass CRD through Terraform.
If you use the community terraform-google-modules/terraform-google-kubernetes-engine module with its enable_default_compute_class plumbing, it requires the google provider ≥ 7.10.

Terraform

I provision the dedicated cluster and its default ComputeClass as code. Three pieces: providers (pinned), the dedicated cluster (with the version floor and autoscaling/NAP settings), and the default ComputeClass manifest.

Providers and version pinning

terraform {
  required_version = ">= 1.5"

  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 7.36"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.30"
    }
  }
}

provider "google" {
  project = var.project_id
  region  = var.region
}

The dedicated cluster — pin the version, turn on autoscaling / NAP

# A cluster built specifically and exclusively for this solution.
resource "google_container_cluster" "solution" {
  name     = "n4-solution-euw3"
  location = var.region

  # Rapid channel keeps us on a version new enough for both the
  # cluster-level default ComputeClass AND ComputeClass-driven
  # node-pool auto-creation without cluster-level NAP.
  release_channel {
    channel = "RAPID"
  }

  # Insist on a control-plane version that supports the cluster-level
  # default ComputeClass and auto-creation. This is the binding floor.
  min_master_version = "1.33.3-gke.1136000"

  remove_default_node_pool = true
  initial_node_count       = 1

  # Cluster-level node auto-provisioning. REQUIRED if your control plane is
  # older than 1.33.3-gke.1136000; optional (but harmless) at/after it.
  cluster_autoscaling {
    enabled = true

    resource_limits {
      resource_type = "cpu"
      minimum       = 4
      maximum       = 512
    }
    resource_limits {
      resource_type = "memory"
      minimum       = 16
      maximum       = 2048
    }

    auto_provisioning_defaults {
      management {
        auto_repair  = true
        auto_upgrade = true
      }
    }
  }
}

# A small autoscaling system pool so the Standard cluster satisfies the
# "at least one node pool has autoscaling enabled" requirement.
resource "google_container_node_pool" "system" {
  name     = "system"
  cluster  = google_container_cluster.solution.id
  location = var.region

  autoscaling {
    min_node_count = 1
    max_node_count = 3
  }

  node_config {
    machine_type = "e2-standard-4"
    oauth_scopes = ["https://www.googleapis.com/auth/cloud-platform"]
  }
}

The cluster-level default ComputeClass as a Terraform-managed manifest

data "google_client_config" "default" {}

provider "kubernetes" {
  host                   = "https://${google_container_cluster.solution.endpoint}"
  cluster_ca_certificate = base64decode(
    google_container_cluster.solution.master_auth[0].cluster_ca_certificate
  )
  token = data.google_client_config.default.access_token
}

resource "kubernetes_manifest" "default_compute_class" {
  manifest = {
    apiVersion = "cloud.google.com/v1"
    kind       = "ComputeClass"
    metadata = {
      # "default" => cluster-wide default. Requires GKE 1.33.1-gke.1744000+.
      name = "default"
    }
    spec = {
      priorities = [
        { machineFamily = "n4", spot = false },
        { machineFamily = "c4", spot = false },
      ]
      # Fires for a CLUSTER-level default; would NOT fire for a namespace default.
      activeMigration = {
        optimizeRulePriority = true
      }
      nodePoolAutoCreation = {
        enabled = true
      }
      whenUnsatisfiable = "DoNotScaleUp"
    }
  }

  depends_on = [google_container_cluster.solution]
}

Gotcha worth knowing: kubernetes_manifest validates against the cluster's API at plan time, so the ComputeClass CRD must already exist on the control plane when you plan. On a brand-new cluster that means a two-phase apply (cluster first, then the manifest), or driving the kubectl apply from a null_resource/local-exec against the rendered YAML. On GKE the CRD ships with the control plane, so once the cluster exists, the manifest applies cleanly.

Because this is the cluster default, there are no workload-side Terraform or manifest changes. Anything scheduled on this dedicated cluster inherits N4→C4→N4 automatically.

What I'd tell my past self

The N4 stockouts were never going to be "fixed" by retrying harder — capacity in a given zone isn't something I control. What I could control was whether a missing N4 node meant a stalled rollout or a transparent fall-through to C4. Building a dedicated cluster with a cluster-level default ComputeClass gave me a declarative, GitOps-friendly way to express "prefer N4, accept C4, return to N4" for the entire solution at once — no per-workload wiring, and no risk of mixing class selection with node selectors.

Three things genuinely mattered in practice: set whenUnsatisfiable explicitly so an upgrade never changes my failure mode behind my back; use a cluster-level default, not a namespace label, because only the cluster-level default actually triggers the active migration back to N4; and pin the GKE version to at least 1.33.3-gke.1136000 on Rapid, because the default-ComputeClass and auto-creation features I'm relying on simply don't exist on older clusters. Get those right and the stockouts stop being incidents and become a non-event.

IoT certificates for AWS with Terraform

Vincent — Sat, 11 Nov 2023 19:02:15 +0000

Today I am trying to simplify the deployment of small projects with little amount of things on AWS. Having worked with AWS in the console to create the certificates, I realized it's an easy way to have that working but using IaC would be a breeze compared to that.

Create certificate requests with Terraform

Using certificates with terraform is pretty simple:

resource "tls_private_key" "signed_1" {
  algorithm = "RSA"
  rsa_bits  = 2048
}

resource "tls_cert_request" "signed_1" {
  private_key_pem = tls_private_key.signed_1.private_key_pem

  subject {
    organization = "iot.rpi.com"
    common_name = "thing1.iot.rpi.com"
  }
}

Get a certificate from AWS IoT

Using the aws_iot_certificate we can get a certificate from the Certificate Signing Request (CSR) generated just before.

resource "aws_iot_certificate" "iot_certificate" {
  active = true
  csr = tls_cert_request.signed_1.cert_request_pem
}

Output the certificates to use them

Then for our IoT thing, we will need the private key used to create our CSR, the certificate itself and the CA used to sign the certificate.

We can output them and define them as sensitive, they will be available in the terraform.tfstate file.

# Get a CA used to sign the certificate
data "tls_certificate" "aws_root_ca" {
  url = "https://www.amazontrust.com/repository/AmazonRootCA1.pem"
}

output "thing_cert_pem" {
  value = aws_iot_certificate.iot_certificate.certificate_pem
  sensitive = true
}

output "thing_key_pem" {
  value = tls_private_key.signed_1.private_key_pem
  sensitive = true
}

output "aws_root_ca_pem" {
  sensitive = true
  value = data.tls_certificate.aws_root_ca.content
}

The result resides in the state file, that we can parse to get the values:

#!/bin/sh

mkdir -r certs

TERRAFORM_STATE_FILE=terraform.tfstate
CERT=$(jq ".outputs.thing_cert_pem.value" < ${TERRAFORM_STATE_FILE} | tr -d '"' )
KEY=$(jq ".outputs.thing_key_pem.value" < ${TERRAFORM_STATE_FILE}| tr -d '"')

# shellcheck disable=SC2039
echo "$CERT"> certs/thing1.crt.pem
# shellcheck disable=SC2039
echo "$KEY" > certs/thing1.key.pem
wget -O certs/AmazonRootCA1.pem "https://www.amazontrust.com/repository/AmazonRootCA1.pem"

The PEM files will then be available in the certs folder and you just have to use them !

If you want to help me, buy me a coffe:

Thank you for reading

DEV Community: Vincent