DEV Community: iapilgrim

GCP The Hard Way — Part 8: Designing a Broken System for Someone Else to Debug

iapilgrim — Tue, 01 Sep 2026 14:39:05 +0000

Introduction

Every prior post in this series involved diagnosing a failure someone
else (in this case, the author) had designed. This final post reverses
that relationship: you design the failure. Constructing a realistic,
diagnosable incident requires a deeper understanding of a system than
fixing one does, because you must accurately predict which symptoms a
given misconfiguration will produce. This exercise is a common
technique in SRE training programs, and this post walks through
applying it to a Google Cloud environment.

Solution overview

┌─────────────────────┐        ┌────────────────────┐
│   Design Phase        │        │    Debug Phase       │
│                        │───────▶│                       │
│  - Inject 3–5 faults   │ hand   │  - Investigate via    │
│  - Write incident      │  off   │    logs and metrics   │
│    ticket (symptoms     │        │  - Isolate root cause │
│    only, no root cause) │        │  - Remediate           │
└─────────────────────┘        └────────────────────┘

Criteria for a well-designed failure scenario

Clear symptom, unclear cause — the observer sees a concrete behavior, not an obvious misconfiguration
Reliably reproducible — the failure occurs consistently, not intermittently by chance
Plausible red herrings present — realistic incidents involve ruling out multiple hypotheses, not confirming the first guess
A definitive verification method exists — once the correct hypothesis is identified, there must be an unambiguous way to confirm it

Walkthrough

Step 1: Establish a known-good baseline

Reuse the three-tier architecture from Part 2 (Load Balancer → MIG →
Cloud SQL), and confirm it functions correctly before introducing any
faults — this establishes a clean baseline to distinguish injected
faults from pre-existing configuration issues.

Step 2: Select and inject 3–5 faults

A representative fault bank, spanning difficulty levels:

Fault	Injection	External symptom
Firewall allows only one of two required health-check ranges	`--source-ranges=130.211.0.0/22` (omitting `35.191.0.0/16`)	Intermittent backend health flapping
Firewall target tag mismatch	Firewall applies `web-server`; VM tagged `app-server`	Connection timeout, resembling an application crash
Cloud SQL connection limit set below expected concurrent load	`--database-flags=max_connections=20`	Failures only under concurrent load, invisible to single-request testing
Read replica under-provisioned relative to write volume	Small tier replica against high write throughput	Stale reads that resemble an application logic bug

Example injection:

gcloud sql instances patch app-db --database-flags=max_connections=20

Choose faults spanning multiple categories (network, IAM, database,
application) rather than concentrating on one area, to keep the
diagnostic exercise varied.

Step 3: Write the incident ticket

The ticket should be written from the perspective of an end user
reporting a problem, deliberately omitting any technical language that
would hint at the root cause:

# Ticket #001: Intermittent login failures during peak hours

## Description
Users report occasional "connection timeout" errors when logging in,
primarily between 9–10 AM. No issues reported in the afternoon.

## Already investigated
- VM restarted — issue persists
- CPU and memory utilization checked — both within normal range

## Access provided
Cloud Console (read-only), Cloud Logging, gcloud CLI

Including one or two already-ruled-out hypotheses ("already
investigated") makes the exercise more representative of real
incident response, where an initial theory is often incorrect.

Step 4: Grant scoped access for the investigator

gcloud projects add-iam-policy-binding <PROJECT_ID> \
  --member="user:<investigator-email>" \
  --role="roles/viewer"

gcloud projects add-iam-policy-binding <PROJECT_ID> \
  --member="user:<investigator-email>" \
  --role="roles/logging.viewer"

Read-only access prevents accidental remediation before the
investigator has confirmed a root cause.

Step 5: Structured investigation approach

Whether debugging as a peer or returning to your own system after a
delay, apply a consistent process:

Reproduce the symptom first — do not make changes before confirming the reported behavior
Narrow scope by elimination — isolate whether the fault sits in the network, application, or database layer by testing each layer independently (e.g., curl localhost on the VM before testing through the load balancer)
Read logs systematically — filter Cloud Logging by severity and correlate timestamps with reported symptom occurrence
Verify each hypothesis definitively before moving to the next, rather than changing multiple variables simultaneously

Comparing outcomes

| Injected fault | Identified? | Time to identify | False leads pursued |
|---|---|---|---|
| Fault 1 | | | |
| Fault 2 | | | |

Clean up resources

gcloud projects delete <PROJECT_ID>

Conclusion

This series has moved through the core operational competencies of
infrastructure engineering: provisioning without code changes,
diagnosing availability failures, auditing access control, choosing
data stores appropriate to workload characteristics, shipping releases
safely, validating disaster recovery assumptions, controlling cost, and
finally, designing failure scenarios well enough to teach them to
others. That last skill — being able to construct a realistic incident,
not just resolve one — is a strong signal of systems-level understanding,
and a capability directly transferable to building internal SRE
training programs.

This concludes the Google Cloud Fundamentals: The Hard Way series.

GCP The Hard Way — Part 7: Finding and Eliminating Cloud Waste

iapilgrim — Tue, 01 Sep 2026 14:38:54 +0000

Introduction

Cloud cost inefficiency rarely comes from a single dramatic error —
it accumulates from small, individually reasonable-looking decisions:
an oversized VM chosen "to be safe," a static IP reserved for a test
that was never cleaned up, a storage bucket without a lifecycle
policy. This post deliberately introduces four common sources of cloud
waste, lets them run long enough to appear in billing data, and walks
through identifying and remediating each using Billing Reports and
Recommender.

Solution overview

┌─────────────────────────────────────────────┐
│                 Sources of waste               │
├─────────────────────────────────────────────┤
│ 1. Oversized Compute Engine instance           │
│ 2. Cloud Storage without lifecycle policy      │
│ 3. Reserved static IP with no attached resource │
│ 4. Autoscaling ceiling set without usage data   │
└─────────────────────────────────────────────┘
                    │
                    ▼
        ┌───────────────────────┐
        │   Billing Reports +     │
        │   Recommender API       │
        └───────────────────────┘

Prerequisites

A Budget Alert configured on the billing account (recommended threshold: $25) before beginning
At least 3–4 days of elapsed time between provisioning and analysis, since billing trend data needs a meaningful window to be useful

Walkthrough

Step 1: Provision the waste sources

Oversized instance for a lightweight workload:

gcloud compute instances create oversized-vm \
  --zone=asia-southeast1-a --machine-type=n2-standard-8 \
  --image-family=debian-12 --image-project=debian-cloud

Storage without a lifecycle policy:

gcloud storage buckets create gs://<PROJECT_ID>-waste-bucket \
  --location=asia-southeast1

Idle static IP:

gcloud compute addresses create unused-static-ip --region=asia-southeast1

Autoscaling ceiling set without usage justification:

gcloud compute instance-groups managed set-autoscaling waste-mig \
  --zone=asia-southeast1-a --max-num-replicas=20 --min-num-replicas=2 \
  --target-cpu-utilization=0.8

Allow this configuration to run for 3–4 days before proceeding to
analysis.

Step 2: Review Billing Reports

In Billing → Reports, filter by the time range covering the
provisioning period and group by Service, then by SKU for finer
granularity. Rank the four sources by actual cost — results frequently
diverge from intuition; for example, an idle static IP typically costs
far less than an oversized VM, but this should be confirmed against
your own data rather than assumed.

Step 3: Query Recommender for automated findings

gcloud recommender recommendations list \
  --project=<PROJECT_ID> --location=asia-southeast1-a \
  --recommender=google.compute.instance.MachineTypeRecommender

gcloud recommender recommendations list \
  --project=<PROJECT_ID> --location=global \
  --recommender=google.compute.address.IdleResourceRecommender

Coverage gap to note: Recommender does not currently surface
missing storage lifecycle policies as a recommendation — this class of
waste requires manual review of bucket configuration and billing
trends, reinforcing that automated tooling should supplement, not
replace, periodic manual cost review.

Step 4: Remediate

gcloud compute instances stop oversized-vm --zone=asia-southeast1-a
gcloud compute instances set-machine-type oversized-vm \
  --zone=asia-southeast1-a --machine-type=e2-small
gcloud compute instances start oversized-vm --zone=asia-southeast1-a

cat > lifecycle.json <<'EOF'
{
  "rule": [
    {"action": {"type": "SetStorageClass", "storageClass": "COLDLINE"},
     "condition": {"age": 30}},
    {"action": {"type": "Delete"}, "condition": {"age": 365}}
  ]
}
EOF
gcloud storage buckets update gs://<PROJECT_ID>-waste-bucket \
  --lifecycle-file=lifecycle.json

gcloud compute addresses delete unused-static-ip --region=asia-southeast1 --quiet

gcloud compute instance-groups managed set-autoscaling waste-mig \
  --zone=asia-southeast1-a --max-num-replicas=4 --min-num-replicas=1 \
  --target-cpu-utilization=0.7

The revised autoscaling ceiling of 4 should be justified by observed
peak usage during the test window, not set arbitrarily — an
unjustifiably high ceiling creates cost exposure risk even without
generating cost directly, since a traffic spike or application bug
could scale the fleet far beyond actual need.

Measuring savings

Allow 1–2 additional days after remediation, then compare Billing
Reports for the same resource categories before and after the change:

Category	Cost before (monthly, extrapolated)	Cost after	% reduction
Compute (VM resize)
Storage (lifecycle policy)
Idle static IP + LB
Autoscaling ceiling

Clean up resources

gcloud compute instances delete oversized-vm --zone=asia-southeast1-a --quiet
gcloud storage rm -r gs://<PROJECT_ID>-waste-bucket

Conclusion

Automated recommenders are a useful first pass, but this exercise
shows they have blind spots — notably storage lifecycle policies — that
still require deliberate, periodic manual review. Building this review
into a regular cadence, rather than a one-time cleanup, is the
practical difference between temporary savings and sustained cost
efficiency.

In Part 8, we close the series by reversing roles: designing a
broken system for someone else — or a future version of yourself — to
diagnose.

GCP The Hard Way — Part 6: Validating Disaster Recovery Assumptions With a Live Drill

iapilgrim — Tue, 01 Sep 2026 14:38:33 +0000

Introduction

Disaster recovery plans are frequently validated on paper and rarely
tested end-to-end, which means the first real test often happens
during an actual incident. This post walks through provisioning
cross-region Cloud SQL replication and dual-region storage, then
deliberately removing the primary region's infrastructure to measure
actual Recovery Time Objective (RTO) and Recovery Point Objective
(RPO) against a documented estimate made in advance.

Solution overview

Region: asia-southeast1 (Primary)      Region: asia-east1 (DR)
┌─────────────────────┐               ┌─────────────────────┐
│  Compute Engine VM    │              │                       │
│  Cloud SQL (Primary)  │──replication▶│  Cloud SQL (Replica)  │
└─────────────────────┘               └─────────────────────┘
         │                                        
         ▼                                        
┌─────────────────────────────────┐
│  Cloud Storage (dual-region: ASIA1) │
└─────────────────────────────────┘

Prerequisites

Two target regions selected for primary and DR roles
Cloud DNS zone already configured for the application domain

Walkthrough

Step 1: Document your recovery estimate before building anything

Before provisioning any infrastructure, record a written estimate:

Estimated RTO: ____ minutes
Estimated RPO: ____ seconds/minutes
Anticipated recovery steps: 1. 2. 3.

Do not revise this document until after the drill — the comparison at
the end is the primary value of this exercise.

Step 2: Provision cross-region replication

gcloud sql instances create app-db-primary \
  --database-version=MYSQL_8_0 --tier=db-g1-small \
  --region=asia-southeast1 --backup --enable-bin-log

gcloud sql instances create app-db-replica-dr \
  --master-instance-name=app-db-primary --region=asia-east1 \
  --tier=db-g1-small

Step 3: Provision dual-region storage

gcloud storage buckets create gs://<PROJECT_ID>-dr-bucket --location=ASIA1

Step 4: Deploy the application and point DNS at the primary region

gcloud compute instances create app-vm-primary \
  --zone=asia-southeast1-a --machine-type=e2-small \
  --image-family=debian-12 --image-project=debian-cloud

gcloud dns record-sets create app.yourdomain-test.com. \
  --zone=app-dr-zone --type=A --ttl=300 \
  --rrdatas=<PRIMARY_VM_IP>

A 300-second TTL is used deliberately — higher TTLs common in default
registrar configurations directly extend recovery time by keeping
stale DNS resolutions cached at clients and resolvers.

Step 5: Simulate the disaster

Record the exact start time before proceeding — this is your RTO
measurement baseline:

date -u +"%Y-%m-%dT%H:%M:%SZ" > disaster-start-time.txt

gcloud compute instances delete app-vm-primary --zone=asia-southeast1-a --quiet
gcloud sql instances delete app-db-primary --quiet

Deleting the primary instance also orphans the read replica — a
realistic condition that must be handled as part of recovery.

Step 6: Execute recovery without referencing the original plan

gcloud sql instances promote-replica app-db-replica-dr

Deploy the application in the DR region:

gcloud compute instances create app-vm-dr \
  --zone=asia-east1-a --machine-type=e2-small \
  --image-family=debian-12 --image-project=debian-cloud

Expected friction point: connection strings hardcoded to the
original instance name will need manual updates once
app-db-replica-dr becomes the new writable primary. This is a direct
argument for externalizing connection configuration (via Secret
Manager or environment variables) rather than embedding it in
application code or images.

Update DNS to point at the DR region, and record the completion time:

gcloud dns record-sets update app.yourdomain-test.com. \
  --zone=app-dr-zone --type=A --ttl=300 --rrdatas=<DR_VM_IP>

date -u +"%Y-%m-%dT%H:%M:%SZ" > recovery-end-time.txt

Verify DNS resolution independently (e.g., with dig) rather than
assuming propagation is instantaneous once the record is updated.

Step 7: Measure RPO

Cloud SQL read replicas replicate asynchronously; any writes to the
primary that had not yet replicated at the moment of deletion are
lost. To quantify this in a follow-up test run, write a record
immediately before deleting the primary and confirm whether it is
present in the promoted replica — this delta is your empirical RPO.

Comparing estimate to reality

Metric	Documented estimate	Measured result	Delta
RTO
RPO

Most teams running this drill for the first time find their initial
estimate omitted at least one dependency — commonly DNS propagation
delay or hardcoded connection configuration.

Clean up resources

gcloud compute instances delete app-vm-dr --zone=asia-east1-a --quiet
gcloud sql instances delete app-db-replica-dr --quiet
gcloud storage rm -r gs://<PROJECT_ID>-dr-bucket
gcloud dns managed-zones delete app-dr-zone --quiet

Conclusion

Replication and backups reduce data loss risk but do not eliminate
it — RPO is bounded by replication lag, not by the mere existence of a
replica. The most valuable output of this exercise isn't the
architecture itself, but the revised recovery runbook you write
immediately afterward, informed by what your original plan missed.

In Part 7, we shift from availability to cost, examining how
common infrastructure choices translate into avoidable cloud spend.

GCP The Hard Way — Part 5: Canary Deployments Against a Delayed-Failure Bug

iapilgrim — Tue, 01 Sep 2026 14:38:16 +0000

Introduction

A deployment that reports success and a release that doesn't harm
users are not the same thing. This distinction matters most when a bug
doesn't manifest immediately — memory leaks, connection pool
exhaustion, and similar failure modes often pass initial health checks
before degrading. This post deploys a workload to Google Kubernetes
Engine with correctly configured liveness and readiness probes,
introduces a version with a 30-second delayed failure, and walks
through rollback and canary rollout as complementary mitigation
strategies.

Solution overview

┌─────────────┐      ┌──────────────────────┐
│   Ingress /   │────▶│   Service: myapp       │
│ Load Balancer │      └──────────┬───────────┘
└─────────────┘                  │
                    ┌─────────────┴─────────────┐
                    ▼                             ▼
          ┌───────────────────┐        ┌───────────────────┐
          │ Deployment: stable  │        │ Deployment: canary  │
          │  (9 replicas, v1)   │        │  (1 replica, v2)     │
          └───────────────────┘        └───────────────────┘

Prerequisites

A GKE Standard cluster with kubectl configured
Artifact Registry for container image storage

Walkthrough

Step 1: Provision the cluster and registry

gcloud services enable container.googleapis.com artifactregistry.googleapis.com

gcloud artifacts repositories create app-repo \
  --repository-format=docker --location=asia-southeast1

gcloud container clusters create app-cluster \
  --zone=asia-southeast1-a --num-nodes=2 --machine-type=e2-medium

gcloud container clusters get-credentials app-cluster --zone=asia-southeast1-a

Step 2: Build an application with a controllable failure mode

from flask import Flask
import time, os

app = Flask(__name__)
START_TIME = time.time()
BUG_MODE = os.environ.get("BUG_MODE", "false") == "true"

@app.route("/health")
def health():
    if BUG_MODE and (time.time() - START_TIME) > 30:
        return "UNHEALTHY", 500
    return "OK", 200

This mirrors real-world delayed failures more closely than a bug that
fails immediately at startup, which would be caught before any traffic
is served.

Step 3: Deploy with production-grade probes

readinessProbe:
  httpGet: {path: /health, port: 8080}
  periodSeconds: 5
  failureThreshold: 2
livenessProbe:
  httpGet: {path: /health, port: 8080}
  periodSeconds: 10
  failureThreshold: 3

kubectl apply -f deployment.yaml
kubectl get pods -w

Step 4: Roll out the faulty version and observe

kubectl apply -f deployment-v2-bugged.yaml
kubectl get pods -w

For the first ~30 seconds, pods report Running and Ready — this is
expected, and is precisely the window in which a naive smoke test
would report success.

Diagnosing probe behavior: After 30 seconds, inspect events:

kubectl describe pod <pod-name>

readinessProbe failures remove a pod from the Service's Endpoints
list without restarting the container; livenessProbe failures cause
the kubelet to kill and restart the container, visible as an
increasing RESTARTS count in kubectl get pods. Because this
example uses the same path for both probes, expect a repeating
restart cycle consistent with CrashLoopBackOff.

Step 5: Roll back

kubectl rollout undo deployment/myapp
kubectl rollout status deployment/myapp

Measure the elapsed time from issuing the rollback to full traffic
stability — this is your empirical Mean Time to Recovery (MTTR) for
this failure class.

Step 6: Implement canary rollout for the next release

Rather than replacing the entire fleet at once, split traffic across
two Deployments sharing a Service selector:

kubectl apply -f deployment-stable.yaml   # 9 replicas, v1
kubectl apply -f deployment-canary.yaml   # 1 replica, v2 (bug fixed)

for i in $(seq 1 50); do
  curl -s http://<EXTERNAL_IP> | grep -o "v[12]-[a-z]*"
done | sort | uniq -c

Confirm the observed traffic split approximates the 9:1 replica ratio
before progressively shifting more replicas to the canary track.

Clean up resources

kubectl delete deployment myapp-stable myapp-canary --ignore-not-found
kubectl delete service myapp-service --ignore-not-found
gcloud container clusters delete app-cluster --zone=asia-southeast1-a --quiet

Conclusion

Readiness and liveness probes solve different problems and produce
different Kubernetes behaviors — conflating the two is a common source
of confusion when debugging rollout issues. Combined with a canary
rollout strategy, this pattern limits the blast radius of exactly the
kind of delayed-failure bug demonstrated here.

In Part 6, we validate disaster recovery assumptions by
deliberately deleting a region's infrastructure and measuring actual
recovery time against a documented plan.

GCP The Hard Way — Part 4: Choosing the Wrong NoSQL Database, and Migrating When It Shows

iapilgrim — Tue, 01 Sep 2026 14:37:52 +0000

Introduction

Database selection guidance often reads like a features checklist:
Firestore for flexible documents and real-time sync, Bigtable for
high-throughput analytical workloads. What that guidance doesn't
always convey is how the wrong choice manifests in practice — as
cost growth, query limitations, or both. This post builds a
high-frequency event-logging workload on Firestore, measures where it
breaks down, and migrates the same workload to Bigtable, including the
row-key design work that Bigtable migrations often get wrong on the
first attempt.

Solution overview

Phase A                          Phase B
┌──────────┐                    ┌──────────┐
│  Event    │   ~200 writes/s    │  Event    │   ~200 writes/s
│  Producer │──────────────────▶│  Producer │──────────────────▶
└──────────┘                    └──────────┘
     │                                │
     ▼                                ▼
┌──────────┐                    ┌──────────┐
│ Firestore │                    │ Bigtable  │
│ (documents)│                   │(wide-column)│
└──────────┘                    └──────────┘

Prerequisites

A Compute Engine VM to run the load-generation script (isolates network variability from your local machine)
Familiarity with basic Python

Walkthrough

Phase A: Firestore under sustained write load

gcloud services enable firestore.googleapis.com
gcloud firestore databases create --location=asia-southeast1 --type=firestore-native

from google.cloud import firestore
import time, uuid, random, threading

db = firestore.Client()
EVENT_TYPES = ["click", "view", "purchase", "error", "login"]

def write_event():
    db.collection("events").document().set({
        "event_id": str(uuid.uuid4()),
        "user_id": f"user_{random.randint(1, 1000)}",
        "event_type": random.choice(EVENT_TYPES),
        "timestamp": firestore.SERVER_TIMESTAMP,
    })

def worker():
    while True:
        write_event()
        time.sleep(0.05)

for _ in range(10):
    threading.Thread(target=worker).start()

Run this for 15–20 minutes to generate roughly 200,000 documents, then
attempt an aggregate query:

docs = db.collection("events").where("event_type", "==", "click").stream()
count = sum(1 for _ in docs)

Observed limitations:

Multi-field queries frequently require a manually created composite index, surfaced through an error message with an index-creation link
Aggregate counts over large result sets incur a read operation per document, which is directly reflected in Billing Reports under Cloud Firestore

Review actual cost in Billing → Reports, filtered to Cloud
Firestore, before proceeding to the migration.

Phase B: Migrating to Bigtable

Row-key design is the primary design decision in this migration.
A row key derived directly from a monotonically increasing value (such
as a raw timestamp) causes writes to concentrate on a single tablet —
a pattern known as hotspotting — because Bigtable stores rows in
lexicographic key order.

A row key that leads with a high-cardinality field, such as
user_id, distributes writes across the keyspace:

row_key = f"{user_id}#{reversed_timestamp}".encode()

Where reversed_timestamp = 9999999999999 - unix_timestamp_ms,
placing the most recent event first within a given user's key range.

gcloud services enable bigtable.googleapis.com bigtableadmin.googleapis.com

gcloud bigtable instances create event-log-bt \
  --cluster=event-log-cluster \
  --cluster-zone=asia-southeast1-b \
  --cluster-num-nodes=1 \
  --instance-type=PRODUCTION

cbt -instance=event-log-bt createtable events
cbt -instance=event-log-bt createfamily events cf1

from google.cloud import bigtable
import time, random

client = bigtable.Client(project="<PROJECT_ID>", admin=True)
table = client.instance("event-log-bt").table("events")

def write_event():
    user_id = f"user_{random.randint(1, 1000)}"
    ts_ms = int(time.time() * 1000)
    row_key = f"{user_id}#{9999999999999 - ts_ms}".encode()
    r = table.direct_row(row_key)
    r.set_cell("cf1", "event_type", random.choice(["click", "view", "purchase"]))
    r.commit()

Validating key distribution: Run the same load twice — once with a
naive timestamp-only key, once with the user_id-prefixed key — and
compare per-node CPU utilization in Bigtable → Monitoring. A
poorly distributed key shows a pronounced skew toward a single node;
a well-distributed key shows roughly even utilization across the
cluster.

Cost comparison

Bigtable bills for provisioned node-hours regardless of traffic
volume, while Firestore bills per operation. For short-duration,
moderate-volume workloads like this walkthrough, Bigtable may appear
more expensive in absolute terms — Bigtable's cost efficiency
advantage typically emerges only at sustained high throughput (commonly
millions of operations per day). Record both figures from Billing
Reports for a direct comparison.

Clean up resources

gcloud bigtable instances delete event-log-bt --quiet

Firestore data must be deleted at the collection level, or by removing
the entire test project.

Conclusion

Neither database is universally superior — the right choice depends on
access patterns and scale, not feature checklists alone. This
walkthrough demonstrates two concrete decision points worth
internalizing: composite index requirements as an early signal that
Firestore's query model may not fit the workload, and row-key design
as the single highest-leverage decision in any Bigtable schema.

In Part 5, we deploy to GKE and introduce a bug that only
manifests after the container has already passed its initial health
check.

GCP The Hard Way — Part 3: Auditing IAM Misconfigurations Before They Become Incidents

iapilgrim — Tue, 01 Sep 2026 14:37:43 +0000

Introduction

IAM misconfigurations rarely announce themselves. An over-permissioned
service account or a publicly readable storage bucket typically
continues functioning without error, which is precisely what makes
these issues dangerous — they persist silently until discovered by an
audit, or by an attacker. This post walks through introducing three
realistic IAM misconfigurations into a test project, then performing a
structured security review to identify and remediate them using native
GCP tooling.

Solution overview

We separate this exercise into two distinct phases to preserve the
integrity of the audit: a configuration phase where common
misconfigurations are introduced, and a review phase performed without
reference to the configuration phase, using only Cloud IAM and Cloud
Asset Inventory tooling.

┌───────────────────┐        ┌────────────────────┐
│  Configuration      │        │      Review          │
│  Phase               │──────▶│      Phase            │
│                      │ 15-30 │                      │
│  - Broad IAM role    │  min  │  - IAM policy audit   │
│  - Public bucket     │ gap   │  - Asset Inventory    │
│  - Exported SA key   │       │  - Key rotation       │
└───────────────────┘        └────────────────────┘

Prerequisites

A dedicated test project (do not perform this exercise against production infrastructure)
The Cloud Asset API enabled: gcloud services enable cloudasset.googleapis.com

Walkthrough

Phase 1: Introduce common misconfigurations

Overly permissive service account role:

gcloud iam service-accounts create sloppy-sa \
  --display-name="Sloppy Dev SA"

gcloud projects add-iam-policy-binding <PROJECT_ID> \
  --member="serviceAccount:sloppy-sa@<PROJECT_ID>.iam.gserviceaccount.com" \
  --role="roles/editor"

Publicly readable storage bucket:

gsutil mb -l asia-southeast1 gs://<PROJECT_ID>-sloppy-bucket
gsutil iam ch allUsers:objectViewer gs://<PROJECT_ID>-sloppy-bucket

Exported service account key:

gcloud iam service-accounts keys create sloppy-key.json \
  --iam-account=sloppy-sa@<PROJECT_ID>.iam.gserviceaccount.com

Deploy a minimal Cloud Run service using this service account to
simulate a realistic dependency:

gcloud run deploy sloppy-service \
  --image=gcr.io/cloudrun/hello \
  --service-account=sloppy-sa@<PROJECT_ID>.iam.gserviceaccount.com \
  --region=asia-southeast1 --allow-unauthenticated

Phase 2: Perform the security review

Identify over-permissioned identities:

gcloud projects get-iam-policy <PROJECT_ID> --format=json

gcloud asset search-all-iam-policies \
  --scope=projects/<PROJECT_ID> \
  --query="policy:roles/editor"

Any service account bound to roles/editor or roles/owner warrants
scrutiny — these broad roles are rarely required for a well-scoped
workload identity.

Identify publicly accessible storage:

gsutil iam get gs://<PROJECT_ID>-sloppy-bucket

gcloud asset search-all-iam-policies \
  --scope=projects/<PROJECT_ID> \
  --query="policy:allUsers"

Identify exported credentials:

gcloud iam service-accounts keys list \
  --iam-account=sloppy-sa@<PROJECT_ID>.iam.gserviceaccount.com

Phase 3: Remediate

gcloud projects remove-iam-policy-binding <PROJECT_ID> \
  --member="serviceAccount:sloppy-sa@<PROJECT_ID>.iam.gserviceaccount.com" \
  --role="roles/editor"

gcloud projects add-iam-policy-binding <PROJECT_ID> \
  --member="serviceAccount:sloppy-sa@<PROJECT_ID>.iam.gserviceaccount.com" \
  --role="roles/storage.objectViewer"

gsutil iam ch -d allUsers:objectViewer gs://<PROJECT_ID>-sloppy-bucket

gcloud iam service-accounts keys delete <KEY_ID> \
  --iam-account=sloppy-sa@<PROJECT_ID>.iam.gserviceaccount.com

Redeploy or invoke the Cloud Run service to confirm it continues to
function correctly under the narrower storage.objectViewer role.

Design consideration: most GCP-managed compute services (Cloud
Run, GKE, Compute Engine) can use a service account's identity
directly without an exported JSON key. Keys should generally only be
created for workloads running outside Google Cloud — unnecessary key
exports are a frequent root cause of credential leakage incidents.

Clean up resources

gcloud run services delete sloppy-service --region=asia-southeast1 --quiet
gsutil rm -r gs://<PROJECT_ID>-sloppy-bucket
gcloud iam service-accounts delete \
  sloppy-sa@<PROJECT_ID>.iam.gserviceaccount.com --quiet

Conclusion

This exercise illustrates why IAM review needs to be a routine,
tool-assisted process rather than a one-time configuration step.
gcloud asset search-all-iam-policies in particular scales to
project- and organization-wide queries that manual policy inspection
cannot practically match.

In Part 4, we turn to data modeling — deliberately selecting the
wrong database for a workload, and migrating once the cost and
performance implications become apparent.

GCP The Hard Way — Part 2: Diagnosing Failures in a Highly Available Three-Tier Architecture

iapilgrim — Tue, 01 Sep 2026 14:36:13 +0000

Introduction

High availability features like managed instance groups (MIGs) and
load balancing are often treated as "set and forget" — configure them
once, and the platform handles the rest. In practice, understanding
when and how quickly these mechanisms respond to failure is
essential for setting realistic availability expectations. This post
walks through provisioning a three-tier architecture on Google Cloud
and deliberately introducing five distinct failure conditions to
observe and measure the platform's actual recovery behavior.

Solution overview

The architecture consists of a global HTTP load balancer distributing
traffic across a managed instance group, backed by a health check that
determines instance eligibility for traffic.

        ┌───────────────────────┐
Client ─▶  Global HTTP(S) LB     │
        └───────────┬───────────┘
                     │
        ┌────────────▼────────────┐
        │   Backend Service         │
        │   (Health Check: /health) │
        └────────────┬────────────┘
                     │
     ┌───────────────┼───────────────┐
     ▼                                ▼
┌─────────┐                    ┌─────────┐
│  VM 1   │                    │  VM 2   │
└─────────┘                    └─────────┘
   Managed Instance Group (min:2, max:4)

Prerequisites

Completion of Part 1, or equivalent familiarity with Compute Engine and firewall rules
A load-testing tool such as hey or Apache Bench installed locally or on a separate VM

Walkthrough

Step 1: Deploy the instance template and managed instance group

gcloud compute instance-templates create app-template \
  --machine-type=e2-small \
  --image-family=debian-12 \
  --image-project=debian-cloud \
  --tags=app-server \
  --metadata-from-file=startup-script=startup.sh

gcloud compute instance-groups managed create app-mig \
  --template=app-template --size=2 --zone=asia-southeast1-a

gcloud compute instance-groups managed set-autoscaling app-mig \
  --zone=asia-southeast1-a \
  --max-num-replicas=4 --min-num-replicas=2 \
  --target-cpu-utilization=0.6 --cool-down-period=60

Step 2: Configure health checks and the load balancer

gcloud compute health-checks create http app-health-check \
  --port=8080 --request-path=/health

gcloud compute firewall-rules create allow-health-check \
  --allow=tcp:8080 \
  --source-ranges=130.211.0.0/22,35.191.0.0/16 \
  --target-tags=app-server

gcloud compute backend-services create app-backend \
  --protocol=HTTP --port-name=http \
  --health-checks=app-health-check --global

gcloud compute backend-services add-backend app-backend \
  --instance-group=app-mig --instance-group-zone=asia-southeast1-a --global

gcloud compute url-maps create app-lb --default-service=app-backend
gcloud compute target-http-proxies create app-http-proxy --url-map=app-lb
gcloud compute forwarding-rules create app-http-rule \
  --global --target-http-proxy=app-http-proxy --ports=80

Allow 5–10 minutes for the backend to transition to a HEALTHY state
before proceeding.

Failure scenario 1: Manual instance termination

gcloud compute instance-groups managed list-instances app-mig \
  --zone=asia-southeast1-a
gcloud compute instances delete <INSTANCE_NAME> \
  --zone=asia-southeast1-a --quiet

Observation: Measure the time between deletion and the appearance
of a replacement instance. If no replacement appears within several
minutes, verify whether autohealing is configured — MIG autoscaling
and autohealing are independent mechanisms, and autoscaling alone does
not guarantee replacement of a manually terminated instance outside of
its target size logic.

Failure scenario 2: Health check path blocked by firewall

gcloud compute firewall-rules delete allow-health-check --quiet
gcloud compute backend-services get-health app-backend --global

Diagnostic approach: From inside an affected VM, run
curl localhost:8080/health directly. A successful local response
combined with a failing get-health status isolates the fault to the
network path rather than the application.

Failure scenario 3: Load-induced autoscaling

hey -z 5m -c 200 http://<LB_IP>/

Observation: Track the interval between CPU utilization crossing
the 60% threshold and new instances becoming HEALTHY and receiving
traffic. Expect a multi-minute delay attributable to three sequential
stages: autoscaler metric evaluation, VM boot and startup script
execution, and health check confirmation before the load balancer
begins routing traffic.

Failure scenario 4: Misconfigured health check path

gcloud compute health-checks update http app-health-check \
  --request-path=/healthz

Diagnostic approach: This produces the same UNHEALTHY status as
Scenario 2 but stems from an application-layer mismatch rather than a
network block — reinforcing that identical symptoms can have distinct
root causes, and log inspection (not assumption) should drive
remediation.

Failure scenario 5: Full instance group depletion

gcloud compute instance-groups managed resize app-mig \
  --zone=asia-southeast1-a --size=0

Restore the health check path and resize back to 2, and measure total
recovery time from zero running instances to a healthy backend.

Clean up resources

gcloud compute forwarding-rules delete app-http-rule --global --quiet
gcloud compute target-http-proxies delete app-http-proxy --quiet
gcloud compute url-maps delete app-lb --quiet
gcloud compute backend-services delete app-backend --global --quiet
gcloud compute instance-groups managed delete app-mig \
  --zone=asia-southeast1-a --quiet
gcloud compute instance-templates delete app-template --quiet

Conclusion

Each of these five scenarios produces a distinct combination of
symptoms and recovery timelines, despite relying on the same
underlying platform features. The key takeaway is that autoscaling,
autohealing, and load balancer health checking are separate,
composable mechanisms — understanding their individual behavior is
what separates confident incident response from guesswork.

In Part 3, we shift focus to IAM, auditing a project for
privilege escalation risks introduced through common configuration
shortcuts.

GCP The Hard Way — Part 1: Migrating a Legacy Application Without Touching Its Code

iapilgrim — Tue, 01 Sep 2026 14:36:06 +0000

Introduction

A common first step in cloud adoption is the "lift-and-shift" migration:
moving an existing application to the cloud with minimal changes to
reduce migration risk and preserve the original release cadence. In
this post, we walk through migrating a application with a hardcoded
localhost database dependency to Google Cloud, without modifying a
single line of application code — relying instead on infrastructure
and configuration to bridge the gap.

By the end of this walkthrough, you will have deployed a Compute Engine
instance connected to a Cloud SQL database over a private network path,
with no public database endpoint exposed.

Solution overview

The target architecture places the application on a Compute Engine VM
and the database on Cloud SQL, connected through the Cloud SQL Auth
Proxy running as a sidecar process on the VM. This preserves the
application's original localhost connection string while routing
traffic securely to the managed database.

┌─────────────────────────────┐
│      Compute Engine VM       │
│  ┌────────┐   ┌───────────┐  │        ┌───────────────┐
│  │  App   │──▶│ Cloud SQL │──┼───────▶│   Cloud SQL    │
│  │        │   │Auth Proxy │  │Private │ (Private IP)   │
│  └────────┘   └───────────┘  │  VPC   └───────────────┘
└─────────────────────────────┘

Prerequisites

A GCP project with the Compute Engine and Cloud SQL Admin APIs enabled
A sample application (Node.js, Flask, or WordPress) that reads its database host from a config file or environment variable set to localhost
The application verified working in a local environment before migration begins

Walkthrough

Step 1: Provision the database and compute resources

gcloud services enable compute.googleapis.com sqladmin.googleapis.com

gcloud sql instances create app-db \
  --database-version=MYSQL_8_0 \
  --tier=db-f1-micro \
  --region=asia-southeast1 \
  --no-assign-ip \
  --network=default

gcloud sql databases create appdb --instance=app-db

gcloud compute instances create app-vm \
  --zone=asia-southeast1-a \
  --machine-type=e2-small \
  --image-family=debian-12 \
  --image-project=debian-cloud

We disable the public IP on the Cloud SQL instance from the start
(--no-assign-ip), reflecting a common enterprise security requirement
that databases never be reachable over the public internet.

Step 2: Establish private connectivity

Cloud SQL private IP requires a VPC peering connection to Google's
managed services range:

gcloud compute addresses create google-managed-services-default \
  --global --purpose=VPC_PEERING --prefix-length=16 --network=default

gcloud services vpc-peerings connect \
  --service=servicenetworking.googleapis.com \
  --ranges=google-managed-services-default \
  --network=default

Step 3: Deploy the Cloud SQL Auth Proxy on the VM

gcloud compute ssh app-vm --zone=asia-southeast1-a

On the VM:

curl -o cloud-sql-proxy \
  https://storage.googleapis.com/cloud-sql-connectors/cloud-sql-proxy/v2.11.0/cloud-sql-proxy.linux.amd64
chmod +x cloud-sql-proxy
./cloud-sql-proxy --private-ip <PROJECT_ID>:asia-southeast1:app-db &

The proxy binds to 127.0.0.1:3306 by default, which is exactly the
endpoint the application already expects.

Step 4: Open the required firewall path

Applications frequently listen on ports other than 80/443. Configure
the firewall to allow only the specific port the application requires:

gcloud compute instances add-tags app-vm --tags=app-server --zone=asia-southeast1-a

gcloud compute firewall-rules create allow-app-port \
  --allow=tcp:3000 \
  --source-ranges=0.0.0.0/0 \
  --target-tags=app-server

Common pitfall: a connection timeout at this stage can originate
from either a missing firewall rule or an application that binds to
127.0.0.1 instead of 0.0.0.0. Verify with curl localhost:3000
from inside the VM before troubleshooting the network path.

Step 5: Attach a custom domain

gcloud dns managed-zones create app-zone \
  --dns-name="yourdomain.com." \
  --description="Lift and shift walkthrough"

Create an A record pointing to the VM's external IP through the
Cloud Console or gcloud dns record-sets create.

Validating the deployment

Confirm the following before considering the migration complete:

gcloud sql instances describe app-db shows no public IP address
The application is reachable via the custom domain, not the raw IP
No port other than 80/443 is exposed to 0.0.0.0/0

Clean up resources

gcloud compute instances delete app-vm --zone=asia-southeast1-a --quiet
gcloud sql instances delete app-db --quiet
gcloud compute firewall-rules delete allow-app-port --quiet
gcloud dns managed-zones delete app-zone --quiet

Conclusion

This walkthrough demonstrated that a lift-and-shift migration is
rarely "zero infrastructure change" even when it is "zero code change."
The Cloud SQL Auth Proxy pattern shown here is a common bridge for
applications that cannot be immediately refactored to use
cloud-native service discovery.

In Part 2, we build on this foundation with a highly available
three-tier architecture — and deliberately break it five different
ways to practice production diagnostics.

Google Cloud Fundamentals — The Hard Way: A Break-and-Fix Learning Series

iapilgrim — Tue, 01 Sep 2026 14:22:08 +0000

Overview

Most cloud onboarding tutorials optimize for one outcome: getting a
resource into a RUNNING state as fast as possible. That approach
teaches syntax, but it doesn't teach the skill that actually matters in
production — diagnosing a system when it's not behaving as expected,
using only logs, metrics, and a hypothesis-driven process of
elimination.

This eight-part series takes the opposite approach. Each post walks
through provisioning a piece of Google Cloud infrastructure using the
gcloud CLI, deliberately introducing a realistic failure mode into
that infrastructure, and then walking through the diagnostic process
needed to identify and remediate the issue — with the actual commands,
log queries, and decision points included.

This series is intended for engineers who already understand IaaS/PaaS
fundamentals and want hands-on, failure-driven practice with core
GCP services, in the spirit of Kubernetes The Hard Way.

What this series covers

Part	Focus area	GCP services
1	Migrating a legacy application without modifying its source	Compute Engine, Cloud SQL, Cloud SQL Auth Proxy, VPC Firewall
2	Diagnosing failures in a highly available three-tier architecture	Managed Instance Groups, Cloud Load Balancing, Health Checks
3	Auditing and remediating IAM misconfigurations	IAM, Service Accounts, Cloud Asset Inventory
4	Choosing (and migrating between) NoSQL data stores	Firestore, Bigtable, row-key design
5	Safe rollout patterns for containerized workloads	GKE, liveness/readiness probes, canary deployment
6	Validating disaster recovery assumptions	Cross-region Cloud SQL replicas, Cloud DNS, RTO/RPO
7	Identifying and eliminating cloud waste	Billing Reports, Recommender, FinOps fundamentals
8	Designing failure scenarios for team training	Incident design, SRE practices

Architecture pattern used throughout the series

Several posts in this series build on a common reference architecture:

                    ┌────────────────────┐
   Client  ──────▶  │ Cloud Load Balancer │
                    └─────────┬──────────┘
                              │
                    ┌─────────▼──────────┐
                    │  Managed Instance    │
                    │  Group (2–4 VMs)     │
                    └─────────┬──────────┘
                              │  Private IP
                    ┌─────────▼──────────┐
                    │     Cloud SQL       │
                    │  (Primary + Replica)│
                    └────────────────────┘

This pattern is intentionally simple — the goal isn't architectural
complexity, it's giving you a stable baseline you can break in
controlled, repeatable ways.

Prerequisites for the series

A Google Cloud project with billing enabled (a fresh project with free trial credit or a dedicated test project is strongly recommended)
The gcloud CLI installed locally, or access to Cloud Shell
Working familiarity with Linux command-line tools and basic log interpretation
A Budget Alert configured on your billing account (instructions in Part 1) to avoid unexpected cost overrun while experimenting

How to use this series

Each post is self-contained but builds conceptually on the ones before
it — Part 8 in particular reuses techniques introduced in Parts 2–4.
We recommend working through them in order, and resisting the urge to
look up the fix for an injected failure for at least 20–30 minutes.
That struggle is where the retained learning happens.

Let's get started with [Part 1: Migrating a Legacy Application Without
Touching Its Code →]

Tutorial: Role-Based Access Control for a Node/Express API, Backed by Keycloak

iapilgrim — Tue, 21 Jul 2026 03:00:20 +0000

This is a hands-on walkthrough for kc-rbac-api — module 04 in a series
building up Keycloak features one small, runnable project at a time. By the
end you'll have a Node/Express API with three roles enforced at the
endpoint level, two interchangeable ways of validating incoming tokens, and
a full test suite proving the denials work as intended, not just the happy
path.

Everything below assumes Docker + Docker Compose, Node.js, curl, and
python3 (used by the setup script for JSON parsing — nothing else).

What you're building

An API with three roles — viewer, editor, admin — where each endpoint
enforces a specific minimum role, not just "logged in or not":

Endpoint	viewer	editor	admin
`GET /api/documents`	✅	✅	✅
`POST /api/documents`	❌	✅	✅
`PUT /api/documents/:id`	❌	✅	✅
`DELETE /api/documents/:id`	❌	❌	✅
`GET /api/admin/stats`	❌	❌	✅
`GET /api/whoami`	✅	✅	✅

Notice DELETE breaks the simple "editor and up" pattern — it's
admin-only. That's deliberate, and it's the detail worth testing carefully
later on.

Step 1 — Start Keycloak and the API

make start   # docker compose up -d --build: Keycloak + this API's container

Give Keycloak about 30 seconds to finish booting, then provision everything:

make setup

This runs keycloak/setup.sh, which — entirely through Keycloak's Admin
REST API, no clicking through the console — creates:

a realm called rbac-demo
three realm roles: viewer, editor, admin
a confidential client kc-rbac-api with an audience mapper (more on that below)
three demo users, one per role, and writes the client secret into .env

Username	Password	Role
alice	alice	viewer
bob	bob	editor
carol	carol	admin

Confirm the API is up:

curl http://localhost:3000/health
# {"status":"ok","validationMode":"local"}

Step 2 — Understand the two ways to validate a token

Every protected route sits behind one of two middlewares, and which one is
active is a config toggle (TOKEN_VALIDATION_MODE in .env) — not a code
change:

src/middleware/jwtAuth.js (local) verifies the token's RS256
signature locally, against Keycloak's JWKS endpoint (cached via
jwks-rsa), then checks iss, aud, and exp:

jwt.verify(token, getSigningKey, {
  algorithms: ["RS256"],
  issuer: config.issuer,
  audience: config.clientId,
}, (err, payload) => { /* ... */ });

Fast — one JWKS fetch, then it's pure CPU per request after that. The
tradeoff: if Keycloak revokes the token or the user logs out server-side,
this middleware has no way to know. The token stays "valid" here until it
naturally expires.

src/middleware/introspectAuth.js (introspect) instead POSTs the
token to Keycloak's /token/introspect endpoint (RFC 7662) on every
request:

const { data } = await axios.post(config.introspectionUrl, params, {
  headers: { "Content-Type": "application/x-www-form-urlencoded" },
});
if (!data.active) {
  return res.status(401).json({ error: "token_inactive", ... });
}

Slower — one extra network round trip per call — but revocation is
immediate: Keycloak reports active: false the moment a token is no
longer valid, regardless of its exp. This mode needs its own client
credentials, since introspection is itself an authenticated endpoint —
only clients that own or trust the token's audience get to ask Keycloak
about it.

Neither one is "the right answer" on its own — it's a latency-vs-freshness
tradeoff. Short access-token lifespans (this realm is set to 300 seconds)
are the usual way to bound local validation's blind spot without paying
the introspection cost on every single call.

Try switching modes and watching /api/whoami reflect it:

TOKEN_VALIDATION_MODE=introspect npm start

TOKEN=$(curl -s -X POST http://localhost:8080/realms/rbac-demo/protocol/openid-connect/token \
  -d "grant_type=password&client_id=kc-rbac-api&client_secret=$CLIENT_SECRET&username=carol&password=carol" \
  | python3 -c "import sys,json;print(json.load(sys.stdin)['access_token'])")

curl http://localhost:3000/api/whoami -H "Authorization: Bearer $TOKEN"
# "authMode": "introspect"

Step 3 — Enforce roles with two middleware shapes

src/middleware/rbac.js gives you two ways to gate a route, and picking
the right one matters:

// Exact allow-list — use when roles AREN'T hierarchical for this endpoint
requireRole("admin")

// Hierarchical — viewer(1) < editor(2) < admin(3)
requireMinRole("editor")

requireMinRole is what powers most of the table above — POST/PUT
use requireMinRole("editor") since editor-and-up should both succeed. But
DELETE uses requireRole("admin") explicitly, not
requireMinRole("admin") treated as "the top of the chain" by accident —
it's named as an exact requirement on purpose. If you ever add a role that
doesn't nest cleanly into the hierarchy (say, an auditor who can read
things editor can't but can't write anything), model it with
requireRole() allow-lists rather than forcing it into the rank table.

Roles themselves come out of the token via src/utils/tokenClaims.js,
which merges Keycloak's realm_access.roles (realm-wide roles) and
resource_access.<clientId>.roles (client-scoped roles) into one flat set
— route middleware doesn't care which bucket a role happened to live in.

Step 4 — Understand why the audience (`aud`) check exists

keycloak/setup.sh adds an oidc-audience-mapper so tokens for this
client carry aud: kc-rbac-api, and both middlewares reject anything
where that's missing:

const aud = Array.isArray(data.aud) ? data.aud : [data.aud];
if (!aud.includes(config.clientId)) {
  return res.status(401).json({ error: "invalid_audience", ... });
}

Without this check, a token minted for a different API in the same
realm — one with broader service-account scope, say — would pass
signature and issuer checks just fine and get treated as valid here too.
That's the "confused deputy" problem: a token issued for one purpose
getting replayed against a resource server it was never meant for. aud
is the check that closes that gap.

Step 5 — Run the full allow/deny matrix

make test

This runs tests/test.sh, a self-contained bash suite: it fetches real
tokens for alice/bob/carol, hits every endpoint as every role, and asserts
both the HTTP status and the JSON error code — so a 403 for the wrong
reason still fails the test.

── 6. DELETE /api/documents/:id (admin only) ──
✓ PASS  [DENY] viewer cannot delete   (got 403, error=insufficient_role)
✓ PASS  [DENY] editor cannot delete   (got 403, error=insufficient_role)
✓ PASS  [ALLOW] admin can delete      (got 204)

════════════════════════════════════════════════
  Results: 25 passed, 0 failed
════════════════════════════════════════════════

It's mode-aware: one assertion checks for invalid_token under local
validation but token_inactive under introspection, since Keycloak's
introspection endpoint doesn't distinguish "malformed" from
"merely-unrecognized" — both are a correct 401, just different
vocabulary. Exits non-zero on any failure, so it's safe to drop straight
into CI.

Prefer clicking through requests instead of reading a script? The same
matrix is in http/requests.http (VS Code REST Client — token requests
save into named variables the later requests reuse automatically) and as
a Postman collection + environment in postman/, with pm.test
assertions on every request.

Step 6 — Poke around the Keycloak Admin Console

http://localhost:8080

Username: admin / Password: admin
Realm: rbac-demo
Client kc-rbac-api → Roles tab has viewer/editor/admin → Client scopes → Dedicated scopes → Mappers has the audience mapper you just saw enforced in code

Worth clicking into a user (say, carol) and looking at Role mapping
to see the realm role assignment that ends up in realm_access.roles on
her token.

Before you take this to production

A few deliberate simplifications worth knowing about:

This demo uses Resource Owner Password Credentials (grant_type=password) purely so setup.sh, the .http file, and Postman can fetch tokens with one curl call per user. Browser-facing apps should use Authorization Code + PKCE instead — keep directAccessGrantsEnabled: false for those clients.
The in-memory documents array resets on restart. Swap it for a real datastore before this goes anywhere near real traffic.
requireMinRole assumes a strictly linear hierarchy. It works great for viewer < editor < admin, but don't force a non-nesting role into it — use requireRole() allow-lists instead.

Where the pieces live

File	Purpose
`keycloak/setup.sh`	Realm, roles, client + audience mapper, 3 demo users — all via Admin REST API
`src/middleware/jwtAuth.js`	Local JWT validation (JWKS)
`src/middleware/introspectAuth.js`	RFC 7662 introspection validation
`src/middleware/rbac.js`	`requireRole()` / `requireMinRole()`
`src/routes/documents.js`	The CRUD resource from the table above
`src/routes/admin.js`	Admin-only endpoint
`src/routes/whoami.js`	Debug endpoint — inspect resolved roles/claims for your own token
`tests/test.sh`	The 25-case bash suite from Step 5
`http/requests.http` / `postman/*.json`	Same matrix, interactive

Recap

                          ┌─────────────┐
  client ── access_token ─►  kc-rbac-api │
                          └──────┬──────┘
                                 │
                validationMode = local          validationMode = introspect
                                 │                            │
                    verify sig vs JWKS              POST /token/introspect
                    check iss/aud/exp               (client_id + client_secret)
                    read roles from claims           check active + aud
                                 │                            │
                                 └─────────► requireMinRole() / requireRole() ◄─────────┘
                                                       │
                                              200 / 201 / 204  or  403 insufficient_role

Two validation strategies, one audience check standing between them and
"any signed token gets in," and a role hierarchy that's explicit about
where it stops being hierarchical. That's the whole module.

Next up in the series: module 05 — Identity Brokering, wiring up two
Keycloak realms in one Compose stack so a "corp" realm can broker into a
"partner" realm via OIDC, without needing an actual external IdP to test
against.

Keycloak from Scratch, Part 2 — Authorization Code + PKCE, Built by Hand

iapilgrim — Mon, 20 Jul 2026 12:42:11 +0000

In Part 1 we logged in by sending a username and password directly
to Keycloak. That's fine for a script, but no browser-facing app should
ever do that — it means the app itself sees the raw password, and it
can't participate in single sign-on. What every real login button should
do instead is redirect the user to Keycloak and get a token handed
back. That's the Authorization Code flow, and for anything that can't
keep a secret (a browser app, a mobile app), it needs a companion called
PKCE.

We're going to implement it by hand — no keycloak-js, no library — so
every step is visible instead of hidden behind keycloak.login().

What we're building

A public Keycloak client (no secret — a browser can't keep one)
A plain HTML + vanilla JS page that performs the full redirect flow
A small Flask API that validates the resulting token independently, without ever talking to Keycloak per request

Prerequisites

Everything from Part 1 (Keycloak running via Docker Compose)
Python 3 with pip install flask flask-cors "pyjwt[crypto]"
A browser

1. Create a new realm and a public client

Admin Console → Create realm → pkce-demo
Clients → Create client → Client ID: pkce-spa-client → Next
Client authentication: Off — this is what makes it a public client. There's no secret to steal because there isn't one.
Capability config: check Standard flow, uncheck Direct access grants (we don't want this client falling back to Part 1's flow) → Save
Settings tab:
- Valid redirect URIs: http://localhost:5173/*
- Web origins: http://localhost:5173
Advanced tab → scroll to Proof Key for Code Exchange Code Challenge Method → set to S256

This last setting is the one that actually enforces PKCE server-side.
Without it, Keycloak will still accept a PKCE challenge if the client
sends one, but it won't require it — leaving the door open for a
client that skips PKCE entirely.

2. Create a role and a user

Realm roles → create spa-user
Users → Add user → username dave, first/last name Dave/Demo (see Part 1 if you're wondering why that matters)
Credentials tab → password dave, Temporary: Off
Role mapping → assign spa-user

3. The concept: what PKCE actually proves

A confidential client proves its identity with a secret. A public client
can't do that — so PKCE proves something different: that the app
redeeming the authorization code is the same app that started the login.

Browser                                          Keycloak
  │  code_verifier = random(64 bytes)                │
  │  code_challenge = SHA256(code_verifier)          │
  ├──── GET /auth?code_challenge=... ───────────────►│
  │◄─── redirect back with ?code=... ─────────────────┤
  ├──── POST /token                                   │
  │        code=..., code_verifier=... ──────────────►│  (recomputes SHA256(code_verifier),
  │◄─── access_token, id_token ───────────────────────┤   compares to the original challenge)

The code_verifier never leaves the browser tab until the final token
request. If an attacker somehow intercepts the redirect and grabs the
code (say, via a malicious app registered against the same custom
URL scheme on mobile), they still can't redeem it — they don't have the
verifier, and they can't derive it from the challenge, because SHA-256
doesn't run backwards.

4. Generating the challenge (app.js)

function base64UrlEncode(bytes) {
  return btoa(String.fromCharCode(...new Uint8Array(bytes)))
    .replace(/\+/g, "-").replace(/\//g, "_").replace(/=+$/, "");
}

function randomString(length = 64) {
  const arr = new Uint8Array(length);
  crypto.getRandomValues(arr);
  return base64UrlEncode(arr).slice(0, length);
}

async function sha256(input) {
  return crypto.subtle.digest("SHA-256", new TextEncoder().encode(input));
}

Nothing exotic — crypto.subtle (native to every modern browser) does
the hashing, and the verifier is just cryptographically random bytes.

5. Kicking off the login

async function login() {
  const codeVerifier = randomString(64);
  const codeChallenge = base64UrlEncode(await sha256(codeVerifier));
  const state = randomString(16);

  // This never leaves the browser until step 6.
  sessionStorage.setItem("pkce_code_verifier", codeVerifier);
  sessionStorage.setItem("pkce_state", state);

  const params = new URLSearchParams({
    client_id: "pkce-spa-client",
    redirect_uri: window.location.origin + "/index.html",
    response_type: "code",
    scope: "openid",
    state,
    code_challenge: codeChallenge,
    code_challenge_method: "S256",
  });

  window.location.href =
    `http://localhost:8080/realms/pkce-demo/protocol/openid-connect/auth?${params}`;
}

The state parameter is a separate, unrelated safety net: a random value
checked on the way back to guard against CSRF — someone tricking your
browser into completing an authorization flow you didn't start.

6. Handling the redirect back

Keycloak sends the browser back to your redirect_uri with ?code=...&state=...
in the URL. This runs on page load:

async function handleCallback(code, returnedState) {
  const expectedState = sessionStorage.getItem("pkce_state");
  if (returnedState !== expectedState) {
    throw new Error("state mismatch — possible CSRF, aborting");
  }

  const codeVerifier = sessionStorage.getItem("pkce_code_verifier");

  const body = new URLSearchParams({
    grant_type: "authorization_code",
    client_id: "pkce-spa-client",
    redirect_uri: window.location.origin + "/index.html",
    code,
    code_verifier: codeVerifier,
  });

  const resp = await fetch(
    "http://localhost:8080/realms/pkce-demo/protocol/openid-connect/token",
    { method: "POST", headers: { "Content-Type": "application/x-www-form-urlencoded" }, body }
  );

  const tokens = await resp.json();
  sessionStorage.setItem("access_token", tokens.access_token);
  sessionStorage.setItem("id_token", tokens.id_token);
  window.history.replaceState({}, document.title, window.location.origin + "/index.html");
}

Keycloak re-derives SHA256(code_verifier) server-side and compares it
to the code_challenge it received in step 5. If they match, and the
code hasn't already been redeemed or expired, you get tokens back.

7. Validating the token on the API side

The API never talks to Keycloak per request — it fetches Keycloak's
public signing keys once (JWKS) and verifies signatures locally:

from flask import Flask, request, jsonify
from flask_cors import CORS
import jwt
from jwt import PyJWKClient

ISSUER = "http://localhost:8080/realms/pkce-demo"
CLIENT_ID = "pkce-spa-client"
jwks_client = PyJWKClient(f"{ISSUER}/protocol/openid-connect/certs")

app = Flask(__name__)
CORS(app, origins=["http://localhost:5173"])

@app.route("/api/me")
def me():
    auth = request.headers.get("Authorization", "")
    token = auth.split(" ", 1)[1] if auth.startswith("Bearer ") else None
    if not token:
        return jsonify({"error": "missing bearer token"}), 401

    signing_key = jwks_client.get_signing_key_from_jwt(token)
    claims = jwt.decode(
        token, signing_key.key, algorithms=["RS256"],
        issuer=ISSUER, options={"verify_aud": False},
    )
    # Keycloak's default audience is "account", not the client id — check
    # azp (authorized party) instead, which is always the requesting client.
    if claims.get("azp") != CLIENT_ID:
        return jsonify({"error": "unexpected azp"}), 401

    return jsonify({
        "preferred_username": claims.get("preferred_username"),
        "roles": claims.get("realm_access", {}).get("roles", []),
    })

Because Keycloak signs tokens asymmetrically (RS256), the API only ever
needs the public half of the key pair — it can validate tokens without
ever holding a shared secret with Keycloak.

8. Run it

python3 api/server.py &            # :5001
python3 -m http.server 5173 &      # :5173, serving the SPA

Open http://localhost:5173, click Log in, authenticate as
dave/dave, and you should land back on the page with decoded ID
token and access token claims printed, plus a working call to
/api/me.

What actually happened here

Public client, no secret — PKCE substitutes possession of a one-time verifier for a shared secret the client can't keep anyway.
state and code_verifier are solving different problems — CSRF vs. authorization-code interception — don't conflate them.
Asymmetric JWT validation means your API and your identity provider never need to share a secret at all, unlike the client-secret approach from Part 1.

Next up, Part 3: back to server-to-server auth — client credentials
grants, service accounts, and when Direct Grant (Part 1) is and isn't
the right tool for machine identities.

Keycloak from Scratch, Part 1 — Realms, Roles, Clients & Users (No Docker Magic, Just Clicks)

iapilgrim — Mon, 20 Jul 2026 12:40:51 +0000

If you've ever inherited a Keycloak instance and had no idea what a
"realm" actually isolates, or why some clients have secrets and others
don't, this post is the ground floor. We're building one small realm by
hand, in the Admin Console — no automation scripts, no realm-export JSON
to skim past. Just enough clicking to make every concept stick.

By the end you'll have:

A dedicated realm
Three roles representing a basic permission tier (admin, editor, viewer)
A confidential client configured for direct login
Three users, each mapped to one role
A working curl command that logs in as each user and shows you their role inside a decoded JWT

Prerequisites

Docker + Docker Compose
curl and python3 (for a one-line JWT decode — no library needed)

1. Start Keycloak

# docker-compose.yml
services:
  keycloak:
    image: quay.io/keycloak/keycloak:26.7.0
    container_name: keycloak-fundamentals
    command: start-dev
    environment:
      KEYCLOAK_ADMIN: admin
      KEYCLOAK_ADMIN_PASSWORD: admin
    ports:
      - "8080:8080"

docker compose up -d

Give it about 30 seconds, then open http://localhost:8080 and log in
with admin / admin.

start-dev is a dev-mode flag — it skips TLS and some production
checks so you can iterate quickly. Never run this flag in production.

2. Create a realm

A realm is Keycloak's top-level tenancy boundary — its own users,
roles, clients, and even its own login theme, fully isolated from every
other realm on the same server. Think of it as a separate mini
directory service you can spin up per environment, per product, or per
customer.

Top-left realm dropdown → Create realm
Realm name: fundamentals
Create

3. Create three realm roles

Roles are how you express "what can this identity do" without hardcoding
usernames into your authorization logic. We're using realm roles
here — visible to every client in the realm, as opposed to client
roles, which are scoped to one specific client. (We'll hit client
roles in a later post when we build a proper RBAC API.)

Left sidebar → Realm roles → Create role
Create three, one at a time: admin, editor, viewer

4. Create a confidential client

Left sidebar → Clients → Create client
Client ID: kc-fundamentals-app → Next
Client authentication: toggle On

This is the setting that makes a client "confidential" — it gets a
secret, and Keycloak requires that secret before it'll hand out a
token. A public client (toggle off) has no secret at all, because
anything shipped to a browser or mobile app can't actually keep one
secret. We'll build a public client in Part 2.

Capability config: check Direct access grants, leave Standard flow on if you like (harmless either way for this demo) → Save
Go to the Credentials tab and copy the Client secret — you'll paste it into the curl command later

5. Create three users

Repeat this for alice, bob, and carol:

Users → Add user
Username: alice
First name: Alice, Last name: Demo

Don't skip this. Keycloak has a built-in "Update Profile" required
action that checks profile completeness dynamically, at login
time — not something you'll see if you inspect the user's stored
data beforehand. An empty first/last name silently blocks login later
with a generic "Account is not fully set up" error that has nothing
to do with the password you set. This one cost me an embarrassing
amount of debugging the first time — save yourself the trouble.

Create
Credentials tab → Set password → alice → toggle Temporary: Off → Save

"Temporary" means the user is forced to change it on next login —
fine for humans clicking through a browser flow, but it'll break a
scripted or API-driven login every time, since there's no browser to
serve the "set a new password" screen. Off, for this demo.

Role mapping tab → Assign role → pick admin

Repeat for bob → editor and carol → viewer.

6. One more thing to check: OTP

Some Keycloak versions ship the built-in direct grant authentication
flow with its conditional-OTP branch misconfigured as Required instead
of Conditional. If that's the case on your instance, every login
attempt from a user without an OTP device configured — which is all
three of ours — gets rejected before the password is even checked.

Authentication (left sidebar) → Flows tab → select direct grant
Find the OTP-related row at the top level (often labeled something like "Direct Grant - Conditional OTP")
If it's not already Conditional or Disabled, set it to Disabled

7. Log in and inspect the token

This is a Direct Access Grant (also called Resource Owner Password
Credentials, or ROPC) — the client trades a username and password
directly for a token, no browser redirect involved. It's convenient for
this fundamentals demo and for machine-to-machine scripts, but it means
the client sees the raw password, so it's a poor fit for anything
user-facing. (Part 2 covers the flow you actually want for a browser app:
Authorization Code + PKCE.)

curl -s -X POST http://localhost:8080/realms/fundamentals/protocol/openid-connect/token \
  -H "Content-Type: application/x-www-form-urlencoded" \
  -d "grant_type=password" \
  -d "client_id=kc-fundamentals-app" \
  -d "client_secret=PASTE_YOUR_SECRET_HERE" \
  -d "username=alice" \
  -d "password=alice" | python3 -m json.tool

If that returns an access_token, decode its payload to see the role
claim in the flesh:

curl -s -X POST http://localhost:8080/realms/fundamentals/protocol/openid-connect/token \
  -d "grant_type=password" -d "client_id=kc-fundamentals-app" \
  -d "client_secret=PASTE_YOUR_SECRET_HERE" -d "username=alice" -d "password=alice" \
  | python3 -c "
import sys, json, base64
token = json.load(sys.stdin)['access_token']
payload = token.split('.')[1]
payload += '=' * (-len(payload) % 4)
print(json.dumps(json.loads(base64.urlsafe_b64decode(payload)), indent=2))
"

Look for:

"realm_access": {
  "roles": ["admin", "default-roles-fundamentals", "offline_access", "uma_authorization"]
}

admin is the one we assigned. The other three are Keycloak's own
housekeeping roles every user gets by default.

Repeat with bob/bob and carol/carol — you should see editor
and viewer respectively.

What actually happened here

Realm isolation: fundamentals never sees or touches any other realm's users, clients, or roles — that's the whole point of realms.
Confidential client + Direct Grant: fine for this demo and for trusted server-side/CLI tooling, wrong for a browser SPA (that's Part 2).
Role claims propagate automatically: once a realm role is assigned to a user, Keycloak embeds it in every token that user gets, with zero extra configuration — this is what your API or gateway will check against later.

Troubleshooting: "Account is not fully set up"

If you hit this, it's one of exactly two causes:

OTP required in the direct grant flow (see step 6 above)
Incomplete user profile — missing first/last name (see step 5). This one is genuinely invisible from the outside: it's evaluated live at login, not stored anywhere you can query beforehand.

Check both. They're independent, and fixing only one can still leave you
stuck.

Next up, Part 2: we swap Direct Grant for the flow every real
browser app should use — Authorization Code with PKCE — and build a
vanilla-JS SPA that implements the whole thing by hand, no auth library,
so every step of the exchange is visible.