DEV Community

iapilgrim
iapilgrim

Posted on

GCP The Hard Way — Part 2: Diagnosing Failures in a Highly Available Three-Tier Architecture

#gcp #loadbalancing #sre #devops

Introduction

High availability features like managed instance groups (MIGs) and
load balancing are often treated as "set and forget" — configure them
once, and the platform handles the rest. In practice, understanding
when and how quickly these mechanisms respond to failure is
essential for setting realistic availability expectations. This post
walks through provisioning a three-tier architecture on Google Cloud
and deliberately introducing five distinct failure conditions to
observe and measure the platform's actual recovery behavior.

Solution overview

The architecture consists of a global HTTP load balancer distributing
traffic across a managed instance group, backed by a health check that
determines instance eligibility for traffic.

        ┌───────────────────────┐
Client ─▶  Global HTTP(S) LB     │
        └───────────┬───────────┘
                     │
        ┌────────────▼────────────┐
        │   Backend Service         │
        │   (Health Check: /health) │
        └────────────┬────────────┘
                     │
     ┌───────────────┼───────────────┐
     ▼                                ▼
┌─────────┐                    ┌─────────┐
│  VM 1   │                    │  VM 2   │
└─────────┘                    └─────────┘
   Managed Instance Group (min:2, max:4)
Enter fullscreen mode Exit fullscreen mode

Prerequisites

  • Completion of Part 1, or equivalent familiarity with Compute Engine and firewall rules
  • A load-testing tool such as hey or Apache Bench installed locally or on a separate VM

Walkthrough

Step 1: Deploy the instance template and managed instance group 

gcloud compute instance-templates create app-template \
  --machine-type=e2-small \
  --image-family=debian-12 \
  --image-project=debian-cloud \
  --tags=app-server \
  --metadata-from-file=startup-script=startup.sh

gcloud compute instance-groups managed create app-mig \
  --template=app-template --size=2 --zone=asia-southeast1-a

gcloud compute instance-groups managed set-autoscaling app-mig \
  --zone=asia-southeast1-a \
  --max-num-replicas=4 --min-num-replicas=2 \
  --target-cpu-utilization=0.6 --cool-down-period=60
Enter fullscreen mode Exit fullscreen mode

Step 2: Configure health checks and the load balancer 

gcloud compute health-checks create http app-health-check \
  --port=8080 --request-path=/health

gcloud compute firewall-rules create allow-health-check \
  --allow=tcp:8080 \
  --source-ranges=130.211.0.0/22,35.191.0.0/16 \
  --target-tags=app-server

gcloud compute backend-services create app-backend \
  --protocol=HTTP --port-name=http \
  --health-checks=app-health-check --global

gcloud compute backend-services add-backend app-backend \
  --instance-group=app-mig --instance-group-zone=asia-southeast1-a --global

gcloud compute url-maps create app-lb --default-service=app-backend
gcloud compute target-http-proxies create app-http-proxy --url-map=app-lb
gcloud compute forwarding-rules create app-http-rule \
  --global --target-http-proxy=app-http-proxy --ports=80
Enter fullscreen mode Exit fullscreen mode

Allow 5–10 minutes for the backend to transition to a HEALTHY state
before proceeding.

Failure scenario 1: Manual instance termination 

gcloud compute instance-groups managed list-instances app-mig \
  --zone=asia-southeast1-a
gcloud compute instances delete <INSTANCE_NAME> \
  --zone=asia-southeast1-a --quiet
Enter fullscreen mode Exit fullscreen mode

Observation: Measure the time between deletion and the appearance
of a replacement instance. If no replacement appears within several
minutes, verify whether autohealing is configured — MIG autoscaling
and autohealing are independent mechanisms, and autoscaling alone does
not guarantee replacement of a manually terminated instance outside of
its target size logic.

Failure scenario 2: Health check path blocked by firewall 

gcloud compute firewall-rules delete allow-health-check --quiet
gcloud compute backend-services get-health app-backend --global
Enter fullscreen mode Exit fullscreen mode

Diagnostic approach: From inside an affected VM, run
curl localhost:8080/health directly. A successful local response
combined with a failing get-health status isolates the fault to the
network path rather than the application.

Failure scenario 3: Load-induced autoscaling 

hey -z 5m -c 200 http://<LB_IP>/
Enter fullscreen mode Exit fullscreen mode

Observation: Track the interval between CPU utilization crossing
the 60% threshold and new instances becoming HEALTHY and receiving
traffic. Expect a multi-minute delay attributable to three sequential
stages: autoscaler metric evaluation, VM boot and startup script
execution, and health check confirmation before the load balancer
begins routing traffic.

Failure scenario 4: Misconfigured health check path 

gcloud compute health-checks update http app-health-check \
  --request-path=/healthz
Enter fullscreen mode Exit fullscreen mode

Diagnostic approach: This produces the same UNHEALTHY status as
Scenario 2 but stems from an application-layer mismatch rather than a
network block — reinforcing that identical symptoms can have distinct
root causes, and log inspection (not assumption) should drive
remediation.

Failure scenario 5: Full instance group depletion 

gcloud compute instance-groups managed resize app-mig \
  --zone=asia-southeast1-a --size=0
Enter fullscreen mode Exit fullscreen mode

Restore the health check path and resize back to 2, and measure total
recovery time from zero running instances to a healthy backend.

Clean up resources 

gcloud compute forwarding-rules delete app-http-rule --global --quiet
gcloud compute target-http-proxies delete app-http-proxy --quiet
gcloud compute url-maps delete app-lb --quiet
gcloud compute backend-services delete app-backend --global --quiet
gcloud compute instance-groups managed delete app-mig \
  --zone=asia-southeast1-a --quiet
gcloud compute instance-templates delete app-template --quiet
Enter fullscreen mode Exit fullscreen mode

Conclusion

Each of these five scenarios produces a distinct combination of
symptoms and recovery timelines, despite relying on the same
underlying platform features. The key takeaway is that autoscaling,
autohealing, and load balancer health checking are separate,
composable mechanisms — understanding their individual behavior is
what separates confident incident response from guesswork.

In Part 3, we shift focus to IAM, auditing a project for
privilege escalation risks introduced through common configuration
shortcuts.

Top comments (0)