DEV Community

iapilgrim
iapilgrim

Posted on

GCP The Hard Way — Part 8: Designing a Broken System for Someone Else to Debug

#gcp #sre #incidentresponse #finale

Introduction

Every prior post in this series involved diagnosing a failure someone
else (in this case, the author) had designed. This final post reverses
that relationship: you design the failure. Constructing a realistic,
diagnosable incident requires a deeper understanding of a system than
fixing one does, because you must accurately predict which symptoms a
given misconfiguration will produce. This exercise is a common
technique in SRE training programs, and this post walks through
applying it to a Google Cloud environment.

Solution overview 

┌─────────────────────┐        ┌────────────────────┐
│   Design Phase        │        │    Debug Phase       │
│                        │───────▶│                       │
│  - Inject 3–5 faults   │ hand   │  - Investigate via    │
│  - Write incident      │  off   │    logs and metrics   │
│    ticket (symptoms     │        │  - Isolate root cause │
│    only, no root cause) │        │  - Remediate           │
└─────────────────────┘        └────────────────────┘
Enter fullscreen mode Exit fullscreen mode

Criteria for a well-designed failure scenario

  1. Clear symptom, unclear cause — the observer sees a concrete behavior, not an obvious misconfiguration
  2. Reliably reproducible — the failure occurs consistently, not intermittently by chance
  3. Plausible red herrings present — realistic incidents involve ruling out multiple hypotheses, not confirming the first guess
  4. A definitive verification method exists — once the correct hypothesis is identified, there must be an unambiguous way to confirm it

Walkthrough

Step 1: Establish a known-good baseline

Reuse the three-tier architecture from Part 2 (Load Balancer → MIG →
Cloud SQL), and confirm it functions correctly before introducing any
faults — this establishes a clean baseline to distinguish injected
faults from pre-existing configuration issues.

Step 2: Select and inject 3–5 faults

A representative fault bank, spanning difficulty levels:

Fault Injection External symptom
Firewall allows only one of two required health-check ranges --source-ranges=130.211.0.0/22 (omitting 35.191.0.0/16) Intermittent backend health flapping
Firewall target tag mismatch Firewall applies web-server; VM tagged app-server Connection timeout, resembling an application crash
Cloud SQL connection limit set below expected concurrent load --database-flags=max_connections=20 Failures only under concurrent load, invisible to single-request testing
Read replica under-provisioned relative to write volume Small tier replica against high write throughput Stale reads that resemble an application logic bug

Example injection:

gcloud sql instances patch app-db --database-flags=max_connections=20
Enter fullscreen mode Exit fullscreen mode

Choose faults spanning multiple categories (network, IAM, database,
application) rather than concentrating on one area, to keep the
diagnostic exercise varied.

Step 3: Write the incident ticket

The ticket should be written from the perspective of an end user
reporting a problem, deliberately omitting any technical language that
would hint at the root cause:

# Ticket #001: Intermittent login failures during peak hours

## Description
Users report occasional "connection timeout" errors when logging in,
primarily between 9–10 AM. No issues reported in the afternoon.

## Already investigated
- VM restarted — issue persists
- CPU and memory utilization checked — both within normal range

## Access provided
Cloud Console (read-only), Cloud Logging, gcloud CLI
Enter fullscreen mode Exit fullscreen mode

Including one or two already-ruled-out hypotheses ("already
investigated") makes the exercise more representative of real
incident response, where an initial theory is often incorrect.

Step 4: Grant scoped access for the investigator 

gcloud projects add-iam-policy-binding <PROJECT_ID> \
  --member="user:<investigator-email>" \
  --role="roles/viewer"

gcloud projects add-iam-policy-binding <PROJECT_ID> \
  --member="user:<investigator-email>" \
  --role="roles/logging.viewer"
Enter fullscreen mode Exit fullscreen mode

Read-only access prevents accidental remediation before the
investigator has confirmed a root cause.

Step 5: Structured investigation approach

Whether debugging as a peer or returning to your own system after a
delay, apply a consistent process:

  1. Reproduce the symptom first — do not make changes before confirming the reported behavior
  2. Narrow scope by elimination — isolate whether the fault sits in the network, application, or database layer by testing each layer independently (e.g., curl localhost on the VM before testing through the load balancer)
  3. Read logs systematically — filter Cloud Logging by severity and correlate timestamps with reported symptom occurrence
  4. Verify each hypothesis definitively before moving to the next, rather than changing multiple variables simultaneously

Comparing outcomes 

| Injected fault | Identified? | Time to identify | False leads pursued |
|---|---|---|---|
| Fault 1 | | | |
| Fault 2 | | | |
Enter fullscreen mode Exit fullscreen mode

Clean up resources 

gcloud projects delete <PROJECT_ID>
Enter fullscreen mode Exit fullscreen mode

Conclusion

This series has moved through the core operational competencies of
infrastructure engineering: provisioning without code changes,
diagnosing availability failures, auditing access control, choosing
data stores appropriate to workload characteristics, shipping releases
safely, validating disaster recovery assumptions, controlling cost, and
finally, designing failure scenarios well enough to teach them to
others. That last skill — being able to construct a realistic incident,
not just resolve one — is a strong signal of systems-level understanding,
and a capability directly transferable to building internal SRE
training programs.

This concludes the Google Cloud Fundamentals: The Hard Way series.

Top comments (0)