DEV Community: Ayush Raj Jha

Real-World Incident Automation Using GCP: How I Cut MTTR by 80%

Ayush Raj Jha — Wed, 18 Mar 2026 20:05:03 +0000

We used to resolve incidents with Slack messages, gut instinct, and heroics. Now most incidents resolve themselves. Here's exactly how I built that.

The Problem with Manual Incident Response

At 2:47am on a Tuesday, our payment service started throwing errors. A senior engineer woke up to a PagerDuty alert, spent 12 minutes just finding the right runbook, another 20 minutes correlating logs across three different dashboards, and finally identified a misconfigured connection pool that had been deployed 6 hours earlier.

Total time to resolve: 51 minutes of customer-impacting downtime.

The fix itself? Four lines of config. The rest was just finding the problem.

I decided to systematically eliminate the detective work. This post covers the automation layer I built on GCP to detect, diagnose, and in many cases auto-remediate incidents before a human ever gets paged.

System Architecture

[Cloud Monitoring]
       |
       | Alert fires (Pub/Sub notification)
       ▼
[Cloud Functions]          ← Incident Orchestrator
       |
       ├── Enrich context  (logs, traces, recent deploys)
       ├── Classify type   (latency / error rate / saturation)
       ├── Check runbook   (known issue? auto-remediate)
       ├── Notify humans   (Slack + PagerDuty if needed)
       └── Create ticket   (Jira with pre-filled context)
             |
             ▼
       [Firestore]          ← Incident state store
       [Cloud Run]          ← Remediation actions
       [BigQuery]           ← Incident history + analytics

Step 1: Alert Routing via Pub/Sub

All Cloud Monitoring alerts route to a single Pub/Sub topic. The alert payload carries enough context to start triage immediately.

# Cloud Monitoring notification channel config (Terraform)
resource "google_monitoring_notification_channel" "pubsub" {
  display_name = "Incident Automation Topic"
  type         = "pubsub"

  labels = {
    topic = "projects/my-project/topics/monitoring-alerts"
  }
}

resource "google_monitoring_alert_policy" "high_error_rate" {
  display_name = "High Error Rate - API Service"
  combiner     = "OR"

  conditions {
    display_name = "Error rate > 1%"
    condition_threshold {
      filter          = "metric.type=\"custom.googleapis.com/sli/error_rate\" resource.type=\"k8s_container\""
      duration        = "120s"
      comparison      = "COMPARISON_GT"
      threshold_value = 0.01

      aggregations {
        alignment_period   = "60s"
        per_series_aligner = "ALIGN_RATE"
      }
    }
  }

  notification_channels = [google_monitoring_notification_channel.pubsub.name]

  alert_strategy {
    auto_close = "604800s"
  }
}

Step 2: The Incident Orchestrator (Cloud Functions)

This is the heart of the system. A Cloud Function subscribes to the alert topic and orchestrates the response:

import functions_framework
import json
import base64
from google.cloud import logging_v2, firestore, pubsub_v1
from datetime import datetime, timedelta
import requests

db = firestore.Client()
log_client = logging_v2.Client()

@functions_framework.cloud_event
def handle_alert(cloud_event):
    """Entry point: triggered by Pub/Sub alert message."""

    # Decode the alert payload
    data = base64.b64decode(cloud_event.data["message"]["data"]).decode()
    alert = json.loads(data)

    incident_id = f"inc_{cloud_event.id[:8]}"

    print(json.dumps({
        "message": "Incident received",
        "incident_id": incident_id,
        "policy": alert.get("incident", {}).get("policy_name"),
        "severity": "INFO"
    }))

    # Store incident state
    incident_ref = db.collection("incidents").document(incident_id)
    incident_ref.set({
        "id": incident_id,
        "alert": alert,
        "status": "investigating",
        "created_at": datetime.utcnow(),
        "timeline": []
    })

    # Run the triage pipeline
    context = enrich_context(alert, incident_id)
    classification = classify_incident(context)
    remediated = attempt_auto_remediation(classification, context, incident_id)

    if not remediated:
        notify_on_call(incident_id, context, classification)

    # Always log to BigQuery for trend analysis
    log_to_bigquery(incident_id, classification, context, remediated)


def enrich_context(alert: dict, incident_id: str) -> dict:
    """Gather all context needed for triage."""

    resource = alert.get("incident", {}).get("resource", {})
    service_name = resource.get("labels", {}).get("container_name", "unknown")

    context = {
        "service": service_name,
        "alert_time": alert.get("incident", {}).get("started_at"),
        "policy": alert.get("incident", {}).get("policy_name"),
        "recent_logs": fetch_recent_errors(service_name),
        "recent_deploys": fetch_recent_deploys(service_name),
        "upstream_health": check_upstream_dependencies(service_name),
    }

    # Add timeline entry
    db.collection("incidents").document(incident_id).update({
        "timeline": firestore.ArrayUnion([{
            "time": datetime.utcnow().isoformat(),
            "action": "context_enriched",
            "data": {"services_checked": list(context.keys())}
        }])
    })

    return context


def fetch_recent_errors(service_name: str) -> list:
    """Pull last 50 error logs from the past 15 minutes."""

    client = log_client
    now = datetime.utcnow()
    window_start = (now - timedelta(minutes=15)).strftime("%Y-%m-%dT%H:%M:%SZ")

    filter_str = f"""
        resource.type="k8s_container"
        resource.labels.container_name="{service_name}"
        severity>=ERROR
        timestamp>="{window_start}"
    """

    entries = list(client.list_entries(filter_=filter_str, max_results=50))

    return [
        {
            "timestamp": entry.timestamp.isoformat(),
            "message": entry.payload if isinstance(entry.payload, str) 
                       else entry.payload.get("message", ""),
            "severity": entry.severity,
        }
        for entry in entries
    ]


def fetch_recent_deploys(service_name: str) -> list:
    """Check Cloud Deploy history for recent changes."""

    # Query your deployment history — this example uses a Firestore deploys collection
    deploys_ref = db.collection("deploys") \
        .where("service", "==", service_name) \
        .where("deployed_at", ">=", datetime.utcnow() - timedelta(hours=4)) \
        .order_by("deployed_at", direction=firestore.Query.DESCENDING) \
        .limit(5)

    return [
        {
            "version": d.get("version"),
            "deployed_at": d.get("deployed_at").isoformat(),
            "deployed_by": d.get("deployed_by"),
        }
        for d in [doc.to_dict() for doc in deploys_ref.stream()]
    ]

Step 3: Incident Classification

Once I have context, I classify the incident type. This drives which remediation playbook to run:

from enum import Enum

class IncidentType(Enum):
    RECENT_DEPLOY_REGRESSION  = "recent_deploy_regression"
    DEPENDENCY_FAILURE        = "dependency_failure"
    RESOURCE_EXHAUSTION       = "resource_exhaustion"
    TRAFFIC_SPIKE             = "traffic_spike"
    UNKNOWN                   = "unknown"


def classify_incident(context: dict) -> dict:
    """
    Classify the incident based on enriched context.
    Returns classification with confidence score.
    """

    recent_deploys = context.get("recent_deploys", [])
    upstream_health = context.get("upstream_health", {})
    recent_logs = context.get("recent_logs", [])

    # Rule 1: Deploy in last 30 mins + errors started → deploy regression
    if recent_deploys:
        latest_deploy_age_mins = _minutes_ago(recent_deploys[0]["deployed_at"])
        if latest_deploy_age_mins < 30:
            return {
                "type": IncidentType.RECENT_DEPLOY_REGRESSION,
                "confidence": 0.85,
                "evidence": f"Deploy {recent_deploys[0]['version']} {latest_deploy_age_mins}m ago",
                "suggested_action": "rollback",
                "rollback_to": recent_deploys[1]["version"] if len(recent_deploys) > 1 else None,
            }

    # Rule 2: Upstream dependency unhealthy
    unhealthy_deps = [k for k, v in upstream_health.items() if not v.get("healthy")]
    if unhealthy_deps:
        return {
            "type": IncidentType.DEPENDENCY_FAILURE,
            "confidence": 0.90,
            "evidence": f"Unhealthy dependencies: {unhealthy_deps}",
            "suggested_action": "notify_dep_team",
            "dependencies": unhealthy_deps,
        }

    # Rule 3: OOM / CPU errors in logs
    oom_logs = [l for l in recent_logs if "OOMKilled" in l.get("message", "")
                                       or "out of memory" in l.get("message", "").lower()]
    if oom_logs:
        return {
            "type": IncidentType.RESOURCE_EXHAUSTION,
            "confidence": 0.80,
            "evidence": f"{len(oom_logs)} OOM events detected",
            "suggested_action": "scale_up",
        }

    return {
        "type": IncidentType.UNKNOWN,
        "confidence": 0.0,
        "suggested_action": "page_human",
    }


def _minutes_ago(iso_timestamp: str) -> float:
    from datetime import timezone
    ts = datetime.fromisoformat(iso_timestamp.replace("Z", "+00:00"))
    return (datetime.now(timezone.utc) - ts).total_seconds() / 60

Step 4: Auto-Remediation Actions

For high-confidence classifications, we attempt automated remediation:

def attempt_auto_remediation(classification: dict, context: dict, incident_id: str) -> bool:
    """
    Attempt automated fix. Returns True if remediation was applied.
    Only acts on high-confidence classifications.
    """

    confidence = classification.get("confidence", 0)
    action = classification.get("suggested_action")

    # Only auto-remediate above 80% confidence
    if confidence < 0.80:
        print(f"Confidence {confidence:.0%} too low for auto-remediation")
        return False

    success = False

    if action == "rollback":
        success = _execute_rollback(
            service=context["service"],
            target_version=classification.get("rollback_to"),
            incident_id=incident_id,
        )

    elif action == "scale_up":
        success = _scale_deployment(
            service=context["service"],
            scale_factor=2,
            incident_id=incident_id,
        )

    if success:
        _add_timeline(incident_id, "auto_remediation_applied", {
            "action": action,
            "confidence": confidence,
        })

        # Notify Slack but don't page — just inform
        _notify_slack(
            channel="#incidents",
            message=f"✅ Auto-remediated: `{action}` applied to `{context['service']}` "
                    f"(confidence: {confidence:.0%}). Monitoring...",
            incident_id=incident_id,
        )

    return success


def _execute_rollback(service: str, target_version: str, incident_id: str) -> bool:
    """Trigger Cloud Run or GKE rollback via Cloud Deploy."""

    import subprocess

    if not target_version:
        print("No rollback target version available")
        return False

    try:
        # GKE rollback via kubectl (in practice, use Cloud Deploy API)
        result = subprocess.run([
            "kubectl", "rollout", "undo",
            f"deployment/{service}",
            f"--to-revision={target_version}",
            "-n", "production"
        ], capture_output=True, text=True, timeout=60)

        if result.returncode == 0:
            print(f"Rollback to {target_version} succeeded for {service}")
            return True
        else:
            print(f"Rollback failed: {result.stderr}")
            return False

    except Exception as e:
        print(f"Rollback exception: {e}")
        return False

Step 5: Smart Notifications

When auto-remediation isn't triggered, we send a pre-filled context package to the on-call engineer instead of a bare alert:

def notify_on_call(incident_id: str, context: dict, classification: dict):
    """
    Send enriched incident brief to Slack + PagerDuty.
    The goal: engineer opens the alert and immediately knows where to look.
    """

    recent_deploy_msg = ""
    if context.get("recent_deploys"):
        d = context["recent_deploys"][0]
        recent_deploy_msg = f"\n• *Recent deploy:* `{d['version']}` by {d['deployed_by']} ({d['deployed_at']})"

    unhealthy = [k for k, v in context.get("upstream_health", {}).items() if not v.get("healthy")]
    dep_msg = f"\n• *Unhealthy deps:* {', '.join(unhealthy)}" if unhealthy else ""

    slack_blocks = [
        {
            "type": "header",
            "text": {"type": "plain_text", "text": f"🚨 Incident {incident_id}"}
        },
        {
            "type": "section",
            "text": {
                "type": "mrkdwn",
                "text": (
                    f"*Service:* `{context['service']}`\n"
                    f"*Classification:* {classification['type'].value} "
                    f"({classification['confidence']:.0%} confidence)\n"
                    f"*Evidence:* {classification.get('evidence', 'N/A')}"
                    f"{recent_deploy_msg}"
                    f"{dep_msg}"
                )
            }
        },
        {
            "type": "actions",
            "elements": [
                {
                    "type": "button",
                    "text": {"type": "plain_text", "text": "📋 View Runbook"},
                    "url": f"https://wiki.internal/runbooks/{classification['type'].value}"
                },
                {
                    "type": "button",
                    "text": {"type": "plain_text", "text": "📊 Open Dashboard"},
                    "url": f"https://console.cloud.google.com/monitoring?project=my-project"
                },
            ]
        }
    ]

    # Post to Slack
    requests.post(
        "https://hooks.slack.com/services/YOUR/WEBHOOK/URL",
        json={"blocks": slack_blocks}
    )

Results After 4 Months

Metric	Before	After
Mean time to detect (MTTD)	38 min	3 min
Mean time to resolve (MTTR)	51 min	10 min
Auto-remediated incidents	0%	42%
Incidents requiring escalation	100%	31%
On-call pages per week	47	11
Engineer time on incidents	~8hr/week	~1.5hr/week

The 42% auto-remediation rate was higher than we expected. Turns out, "recent deploy broke something" and "dependency flaked" together account for most incidents — and both are very automatable.

Safety Guardrails We Added

Automation at this level requires careful guardrails:

Confidence threshold: Never auto-remediate below 80% confidence. Human judgment kicks in for ambiguous cases.
Blast radius limits: Auto-rollback only if the deployment was in the last 30 minutes. Older issues get human review.
Audit trail: Every automated action writes to Firestore and BigQuery. Full immutable history.
Circuit breaker on the automation itself: If auto-remediation fails twice in a row for the same service, disable it and page a human.
Business hours sensitivity: During peak business hours, we lower the confidence threshold to 90% (more conservative). At 3am, 80% is acceptable.

What I'd Do Differently

Invest in runbook quality before automation. You're essentially codifying your runbooks. If the runbooks are bad, the automation is bad.
Start with notification enrichment, not remediation. Even just sending pre-filled context to on-call engineers cut our MTTR by 40% before we automated a single action.
Build the BigQuery incident history early. Trend analysis on incident types drove which automations to prioritize. Without data, we would have automated the wrong things first.

Resources

What percentage of your incidents are auto-remediated? I'd love to compare numbers — drop a comment below.

Tags: #sre #googlecloud #incidentmanagement #automation #gcp #devops #platform

Scaling SRE Systems with GCP + Kubernetes: Lessons from Running at 10x Traffic

Ayush Raj Jha — Wed, 18 Mar 2026 20:00:24 +0000

How we redesigned our reliability engineering stack on Google Kubernetes Engine — and the SLO framework that changed how our team thinks about uptime.

Context

Eighteen months ago, our SRE team was managing reliability the old-fashioned way: dashboards nobody looked at, on-call rotations driven by alert fatigue, and postmortems that produced action items nobody followed up on.

Then we 10x'd our traffic during a product launch. Everything broke, not catastrophically, but in that slow, grinding way that's actually worse. Cascading latency. Partial outages. Customer-facing errors that took us 40 minutes to even notice.

We rebuilt everything. This is what we built, and more importantly, why.

The Stack

Layer	Technology	Purpose
Orchestration	GKE Autopilot	Container management, autoscaling
Service Mesh	Istio on GKE	Traffic management, mTLS, observability
Metrics	Cloud Monitoring + Prometheus	SLI/SLO tracking
Tracing	Cloud Trace + OpenTelemetry	Distributed request tracing
Logging	Cloud Logging + Log-based metrics	Structured logs, alerting
Incident Management	PagerDuty + custom runbooks	On-call, escalation

Part 1: Defining SLOs That Actually Mean Something

The most important thing we did wasn't technical, it was agreeing on what "reliable" means.

We adopted the Google SRE Book's framework: define SLIs (what you measure), set SLOs (your target), track your error budget (how much you can fail).

Here's our core SLI/SLO definitions in code, using the Prometheus recording rules format we deploy to GKE:

# slo-rules.yaml
groups:
  - name: slo_rules
    interval: 30s
    rules:

      # SLI: Availability (% of requests that succeed)
      - record: sli:availability:ratio_rate5m
        expr: |
          sum(rate(http_requests_total{status!~"5.."}[5m]))
          /
          sum(rate(http_requests_total[5m]))

      # SLI: Latency (% of requests under 300ms)
      - record: sli:latency:ratio_rate5m
        expr: |
          sum(rate(http_request_duration_seconds_bucket{le="0.3"}[5m]))
          /
          sum(rate(http_request_duration_seconds_count[5m]))

      # Error budget remaining (SLO = 99.9% availability over 30 days)
      - record: slo:error_budget:remaining
        expr: |
          1 - (
            (1 - sli:availability:ratio_rate5m) / (1 - 0.999)
          )

And the corresponding alerting policy:

# alert-rules.yaml
groups:
  - name: slo_alerts
    rules:

      # Burn rate alert: fast burn (2% budget in 1 hour)
      - alert: HighErrorBudgetBurn
        expr: |
          slo:error_budget:remaining < 0.02
          and
          slo:error_budget:remaining offset 1h > 0.02
        for: 2m
        labels:
          severity: critical
          team: sre
        annotations:
          summary: "Burning error budget at critical rate"
          description: "2% of monthly error budget consumed in 1 hour. Current availability: {{ $value | humanizePercentage }}"
          runbook: "https://wiki.internal/runbooks/high-error-budget-burn"

      # Slow burn alert: 5% budget in 6 hours
      - alert: ElevatedErrorBudgetBurn
        expr: |
          slo:error_budget:remaining < 0.05
          and
          slo:error_budget:remaining offset 6h > 0.05
        for: 15m
        labels:
          severity: warning
          team: sre
        annotations:
          summary: "Elevated error budget consumption"
          runbook: "https://wiki.internal/runbooks/elevated-burn"

💡 Key insight: Before SLOs, we'd page on-call for any spike in errors, even if it lasted 30 seconds. Burn-rate alerting means we only wake people up when budget consumption threatens our monthly target. On-call load dropped 70%.

Part 2: GKE Autopilot for Hands-Off Infrastructure

We moved from self-managed GKE Standard to GKE Autopilot, and it was the right call for our scale.

Autopilot handles node provisioning, sizing, and security hardening automatically. You define what you need; GKE figures out how to run it.

Here's our typical service deployment manifest:

# service-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-service
  namespace: production
  labels:
    app: api-service
    version: v2.3.1
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api-service
  template:
    metadata:
      labels:
        app: api-service
        version: v2.3.1
      annotations:
        # Prometheus scraping
        prometheus.io/scrape: "true"
        prometheus.io/port: "8080"
        prometheus.io/path: "/metrics"
    spec:
      containers:
        - name: api-service
          image: gcr.io/my-project/api-service:v2.3.1
          ports:
            - containerPort: 8080

          # Resource requests drive Autopilot node sizing
          resources:
            requests:
              cpu: "500m"
              memory: "512Mi"
            limits:
              cpu: "2000m"
              memory: "2Gi"

          # Readiness/liveness probes are non-negotiable
          readinessProbe:
            httpGet:
              path: /healthz/ready
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 5
            failureThreshold: 3

          livenessProbe:
            httpGet:
              path: /healthz/live
              port: 8080
            initialDelaySeconds: 30
            periodSeconds: 10
            failureThreshold: 3

          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace

---
# Horizontal Pod Autoscaler
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-service-hpa
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api-service
  minReplicas: 3
  maxReplicas: 50
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 70
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 60
      policies:
        - type: Percent
          value: 100
          periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300   # 5 min cooldown before scaling down

💡 Lesson: Set scaleDown.stabilizationWindowSeconds to at least 300 seconds. We had premature scale-down during traffic waves that caused latency spikes as pods restarted. The 5-minute cooldown eliminated this.

Part 3: Istio for Traffic Management and Reliability Patterns

Istio gives us reliability primitives such as circuit breaking, retries, timeouts, at the infrastructure layer, so services don't have to implement them individually.

# istio-destinationrule.yaml — circuit breaking
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: api-service
  namespace: production
spec:
  host: api-service
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 100
      http:
        http1MaxPendingRequests: 100
        http2MaxRequests: 1000

    # Circuit breaker
    outlierDetection:
      consecutiveGatewayErrors: 5
      interval: 30s
      baseEjectionTime: 30s
      maxEjectionPercent: 50
      minHealthPercent: 30

---
# istio-virtualservice.yaml — retries + timeouts
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: api-service
  namespace: production
spec:
  hosts:
    - api-service
  http:
    - route:
        - destination:
            host: api-service
            port:
              number: 8080
      timeout: 10s
      retries:
        attempts: 3
        perTryTimeout: 3s
        retryOn: "gateway-error,connect-failure,retriable-4xx"

💡 Lesson: Be careful with retry configuration. Retrying on 5xx without thought can amplify load on an already-struggling downstream service. We scope retries to idempotent operations only and never retry POST endpoints that modify state.

Part 4: Structured Logging for Faster Incident Response

When you're in an incident, log quality is everything. We standardized on structured JSON logs emitted via our shared logging library:

import json
import logging
import time
from contextvars import ContextVar

trace_id_var: ContextVar[str] = ContextVar("trace_id", default="")
span_id_var: ContextVar[str] = ContextVar("span_id", default="")

class StructuredLogger:
    def __init__(self, service_name: str):
        self.service = service_name

    def _log(self, level: str, message: str, **kwargs):
        entry = {
            "timestamp": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
            "severity": level,
            "message": message,
            "service": self.service,
            "trace_id": trace_id_var.get(),
            "span_id": span_id_var.get(),
            **kwargs
        }
        print(json.dumps(entry))

    def info(self, message: str, **kwargs):
        self._log("INFO", message, **kwargs)

    def error(self, message: str, **kwargs):
        self._log("ERROR", message, **kwargs)

    def warn(self, message: str, **kwargs):
        self._log("WARNING", message, **kwargs)


# Usage
log = StructuredLogger("payment-service")

log.info(
    "Payment processed",
    user_id="u_12345",
    amount_cents=4999,
    currency="USD",
    duration_ms=142,
)

log.error(
    "Payment gateway timeout",
    user_id="u_12345",
    gateway="stripe",
    timeout_ms=3000,
    retry_attempt=2,
)

Cloud Logging automatically parses these JSON entries. We then create log-based metrics and wire them into our SLO dashboards:

# Create a log-based metric for payment errors
gcloud logging metrics create payment_errors \
  --description="Count of payment processing errors" \
  --log-filter='
    resource.type="k8s_container"
    jsonPayload.service="payment-service"
    jsonPayload.severity="ERROR"
  '

Results

After 6 months running this stack at 10x traffic:

Metric	Before	After
Mean time to detect (MTTD)	38 minutes	4 minutes
Mean time to resolve (MTTR)	2.1 hours	22 minutes
On-call pages per week	47	9
Monthly availability	99.2%	99.94%
Error budget consumed (avg)	N/A	18%

What I'd Do Differently

Adopt OpenTelemetry from day one. We had a mix of Cloud Trace and a legacy Zipkin setup. Migrating mid-scale was painful.
Define SLOs before writing a line of infra code. The technical decisions flow naturally once you know what you're optimizing for.
Invest in runbook quality early. Our fastest incident resolutions happen when the runbook is good enough that a junior engineer can follow it at 3am without escalating.

Resources

What's your team's error budget policy? Do you freeze deploys when budget hits a threshold? I'd love to compare notes in the comments.

Tags: #sre #kubernetes #gke #googlecloud #devops #platform #reliability

How I Built a Real-Time Data Pipeline Using Google Cloud (And What I Learned the Hard Way)

Ayush Raj Jha — Wed, 18 Mar 2026 19:56:26 +0000

A practical walkthrough of building a production-grade streaming pipeline with Pub/Sub, Dataflow, and BigQuery — including the mistakes I made so you don't have to.

The Problem

Our platform was generating millions of user events per day — clicks, purchases, errors, session data. We were batch-processing everything nightly with a cron job and an aging ETL script. By the time analysts ran their morning reports, the "live" data was already 8 hours stale.

The business need was clear: we needed real-time insights. A customer cancels their subscription — we want to trigger a retention workflow now, not tomorrow morning.

This is the story of how I built that pipeline on Google Cloud, what worked, what blew up in production, and what I'd do differently.

Architecture Overview

[App Servers]
     |
     ▼
[Cloud Pub/Sub]              ← ingestion layer, durable buffer
     |
     ▼
[Dataflow (Apache Beam)]     ← stream processing, transformations
     |
     ├──► [BigQuery]         → analytics warehouse
     ├──► [Cloud Bigtable]   → low-latency lookups
     └──► [Pub/Sub (out)]    → trigger downstream systems

Each layer has a single responsibility. Let me walk through each one.

Step 1: Event Ingestion with Cloud Pub/Sub

Pub/Sub is Google's managed messaging service — think fully-managed Kafka without the operational overhead.

from google.cloud import pubsub_v1
import json

publisher = pubsub_v1.PublisherClient()
topic_path = publisher.topic_path("my-project", "user-events")

def publish_event(event: dict):
    data = json.dumps(event).encode("utf-8")
    future = publisher.publish(
        topic_path,
        data,
        event_type=event.get("type", "unknown"),
        source_service=event.get("service", "unknown"),
    )
    return future.result()

# Example
publish_event({
    "user_id": "u_12345",
    "type": "subscription_cancelled",
    "timestamp": "2024-03-15T10:23:45Z",
    "plan": "pro",
    "reason": "too_expensive"
})

💡 Lesson #1: Always set message attributes. You'll want to filter downstream without deserializing the full payload.

💡 Lesson #2: Pub/Sub guarantees at-least-once delivery. Your consumers must be idempotent. We got burned when a retry storm caused duplicate records in BigQuery.

Step 2: Stream Processing with Dataflow

Dataflow is Google's managed Apache Beam runner. You write the pipeline once; Dataflow handles autoscaling and fault tolerance.

import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions
from apache_beam.io.gcp.pubsub import ReadFromPubSub
from apache_beam.io.gcp.bigquery import WriteToBigQuery
import json
from datetime import datetime

class ParseAndEnrich(beam.DoFn):
    def process(self, message):
        try:
            event = json.loads(message.data.decode("utf-8"))
            event["ingestion_timestamp"] = datetime.utcnow().isoformat()
            event["pipeline_version"] = "2.1.0"

            if not event.get("user_id"):
                beam.metrics.Metrics.counter("pipeline", "invalid_events").inc()
                return

            yield event
        except json.JSONDecodeError:
            yield beam.pvalue.TaggedOutput("dead_letter", message)


def run_pipeline():
    options = PipelineOptions([
        "--project=my-project",
        "--runner=DataflowRunner",
        "--region=us-central1",
        "--streaming",
        "--enable_streaming_engine",
    ])

    with beam.Pipeline(options=options) as p:
        events = (
            p
            | "Read Pub/Sub" >> ReadFromPubSub(
                subscription="projects/my-project/subscriptions/user-events-sub",
                with_attributes=True,
            )
            | "Parse & Enrich" >> beam.ParDo(ParseAndEnrich()).with_outputs(
                "dead_letter", main="valid"
            )
        )

        (
            events.valid
            | "1-min Windows" >> beam.WindowInto(beam.transforms.window.FixedWindows(60))
            | "Write BQ" >> WriteToBigQuery(
                table="my-project:analytics.user_events",
                schema="AUTO_DETECT",
                write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND,
                method=WriteToBigQuery.Method.STREAMING_INSERTS,
            )
        )

        (
            events.dead_letter
            | "Write DLQ" >> beam.io.WriteToPubSub(
                topic="projects/my-project/topics/dead-letter"
            )
        )

💡 Lesson #3: Always implement a dead-letter queue. In week one, ~0.3% of events had malformed timestamps from a mobile client bug. Without a DLQ, they would have silently failed.

💡 Lesson #4: Windowing is a latency/cost tradeoff. Moving from 5-minute to 1-minute windows raised our Dataflow costs ~40%. Know your SLA before choosing.

Step 3: BigQuery as the Analytics Layer

CREATE TABLE analytics.user_events (
    event_id        STRING NOT NULL,
    user_id         STRING NOT NULL,
    event_type      STRING NOT NULL,
    timestamp       TIMESTAMP NOT NULL,
    ingestion_ts    TIMESTAMP NOT NULL,
    session_id      STRING,
    properties      JSON
)
PARTITION BY DATE(timestamp)
CLUSTER BY event_type, user_id;

💡 Lesson #5: Partition + cluster from day one. Our query costs dropped 60% after adding proper clustering.

💡 Lesson #6: Use insertId for deduplication — BigQuery's dedup window is only ~1 minute:

row_to_insert = {
    "insertId": event["event_id"],
    "json": event
}

Step 4: Observability

class ProcessEvents(beam.DoFn):
    def __init__(self):
        self.events_processed = beam.metrics.Metrics.counter("pipeline", "events_processed")
        self.latency = beam.metrics.Metrics.distribution("pipeline", "event_latency_ms")

    def process(self, event):
        import time
        start = time.time()
        # ... processing logic ...
        self.latency.update(int((time.time() - start) * 1000))
        self.events_processed.inc()
        yield event

Alert	Threshold	Action
Pub/Sub oldest message age	> 5 minutes	Page on-call
Dataflow worker errors	> 1% of events	Slack alert
BigQuery streaming errors	Any	PagerDuty
Dead-letter queue depth	> 1000 messages	Investigate

Results After 3 Months in Production

Latency: 8 hours → under 90 seconds end-to-end
Reliability: 99.97% of events land in BigQuery successfully
Cost: ~$340/month for ~50M events/day
Developer velocity: analysts self-serve on real-time data without waiting for nightly runs

What I'd Do Differently

Start with a schema registry from day one. Multiple teams publishing inconsistent schemas cost weeks of debugging.
Use Dataflow Flex Templates earlier. Classic Templates hit limitations fast.
Set Pub/Sub retention to 7 days minimum. We had it at 1 day in staging; when a bug caused a processing halt, we lost messages we could have replayed.

Resources

Have you built something similar? I'd love to hear how you approached deduplication — drop a comment below.

Tags: #googlecloud #dataengineering #gcp #bigquery #dataflow #pubsub #streaming

The Healthcare Cloud Reliability Crisis Nobody Wants to Talk About (And What I Learned Fixing It)

Ayush Raj Jha — Sat, 14 Mar 2026 18:22:25 +0000

Why 99.91% Uptime Is a Clinical Problem: Redesigning Healthcare Cloud Infrastructure

By Ayush Raj Jha, Senior Software Engineer, Oracle Health

Let me start with a number that should bother you: 99.91% uptime.

On its face, that sounds impressive. Nine-nines adjacent. Marketing teams love it. But let me translate it into clinical language: at 99.91% availability, a healthcare system running 24/7 experiences roughly 8 hours of unplanned downtime per year. Eight hours where a nurse cannot pull a medication history. Eight hours where a physician cannot access lab results during an active patient care decision. Eight hours where the EHR that the entire care delivery workflow depends on simply does not respond.

That was the baseline when I started redesigning Oracle Health's cloud infrastructure. It was, per Oracle's own SLA, below the contractually required threshold. And it was, according to a surprising number of healthcare CIOs I have spoken with, considered roughly normal for enterprise EHR cloud deployments.

I want to talk about why that happened, what the actual technical failure modes look like, and what it took to get from 99.91% to 99.98% — which sounds like a rounding error but represents a more than 7x reduction in downtime incidents when you do the math on a system serving 200+ hospitals and an estimated 8–10 million patients annually.

More importantly, I want to talk about the architectural assumptions baked into most healthcare cloud infrastructure that make this problem worse than it needs to be, and why the path forward is harder than most vendors are willing to admit.

Why Healthcare Cloud Is Structurally Different (And Why Most Cloud Architects Get This Wrong)

I spent three years before Oracle working at Motorola Solutions on real-time public safety systems — specifically, building the algorithms that govern how law enforcement devices communicate with dispatch centers in time-critical operational conditions. That background shaped how I think about uptime in a way that a pure cloud engineering background does not.

In public safety, system availability is not a SLA metric. It is an operational constraint. A dispatch system that goes down during an active incident does not generate a help-desk ticket — it potentially costs lives. The architectural philosophy that comes from working in that environment is fundamentally different from the one that comes from building, say, a high-traffic e-commerce platform.

Healthcare infrastructure should work the same way. It largely does not.

The core problem is that most healthcare cloud architectures were not designed from first principles for clinical availability requirements. They were designed by cloud engineers who are excellent at building scalable, cost-effective distributed systems, and then adapted for healthcare by layering HIPAA compliance controls on top. That sequence matters enormously.

When you design for scalability and cost-efficiency first, you make a set of architectural tradeoffs that are perfectly reasonable for most enterprise software: you accept some tolerance for transient failures, you design for eventual consistency in exchange for throughput, you use asynchronous replication because synchronous replication is expensive and adds latency. Then you try to bolt clinical availability requirements on afterward, and you discover that those earlier tradeoffs have become load-bearing walls.

The specific failure modes I encountered at Oracle Health were predictable once you understand this pattern.

The Three Core Failure Modes

Failure Mode 1: Single-Region Deployment with Manual Failover

When I started working on Oracle Health's disaster recovery architecture, the recovery process for a major regional failure involved manual intervention steps measured in hours. The documented RTO was technically within contract bounds, but the actual time-to-recovery in real-world failover exercises was routinely 4–6 hours.

That gap between documented and actual recovery time is extraordinarily common in healthcare cloud environments, and almost nobody talks about it publicly because nobody wants to be the one holding that comparison.

The reason for the gap is almost always the same: the documented process assumes everything works as expected during the failure event. Real failures are messy. Runbooks have steps that were written six months ago and have not been tested against the current infrastructure state. The engineer who designed the failover procedure left the company. The third-party dependency that the procedure assumed would be available during a regional outage turns out to also be regional.

Failure Mode 2: Deployment Processes That Require Extended Maintenance Windows

When I audited Oracle Health's instance deployment process, it was taking 14–16 hours. This is not unusual — I have seen healthcare cloud vendors with deployment windows that run longer. The reason matters: healthcare cloud deployments are complex. They involve data migration, schema changes, compliance validation, FHIR endpoint reconfiguration, integration engine updates, and a cascade of dependency checks that each have to complete in sequence.

The clinical consequence of a 14–16 hour deployment window is that hospitals are either taking extended planned downtime for upgrades, or they are deferring upgrades and running on older software versions to avoid the disruption. Both are bad. Extended planned downtime is genuinely painful for clinical operations. Deferred upgrades mean hospitals are not getting security patches and clinical improvement updates in anything like a reasonable timeframe.

Failure Mode 3: Infrastructure Cost Structures That Punish Reliability

This is the one that I find most interesting from a systems design perspective, because it creates a perverse incentive at the architectural level.

High-availability infrastructure is expensive. Multi-region active-passive setups require you to provision and maintain a standby environment that sits largely idle under normal conditions. Synchronous replication adds latency and compute cost. Automated failover systems require continuous testing and maintenance overhead.

The consequence in practice: healthcare cloud vendors face constant pressure to reduce infrastructure costs, and the easiest place to cut is the reliability infrastructure that only matters when something goes wrong. You can defer the investment in proper disaster recovery for a long time before it visibly costs you anything — right until the moment a regional AWS outage hits during a shift change at a major academic medical center.

What I Actually Built: A Technical Walkthrough

Automated Deployment: Getting From 14 Hours to 2 Hours

The 14–16 hour deployment time was not a single problem. It was twelve smaller problems that had accumulated over several years of the deployment process growing organically without a systematic redesign.

The core architectural change was decomposing the monolithic deployment pipeline into parallelized stages with dependency-aware orchestration. The original process was essentially a sequential script — step 1 must complete before step 2 begins, regardless of whether step 2 actually depends on step 1's output. When you map the actual dependency graph, you discover that a significant portion of the steps can run in parallel.

The second major change was separating stateful from stateless deployment components. Schema migrations and data transformations are stateful — they must complete and be validated before clinical workloads can use the updated schema. Most other configuration and service deployment steps are stateless and can proceed independently. Once you have made that separation explicit in your deployment orchestration, you can run the stateless components in parallel with the stateful ones rather than waiting for everything to serialize.

The third change was building idempotent deployment steps. This sounds obvious in 2025, but a surprising number of the original deployment steps were not idempotent — a failed deployment at step 8 of 40 left the environment in a partially updated state that required manual remediation before the deployment could be retried. Idempotent steps mean that a deployment failure is a retry problem, not a remediation problem.

Result: deployment time from 14–16 hours to 2–3 hours — an 85% reduction achieved almost entirely through orchestration redesign rather than hardware investment.

The cost implication is $2.8 million in annual savings from reduced deployment infrastructure overhead and engineering time. But the clinical implication matters more: hospitals that were previously scheduling 12-hour maintenance windows for upgrades can now upgrade during a 3-hour overnight window with significantly less disruption to operations.

Multi-Region Disaster Recovery: The Architecture of Sub-15-Minute RTO

The recovery time objective I was asked to achieve was aggressive: under 15 minutes from failure detection to full clinical system availability. At the time, the healthcare IT community's conventional wisdom was that sub-15-minute cloud RTO for EHR systems was not achievable at acceptable cost.

The conventional wisdom was wrong, but it was wrong for an interesting reason: it was based on the cost structure of active-passive disaster recovery architectures, which are genuinely expensive to operate correctly. The path to sub-15-minute RTO at acceptable cost runs through a different architectural model.

The key architectural insight is that you do not need a fully provisioned standby environment to achieve fast failover — you need a pre-warmed, partial environment that can be rapidly scaled to full capacity once a failover is triggered. Full standby provisioning means paying for idle compute 100% of the time, while pre-warmed partial provisioning means paying for a smaller baseline that scales on-demand.

The specific implementation involves three components working together:

Component 1: Continuous Asynchronous Replication with Synchronous Checkpointing for Critical Data

Not all clinical data has the same RPO requirement. Medication administration records, active order sets, and real-time monitoring data need near-zero RPO — you cannot lose recent medication administration records in a failover event. Historical imaging data, archived encounter notes, and reporting data can tolerate a longer RPO.

The architecture separates these tiers explicitly, applying synchronous replication only to the zero-RPO tier (accepting the latency and cost overhead where it clinically matters) and asynchronous replication to everything else. This requires careful data classification work to implement correctly, and the classification has to be maintained as the application evolves — an ongoing operational discipline, not a one-time architectural decision.

Component 2: Automated Failure Detection with Graduated Response Tiers

The difference between a 15-minute RTO and a 4-hour RTO is often not the failover mechanism itself — it is how long it takes to detect that a failover is necessary and make the decision to execute it. In many healthcare environments, the detection-to-decision path runs through multiple human approval steps that each add minutes.

The automated detection system uses layered health checks at the infrastructure, application, and clinical workflow levels, with predefined escalation tiers that trigger automated failover for unambiguous failure conditions and page on-call engineers for ambiguous conditions. The trigger criteria are defined in advance and tested regularly, so the decision to failover is not made under pressure during an active incident — it is made in advance and executed automatically.

Component 3: Infrastructure-as-Code with Immutable Deployment Artifacts

One of the consistent failure modes in disaster recovery exercises is configuration drift: the standby environment that was provisioned three months ago no longer matches the production environment because production has been updated and the standby has not. When you try to fail over to it, the configuration mismatch causes failures that were not present in the last DR test.

The solution is treating the standby environment as an immutable artifact that is rebuilt from code on a defined schedule rather than maintained through incremental updates. Every production infrastructure change committed to version control automatically triggers an update to the standby environment definition. Failover, when it happens, is deploying a known-good artifact to a pre-warmed environment — not trying to bring a potentially stale environment up to current state under time pressure.

Achieved RTO in production failover exercises: consistently under 12 minutes. RPO for zero-RPO tier: under 3 minutes.

Infrastructure Cost Optimization: The 22% Reduction

The 22% annual reduction came primarily from three sources specific to healthcare cloud infrastructure at scale.

Source 1: Workload-Aware Resource Scheduling

Healthcare EHR workloads have highly predictable temporal patterns. Clinical workflows are heavily concentrated during morning rounds, shift changes, and medication administration peaks — and much lighter overnight and on weekends. Standard cloud auto-scaling responds to these patterns reactively, scaling up after demand has already increased.

Workload-aware scheduling uses historical pattern data to pre-scale proactively: provision for the expected morning peak before it arrives, and de-provision during known low-utilization windows. The model requires careful calibration to avoid under-provisioning during unexpected demand spikes — you need statistical confidence intervals on the demand forecasts, not just point estimates.

Source 2: Network Topology Optimization

Data transfer costs are consistently underestimated in healthcare cloud cost modeling because they are often not visible until they show up on the monthly invoice. Healthcare systems move a lot of data: HL7 and FHIR messages, DICOM imaging studies, audit logs, backup streams. In a naively architected multi-region environment, a significant fraction of this data crosses region boundaries unnecessarily.

The optimization involves auditing the actual data flow patterns (not the assumed ones), identifying data being transferred cross-region but that could be served from a regional cache or replica, and restructuring the routing accordingly.

Source 3: Storage Tier Rationalization

Clinical data has a well-defined lifecycle: active patient encounter data has very different access patterns from historical encounter data from five years ago, which has different patterns from archived data from fifteen years ago. Storage costs drop dramatically as you move down the tiers, and the performance difference is clinically irrelevant for data that is accessed rarely.

The work here is not technically complex — it is operationally complex. You have to get agreement on retention policies, validate that the tiering logic respects HIPAA's minimum retention requirements, test that retrieval from lower-cost storage tiers meets the latency requirements for the workflows that occasionally need it, and build monitoring to catch cases where data is incorrectly tiered. That operational complexity is why this work tends not to get done despite being straightforward in principle.

The Uncomfortable Conclusion: This Is a People Problem as Much as a Technology Problem

Here is what I did not expect when I started this work: the hardest part was not the architecture. The hardest part was the organizational dynamics around accepting that the existing architecture was inadequate.

Healthcare organizations are, reasonably, extremely risk-averse about infrastructure changes. The existing systems, whatever their flaws, are known quantities. The IT team knows how they fail, has runbooks for the common failure modes, and has learned to work around the limitations. A proposal to redesign the disaster recovery architecture from scratch is a proposal to take on a large, complex project that will introduce new failure modes that nobody has seen before — in an environment where a failure has direct clinical consequences.

The way I navigated this was through incremental validation: designing the new architecture to be testable in isolation before being deployed in production, running parallel operation for a defined period with automatic rollback capability, and being extremely transparent about the failure modes we were introducing alongside the ones we were eliminating.

The sub-15-minute RTO did not get deployed because I convinced a room full of healthcare CIOs that my architecture was theoretically sound. It got deployed because we ran 47 failover exercises in the test environment, documented the failure modes we encountered and how we resolved them, and built a track record of consistent behavior before asking anyone to trust it in production.

That process takes time. It takes organizational patience. It is the part of this work that does not fit in a conference talk or an architecture diagram. But it is the part that actually determines whether the technology works in the environment it was designed for.

Where This Is Heading: The Problems That Are Not Solved Yet

I want to be honest about the frontier here, because I think the field tends toward premature declarations of solved problems.

The zero-RPO problem for real-time clinical applications is not solved at acceptable cost. Sub-5-minute RPO is achievable. Zero RPO — no data loss under any failure condition — requires synchronous multi-region replication for all clinical data, which adds latency that is clinically problematic for real-time applications like surgical monitoring and ICU alerting. The engineering path to zero RPO with acceptable latency is not obvious, and I do not think anyone has solved it in production at scale.

Predictive failure detection is immature. The current state of the art in healthcare cloud is reactive: we detect failures when they occur and respond quickly. Predicting infrastructure failures before they cause clinical impact — using ML against telemetry data to identify failure precursors — is a research problem that has not translated into production healthcare cloud systems in any serious way. This is the next frontier, and it is genuinely hard.

The compliance-reliability tension is getting worse, not better. Regulatory requirements in healthcare cloud are accumulating faster than the infrastructure tooling to satisfy them efficiently. Automated compliance posture management at the scale of a large EHR cloud deployment remains an unsolved operational problem, and the cost of manual compliance monitoring is substantial and growing.

The 8 hours of annual downtime that I started with is not a technical inevitability. It is the consequence of architectural decisions that prioritized cost and scalability over clinical availability, made at a time when the healthcare IT industry had not yet internalized what cloud reliability requirements actually mean for care delivery.

Fixing it requires being honest about the failure modes — including the organizational ones — and being willing to do work that is genuinely hard and does not have a clean vendor-supplied answer.

The number I am working toward is not 99.99%. It is zero unplanned downtime events. That is a different goal, and it requires a different architectural philosophy: one that treats availability as a constraint to be satisfied rather than a metric to be maximized.

We are not there yet. But the gap between where we are and where we need to be is engineering, not magic.

Ayush Raj Jha is a Senior Software Engineer at Oracle Health and a former Software Engineer at Motorola Solutions, where he built real-time optimization systems for public safety infrastructure. He holds IEEE Senior Member status and has published peer-reviewed work on applied ML, Internet of Things (IoT), and cloud infrastructure at IEEE-affiliated venues. He writes about the gap between how mission-critical systems are designed and how they actually behave under operational conditions.

Building an AI-Powered SRE Incident Response Workflow With AWS Strands Agents

Ayush Raj Jha — Sat, 14 Mar 2026 18:11:30 +0000

By Ayush Raj Jha, Senior Software Engineer, Oracle Health | Former Software Engineer, Motorola Solutions

Learn how to automate CloudWatch alerts, Kubernetes remediation, and incident reporting using multi-agent AI workflows with the AWS Strands Agents SDK.

The SRE Incident Response Agent is a multi-agent sample that ships with the AWS Strands Agents SDK. It automatically discovers active CloudWatch alarms, performs AI-powered root cause analysis using Claude Sonnet 4 on Amazon Bedrock, proposes Kubernetes or Helm remediations, and posts a structured incident report to Slack.

This guide covers everything you need to clone the repo and run it yourself.

Prerequisites

Before you begin, make sure the following are in place:

Python 3.11+ installed on your machine
AWS credentials configured (aws configure or an active IAM role)
Amazon Bedrock access enabled for Claude Sonnet 4 in your target region
kubectl and helm v3 installed — only required if you plan to run live remediations. Dry-run mode works without them.

Step 1: Clone the Repository

The sample lives inside the strands-agents/samples open source repository. Clone it and navigate to the SRE agent directory:

git clone https://github.com/strands-agents/samples.git
cd samples/02-samples/sre-incident-response-agent

The directory contains the following files:

sre-incident-response-agent/
├── sre_agent.py           # Main agent: 4 agents + 8 tools
├── test_sre_agent.py      # Pytest unit tests (12 tests, mocked AWS)
├── requirements.txt
├── .env.example
└── README.md

Step 2: Create a Virtual Environment and Install Dependencies

python -m venv .venv
source .venv/activate        # Windows: .venv\Scripts\activate
pip install -r requirements.txt

The requirements.txt pins the core dependencies:

strands-agents>=0.1.0
strands-agents-tools>=0.1.0
boto3>=1.38.0
botocore>=1.38.0

Step 3: Configure Environment Variables

Copy .env.example to .env and fill in your values:

cp .env.example .env

Open .env and set the following:

# AWS region where your CloudWatch alarms live
AWS_REGION=us-east-1

# Amazon Bedrock model ID (Claude Sonnet 4 is the default)
BEDROCK_MODEL_ID=us.anthropic.claude-sonnet-4-20250514-v1:0

# DRY_RUN=true means kubectl/helm commands are printed, not executed.
# Set to false only when you are ready for live remediations.
DRY_RUN=true

# Optional: post the incident report to Slack.
# Leave blank to print to stdout instead.
SLACK_WEBHOOK_URL=

Step 4: Grant IAM Permissions

The agent needs read-only access to CloudWatch alarms, metric statistics, and log events. No write permissions to CloudWatch are required. Attach the following policy to the IAM role or user running the agent:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": [
      "cloudwatch:DescribeAlarms",
      "cloudwatch:GetMetricStatistics",
      "logs:FilterLogEvents",
      "logs:DescribeLogGroups"
    ],
    "Resource": "*"
  }]
}

Step 5: Run the Agent

There are two ways to trigger the agent.

Option A: Automatic Alarm Discovery

Let the agent discover all active CloudWatch alarms on its own. This is the recommended mode for a real on-call scenario:

python sre_agent.py

Option B: Targeted Investigation

Pass a natural-language description of the triggering event. The agent will focus its investigation on the service and symptom you describe:

python sre_agent.py "High CPU alarm fired on ECS service my-api in prod namespace"

Example Output

Running the targeted trigger above produces output similar to the following:

Starting SRE Incident Response
   Trigger: High CPU alarm fired on ECS service my-api in prod namespace

[cloudwatch_agent] Fetching active alarms...
  Found alarm: my-api-HighCPU (CPUUtilization > 85% for 5m)
  Metric stats: avg 91.3%, max 97.8% over last 30 min
  Log events: 14 OOMKilled events in /ecs/my-api

[rca_agent] Performing root cause analysis...
  Root cause: Memory leak causing CPU spike as GC thrashes
  Severity: P2 - single service, <5% of users affected
  Recommended fix: Rolling restart to clear heap; monitor for recurrence

[remediation_agent] Applying remediation...
  [DRY-RUN] kubectl rollout restart deployment/my-api -n prod

================================================================
*[P2] SRE Incident Report - 2025-10-14 09:31 UTC*

What happened: CloudWatch alarm my-api-HighCPU fired at 09:18 UTC.
CPU reached 97.8% (threshold 85%). 14 OOMKilled events in 15 min.

Root cause: Memory leak in application heap leading to aggressive GC,
causing CPU saturation. Likely introduced in the last deployment.

Remediation: Rolling restart of deployment/my-api in namespace prod
initiated (dry-run). All pods will be replaced with fresh instances.

Follow-up:
  - Monitor CPUUtilization for next 30 min
  - Review recent commits for memory allocation changes
  - Consider setting memory limits in the Helm chart
================================================================

Running the Tests (No AWS Credentials Required)

The sample ships with 12 pytest unit tests that mock boto3 entirely. You can run the full test suite in any environment, including CI, without any AWS credentials:

pip install pytest pytest-mock
pytest test_sre_agent.py -v

# Expected: 12 passed

Enabling Live Remediation

Once you have validated the agent's behaviour in dry-run mode and are satisfied with the decisions it makes, you can enable live kubectl and helm execution by setting DRY_RUN=false in your .env file:

DRY_RUN=false

Conclusion

In under five minutes of setup, the AWS Strands Agents SDK gives you a working multi-agent incident response loop: alarm discovery, AI-powered root cause analysis, Kubernetes remediation, and a structured incident report, all driven by a single python sre_agent.py command. The dry-run default means there is no risk in running it against a real environment while you evaluate its reasoning.

From here, the natural next steps are connecting a Slack webhook for team notifications, adding a PagerDuty tool for incident tracking, or extending the RCA agent with a vector store of past postmortems. All of that is a tool definition away.

Ayush Raj Jha is a Senior Software Engineer at Oracle Health and former Software Engineer at Motorola Solutions, where he built real-time optimization and device intelligence systems for public safety at scale. He holds IEEE Senior Member status and publishes peer-reviewed research on Healthcare, IoT, Cloud Infrastructure and mission-critical systems at IEEE-affiliated venues.