DEV Community: Samson Tanimawo

Cost-Effective Observability: The 80/20 Stack for Startups

Samson Tanimawo — Tue, 01 Sep 2026 13:26:41 +0000

You Don't Need Datadog (Yet)

I see startups spending $5,000/month on Datadog with 8 engineers. That's $625 per engineer per month for monitoring. At that stage, you need that money for product development.

Here's the observability stack that costs under $200/month and covers 80% of what you need.

The Stack

Metrics:    Prometheus + Grafana (free, self-hosted on K8s)
Logs:       Loki (free, self-hosted) or CloudWatch ($)
Tracing:    OpenTelemetry → Jaeger (free, self-hosted)
Alerting:   Alertmanager → PagerDuty free tier
Status:     Upptime (free, GitHub-based)
─────────────────────────────────────────────
Total:      ~$150/month (PagerDuty + infrastructure)

Setup: 4 Hours, Not 4 Weeks

Step 1: Prometheus + Grafana (1 hour)

# helm install
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install monitoring prometheus-community/kube-prometheus-stack \
  --set prometheus.prometheusSpec.retention=7d \
  --set prometheus.prometheusSpec.resources.requests.memory=512Mi \
  --set prometheus.prometheusSpec.resources.limits.memory=1Gi

This gives you:

Node metrics (CPU, memory, disk)
Pod metrics
K8s state metrics
Pre-built Grafana dashboards

Step 2: Application Metrics (30 minutes)

# Add to your Python app
from prometheus_client import Counter, Histogram, start_http_server

REQUESTS = Counter('http_requests_total', 'Total requests', ['method', 'endpoint', 'status'])
LATENCY = Histogram('http_request_duration_seconds', 'Request latency', ['endpoint'])

@app.middleware
async def metrics_middleware(request, call_next):
    start = time.time()
    response = await call_next(request)
    LATENCY.labels(endpoint=request.url.path).observe(time.time() - start)
    REQUESTS.labels(
        method=request.method,
        endpoint=request.url.path,
        status=response.status_code
    ).inc()
    return response

# Expose /metrics endpoint
start_http_server(9090)

Step 3: Log Aggregation with Loki (1 hour)

helm install loki grafana/loki-stack \
  --set loki.persistence.enabled=true \
  --set loki.persistence.size=10Gi \
  --set promtail.enabled=true

Loki stores logs indexed by labels (like Prometheus for logs). Way cheaper than Elasticsearch.

Step 4: Essential Alerts (1 hour)

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: startup-essential-alerts
spec:
  groups:
    - name: essential
      rules:
        # Is the app up?
        - alert: ServiceDown
          expr: up{job="my-app"} == 0
          for: 2m
          labels: { severity: critical }

        # Is it slow?
        - alert: HighLatency
          expr: histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) > 1
          for: 5m
          labels: { severity: warning }

        # Is it erroring?
        - alert: HighErrorRate
          expr: rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) > 0.05
          for: 5m
          labels: { severity: critical }

        # Is the disk filling?
        - alert: DiskAlmostFull
          expr: (node_filesystem_avail_bytes / node_filesystem_size_bytes) < 0.1
          for: 10m
          labels: { severity: warning }

        # Is memory tight?
        - alert: HighMemoryUsage
          expr: container_memory_working_set_bytes / container_spec_memory_limit_bytes > 0.9
          for: 5m
          labels: { severity: warning }

Five alerts. That's all you need to start. Add more as you learn what breaks.

When to Upgrade

Stage          Team Size    Stack                    Monthly Cost
─────────────  ─────────    ──────────────────       ────────────
Pre-product    1-5 eng      Prometheus+Grafana+Loki  ~$150
Product-market 5-15 eng     Add Jaeger+PagerDuty     ~$500
Scaling        15-30 eng    Consider managed service  ~$2,000
Growth         30-50 eng    Datadog/New Relic/etc     ~$5,000+
Enterprise     50+ eng      Full platform             $10,000+

Don't skip stages. Each stage's stack is right for that stage.

The Anti-Pattern

Don't build a custom monitoring platform. I've seen three startups try this. All three eventually bought Datadog anyway, having wasted 6+ months of engineering time.

Use off-the-shelf tools. Configure them well. Move on to building your product.

If you want production-grade observability without the enterprise price tag, check out what we're building at Nova AI Ops.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Incident Communication: The Status Page That Builds Trust

Samson Tanimawo — Tue, 01 Sep 2026 00:25:34 +0000

Silence Destroys Trust

During our worst outage, we went 35 minutes without updating the status page. Twitter filled the void. Theories ranged from data breach to bankruptcy. A customer posted "They're probably shutdown" which got 200 retweets.

The incident was a 20-minute database migration gone wrong. The reputation damage lasted months.

The Update Cadence

status_page_update_cadence:
  P1_customer_facing:
    first_update: within 5 minutes of detection
    subsequent_updates: every 15 minutes
    post_resolution: within 30 minutes

  P2_degraded_service:
    first_update: within 15 minutes
    subsequent_updates: every 30 minutes
    post_resolution: within 2 hours

  P3_minor_impact:
    first_update: within 30 minutes
    subsequent_updates: every hour
    post_resolution: next business day

The rule: If there's nothing new to say, say "We're still investigating. Next update in 15 minutes." Silence is worse than no news.

Writing Good Status Updates

What NOT to Write

Bad examples:
  "We're experiencing issues."              (vague)
  "Our database is overloaded."              (too technical)
  "A junior engineer deployed bad code."     (blaming)
  "Everything should be fine soon."           (no commitment)
  "We're working on it."                      (says nothing)

What TO Write

Good example:
  "Some users are experiencing slow page loads when 
  accessing their dashboard. Our team has identified 
  the cause and is deploying a fix. We expect full 
  resolution within 20 minutes. API access and core 
  functionality are not affected."

Structure:
  1. What users see (symptom)
  2. What we're doing (action)
  3. When it'll be fixed (ETA)
  4. What's NOT affected (scope)

The Template System

We pre-built templates for common scenarios:

templates = {
    'investigating': {
        'title': '{service} — Investigating {symptom}',
        'body': (
            'We are investigating reports of {symptom} '
            'affecting {service}. Some users may experience '
            '{user_impact}. We are actively working to identify '
            'the cause and will provide an update within '
            '{next_update_minutes} minutes.'
        )
    },
    'identified': {
        'title': '{service} — Issue Identified',
        'body': (
            'We have identified the cause of {symptom} '
            'affecting {service}. {root_cause_plain}. '
            'Our team is implementing a fix. We expect '
            'resolution by {eta}. {scope_statement}'
        )
    },
    'monitoring': {
        'title': '{service} — Fix Deployed, Monitoring',
        'body': (
            'A fix for {symptom} has been deployed. '
            'We are monitoring to confirm full resolution. '
            '{service} should be functioning normally. '
            'If you continue to experience issues, please '
            'contact support. We will provide a final update '
            'within {next_update_minutes} minutes.'
        )
    },
    'resolved': {
        'title': '{service} — Resolved',
        'body': (
            'The issue affecting {service} has been fully resolved. '
            'Total impact duration: {duration}. {affected_count} '
            'were affected. We will publish a detailed incident '
            'report within {report_timeline}. We apologize for '
            'any inconvenience.'
        )
    }
}

Internal vs External Communication

Internal (Slack/war room):    External (status page/Twitter):
─────────────────────────     ──────────────────────────────
Technical details              User-facing impact
"DB connection pool exhausted" "Some users may see errors"
Specific error messages        Expected resolution time
Code-level discussion          What's NOT affected
Blame-free technical discussion Empathetic, professional tone

The Post-Incident Report

Publish a public post-incident report within 48 hours for P1/P2:

# Incident Report: Dashboard Loading Issues — March 15, 2024

## Summary
On March 15 from 2:15 PM to 2:42 PM UTC, approximately 30% of 
users experienced slow or failed dashboard loading. The issue 
was caused by a database configuration change that reduced 
connection pool capacity.

## Timeline
- 2:15 PM UTC — Monitoring detected elevated error rates
- 2:18 PM — Engineering team began investigation
- 2:25 PM — Root cause identified (connection pool misconfiguration)
- 2:32 PM — Fix deployed
- 2:42 PM — Full resolution confirmed

## Impact
- Duration: 27 minutes
- Users affected: ~30% (dashboard only)
- Data loss: None
- Other services: Not affected

## Root Cause
A planned configuration update inadvertently reduced the database
connection pool from 100 to 10 connections.

## What We're Doing to Prevent This
1. Adding automated validation for configuration changes
2. Implementing canary deployment for config updates
3. Adding connection pool monitoring with alerting threshold

We sincerely apologize for the inconvenience and are committed 
to maintaining the reliability you expect from us.

Transparency builds trust. Every company that handles incidents well publishes honest post-mortems.

If you want automated incident communication with AI-generated status updates, check out what we're building at Nova AI Ops.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Load Testing in Production: How We Do It Safely

Samson Tanimawo — Mon, 31 Aug 2026 14:02:17 +0000

Why Staging Load Tests Lie

We ran perfect load tests in staging. 10,000 requests per second, sub-100ms latency, zero errors. Then we launched and fell over at 3,000 RPS in production.

Why? Staging had different data (100 rows vs 10 million), different traffic patterns (uniform vs Zipfian), and different infrastructure (single AZ vs multi-AZ).

Staging load tests are theater. Production load tests find real problems.

The Safety Framework

production_load_test_rules:
  # Rule 1: Always use real traffic patterns, not uniform
  traffic_pattern: "replay_production_logs"  # Not synthetic

  # Rule 2: Gradual ramp-up, never step function
  ramp_up:
    start: "10% of current traffic"
    increment: "10% every 5 minutes"
    max: "200% of current traffic"

  # Rule 3: Automatic abort conditions
  abort_if:
    error_rate: "> 1%"
    p99_latency: "> 2x baseline"
    cpu_utilization: "> 90%"
    any_circuit_breaker: "open"

  # Rule 4: Business hours only
  allowed_window: "Tue-Thu 10am-2pm CT"

  # Rule 5: Team readiness
  requires:
    - ic_designated: true
    - rollback_plan: documented
    - stakeholders_notified: true

Our Load Testing Tool: k6

// k6 load test script
import http from 'k6/http';
import { check, sleep } from 'k6';
import { Rate, Trend } from 'k6/metrics';

const errorRate = new Rate('errors');
const latency = new Trend('request_latency');

export const options = {
  stages: [
    { duration: '5m', target: 100 },   // Ramp to 100 RPS
    { duration: '5m', target: 200 },   // Ramp to 200 RPS
    { duration: '5m', target: 500 },   // Ramp to 500 RPS
    { duration: '10m', target: 500 },  // Hold at 500 RPS
    { duration: '5m', target: 1000 },  // Push to 1000 RPS
    { duration: '10m', target: 1000 }, // Hold at 1000 RPS
    { duration: '5m', target: 0 },     // Ramp down
  ],
  thresholds: {
    'http_req_duration': ['p(99)<500'],  // p99 under 500ms
    'errors': ['rate<0.01'],              // Error rate under 1%
  },
};

export default function () {
  // Simulate real user behavior with weighted endpoints
  const endpoints = [
    { url: '/api/products', weight: 40 },
    { url: '/api/search?q=shoes', weight: 25 },
    { url: '/api/cart', weight: 20 },
    { url: '/api/checkout', weight: 10 },
    { url: '/api/user/profile', weight: 5 },
  ];

  const rand = Math.random() * 100;
  let cumWeight = 0;

  for (const endpoint of endpoints) {
    cumWeight += endpoint.weight;
    if (rand <= cumWeight) {
      const res = http.get(`https://api.production.com${endpoint.url}`);
      check(res, {
        'status is 200': (r) => r.status === 200,
        'latency < 500ms': (r) => r.timings.duration < 500,
      });
      errorRate.add(res.status !== 200);
      latency.add(res.timings.duration);
      break;
    }
  }

  sleep(Math.random() * 2);  // Random think time
}

The Abort System

def monitor_load_test(test_id):
    """Continuously monitor and auto-abort if thresholds breached."""
    while test_is_running(test_id):
        metrics = get_current_metrics()

        if metrics['error_rate'] > 0.01:
            abort_load_test(test_id, reason="Error rate exceeded 1%")
            alert_team(f"Load test aborted: error rate {metrics['error_rate']:.2%}")
            return

        if metrics['p99_latency'] > metrics['baseline_p99'] * 2:
            abort_load_test(test_id, reason="Latency 2x above baseline")
            alert_team(f"Load test aborted: p99 {metrics['p99_latency']}ms")
            return

        if any(cb.state == 'OPEN' for cb in get_circuit_breakers()):
            abort_load_test(test_id, reason="Circuit breaker opened")
            alert_team(f"Load test aborted: circuit breaker tripped")
            return

        time.sleep(10)  # Check every 10 seconds

What We Learned

Production load tests revealed:

Connection pool exhaustion at 800 RPS — Invisible in staging (different pool settings)
Cache stampede during cold start — Staging cache was always warm
Database lock contention on inventory updates — 100 rows vs 10M rows
CDN rate limiting at 500 RPS from single AZ — Never tested multi-origin
Memory leak under sustained load — Only visible after 30+ minutes

None of these showed up in staging. All of them would have caused production outages.

Results

Before production load testing:
  Incidents caused by scale issues: 3-4/quarter
  Time to find scaling bottleneck: Days (post-incident)

After:
  Incidents caused by scale issues: 0-1/quarter
  Time to find scaling bottleneck: Hours (proactive)

If you want AI-powered load testing that predicts breaking points before they happen, check out what we're building at Nova AI Ops.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Effective On-Call Rotations: Lessons From Building Fair Schedules

Samson Tanimawo — Sun, 30 Aug 2026 23:30:31 +0000

The Rotation Nobody Wants

Our on-call rotation was a spreadsheet. Updated manually. Someone always got scheduled during their vacation. Two people occasionally got double-booked. Holidays were a battleground.

Designing Fair Rotations

Principle 1: Equal Burden Distribution

Track total on-call hours, not just shift count:

def calculate_oncall_burden(engineer, period_days=90):
    shifts = get_shifts(engineer, period_days)
    return {
        'total_hours': sum(s.duration_hours for s in shifts),
        'weekend_hours': sum(s.duration_hours for s in shifts if s.is_weekend),
        'holiday_hours': sum(s.duration_hours for s in shifts if s.is_holiday),
        'night_hours': sum(s.duration_hours for s in shifts if s.is_night),
        'pages_received': sum(s.page_count for s in shifts),
        'burden_score': calculate_weighted_score(shifts)
    }

def calculate_weighted_score(shifts):
    """Weight different types of on-call differently."""
    score = 0
    for s in shifts:
        base = s.duration_hours
        if s.is_weekend: base *= 1.5
        if s.is_holiday: base *= 2.0
        if s.is_night: base *= 1.3
        score += base
    return round(score, 1)

Principle 2: Respect Preferences

onCall_preferences:
  alice:
    blackout_dates: ["2024-03-25", "2024-04-01:2024-04-05"]  # Vacation
    preferred_days: ["Mon", "Tue", "Wed"]  # Family on weekends
    max_consecutive_days: 3

  bob:
    blackout_dates: ["2024-04-10"]
    preferred_days: ["any"]
    max_consecutive_days: 7
    prefers_weekends: true  # Weekend differential pay

Principle 3: Minimum Pool Size

The math on sustainable rotations:

Pool size    Frequency           Burnout risk
3 people     1 week on / 2 off   HIGH — unsustainable
4 people     1 week on / 3 off   MEDIUM — barely okay
5 people     1 week on / 4 off   LOW — comfortable
6+ people    1 week on / 5+ off  MINIMAL — ideal

Rule: Minimum 5 people per rotation.
If you have fewer, reduce on-call scope or hire.

Principle 4: Escalation Tiers

escalation_chain:
  tier_1:  # Primary on-call
    response_time: 5 minutes
    scope: all pages

  tier_2:  # Secondary on-call (backup)
    response_time: 15 minutes
    scope: escalated or unacknowledged

  tier_3:  # Engineering manager
    response_time: 30 minutes
    scope: P1 only or when both T1+T2 unavailable

  tier_4:  # CTO/VP Engineering
    response_time: 60 minutes
    scope: Extended P1 (>1 hour), customer escalation

The Override System

Life happens. Make swaps easy:

def request_swap(requesting_engineer, target_date, volunteer=None):
    """Allow easy on-call swaps."""

    if volunteer:
        # Direct swap: Alice asks Bob to cover
        execute_swap(requesting_engineer, volunteer, target_date)
        notify_team(f"{requesting_engineer} swapped with {volunteer} for {target_date}")
    else:
        # Open request: Alice needs coverage, anyone can take it
        post_to_channel(
            f"{requesting_engineer} needs coverage for {target_date}. "
            f"Reply to volunteer. Comp: standard on-call rate."
        )

# Key: NO manager approval needed for swaps
# This reduces friction dramatically

Holiday Fairness

The holiday rotation is separate and tracked year-over-year:

holidays_2024 = [
    'New Years', 'MLK Day', 'Presidents Day', 'Memorial Day',
    'July 4th', 'Labor Day', 'Thanksgiving', 'Christmas'
]

def assign_holidays(team, year):
    # Get historical holiday assignments
    history = get_holiday_history(team, years=3)

    # Sort by who has covered the FEWEST holidays recently
    sorted_team = sorted(team, key=lambda e: history.get(e, 0))

    assignments = {}
    for i, holiday in enumerate(holidays_2024):
        engineer = sorted_team[i % len(sorted_team)]
        assignments[holiday] = engineer

    return assignments

Metrics We Track

Metric	Target	Current
Burden score variance	< 15%	8%
Swap request fulfillment	> 95%	98%
Pages per shift (average)	< 3	1.8
NPS for on-call experience	> 0	+32
Holiday coverage fairness	< 1 shift variance	0.5

If you want AI-powered on-call scheduling that optimizes for fairness automatically, check out what we're building at Nova AI Ops.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

GitOps for Infrastructure: How We Deploy With Zero SSH

Samson Tanimawo — Sun, 30 Aug 2026 13:15:52 +0000

The Last Time I Used SSH

I haven't SSH'd into a production server in 14 months. Not because I'm lazy — because our infrastructure doesn't require it.

GitOps changed everything.

What GitOps Actually Means

GitOps = Git is the single source of truth for your infrastructure. Every change goes through a PR. No manual kubectl, no SSH, no ClickOps.

Traditional:     Developer → kubectl apply → Cluster
GitOps:          Developer → Git PR → CI Review → ArgoCD → Cluster

Our Setup

Repositories:
├── app-service-a/          # Application code + Dockerfile
├── app-service-b/          # Application code + Dockerfile
└── infrastructure/          # All K8s manifests
    ├── base/               # Shared configurations
    │   ├── namespaces/
    │   ├── network-policies/
    │   └── rbac/
    ├── services/
    │   ├── api-service/
    │   │   ├── deployment.yaml
    │   │   ├── service.yaml
    │   │   ├── hpa.yaml
    │   │   └── kustomization.yaml
    │   └── payment-service/
    └── environments/
        ├── staging/
        │   └── kustomization.yaml
        └── production/
            └── kustomization.yaml

ArgoCD Configuration

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: production-services
spec:
  project: default
  source:
    repoURL: https://github.com/org/infrastructure
    targetRevision: main
    path: environments/production
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true        # Delete resources removed from Git
      selfHeal: true     # Revert manual changes
    syncOptions:
      - CreateNamespace=true

The key: selfHeal: true. If someone manually changes something, ArgoCD reverts it within 3 minutes. Git is the truth.

The Deployment Flow

# 1. Developer pushes code to app repo
# 2. CI builds image: api-service:sha-abc123
# 3. CI updates infrastructure repo:

# Automated PR to infrastructure repo
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-service
spec:
  template:
    spec:
      containers:
        - name: api
          image: registry/api-service:sha-abc123  # Updated by CI

# GitHub Actions: Update image tag
- name: Update manifest
  run: |
    cd infrastructure
    kustomize edit set image api-service=registry/api-service:${{ github.sha }}
    git add .
    git commit -m "deploy: api-service ${{ github.sha }}"
    git push

Benefits We've Seen

1. Complete Audit Trail

# Who changed what and when?
git log --oneline environments/production/
# a1b2c3d deploy: api-service sha-abc123 (2024-03-15)
# d4e5f6g feat: add rate limiting to api (2024-03-14)
# g7h8i9j fix: increase memory limit for payment (2024-03-13)

2. Easy Rollbacks

# Rollback = revert a commit
git revert HEAD
git push
# ArgoCD detects change, reverts cluster to previous state
# Total time: ~90 seconds

3. Environment Parity

# environments/staging/kustomization.yaml
bases:
  - ../../services/api-service
patchesStrategicMerge:
  - replicas-patch.yaml  # 1 replica instead of 3

# environments/production/kustomization.yaml  
bases:
  - ../../services/api-service
patchesStrategicMerge:
  - replicas-patch.yaml  # 3 replicas
  - resources-patch.yaml # More CPU/memory

Same base, different overlays. Drift between environments becomes impossible.

4. Disaster Recovery

# Cluster dies? No problem.
# 1. Provision new cluster
# 2. Install ArgoCD
# 3. Point ArgoCD at Git repo
# 4. Everything reconverges automatically

# Recovery time: ~15 minutes (cluster provisioning)
# Data loss: zero (Git has everything)

The Cultural Shift

The hardest part wasn't technical. It was convincing engineers to stop using kubectl directly.

Old way: "I'll just quickly fix this in production" (kubectl edit)
New way: "I'll open a PR to fix this" (5 minutes longer, 100% safer)

After 3 months, nobody missed kubectl. The safety and audit trail are worth the extra 5 minutes.

If you want GitOps with AI-powered drift detection and automated remediation, check out what we're building at Nova AI Ops.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Prometheus at Scale: Surviving the Cardinality Cliff

Samson Tanimawo — Sun, 30 Aug 2026 01:15:43 +0000

The Day Prometheus Fell Over

Prometheus memory usage spiked from 8GB to 32GB overnight. OOM-killed. Monitoring was down for 20 minutes while we scrambled.

The cause? Someone added a request_path label to an HTTP metric. With 50,000 unique paths, that one label created 500,000 new time series.

This is the cardinality cliff.

Understanding Cardinality

Cardinality = number of unique time series. Every unique combination of metric name + label values = one time series.

http_requests_total{method="GET", status="200", service="api"}
http_requests_total{method="GET", status="404", service="api"}
http_requests_total{method="POST", status="200", service="api"}

That's 3 time series. Manageable. But:

http_requests_total{method="GET", status="200", service="api", user_id="u1"}
http_requests_total{method="GET", status="200", service="api", user_id="u2"}
... (100,000 users)

That's 100,000 time series from ONE metric. Multiply by methods, status codes, and services — you're at millions.

Detection: Finding High-Cardinality Metrics

# Top 20 metrics by series count
topk(20, count by (__name__) ({__name__=~".+"}))

Or from the Prometheus API:

curl -s http://prometheus:9090/api/v1/status/tsdb | \
  jq '.data.seriesCountByMetricName | sort_by(.value | tonumber) | reverse | .[0:20]'

Prevention: The Label Policy

We enforce a label policy:

label_policy:
  allowed_high_cardinality:
    - pod_name       # Bounded by cluster size
    - container_name # Bounded by deployments
    - node_name      # Bounded by cluster size
    - namespace      # Bounded and controlled

  forbidden_labels:
    - user_id        # Unbounded
    - request_id     # Unbounded  
    - email          # Unbounded
    - ip_address     # Unbounded
    - url_path       # Semi-unbounded

  max_cardinality_per_metric: 10000
  review_required_above: 5000

Mitigation: Relabeling Rules

# prometheus.yml
scrape_configs:
  - job_name: 'api-service'
    metric_relabel_configs:
      # Drop high-cardinality labels before ingestion
      - source_labels: [__name__]
        regex: 'http_request_duration_.*'
        action: labeldrop
        regex: 'url_path'

      # Aggregate URL paths into categories
      - source_labels: [url_path]
        regex: '/api/v1/users/.*'
        target_label: url_path
        replacement: '/api/v1/users/:id'

      # Drop metrics we don't need at all
      - source_labels: [__name__]
        regex: 'go_gc_.*|process_virtual_.*'
        action: drop

Scaling Strategy: Federation + Thanos

Architecture:

  Service A ──→ Prometheus-1 ──→ Thanos Sidecar ──→ Object Storage
  Service B ──→ Prometheus-1                            ↓
  Service C ──→ Prometheus-2 ──→ Thanos Sidecar ──→ Thanos Store
  Service D ──→ Prometheus-2                            ↓
                                                   Thanos Query
                                                        ↓
                                                    Grafana

# Thanos sidecar on each Prometheus
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus-with-thanos
spec:
  template:
    spec:
      containers:
        - name: prometheus
          args:
            - '--storage.tsdb.retention.time=6h'  # Short local retention
            - '--storage.tsdb.min-block-duration=2h'
            - '--storage.tsdb.max-block-duration=2h'
        - name: thanos-sidecar
          args:
            - 'sidecar'
            - '--objstore.config-file=/etc/thanos/objstore.yml'

Local Prometheus keeps 6 hours. Thanos handles long-term storage in S3. Query across all Prometheus instances through Thanos Query.

Recording Rules: The Performance Multiplier

# Pre-compute expensive queries
groups:
  - name: service-level-aggregations
    interval: 30s
    rules:
      - record: service:http_request_duration:p99_5m
        expr: histogram_quantile(0.99, sum(rate(http_request_duration_bucket[5m])) by (service, le))

      - record: service:http_error_rate:5m
        expr: sum(rate(http_requests_total{status=~"5.."}[5m])) by (service) / sum(rate(http_requests_total[5m])) by (service)

      - record: service:http_requests_rate:5m
        expr: sum(rate(http_requests_total[5m])) by (service)

Dashboards query recording rules (instant) instead of raw metrics (slow).

If you want monitoring that handles scale automatically without cardinality headaches, check out what we're building at Nova AI Ops.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Database Reliability: The SRE Approach to Keeping Data Safe

Samson Tanimawo — Sat, 29 Aug 2026 13:41:35 +0000

The Backup That Wasn't

We had backups. Daily snapshots to S3. Perfectly configured. Never tested.

When we needed to restore after a data corruption incident, we discovered the backups had been silently failing for 3 weeks. The S3 bucket policy had changed, and nobody noticed.

Never again.

Rule 1: Test Your Restores

Backups don't matter. Restores matter.

#!/bin/bash
# weekly-restore-test.sh — runs every Sunday at 3am

TEST_DB="restore_test_$(date +%Y%m%d)"

# Step 1: Download latest backup
aws s3 cp s3://backups/prod-db/latest.sql.gz /tmp/restore-test.sql.gz
if [ $? -ne 0 ]; then
  alert "CRITICAL: Cannot download backup from S3"
  exit 1
fi

# Step 2: Restore to test database
gunzip -c /tmp/restore-test.sql.gz | psql -h test-db-host -U admin -d $TEST_DB
if [ $? -ne 0 ]; then
  alert "CRITICAL: Backup restore failed"
  exit 1
fi

# Step 3: Validate data
PROD_COUNT=$(psql -h prod-db -t -c "SELECT count(*) FROM users")
TEST_COUNT=$(psql -h test-db-host -d $TEST_DB -t -c "SELECT count(*) FROM users")

DIFF=$(( PROD_COUNT - TEST_COUNT ))
if [ $DIFF -gt 100 ]; then
  alert "WARNING: Restored DB has $DIFF fewer rows than production"
fi

# Step 4: Cleanup
psql -h test-db-host -c "DROP DATABASE $TEST_DB"
rm /tmp/restore-test.sql.gz

notify "Backup restore test PASSED. Rows: prod=$PROD_COUNT test=$TEST_COUNT"

We run this weekly. It's caught 4 backup issues in the past year.

Rule 2: Monitor Replication Lag

-- PostgreSQL replication lag
SELECT 
  client_addr,
  state,
  pg_wal_lsn_diff(pg_current_wal_lsn(), sent_lsn) AS send_lag_bytes,
  pg_wal_lsn_diff(sent_lsn, write_lsn) AS write_lag_bytes,
  pg_wal_lsn_diff(write_lsn, flush_lsn) AS flush_lag_bytes,
  pg_wal_lsn_diff(flush_lsn, replay_lsn) AS replay_lag_bytes
FROM pg_stat_replication;

Alert on:

Replication lag > 1 second (warning)
Replication lag > 10 seconds (critical)
Replica disconnected (critical)

Rule 3: Connection Pool Management

# pgbouncer.ini
[pgbouncer]
pool_mode = transaction     # Not session!
max_client_conn = 1000
default_pool_size = 25
reserve_pool_size = 5
reserve_pool_timeout = 3
server_idle_timeout = 600

# Monitoring
stats_period = 60

Without a connection pooler, every application connection holds a full PostgreSQL process. With 10 services each opening 20 connections, that's 200 PostgreSQL backends. With PgBouncer in transaction mode, you need maybe 25.

Rule 4: Query Performance Monitoring

-- Find the worst queries
SELECT 
  queryid,
  calls,
  round(total_exec_time::numeric, 2) AS total_ms,
  round(mean_exec_time::numeric, 2) AS avg_ms,
  round((100 * total_exec_time / sum(total_exec_time) OVER())::numeric, 2) AS pct,
  left(query, 100) AS query_preview
FROM pg_stat_statements
ORDER BY total_exec_time DESC
LIMIT 10;

Run this daily. The top 3 queries usually account for 60%+ of database load.

Rule 5: Safe Schema Migrations

# Dangerous: Locks table for entire ALTER
ALTER TABLE orders ADD COLUMN tracking_id VARCHAR(100);

# Safe: Use CREATE INDEX CONCURRENTLY
CREATE INDEX CONCURRENTLY idx_orders_tracking ON orders(tracking_id);

# Safe migration pattern:
# 1. Add nullable column (instant, no lock)
ALTER TABLE orders ADD COLUMN tracking_id VARCHAR(100);

# 2. Backfill in batches (no lock)
UPDATE orders SET tracking_id = generate_tracking_id(id) 
WHERE id BETWEEN 1 AND 10000;

# 3. Add NOT NULL constraint when ready (requires ACCESS EXCLUSIVE briefly)
ALTER TABLE orders ALTER COLUMN tracking_id SET NOT NULL;

The Database SRE Dashboard

Panel 1: Active connections vs max_connections (saturation)
Panel 2: Query latency p50/p95/p99
Panel 3: Replication lag (seconds)
Panel 4: Transactions per second
Panel 5: Cache hit ratio (should be > 99%)
Panel 6: Disk usage + growth rate + days until full

If your cache hit ratio drops below 99%, you either need more memory or have a query pattern problem.

If you want AI-powered database monitoring that predicts issues before they impact users, check out what we're building at Nova AI Ops.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Container Security for SREs: The Practical Checklist

Samson Tanimawo — Sat, 29 Aug 2026 01:20:27 +0000

Security Is Part of Reliability

SREs think about availability, latency, and throughput. But a security breach is just another type of incident — often the worst kind. Here's the container security checklist I use.

The Base Image Problem

# Bad: 800MB image with everything including gcc
FROM ubuntu:22.04
RUN apt-get update && apt-get install -y python3 python3-pip
COPY . /app
RUN pip install -r requirements.txt

# Good: 50MB image with only what's needed
FROM python:3.11-slim AS builder
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

FROM python:3.11-slim
COPY --from=builder /usr/local/lib/python3.11/site-packages /usr/local/lib/python3.11/site-packages
COPY . /app
USER nobody

Smaller image = smaller attack surface. The multi-stage build removes build tools from the final image.

The Security Checklist

1. Image Scanning

# GitHub Actions: Scan before pushing
- name: Scan image for vulnerabilities
  uses: aquasecurity/trivy-action@master
  with:
    image-ref: 'myapp:${{ github.sha }}'
    format: 'table'
    exit-code: '1'           # Fail build on HIGH/CRITICAL
    severity: 'HIGH,CRITICAL'
    ignore-unfixed: true      # Only fail on fixable vulns

2. Non-Root Container

# Always run as non-root
RUN addgroup --system app && adduser --system --ingroup app app
USER app

# Kubernetes: Enforce non-root
securityContext:
  runAsNonRoot: true
  runAsUser: 1000
  readOnlyRootFilesystem: true
  allowPrivilegeEscalation: false
  capabilities:
    drop: ["ALL"]

3. Network Policies

# Default deny all traffic
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress

---
# Allow only specific traffic
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-service-policy
spec:
  podSelector:
    matchLabels:
      app: api-service
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: nginx-ingress
      ports:
        - port: 8080
  egress:
    - to:
        - podSelector:
            matchLabels:
              app: postgres
      ports:
        - port: 5432

4. Secrets Management

# Bad: Secrets in environment variables
env:
  - name: DB_PASSWORD
    value: "super-secret-password"  # Visible in pod spec!

# Good: Secrets from external vault
env:
  - name: DB_PASSWORD
    valueFrom:
      secretKeyRef:
        name: db-credentials
        key: password

# Best: Secrets injected from Vault
annotations:
  vault.hashicorp.com/agent-inject: "true"
  vault.hashicorp.com/role: "api-service"
  vault.hashicorp.com/agent-inject-secret-db: "secret/data/db"

5. Resource Limits (Security Aspect)

# Without limits, a compromised container can consume all resources
resources:
  requests:
    cpu: 100m
    memory: 128Mi
  limits:
    cpu: 500m
    memory: 256Mi
    ephemeral-storage: 100Mi  # Prevent disk filling attacks

6. Pod Security Standards

# Enforce restricted security standard at namespace level
apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

The Audit Automation

I run this weekly:

#!/bin/bash
# weekly-security-audit.sh

echo "=== Image Age Check ==="
kubectl get pods -A -o json | jq -r '.items[] | .spec.containers[] | .image' | sort -u | while read img; do
  age=$(skopeo inspect docker://$img 2>/dev/null | jq -r '.Created')
  echo "$img — built: $age"
done

echo "=== Privileged Containers ==="
kubectl get pods -A -o json | jq -r '.items[] | select(.spec.containers[].securityContext.privileged == true) | .metadata.namespace + "/" + .metadata.name'

echo "=== Containers Running as Root ==="
kubectl get pods -A -o json | jq -r '.items[] | select(.spec.securityContext.runAsNonRoot != true) | .metadata.namespace + "/" + .metadata.name'

echo "=== Missing Resource Limits ==="
kubectl get pods -A -o json | jq -r '.items[] | select(.spec.containers[].resources.limits == null) | .metadata.namespace + "/" + .metadata.name'

Every finding becomes a ticket. No exceptions.

If you want automated security monitoring for your container infrastructure, check out what we're building at Nova AI Ops.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

The Incident Commander Role: Running Incidents Without Chaos

Samson Tanimawo — Fri, 28 Aug 2026 13:11:17 +0000

Everyone's Debugging, Nobody's Leading

Five engineers in an incident channel. All debugging independently. Nobody coordinating. Three people checking the same dashboard. Two trying conflicting fixes. Customers waiting.

This is what incidents look like without an Incident Commander.

What the IC Does

The IC doesn't debug. They coordinate.

IC Responsibilities:
  ✓ Declare incident severity
  ✓ Assign roles (debugger, communicator, scribe)
  ✓ Coordinate investigation streams
  ✓ Make decisions (rollback? escalate? wait?)
  ✓ Manage communication (status page, stakeholders)
  ✓ Call for help when needed
  ✓ Declare all-clear

IC Does NOT:
  ✗ Write code
  ✗ Run queries
  ✗ SSH into servers
  ✗ Debug the issue

The IC Playbook

Minute 0-5: Declaration

1. Acknowledge the page
2. Open incident channel: #inc-YYYY-MM-DD-description
3. Post severity declaration:

   "I'm IC for this incident.
   Severity: P1 - Customer-facing checkout is down
   Impact: ~30% of checkout attempts failing

   Roles:
   - @alice: Primary debugger
   - @bob: Comms (status page + Slack updates)
   - @charlie: Scribe (timeline)

   First actions:
   - @alice: Check last deploy and error logs
   - @bob: Post initial status page update
   - I'll update every 10 minutes."

Minute 5-15: Investigation

The IC runs a structured investigation loop:

Every 5 minutes:
  1. "@alice, what have you found?"
  2. Synthesize information
  3. Decide next action
  4. Assign next task
  5. Update channel: "Current theory: [X]. Testing: [Y]."

Minute 15+: Decision Points

def ic_decision_tree(situation):
    if situation.root_cause_known:
        if situation.fix_available:
            return "Deploy fix with canary"
        else:
            return "Rollback to last known good"

    if situation.duration > 15 and not situation.making_progress:
        return "Escalate: bring in additional expertise"

    if situation.customer_impact_growing:
        return "Escalate severity + enable fallback"

    return "Continue investigation, update in 5 min"

Communication Templates

Pre-written templates save precious minutes:

templates:
  internal_update:
    format: |
      **Incident Update [{severity}] — {time} UTC**
      Status: {investigating|identified|monitoring|resolved}
      Impact: {impact_description}
      Current action: {what_we_are_doing}
      Next update: {time_of_next_update}

  status_page_update:
    format: |
      We are {status} an issue affecting {service}.
      Some users may experience {symptom}.
      Our team is actively working on a resolution.
      Next update in {minutes} minutes.

  executive_escalation:
    format: |
      P1 Incident: {title}
      Duration: {duration} minutes
      Customer impact: {impact}
      Revenue impact: ~${revenue}/hour
      Current status: {status}
      ETA to resolution: {eta}

Training New ICs

We use game days to train ICs:

Week 1: Shadow an experienced IC during a game day
Week 2: IC a simulated P2 incident (game day)
Week 3: IC a simulated P1 incident (game day)
Week 4: IC a real P3/P4 incident with a mentor observing
Week 5+: IC rotation for all severities

The IC Rotation

ic_rotation:
  schedule: weekly
  pool_size: 6  # Minimum for sustainable rotation
  requirements:
    - Completed IC training program
    - At least 6 months on the team
    - Shadowed 3+ real incidents
  compensation:
    - Same as on-call compensation
    - IC counts as on-call time

Before and After

Metric	Without IC	With IC
MTTR (P1)	67 min	28 min
Communication gaps	Frequent	Rare
Duplicate work	~40%	~5%
Stakeholder satisfaction	Low	High
Post-mortem quality	Incomplete	Thorough

The IC doesn't make incidents shorter because they're smarter. They make incidents shorter because someone is actually managing the response.

If you want AI-assisted incident coordination that makes every engineer an effective IC, check out what we're building at Nova AI Ops.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Terraform at Scale: Lessons from Managing 500+ Resources

Samson Tanimawo — Thu, 27 Aug 2026 23:18:59 +0000

When Terraform Gets Slow

Our Terraform state file grew to 500+ resources. Plans took 8 minutes. Applies timed out. State locking conflicts were daily. Something had to change.

Here's how we tamed it.

Problem 1: Monolithic State

Everything was in one state file. VPCs, databases, Kubernetes clusters, DNS, IAM — all in one giant blob.

Before: 1 state file, 500+ resources
  terraform plan: 8 minutes
  terraform apply: timeout risk
  blast radius: everything

Solution: State Decomposition

infrastructure/
├── network/          # VPCs, subnets, security groups
├── data/             # RDS, ElastiCache, S3
├── compute/          # EKS, ASGs, Launch templates
├── dns/              # Route53 zones and records
├── iam/              # Roles, policies, users
└── monitoring/       # CloudWatch, SNS topics

Each directory = separate state file. Use data sources to reference across boundaries:

# compute/main.tf
data "terraform_remote_state" "network" {
  backend = "s3"
  config = {
    bucket = "terraform-state"
    key    = "network/terraform.tfstate"
    region = "us-east-1"
  }
}

resource "aws_eks_cluster" "main" {
  vpc_config {
    subnet_ids = data.terraform_remote_state.network.outputs.private_subnet_ids
  }
}

Result: 6 state files, 60-100 resources each. Plan time: 45 seconds.

Problem 2: Environment Drift

Dev, staging, and prod drifted constantly because each was copy-pasted.

Solution: Modules + Terragrunt

modules/
├── eks-cluster/
│   ├── main.tf
│   ├── variables.tf
│   └── outputs.tf
└── rds-instance/
    ├── main.tf
    ├── variables.tf
    └── outputs.tf

environments/
├── dev/
│   └── terragrunt.hcl
├── staging/
│   └── terragrunt.hcl
└── prod/
    └── terragrunt.hcl

# environments/prod/terragrunt.hcl
terraform {
  source = "../../modules/eks-cluster"
}

inputs = {
  cluster_name    = "prod-main"
  node_count      = 10
  instance_type   = "m5.2xlarge"
  multi_az        = true
}

Problem 3: Dangerous Applies

Anyone could terraform apply to production from their laptop.

Solution: CI/CD Only

# .github/workflows/terraform.yml
name: Terraform
on:
  pull_request:
    paths: ['infrastructure/**']
  push:
    branches: [main]
    paths: ['infrastructure/**']

jobs:
  plan:
    runs-on: ubuntu-latest
    steps:
      - uses: hashicorp/setup-terraform@v3
      - run: terraform init
      - run: terraform plan -out=plan.tfplan
      - run: terraform show -json plan.tfplan > plan.json
      # Post plan as PR comment
      - uses: actions/github-script@v7
        with:
          script: |
            const plan = require('./plan.json');
            const adds = plan.resource_changes.filter(c => c.change.actions.includes('create')).length;
            const changes = plan.resource_changes.filter(c => c.change.actions.includes('update')).length;
            const deletes = plan.resource_changes.filter(c => c.change.actions.includes('delete')).length;
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              body: `## Terraform Plan\n+${adds} ~${changes} -${deletes}\n\n${deletes > 0 ? '⚠ RESOURCES WILL BE DESTROYED' : ''}`
            });

  apply:
    needs: plan
    if: github.ref == 'refs/heads/main'
    environment: production  # Requires approval
    steps:
      - run: terraform apply plan.tfplan

Problem 4: State Locks

Multiple engineers running plan simultaneously caused state lock conflicts.

Solution: Remote State with DynamoDB Locking

terraform {
  backend "s3" {
    bucket         = "terraform-state"
    key            = "network/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "terraform-locks"
    encrypt        = true
  }
}

Plus: only CI/CD runs apply. Humans run plan locally with -lock=false for quick checks.

Results

Metric	Before	After
Plan time	8 min	45 sec
Apply failures	3/week	0.5/week
State conflicts	Daily	Never
Env drift incidents	Monthly	None in 6 months
Time to provision new env	2 days	30 minutes

If you want AI-powered infrastructure management that catches drift before it causes outages, check out what we're building at Nova AI Ops.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Why Your Microservices Need Circuit Breakers (And How to Add Them)

Samson Tanimawo — Thu, 27 Aug 2026 13:22:09 +0000

The Cascading Failure That Took Down Everything

Our payment service went down for 3 minutes. No big deal, right? Except every service that called payments kept retrying. The retry storms consumed all available connections. Within 10 minutes, all 12 services were down.

3 minutes of one service failing became 45 minutes of total outage.

Circuit breakers prevent this.

How Circuit Breakers Work

State Machine:

  CLOSED ──(failures exceed threshold)──→ OPEN
    ↑                                       │
    │                                       │
    └──(success)──← HALF-OPEN ←──(timeout)──┘

CLOSED:    Normal operation. Requests pass through.
           Track failure rate.

OPEN:      Requests fail immediately (fast failure).
           No traffic to the struggling service.
           Wait for timeout period.

HALF-OPEN: Allow one test request through.
           If it succeeds → CLOSED
           If it fails → OPEN

Implementation in Python

import time
from enum import Enum
from threading import Lock

class CircuitState(Enum):
    CLOSED = "closed"
    OPEN = "open"
    HALF_OPEN = "half_open"

class CircuitBreaker:
    def __init__(self, failure_threshold=5, recovery_timeout=30, 
                 success_threshold=3):
        self.failure_threshold = failure_threshold
        self.recovery_timeout = recovery_timeout
        self.success_threshold = success_threshold
        self.state = CircuitState.CLOSED
        self.failure_count = 0
        self.success_count = 0
        self.last_failure_time = None
        self.lock = Lock()

    def call(self, func, *args, **kwargs):
        with self.lock:
            if self.state == CircuitState.OPEN:
                if time.time() - self.last_failure_time > self.recovery_timeout:
                    self.state = CircuitState.HALF_OPEN
                    self.success_count = 0
                else:
                    raise CircuitBreakerOpenError(
                        f"Circuit breaker is OPEN. Retry after "
                        f"{self.recovery_timeout}s"
                    )

        try:
            result = func(*args, **kwargs)
            self._on_success()
            return result
        except Exception as e:
            self._on_failure()
            raise

    def _on_success(self):
        with self.lock:
            if self.state == CircuitState.HALF_OPEN:
                self.success_count += 1
                if self.success_count >= self.success_threshold:
                    self.state = CircuitState.CLOSED
                    self.failure_count = 0
            self.failure_count = 0

    def _on_failure(self):
        with self.lock:
            self.failure_count += 1
            self.last_failure_time = time.time()
            if self.failure_count >= self.failure_threshold:
                self.state = CircuitState.OPEN
            if self.state == CircuitState.HALF_OPEN:
                self.state = CircuitState.OPEN

# Usage
payment_breaker = CircuitBreaker(failure_threshold=5, recovery_timeout=30)

def process_payment(order):
    try:
        return payment_breaker.call(payment_service.charge, order)
    except CircuitBreakerOpenError:
        return queue_for_retry(order)  # Graceful degradation

What to Do When the Circuit Opens

The circuit breaker buys you time. Use it wisely:

def handle_open_circuit(service_name, request):
    strategies = {
        'payment': lambda r: queue_for_retry(r),           # Retry later
        'recommendations': lambda r: return_cached(r),      # Serve stale data
        'analytics': lambda r: drop_silently(r),            # Non-critical, skip
        'auth': lambda r: allow_with_cached_token(r),       # Cached auth
        'search': lambda r: return_popular_results(r),      # Fallback results
    }
    return strategies.get(service_name, lambda r: return_error(r))(request)

Monitoring Circuit Breakers

circuit_breaker_metrics:
  - name: circuit_breaker_state
    type: gauge
    labels: [service, target]
    # 0=closed, 1=open, 2=half_open

  - name: circuit_breaker_failures_total
    type: counter
    labels: [service, target]

  - name: circuit_breaker_rejected_total
    type: counter
    labels: [service, target]
    # Requests rejected while circuit is open

alerts:
  - alert: CircuitBreakerOpen
    expr: circuit_breaker_state == 1
    for: 1m
    severity: warning
    message: "Circuit breaker for {{ $labels.target }} is OPEN"

The Configuration That Matters

circuit_breakers:
  payment-service:
    failure_threshold: 5
    recovery_timeout: 30s
    success_threshold: 3
    timeout_per_request: 5s

  search-service:
    failure_threshold: 10   # More tolerant
    recovery_timeout: 15s   # Recover faster
    success_threshold: 2
    timeout_per_request: 2s

  auth-service:
    failure_threshold: 3    # Less tolerant (critical)
    recovery_timeout: 10s   # Recover very fast
    success_threshold: 1
    timeout_per_request: 1s

Critical services get lower thresholds (less tolerance) and faster recovery.

If you want AI-powered circuit breaker tuning and cascading failure prevention, check out what we're building at Nova AI Ops.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

The On-Call Handoff That Prevents Dropped Incidents

Samson Tanimawo — Wed, 26 Aug 2026 22:29:06 +0000

The Monday Morning Disaster

Every Monday, the same story: the incoming on-call engineer has no idea what happened over the weekend. The outgoing engineer left a cryptic Slack message at 11pm and went to bed.

We lost 2 hours every Monday rebuilding context.

The Structured Handoff

We built a handoff template that takes 15 minutes to write and saves hours of confusion:

# On-Call Handoff: [DATE] → [DATE]
## Outgoing: @engineer_a | Incoming: @engineer_b

### Active Issues
| Issue | Status | Next Step | ETA |
|-------|--------|-----------|-----|
| DB replication lag | Monitoring | Auto-resolves if < 5s | Check at noon |
| Cert expiry api.prod | Fix scheduled | Deploy cert-bot PR #234 | Tuesday AM |

### Incidents This Shift
1. **[P2] Payment timeout spike** — 2024-03-15 02:30 UTC
   - Resolved: Increased connection pool from 20→50
   - Post-mortem: Scheduled for Wednesday
   - Lingering risk: Pool size is a band-aid, need connection pooler

### Upcoming Risks
- Major deploy of auth-service v3 on Tuesday
- Black Friday load test on Thursday
- AWS maintenance window Friday 2-6am UTC

### Helpful Context
- The cache-service has been flaky — restart fixes it (known bug, JIRA-456)
- New on-call runbook for search-service is at [link]
- PagerDuty schedule was updated — check your shifts

### Metrics to Watch
- DB replication lag: should be < 1s (currently 0.8s)
- Payment success rate: should be > 99.8% (currently 99.7%)
- API error rate: baseline is 0.05% (currently 0.04%)

Automating the Handoff

We automated 80% of this with a bot:

def generate_handoff_report(outgoing_shift_start, outgoing_shift_end):
    report = {
        'incidents': get_incidents(outgoing_shift_start, outgoing_shift_end),
        'active_alerts': get_active_alerts(),
        'recent_deploys': get_deploys(hours=48),
        'upcoming_maintenance': get_maintenance_windows(days=7),
        'slo_status': get_slo_status(),
        'open_tickets': get_oncall_tickets(status='open')
    }

    # Auto-generate summary
    summary = []
    if report['incidents']:
        summary.append(f"{len(report['incidents'])} incidents during shift")
    if report['active_alerts']:
        summary.append(f"{len(report['active_alerts'])} active alerts to monitor")
    if any(slo['budget_remaining'] < 30 for slo in report['slo_status']):
        summary.append("WARNING: SLO budget low for some services")

    return format_handoff(report, summary)

The 15-Minute Handoff Call

The bot generates the report. The humans spend 15 minutes on video:

0-5 min:  Outgoing reviews active issues and incidents
5-10 min: Walk through upcoming risks and context
10-15 min: Incoming asks questions, confirms understanding

Critical rule: The outgoing engineer is NOT released until the incoming engineer says "I'm good."

The Handoff Score

We rate every handoff:

handoff_score:
  report_completed: +1
  call_happened: +1
  all_incidents_documented: +1
  active_issues_listed: +1
  upcoming_risks_noted: +1
  metrics_baseline_included: +1

  max_score: 6
  target: >= 5

We track this weekly. Teams that score consistently above 5 have 60% fewer "lost context" incidents.

Results

Metric	Before	After
Monday morning incidents due to lost context	3-4/month	0-1/month
Time to rebuild context	2 hours	15 minutes
Incoming on-call confidence (1-5)	2.3	4.6
Escalations due to missing info	8/month	1/month

The best part: engineers actually look forward to handoffs now because they're quick and useful instead of stressful.

If you want AI-generated on-call handoff reports that capture everything automatically, check out what we're building at Nova AI Ops.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com