DEV Community: Samson Tanimawo

Service Maps: The Architectural Clarity Your Team Is Missing

Samson Tanimawo — Fri, 03 Jul 2026 00:30:12 +0000

"What Calls What?"

The question that launches a thousand Slack threads. In every microservices architecture I've worked with, nobody has a complete picture of how services interact.

Service maps fix this.

The Problem With Static Diagrams

Every team has architecture diagrams. They're all wrong. They were accurate on the day they were created, which was 18 months ago.

Reality of static diagrams:
  Created: January 2023 (accurate)
  Updated: March 2023 (still mostly accurate)
  Last touched: March 2023
  Current accuracy: ~40%
  New services since: 12
  Removed services: 3
  Changed dependencies: 27

Static diagrams are documentation debt.

Auto-Generated Service Maps

The solution is maps generated from actual traffic. There are three data sources:

1. Network Traffic (Infrastructure Level)

# Parse Kubernetes network policies + actual traffic
def build_network_map(k8s_client):
    services = k8s_client.list_namespaced_service('production')
    connections = []

    for svc in services.items:
        # Get pods for this service
        pods = k8s_client.list_namespaced_pod(
            'production',
            label_selector=','.join(f"{k}={v}" for k, v in svc.spec.selector.items())
        )
        # Check established connections
        for pod in pods.items:
            conns = get_pod_connections(pod.metadata.name)  # from /proc/net/tcp
            for conn in conns:
                target_svc = resolve_ip_to_service(conn.remote_ip)
                if target_svc:
                    connections.append({
                        'source': svc.metadata.name,
                        'target': target_svc,
                        'port': conn.remote_port
                    })

    return connections

2. Distributed Traces (Application Level)

-- Query trace data for service dependencies
SELECT 
  parent_service,
  child_service,
  COUNT(*) as call_count,
  AVG(duration_ms) as avg_latency,
  SUM(CASE WHEN status = 'ERROR' THEN 1 ELSE 0 END)::float / COUNT(*) as error_rate
FROM traces
WHERE timestamp > NOW() - INTERVAL '24 hours'
GROUP BY parent_service, child_service
ORDER BY call_count DESC;

3. API Gateway Logs (Edge Level)

API Gateway → Which external services call which internal services
Load Balancer → Traffic distribution and routing rules

What a Good Service Map Shows

Beyond just "A calls B," a useful service map includes:

service_map_edge:
  source: checkout-api
  target: payment-service
  metadata:
    requests_per_second: 150
    error_rate: 0.2%
    p99_latency: 120ms
    protocol: gRPC
    circuit_breaker: enabled
    retry_policy: 3x with exponential backoff
    owner_team: payments
    tier: critical
    last_deploy: 2024-03-14T15:30:00Z

Color-code by health:

Green: < 1% error rate, latency within SLO
Yellow: Approaching SLO limits
Red: SLO breached

The Incident Response Advantage

During an incident, a live service map immediately answers:

Impact scope: What services depend on the broken one?
Blast radius: How many users are affected?
Root cause direction: Is the problem upstream or downstream?
Recovery order: Which services need to come back first?

Before service maps (incident response):
  "Is anyone else affected?" → 15 minutes of Slack investigation

After service maps:
  *click on broken service* → instantly see all dependents
  Answer: 4 downstream services, 3 are degraded

Start Simple

You don't need a fancy tool to get started:

Export your K8s services and network policies
Add trace-based dependencies
Render with a simple graph library
Update automatically via cron job

A basic but accurate map beats a fancy but outdated one every time.

If you want auto-generated, always-accurate service maps with AI-powered impact analysis, check out what we're building at Nova AI Ops.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

AI in Incident Response: Hype vs. Reality in 2024

Samson Tanimawo — Thu, 02 Jul 2026 13:21:33 +0000

Every Vendor Claims AI Magic

Open any monitoring vendor's website and you'll see: "AI-powered incident detection!" "ML-driven root cause analysis!" "Intelligent alerting!"

After evaluating a dozen AI ops tools and running three in production, here's what actually works and what's snake oil.

What Works: Anomaly Detection

ML-based anomaly detection genuinely helps with metrics that have predictable patterns:

Good candidates for ML anomaly detection:
  - Request rate (daily/weekly seasonality)
  - CPU usage (follows traffic patterns)
  - Database connections (predictable daily cycles)
  - Error counts (should be near-zero baseline)

Bad candidates:
  - Deployment metrics (irregular by nature)
  - Batch job durations (vary by data volume)
  - Cache hit rates (depends on traffic mix)
  - Anything with frequent step changes

The key is training on enough data. You need at least 2-3 weeks of data for daily patterns, and 6-8 weeks for weekly seasonality.

What Works: Alert Correlation

This is where AI delivers real value. When 15 alerts fire simultaneously, AI can group them and identify the probable root cause:

Raw alerts (what the human sees):
  01:15:03  CRITICAL  api-server p99 latency > 2s
  01:15:07  WARNING   postgres connection pool 90%
  01:15:12  CRITICAL  checkout error rate > 5%
  01:15:15  WARNING   redis response time > 100ms
  01:15:18  CRITICAL  payment-service timeout
  01:15:22  WARNING   cart-service p99 > 1s

AI-correlated (what the human should see):
  01:15:03  INCIDENT  Database connection pool exhaustion
    Impact: checkout, payment, cart services degraded
    Probable cause: postgres connection pool at 90%
    Related alerts: 6 (grouped)
    Suggested action: Check for connection leaks, consider pool size increase

This is the difference between a 45-minute investigation and a 5-minute fix.

What Doesn't Work (Yet): Autonomous Remediation

Vendors love to demo "AI automatically fixed the issue!" In reality:

Auto-scaling works great (but that's not really AI)
Auto-rollback works great (also not really AI)
Actual autonomous root cause analysis and fix? Not reliable enough for production.

I tested three autonomous remediation products. Results:

Correct diagnosis: 72%
Correct remediation: 45%
Made things worse: 8%
Did nothing useful: 47%

A 45% success rate isn't good enough for production systems. But it IS good enough for suggesting actions to a human.

The AI-Assisted Sweet Spot

Human only:         AI suggests + Human decides:    AI autonomous:
───────────         ────────────────────────────    ──────────────
Slow, error-prone   Fast, accurate                  Fast, risky
at 3am              at 3am                          at 3am

                    ← THIS IS WHERE WE SHOULD BE

The best approach today:

AI detects the anomaly
AI correlates related alerts
AI suggests probable root cause
AI recommends remediation steps
Human approves the action
Automation executes

What to Evaluate

When looking at AI ops tools, ask:

What data do you need? (If they need 6 months of data to start, that's a red flag)
What's the false positive rate? (Anything > 10% will be ignored by your team)
Can I see the reasoning? (Black-box AI is useless for incident response)
Does it integrate with my existing tools? (If it requires rip-and-replace, walk away)
What happens when the AI is wrong? (Good tools show confidence scores)

My Prediction

In 2-3 years, AI will handle 80% of incidents autonomously. The remaining 20% — novel failures, complex cascading issues — will still need human judgment. But that's fine. Those are the interesting problems.

If you want to see how AI-assisted incident response actually works in practice, check out what we're building at Nova AI Ops.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Monitoring Costs Are Out of Control — Here's How to Fix It

Samson Tanimawo — Thu, 02 Jul 2026 00:06:24 +0000

The $50K/Month Monitoring Bill

I audited our monitoring stack last quarter. The total cost across all tools: $52,000/month. For a company with 200 engineers. That's $260 per engineer per month just to watch our systems.

Something had to change.

Where the Money Goes

Breakdown of $52K/month:
  Custom metrics ingestion:  $18,000 (35%)
  Log storage & search:      $14,000 (27%)
  APM/Tracing:               $9,000  (17%)
  Alerting platform:         $4,000  (8%)
  Synthetic monitoring:      $3,000  (6%)
  Dashboards & visualization:$2,000  (4%)
  Status page:               $2,000  (4%)

The top two — metrics and logs — were 62% of the bill.

Strategy 1: Metrics Cardinality Audit

High-cardinality metrics are the #1 cost driver. One bad label can 10x your bill:

# This is fine: ~100 time series
http_requests_total{method="GET", status="200", service="api"}

# This is expensive: ~1,000,000 time series
http_requests_total{method="GET", status="200", service="api", user_id="..."}
#                                                              ^^^^^^^^^
#                                                              100K unique values

I wrote a script to find high-cardinality offenders:

import requests

def find_high_cardinality_metrics(prometheus_url, threshold=10000):
    # Get all metric names
    r = requests.get(f"{prometheus_url}/api/v1/label/__name__/values")
    metrics = r.json()['data']

    expensive = []
    for metric in metrics:
        # Count time series per metric
        r = requests.get(f"{prometheus_url}/api/v1/series", 
                        params={'match[]': metric})
        count = len(r.json()['data'])
        if count > threshold:
            expensive.append({'metric': metric, 'series_count': count})

    return sorted(expensive, key=lambda x: x['series_count'], reverse=True)

# Found: request_duration_bucket had 2.3M series due to URL path label

We found three metrics responsible for 60% of our series count. Fixed them in a day.

Strategy 2: Aggregation at Collection

Instead of sending raw metrics and aggregating at query time, aggregate at collection:

# prometheus.yml - recording rules
groups:
  - name: aggregations
    rules:
      # Pre-aggregate p99 latency per service (not per pod)
      - record: service:http_request_duration:p99
        expr: histogram_quantile(0.99, sum(rate(http_request_duration_bucket[5m])) by (service, le))

      # Pre-aggregate error rate per service
      - record: service:http_errors:rate5m
        expr: sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)

Pre-aggregated metrics are cheaper to store AND faster to query.

Strategy 3: Right-Size Your Retention

Do you really need 13 months of 15-second resolution metrics? Probably not.

retention_policy:
  raw_metrics:
    resolution: 15s
    retention: 7 days

  5_minute_rollups:
    resolution: 5m
    retention: 30 days

  1_hour_rollups:
    resolution: 1h
    retention: 13 months

  1_day_rollups:
    resolution: 1d
    retention: 5 years

Strategy 4: Eliminate Unused Dashboards

We had 340 dashboards. I checked access logs:

340 total dashboards
 82 viewed in last 30 days (24%)
 31 viewed in last 90 days (9%)
227 never viewed in 6 months (67%)

67% of our dashboards were zombie dashboards. Nobody looked at them, but they drove metric queries.

We archived everything not viewed in 90 days. Savings: $3,200/month from reduced query load.

Results

Before: $52,000/month
  Metrics cardinality fix:    -$11,000
  Log tier/sampling changes:  -$8,000
  Dashboard cleanup:          -$3,200
  Retention right-sizing:     -$5,800
  Duplicate tool consolidation:-$6,000
After: $18,000/month

Annual savings: $408,000

And we actually improved our observability because the remaining data was higher quality.

If you want monitoring that's cost-effective by design, check out what we're building at Nova AI Ops.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Hiring SREs: What I Look For After Interviewing 100+ Candidates

Samson Tanimawo — Wed, 01 Jul 2026 13:01:47 +0000

The SRE Hiring Problem

SRE roles are notoriously hard to fill. The intersection of software engineering, systems administration, and operational wisdom is narrow. After interviewing over 100 candidates across three companies, here's what I've learned.

What I Don't Care About

Years of experience with a specific tool
Certifications (CKA, AWS SA, etc.)
Memorized answers about CAP theorem
Whether they've used Terraform or Pulumi

Tools change. Fundamentals don't.

What I Actually Look For

1. Systematic Debugging (The #1 Skill)

I give every candidate the same scenario:

"Users report the app is slow. What do you do?"

Bad answer: "I'd check the database."

Good answer: "First, I'd define 'slow' — which endpoints, which users, since when? Then I'd check the golden signals: is latency up? Error rate up? Traffic unusual? Saturation? I'd compare current metrics to baseline and narrow down from there."

The best candidates think in systems, not components.

2. Comfort With Ambiguity

SRE work is fundamentally ambiguous. The page fires, you don't know why, and people are waiting.

I ask: "Tell me about a time you had to make a decision with incomplete information."

I want to hear about:

How they scoped the unknowns
What heuristics they used
How they communicated uncertainty
Whether they revised their approach as they learned more

3. Writing Ability

SREs write post-mortems, runbooks, design docs, and incident updates. If you can't write clearly under pressure, you'll struggle.

I include a writing exercise: "Write a 3-sentence status update for a customer-facing outage."

Good example:

"We identified an issue affecting login for approximately 15% of users starting at 2:30 PM UTC. Our team has identified the root cause and deployed a fix. We're monitoring to confirm full resolution, which we expect within the next 10 minutes."

4. Automation Mindset

I ask: "What's the last thing you automated, and why?"

The "why" matters more than the "what." I want to hear about:

Identifying repetitive toil
Calculating the ROI of automation
Choosing the right level of automation
Knowing when NOT to automate

5. Empathy

Surprising for a technical role? SREs are the bridge between development, operations, and business. Empathy means:

Understanding why a developer shipped that sketchy PR (deadline pressure)
Knowing that a product manager asking "when will it be fixed?" is scared, not annoying
Recognizing that the junior engineer who caused the outage feels terrible already

My Interview Structure

Round 1 (45 min): Technical screen
  - Systems design: "Design monitoring for a checkout service"
  - Debugging scenario: "Walk me through investigating latency"

Round 2 (60 min): Incident simulation
  - Live scenario: simulated page with fake dashboards
  - Evaluate: systematic thinking, communication, tool usage

Round 3 (45 min): Culture and collaboration
  - Post-mortem discussion: review a real (anonymized) incident
  - Conflict resolution: "Dev team pushes back on your SLO proposal"

Round 4 (30 min): Writing exercise
  - Write a runbook for a given scenario
  - Write a status update for a given incident

Red Flags

Blames people instead of systems
Can't explain things simply
Never says "I don't know"
Only talks about tools, never principles
No interest in learning from incidents

If you're building an SRE team and want AI to handle the routine work so your engineers focus on what matters, check out what we're building at Nova AI Ops.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Log Management at Scale: How We Cut Costs 70% Without Losing Signal

Samson Tanimawo — Wed, 01 Jul 2026 00:55:38 +0000

$12,000/Month for Logs Nobody Reads

Our logging bill was $12,000/month. We were ingesting 2TB/day. When I asked the team what percentage of logs they actually looked at during incidents, the answer was embarrassing: about 5%.

We were paying to store 95% noise.

The Log Audit

First, I categorized all log sources by value:

High value (always need during incidents):
  Application errors (stack traces)
  Authentication events
  Business transactions
  External API calls with responses
  Health check failures

Medium value (sometimes useful):
  Request/response logs (sampled)
  Performance metrics in logs
  Deployment events
  Configuration changes

Low value (almost never needed):
  Debug/trace level logs
  Health check successes
  Static asset requests
  Heartbeat messages
  Verbose framework logs

Strategy 1: Log Levels as a Service

We made log levels dynamic. In production, default is WARN. During incidents, flip to DEBUG for the affected service:

import os
import logging

# Log level from environment variable, changeable at runtime
LOG_LEVEL = os.environ.get('LOG_LEVEL', 'WARNING')
logging.basicConfig(level=getattr(logging, LOG_LEVEL))

# Endpoint to change log level without restart
@app.post('/admin/log-level')
async def set_log_level(level: str):
    logging.getLogger().setLevel(getattr(logging, level.upper()))
    return {'status': 'ok', 'level': level}

In Kubernetes:

# Normal operation
kubectl set env deployment/api LOG_LEVEL=WARNING

# During incident
kubectl set env deployment/api LOG_LEVEL=DEBUG

# After incident
kubectl set env deployment/api LOG_LEVEL=WARNING

Strategy 2: Tiered Retention

retention_policy:
  hot_storage:  # Fast search, expensive
    duration: 7 days
    filter: "level >= WARN OR tag:business_event"

  warm_storage:  # Slower search, cheaper
    duration: 30 days
    filter: "level >= INFO"

  cold_storage:  # Archive only, cheapest
    duration: 365 days
    filter: "tag:audit OR tag:compliance"

  drop:  # Don't store at all
    filter: "level = DEBUG OR source:health_check"

Strategy 3: Structured Logging

Unstructured logs are expensive to parse. Structured logs are cheap to query:

# Bad: Unstructured
logger.info(f"User {user_id} purchased {product_id} for ${amount}")
# Parsing this requires regex, which costs compute

# Good: Structured
logger.info("purchase_completed", extra={
    'user_id': user_id,
    'product_id': product_id, 
    'amount': amount,
    'currency': 'USD'
})
# Output: {"message": "purchase_completed", "user_id": "u123", ...}
# Queryable without parsing

Strategy 4: Sample Verbose Logs

import random

def should_log_request(request):
    # Always log errors
    if request.status_code >= 400:
        return True
    # Always log slow requests
    if request.duration_ms > 1000:
        return True
    # Sample 10% of successful requests
    return random.random() < 0.10

The Results

Before:
  Daily ingestion: 2 TB
  Monthly cost: $12,000
  Useful data: ~5%

After:
  Daily ingestion: 400 GB
  Monthly cost: $3,600
  Useful data: ~70%

We cut costs by 70% AND improved signal quality. Searches are faster because there's less noise. Incidents resolve quicker because relevant logs surface immediately.

The Rule

Before adding a log statement, ask: "Will someone look at this during an incident?" If the answer is no, it's DEBUG level at most.

If you're spending too much on logs and want smarter log management, check out what we're building at Nova AI Ops.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Canary Deployments: The Pattern That Cut Our Rollback Rate by 80%

Samson Tanimawo — Tue, 30 Jun 2026 13:27:00 +0000

Deploy and Pray

Our deployment strategy used to be: merge to main, deploy to all pods, watch Slack for complaints. Professional? No. Common? Absolutely.

After a particularly bad deploy took down checkout for 23 minutes, we implemented canary deployments.

What Canary Deployments Actually Mean

A canary deployment routes a small percentage of traffic to the new version while monitoring for problems:

Traffic flow:
  Users ──→ Load Balancer ──→ 95% → v1.2.3 (current)
                            └──→  5% → v1.2.4 (canary)

If the canary looks healthy after N minutes, gradually increase traffic. If it looks bad, kill it. Zero impact on 95% of users.

Our Canary Pipeline

# .github/workflows/canary-deploy.yml
canary_deploy:
  steps:
    - name: Deploy canary (5%)
      run: |
        kubectl set image deployment/api-canary api=api:${{ github.sha }}
        kubectl scale deployment/api-canary --replicas=1
        # Configure traffic split
        kubectl apply -f - <<EOF
        apiVersion: split.smi-spec.io/v1alpha1
        kind: TrafficSplit
        metadata:
          name: api-canary
        spec:
          service: api
          backends:
          - service: api-stable
            weight: 95
          - service: api-canary
            weight: 5
        EOF

    - name: Wait and analyze (10 minutes)
      run: |
        sleep 600
        # Check canary health
        ERROR_RATE=$(curl -s prometheus/api/v1/query?query=rate(http_errors{version="canary"}[5m]) | jq '.data.result[0].value[1]')
        LATENCY=$(curl -s prometheus/api/v1/query?query=histogram_quantile(0.99,rate(http_duration_bucket{version="canary"}[5m])) | jq '.data.result[0].value[1]')

        echo "Canary error rate: $ERROR_RATE"
        echo "Canary p99 latency: $LATENCY"

        if (( $(echo "$ERROR_RATE > 0.01" | bc -l) )); then
          echo "CANARY FAILED: Error rate too high"
          exit 1
        fi

    - name: Promote to 50%
      run: |
        kubectl apply -f traffic-split-50.yaml
        sleep 600  # Wait another 10 min

    - name: Full rollout
      run: |
        kubectl set image deployment/api-stable api=api:${{ github.sha }}
        kubectl delete deployment api-canary
        kubectl delete trafficsplit api-canary

The Canary Checklist

What we check during the canary window:

CANARY_CHECKS = {
    'error_rate': {
        'query': 'rate(http_5xx_total{version="canary"}[5m]) / rate(http_requests_total{version="canary"}[5m])',
        'threshold': 0.01,  # Max 1% errors
        'comparison': 'less_than'
    },
    'latency_p99': {
        'query': 'histogram_quantile(0.99, rate(http_duration_bucket{version="canary"}[5m]))',
        'threshold': 0.5,  # Max 500ms
        'comparison': 'less_than'
    },
    'success_rate': {
        'query': 'rate(http_2xx_total{version="canary"}[5m]) / rate(http_requests_total{version="canary"}[5m])',
        'threshold': 0.99,  # Min 99% success
        'comparison': 'greater_than'
    },
    'memory_usage': {
        'query': 'container_memory_working_set_bytes{version="canary"}',
        'threshold': 512 * 1024 * 1024,  # Max 512MB
        'comparison': 'less_than'
    }
}

Results After 6 Months

Metric	Before	After
Rollback rate	15% of deploys	3% of deploys
Mean time to detect bad deploy	25 min	8 min
Customer-facing incidents from deploys	4/month	0.5/month
Deploy frequency	1x/day (afraid)	5x/day (confident)

The counterintuitive result: we deploy MORE often now because we're less afraid. And because each deploy is smaller, issues are easier to find.

Start Simple

You don't need Istio or a service mesh for canary deploys. Start with:

Two deployment objects (stable + canary)
A load balancer that supports weighted routing
A script that checks error rates after deploy
A human who decides whether to promote or rollback

Automate from there.

If you want AI-powered canary analysis that automatically promotes or rolls back, check out what we're building at Nova AI Ops.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Platform Engineering: Building an Internal Developer Platform That Teams Actually Use

Samson Tanimawo — Tue, 30 Jun 2026 00:53:49 +0000

The "Build It and They Won't Come" Problem

Our platform team spent 6 months building an internal developer platform. Beautiful service catalog, automated provisioning, self-service databases. Nobody used it.

Here's what we learned.

Why Platforms Fail

Most internal platforms fail for the same reason: they're built top-down instead of bottom-up.

Top-down: "We decided every team should use this standardized deployment pipeline."
Bottom-up: "We noticed 8 teams solving the same problem differently, so we built a shared solution."

The Paved Road Approach

Instead of mandating tools, offer a paved road. Make the right thing the easy thing.

Paved road (easy):        Off-road (hard but allowed):
─────────────────         ──────────────────────────
Standard CI/CD template   Custom pipeline
Managed Postgres          Self-managed DB
Shared observability       Own monitoring stack
Pre-configured K8s         Custom infrastructure

The key: off-road is allowed but unsupported. You break it, you own it.

What Our IDP Looks Like

# service.yaml — the only file developers need to create
apiVersion: platform/v1
kind: Service
metadata:
  name: checkout-api
  team: payments
  tier: critical
spec:
  language: python
  framework: fastapi

  dependencies:
    - postgres:14
    - redis:7
    - rabbitmq:3

  scaling:
    min: 3
    max: 20
    metric: cpu
    target: 70

  environments:
    staging:
      replicas: 1
    production:
      replicas: 3
      multi_az: true

From this single file, the platform provisions:

Git repo with CI/CD pipeline
Kubernetes namespace and RBAC
Database and connection secrets
Monitoring dashboards (golden signals)
Alerting rules
Log aggregation
Service mesh entry

The Developer Experience Metrics

We track these to know if the platform is working:

Time from idea to production deploy:    Before: 2 weeks  After: 4 hours
Time to provision a new environment:    Before: 3 days   After: 12 minutes
Deploy frequency:                       Before: weekly    After: 5x/day
Change failure rate:                    Before: 18%       After: 4%
Developer satisfaction (quarterly NPS): Before: -10       After: +52

The Self-Service Portal

Our portal has exactly four actions:

Create Service — Generates everything from service.yaml
View My Services — Dashboard of health, deploys, costs
Request Resource — Database, queue, cache (auto-provisioned)
Get Help — Links to docs + Slack channel

That's it. Four buttons. If you need more than four buttons, your platform is too complex.

Adoption Strategy

We didn't mandate adoption. We seduced teams into it:

Week 1-4: Pilot with the friendliest team. Fix everything.
Week 5-8: Add two more teams. Fix more things.
Week 9-12: Success stories in engineering all-hands.
Week 13+: Other teams start asking to join.

By month 6, 80% of teams had migrated voluntarily. The remaining 20% had legitimate edge cases we accommodated.

What Not to Build

Don't build a service mesh if you have < 20 services
Don't build a custom scheduler if standard K8s works
Don't build a custom secret manager — use Vault or cloud-native
Don't build a custom CI system — use GitHub Actions/GitLab CI

Build the glue, not the tools.

If you want a platform that includes AI-powered operations from day one, check out what we're building at Nova AI Ops.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Chaos Engineering for Teams That Aren't Netflix

Samson Tanimawo — Mon, 29 Jun 2026 15:44:45 +0000

You Don't Need Chaos Monkey

Every chaos engineering talk starts with Netflix and Chaos Monkey. Cool story. You're not Netflix. You probably have 5-50 services, not 500. You don't need a sophisticated chaos platform.

You need a methodology.

Start With Game Days

Before injecting failures into production, run tabletop exercises:

## Game Day: Database Failover
Date: 2024-03-15
Scope: Primary DB goes down
Participants: SRE team + Backend leads

### Scenario
At 10:00 AM, the primary database becomes unreachable.

### Questions to answer:
1. How does the application behave?
2. Does the replica promote automatically?
3. What's the expected failover time?
4. What manual steps are needed?
5. How do we verify data consistency after failover?

### Pre-requisites
- [ ] Backup verified within last 24 hours
- [ ] Runbook for DB failover reviewed
- [ ] Rollback plan documented
- [ ] All participants in incident channel

Game days cost nothing and reveal 80% of the gaps.

Your First Real Chaos Experiment

Start small. Really small.

# Experiment 1: Kill a single pod
# Hypothesis: Traffic shifts to remaining pods with zero errors

# Before
kubectl get pods -l app=api-service
# NAME                          READY   STATUS
# api-service-7d8f9b6c4-abc12   1/1     Running
# api-service-7d8f9b6c4-def34   1/1     Running
# api-service-7d8f9b6c4-ghi56   1/1     Running

# The experiment
kubectl delete pod api-service-7d8f9b6c4-abc12

# Observe
# - Did error rate spike?
# - Did latency increase?
# - Did K8s reschedule the pod?
# - How long until back to 3 replicas?

If killing one pod causes errors, you have a serious problem that's better to find now.

The Chaos Experiment Template

experiment:
  name: "Network latency to payment service"
  date: "2024-03-20"

  hypothesis:
    steady_state: "P99 checkout latency < 500ms, error rate < 0.1%"
    expected_behavior: >
      Circuit breaker activates within 5 seconds.
      Checkout falls back to cached payment validation.
      Users see a 'retry' message, not an error.

  method:
    tool: "tc (traffic control)"
    injection: "200ms latency on port 443 to payment-service"
    duration: "5 minutes"
    scope: "Single availability zone"

  abort_conditions:
    - "Error rate exceeds 5%"
    - "Checkout success rate drops below 90%"
    - "Any data inconsistency detected"

  rollback:
    command: "tc qdisc del dev eth0 root"
    verification: "Check latency returns to baseline"

  results:
    actual_behavior: "[filled in after experiment]"
    hypothesis_confirmed: true/false
    action_items: []

Progressive Chaos Levels

Level 1: Kill a pod (Week 1)
Level 2: Kill all pods in one AZ (Week 4)
Level 3: Inject latency to a dependency (Week 8)
Level 4: Simulate full dependency outage (Week 12)
Level 5: Multi-failure scenario (Week 16+)

Don't skip levels. Each one builds confidence and reveals issues.

What We Found

After 6 months of regular chaos experiments:

12 missing circuit breakers discovered
3 services with no health checks
5 services with incorrect timeout configurations
2 services with hard-coded dependency URLs (no DNS)
1 service that crashed when its cache was unavailable

All of these would have caused outages eventually. We found them on our terms, during business hours, with everyone ready.

The Business Case

Prevented outages (estimated): 4 per quarter
Average outage cost: $15,000
Chaos engineering cost: ~20 hours/quarter of eng time

ROI: ($60,000 saved - $4,000 cost) = $56,000/quarter

If you want to run chaos experiments with AI-guided blast radius analysis, check out what we're building at Nova AI Ops.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Distributed Tracing: The Missing Piece of Your Observability Stack

Samson Tanimawo — Mon, 29 Jun 2026 00:17:36 +0000

When Logs and Metrics Aren't Enough

You have great dashboards. Your log aggregation is solid. But when a user reports "the checkout page is slow," you still spend 30 minutes jumping between services trying to find the bottleneck.

That's the gap distributed tracing fills.

What Tracing Actually Shows You

A trace is a complete picture of a single request as it flows through your system:

User Request → API Gateway → Auth Service → Product Service → DB → Cache → Response
                  5ms          12ms           45ms       120ms  3ms
                                                          ^
                                              This is your bottleneck

Without tracing, you'd see:

API Gateway: latency looks fine
Auth Service: latency looks fine
Product Service: latency is HIGH but why?

With tracing, you see the exact DB query inside Product Service that's taking 120ms.

Getting Started with OpenTelemetry

OpenTelemetry is the standard. Here's a minimal setup:

# Python example with Flask
from opentelemetry import trace
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from opentelemetry.instrumentation.sqlalchemy import SQLAlchemyInstrumentor
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

# Setup
provider = TracerProvider()
provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="http://otel-collector:4317"))
)
trace.set_tracer_provider(provider)

# Auto-instrument everything
FlaskInstrumentor().instrument_app(app)
RequestsInstrumentor().instrument()
SQLAlchemyInstrumentor().instrument(engine=db.engine)

That's it. Three auto-instrumentations cover 80% of what you need.

Custom Spans for the Other 20%

Auto-instrumentation gives you HTTP calls and DB queries. Add custom spans for business logic:

tracer = trace.get_tracer(__name__)

def process_order(order):
    with tracer.start_as_current_span("process_order") as span:
        span.set_attribute("order.id", order.id)
        span.set_attribute("order.total", order.total)

        with tracer.start_as_current_span("validate_inventory"):
            validate_inventory(order.items)

        with tracer.start_as_current_span("charge_payment"):
            charge_payment(order.payment_method, order.total)

        with tracer.start_as_current_span("send_confirmation"):
            send_email(order.customer_email)

Sampling Strategy

You can't trace every request in production. Well, you can, but your bill will be astronomical.

# otel-collector-config.yaml
processors:
  probabilistic_sampler:
    sampling_percentage: 10  # Sample 10% of requests

  tail_sampling:
    policies:
      # Always keep errors
      - name: errors
        type: status_code
        status_code: {status_codes: [ERROR]}
      # Always keep slow requests
      - name: slow-requests  
        type: latency
        latency: {threshold_ms: 1000}
      # Sample 5% of everything else
      - name: default
        type: probabilistic
        probabilistic: {sampling_percentage: 5}

Tail sampling is the key. It lets you keep 100% of interesting traces and only 5% of boring ones.

The Three Queries That Matter

Once you have tracing data, these three queries solve 90% of debugging:

1. "Show me the slowest traces in the last hour"
   → Finds performance regressions

2. "Show me traces with errors, grouped by service"
   → Finds which service is failing

3. "Show me traces for user X's request at time T"
   → Reproduces specific customer issues

Common Mistakes

Not propagating trace context — If service A calls service B but doesn't pass the trace ID, you get broken traces
Over-sampling in production — Start at 1-5%, increase as needed
Not adding business context — Adding user.id, order.id, etc. to spans makes traces actually useful
Ignoring async operations — Queues break trace propagation unless you explicitly pass context

If you want AI-powered trace analysis that automatically finds bottlenecks, check out what we're building at Nova AI Ops.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

The Golden Signals: A Practical Implementation Guide

Samson Tanimawo — Sun, 28 Jun 2026 13:18:06 +0000

Four Metrics to Rule Them All

Google's SRE book introduced the four golden signals: Latency, Traffic, Errors, and Saturation. Simple concept, but I've seen teams struggle with implementation.

Here's a practical guide from someone who's implemented them across 50+ services.

Signal 1: Latency

Not all latency is equal. You need to track successful requests and error requests separately.

# Bad: Average latency
latency = total_request_time / total_requests  # Useless

# Good: Percentile latency, separated by status
from prometheus_client import Histogram

REQUEST_LATENCY = Histogram(
    'http_request_duration_seconds',
    'Request latency',
    ['method', 'endpoint', 'status_class'],
    buckets=[.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10]
)

@app.middleware
async def track_latency(request, call_next):
    start = time.time()
    response = await call_next(request)
    duration = time.time() - start
    status_class = f"{response.status_code // 100}xx"
    REQUEST_LATENCY.labels(
        method=request.method,
        endpoint=request.url.path,
        status_class=status_class
    ).observe(duration)
    return response

Alert on p99, not p50. Your happiest users don't need help.

- alert: HighLatencyP99
  expr: histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) > 0.5
  for: 5m
  labels:
    severity: warning

Signal 2: Traffic

Traffic tells you "is this normal?" It's the context for every other signal.

# Current request rate
rate(http_requests_total[5m])

# Compare to same time last week
rate(http_requests_total[5m]) 
  / 
rate(http_requests_total[5m] offset 7d)

# Alert on sudden drops (possible outage nobody noticed)
- alert: TrafficDrop
  expr: >
    rate(http_requests_total[5m]) 
    < 
    (rate(http_requests_total[5m] offset 1h) * 0.5)
  for: 10m
  annotations:
    summary: "Traffic dropped >50% compared to 1 hour ago"

Traffic drops are often more concerning than traffic spikes.

Signal 3: Errors

Track error rate as a percentage, not absolute count:

# Error rate percentage
(
  sum(rate(http_requests_total{status=~"5.."}[5m]))
  /
  sum(rate(http_requests_total[5m]))
) * 100

But also track error types separately:

error_categories:
  - 5xx: "Server errors (our fault)"
  - 4xx_excluding_404: "Client errors (possible API issue)"
  - timeout: "Request timeouts"
  - circuit_breaker: "Dependency failures"

Signal 4: Saturation

The most underrated signal. Saturation answers: "how close are we to full?"

# CPU saturation
process_cpu_seconds_total / container_spec_cpu_quota

# Memory saturation
container_memory_working_set_bytes / container_spec_memory_limit_bytes

# Connection pool saturation
active_connections / max_connections

# Queue saturation (the one everyone forgets)
message_queue_depth / message_queue_capacity

Alert before you hit 100%. I use 80% as the threshold for warning and 95% for critical.

Putting It All Together

Every service gets a standard dashboard with four rows:

Row 1: Latency   [p50] [p90] [p99] [error latency]
Row 2: Traffic    [rate] [vs last week] [by endpoint]
Row 3: Errors     [rate %] [by type] [by endpoint]
Row 4: Saturation [CPU] [Memory] [Connections] [Queue]

This fits on one screen. No scrolling. Any engineer can assess service health in 10 seconds.

The Anti-Pattern

Don't build a golden signals dashboard per service manually. Template it:

{
  "dashboard": {
    "title": "Golden Signals: {{ service_name }}",
    "templating": {
      "list": [
        { "name": "service", "type": "query" },
        { "name": "environment", "type": "custom", "options": ["prod", "staging"] }
      ]
    }
  }
}

One template, 50 dashboards. Update once, apply everywhere.

If you want golden signal monitoring that sets itself up automatically, check out what we're building at Nova AI Ops.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Kubernetes Observability: What to Monitor and Why

Samson Tanimawo — Sun, 28 Jun 2026 01:17:57 +0000

The Kubernetes Monitoring Maze

Kubernetes gives you a thousand metrics out of the box. Most teams monitor all of them and understand none of them.

After running K8s in production for four years, here's what actually matters.

The Three Layers

Kubernetes observability has three distinct layers, and you need different strategies for each:

Layer 1: Cluster Health (infrastructure)
Layer 2: Workload Health (your apps)
Layer 3: Application Performance (user experience)

Layer 1: Cluster Health

These are your "is the platform working?" metrics:

critical_cluster_metrics:
  nodes:
    - node_ready_status        # Are all nodes healthy?
    - node_cpu_utilization     # Alert at 85%
    - node_memory_utilization  # Alert at 90%
    - node_disk_pressure       # Boolean alert
    - node_pid_pressure        # Rarely fires, always critical

  control_plane:
    - apiserver_request_latency_p99  # Alert > 1s
    - etcd_disk_wal_fsync_duration   # Alert > 100ms
    - scheduler_pending_pods         # Alert if > 0 for 5min
    - controller_manager_queue_depth # Alert if growing

Pro tip: Don't alert on individual node CPU. Alert on cluster-level capacity:

# Alert when cluster is 80% utilized
(
  sum(node_cpu_seconds_total{mode!="idle"}) 
  / 
  sum(node_cpu_seconds_total)
) > 0.80

Layer 2: Workload Health

This is where most teams get it wrong. They monitor pods instead of workloads.

critical_workload_metrics:
  deployments:
    - available_replicas < desired_replicas  # For > 5min
    - deployment_generation != observed_generation  # Stuck rollout

  pods:
    - restart_count increasing       # CrashLoopBackOff detection
    - container_oom_killed            # Memory limits too low
    - pod_pending_duration > 2min     # Scheduling issues

  hpa:
    - current_replicas == max_replicas  # Scale ceiling hit
    - cpu_utilization_vs_target         # Consistently above target

The most valuable alert I ever wrote:

# Detect pods stuck in CrashLoopBackOff
alert: PodCrashLooping
expr: rate(kube_pod_container_status_restarts_total[15m]) > 0
for: 15m
labels:
  severity: warning
annotations:
  summary: "Pod {{ $labels.pod }} is crash-looping"
  description: "{{ $labels.pod }} in {{ $labels.namespace }} has restarted {{ $value }} times in 15 minutes"

Layer 3: Application Performance

This is what your users actually care about:

application_metrics:
  red_method:  # Rate, Errors, Duration
    - request_rate_per_second
    - error_rate_percentage        # Alert > 1%
    - request_duration_p99         # Alert > 500ms

  use_method:  # Utilization, Saturation, Errors
    - cpu_request_vs_limit_ratio
    - memory_request_vs_limit_ratio
    - network_receive_bytes_rate

The Dashboard That Saves Us

We built a single "K8s Health" dashboard with four panels:

Cluster capacity — CPU/Memory/Disk utilization per node pool
Workload status — Table of all deployments with health status
Error rates — All services, sorted by error rate
Recent events — K8s events filtered to warnings and errors

This one dashboard answers 90% of "is something wrong?" questions.

Common Mistakes

Monitoring pods instead of services — Pods are ephemeral, services are what matter
Not setting resource requests — Without requests, your metrics are meaningless
Alerting on resource usage instead of SLOs — High CPU isn't a problem if latency is fine
Ignoring the control plane — An unhealthy API server affects everything

If you want unified Kubernetes observability without the complexity, check out what we're building at Nova AI Ops.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

On-Call Wellness: Protecting Your Engineers from Burnout

Samson Tanimawo — Sat, 27 Jun 2026 13:21:24 +0000

The On-Call Burnout Epidemic

I watched three senior SREs leave our team in six months. Exit interviews all said the same thing: on-call was unsustainable.

We were spending $500K+ recruiting replacements for a problem that could have been fixed with $0 and better practices.

The Warning Signs

Before someone quits, they show these signals:

Cynicism in post-mortems — "This will never get fixed"
Alert numbness — Slow to respond, missed pages
Vacation avoidance — "I can't take time off, who would cover?"
Scope creep rejection — "That's not my problem"
Meeting silence — Previously engaged, now checked out

If you see three or more of these in someone on your team, they're already halfway out the door.

What We Changed

1. Hard Cap on Pages

We set a maximum of 2 pages per 8-hour on-call shift. If someone gets paged more than that, the secondary automatically takes over and the incident is escalated as a process failure.

on_call_policy:
  max_pages_per_shift: 2
  shift_duration_hours: 8
  overflow_action: "escalate_to_secondary"
  overflow_review: "weekly_ops_review"

2. Follow-the-Sun Rotation

We stopped asking people to be on-call at 3am. With team members across US timezones, we created overlapping business-hours shifts:

Shift A (Eastern):  6am - 2pm ET
Shift B (Central):  11am - 7pm CT  
Shift C (Pacific):  2pm - 10pm PT
Overnight:          Managed by alert automation + escalation

Nobody gets paged between 10pm and 6am unless it's a true P1.

3. On-Call Compensation

We implemented:

$500 flat fee per on-call week
$200 per off-hours page
Comp day after any overnight incident > 30 minutes
On-call swaps require zero management approval

4. The "Toil Budget"

Each engineer gets a toil budget: maximum 30% of their time on operational work. If toil exceeds 30%, they're pulled from on-call until the team automates the excess.

Weekly toil tracking:
  Alert response:     4 hours
  Manual deployments: 2 hours  
  Config updates:     1 hour
  Ad-hoc debugging:   3 hours
  ─────────────────────────
  Total:              10 hours (25% of 40hr week) ✓

5. Quarterly On-Call Reviews

Every quarter, we review:

Pages per person
Off-hours disruptions
Toil percentages
Team sentiment survey
Attrition risk signals

The Results

Metric	Before	After (6 months)
Attrition rate	40%/year	8%/year
Pages per shift	4.7	1.2
Off-hours pages	12/week	2/week
Team NPS	-15	+45
Recruitment cost saved	-	~$400K/year

The Key Insight

On-call wellness isn't a perk. It's a business decision. Replacing a senior SRE costs $150-200K in recruiting, onboarding, and lost productivity. Preventing burnout costs almost nothing.

If you're looking to reduce on-call toil and protect your team from burnout, check out what we're building at Nova AI Ops.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com