DEV Community: Samson Tanimawo

Incident Severity Levels: SEV-1 to SEV-5 Calibration

Samson Tanimawo — Sun, 03 May 2026 14:27:49 +0000

Why Severity Is Broken at Most Companies

Everyone has severity levels. Almost nobody agrees on what they mean.

Ask ten engineers what SEV-2 means and you'll get eight different answers. This causes:

Under-paged incidents (people thought SEV-3 meant "no rush")
Over-paged incidents (everything is SEV-1)
Exhausted on-call (false alarms)
Missed SLOs (incidents not escalated in time)

Calibration matters. Here's a definition that actually works.

The Five Levels

SEV-1: Critical

Primary product is completely down for all users
Active data loss
Security breach in progress
Core business stopped (can't process payments, can't log in)

Target response: 5 minutes
Escalation: Immediate, all hands
Post-mortem: Required, public within 5 days

SEV-2: High

Primary product is degraded for most users
Core feature unavailable for a subset
Significant customer impact but workaround exists
Performance significantly degraded (>5x normal latency)

Target response: 15 minutes
Escalation: Page primary on-call, notify secondary
Post-mortem: Required, internal within 5 days

SEV-3: Medium

Non-critical feature broken
Affects a small percentage of users
Degraded performance within tolerance
Bug in new feature rollout

Target response: 1 hour
Escalation: Page during business hours, ticket overnight
Post-mortem: Recommended

SEV-4: Low

Minor bug with workaround
Internal tooling broken
Non-customer-facing issue
Cosmetic problems

Target response: 1 business day
Escalation: Ticket only
Post-mortem: Not required

SEV-5: Informational

Not actually broken
Preemptive warning
"This might become a problem"
Observed anomaly without impact

Target response: Backlog
Escalation: None
Post-mortem: Not required

The Calibration Problem

Levels written on paper are useless. What matters is consistent application.

Run this exercise: take your last 50 incidents. Ask three SRE leads to independently assign severity levels. Compare.

If more than 20% disagree by at least one level, your definitions aren't calibrated. Run training.

The "When In Doubt" Rules

When severity is ambiguous, default to higher severity and downgrade if wrong.

Better to over-escalate and apologize than under-escalate and miss a SEV-1 for 4 hours.

Specific rules:

User data loss → always SEV-1 or SEV-2, never lower
Security issue → always SEV-1 or SEV-2
Revenue impact → SEV-2 minimum if measurable
Uncertain scope → start at higher severity, downgrade when scope is clear

Customer Impact Matrix

For fast calibration, use a matrix:

| <1% users | 1-10% users | 10-50% | >50% users
Product Down | SEV-2 | SEV-1 | SEV-1 | SEV-1
Major Degraded | SEV-3 | SEV-2 | SEV-2 | SEV-1
Minor Degraded | SEV-4 | SEV-3 | SEV-2 | SEV-2
Workaround Exists| SEV-4 | SEV-4 | SEV-3 | SEV-2

This gives you a fast severity assignment without relying on intuition.

Time-Based Escalation

Severity isn't fixed for the incident lifetime. It escalates:

sev_2:
auto_escalate_to_sev_1:
- if_not_resolved_in: 60_minutes
- if_user_impact_grows: above_10_percent
- if_revenue_loss_exceeds: $10000/hour

Start at SEV-2, auto-escalate if things worsen. Don't let an incident linger at the same severity if the impact is growing.

The Downgrade Rule

Downgrading is allowed but must be justified in writing in the incident channel.

"Downgrading from SEV-1 to SEV-2 at 10:23. Initial reports of
total outage were incorrect. Real impact is ~5% of users in
us-west-2 only. Ticket: INC-1234"

This prevents silent downgrades that understate severity for retro analysis.

SLO Integration

Your SLOs and severity levels should align:

SLO: 99.95% availability (21.6 min/month budget)

If this month's error budget burned:
<25% → normal operations
25-50% → no SEV-3 burn-down deploys
50-75% → SEV-2 threshold lowered
>75% → any degradation is SEV-1

When you're running low on error budget, everything gets more severe.

Practical Incident Categories

Beyond numeric severity, label incidents by type:

INCIDENT_TYPES:
- infrastructure (AWS, networking)
- application (code bug)
- deployment (bad release)
- capacity (scaling failure)
- data (corruption, loss)
- security (breach, exposure)
- external (3rd-party dependency)

Severity tells you how urgent. Type tells you who to page.

The Monthly Review

Once a month, review:

All SEV-1s and SEV-2s
Any SEV-3 that should have been SEV-2
Any SEV-2 that should have been SEV-3
Average time from incident open to correct severity assignment

Adjust the definitions based on what you learn. Severity is a living standard.

Common Mistakes

Pet severity every team invents their own. Standardize company-wide.
SEV-0 don't add levels above SEV-1. Just use "SEV-1 all hands."
Severity inflation if every incident is SEV-2, nobody takes SEV-2 seriously
Severity deflation pressure to avoid post-mortems leads to fake SEV-4s
Unchanging severity escalation is a tool, use it

The Goal

Severity should mean the same thing to every person in the org. Engineers, PMs, execs, customer support.

When someone says "SEV-1," everyone should know what that means, how urgent it is, and what the response looks like.

When you achieve that, incident response gets dramatically better.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Memory Leak Detection in Long-Running Services

Samson Tanimawo — Sat, 02 May 2026 14:27:34 +0000

The Slowest Incident to Diagnose

Memory leaks are sneaky. The service runs fine for hours. Then, slowly, it gets worse. Slower responses, more GC pauses, eventual OOM kills.

And when you look at the first 30 minutes of metrics, everything looks normal.

The Three Flavors of Memory Growth

1. True leaks

Objects allocated but never freed
Classic in C/C++, rare in Go/Java with GC
Grows linearly forever until OOM

2. Unbounded caches

Cache adds entries but never evicts
Common in Node.js, Python, Go
Grows until memory pressure triggers other issues

3. Memory fragmentation

Heap is large but not usable
Happens in long-running Java, Go,.NET services
Not really a "leak" but behaves like one

All three cause the same symptom: memory grows over time. Treatment is different for each.

Detection Without Heap Dumps

Before you reach for pprof or heap dumps, the fastest diagnosis is graph-watching:

# Is memory growing linearly over the last 24 hours?
deriv(container_memory_working_set_bytes{service="api"}[24h]) > 0

# Is GC pause time increasing?
rate(jvm_gc_pause_seconds_sum[1h]) > 0.05

If memory is growing by ~500MB/day and GC pauses are increasing, you have a leak. Diagnosis complete.

The question is where.

Go Memory Profiling

Go makes this relatively easy:

import _ "net/http/pprof"

// In main():
go func() {
http.ListenAndServe(":6060", nil)
}()

Then:

# Get a heap profile
go tool pprof http://localhost:6060/debug/pprof/heap

# In the pprof shell:
(pprof) top
(pprof) list suspiciousFunction
(pprof) web # generates a SVG callgraph

Look for:

Objects with high inuse_space
Objects with growing counts over time
Unexpected large maps or slices

Key trick: take two heap profiles 1 hour apart and diff them:

go tool pprof -base heap1.pprof heap2.pprof

What shows up as "new" allocations in the diff is almost certainly your leak.

Java Memory Profiling

Java is harder because the JVM adds layers:

# Dump the heap
jmap -dump:format=b,file=heap.hprof <pid>

# Analyze with Eclipse MAT or JVisualVM

In MAT, look for:

Leak Suspects report (automatic)
Dominator tree (what's holding the most memory)
GC roots path (what's preventing garbage collection)

Common Java culprits:

Static collections (especially static Map)
ThreadLocal values without cleanup
Listeners/callbacks registered but never unregistered
finalize() methods delaying collection

Node.js Memory Profiling

// Enable the inspector
node --inspect app.js

// Then in Chrome DevTools → Memory → Heap Snapshot
// Take 3 snapshots: baseline, after 10 min, after 20 min
// Compare to find retained objects

Common Node culprits:

Event emitter listeners that accumulate
Closures holding references to large objects
Unbounded caches (remember, Node has no built-in LRU)
Stream buffers not being drained

Python Memory Profiling

import tracemalloc
tracemalloc.start()

#... run the leaky operation...

snapshot = tracemalloc.take_snapshot()
top_stats = snapshot.statistics('lineno')
for stat in top_stats[:10]:
print(stat)

Or use memory_profiler:

from memory_profiler import profile

@profile
def suspect_function():
# code here

Common Python culprits:

Global lists/dicts growing unbounded
Reference cycles with __del__ methods
C extensions leaking (hardest to find)
Pandas DataFrames kept around too long

The Cache Leak Special Case

The most common "leak" isn't a leak at all. It's a cache without eviction.

# BAD: unbounded
cache = {}
def get_user(user_id):
if user_id not in cache:
cache[user_id] = fetch_from_db(user_id)
return cache[user_id]

# GOOD: bounded LRU
from functools import lru_cache

@lru_cache(maxsize=10000)
def get_user(user_id):
return fetch_from_db(user_id)

Always bound your caches. Always.

Fragmentation in Go

Go's garbage collector can leave the heap fragmented. You see:

Runtime memory is high
Heap profile shows low allocations
runtime.GC() doesn't reduce usage much

Solution: tune GOGC or force memory release:

import "runtime/debug"
debug.SetGCPercent(20) // More aggressive GC
debug.FreeOSMemory() // Return memory to OS

The Long-Running Service Pattern

Services that run for weeks without restart accumulate cruft. Even without leaks.

We use this pattern:

deployment_policy:
max_uptime: 7d
restart_schedule: "rolling restart every 7 days"

Every pod gets restarted weekly during a quiet window. Eliminates slow memory growth as a class of problem.

This isn't defeat. It's acknowledging that long-running processes in any language eventually accumulate state you don't want.

The Diagnostic Checklist

When a service is suspected of leaking:

Is memory growing linearly or logarithmically? (linear = real leak)
Is GC frequency/duration increasing? (yes = real pressure)
Are request rates growing proportionally? (yes = normal growth, not leak)
Take heap profile, save baseline
Wait 1 hour, take second profile, diff
Look for unexpected high-count objects
Trace back to allocation site
Fix the leak, deploy, watch metrics for 24h

Rinse and repeat. Memory leaks are annoying but systematically fixable.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

CI/CD Reliability: When Your Deploy Pipeline is Your SPOF

Samson Tanimawo — Fri, 01 May 2026 14:23:02 +0000

The Invisible SPOF

Every engineering org has a single point of failure that nobody lists on their risk registry: the deploy pipeline itself.

When CI/CD breaks, you can't ship features. You can't deploy hotfixes. You can't roll back a broken release. Your production doesn't go down, but your ability to fix production does.

We had a 4-hour outage last year caused by a GitHub Actions incident. Not a single server went down. We just couldn't deploy the fix.

Categorizing the Risk

Your pipeline consists of:

source_control: # GitHub, GitLab, Bitbucket
failure_mode: "can't merge PRs"

ci_runners: # GitHub Actions, CircleCI, self-hosted
failure_mode: "builds don't run"

artifact_storage: # ECR, Artifactory, S3
failure_mode: "images don't build or push"

deployment_controller: # ArgoCD, Flux, Spinnaker
failure_mode: "deploys don't happen"

cluster_api: # k8s API, cloud provider API
failure_mode: "resources don't change"

Each layer is a failure domain. A serious pipeline needs fallback plans for each.

The Manual Escape Hatch

Rule #1: You must have a documented path to deploy manually.

Not for daily use for emergencies. Every team should know:

How to build the image locally
How to push to the registry
How to update the cluster without the normal pipeline
Who has permission to do this in production

We test this quarterly. Every SRE must deploy one service manually, end-to-end, in under 10 minutes, without the pipeline.

Hardening the Pipeline Itself

1. Pin your dependencies

# BAD
uses: actions/checkout@main

# GOOD
uses: actions/checkout@v4.1.1

If actions/checkout@main breaks, your deploys break. Pin to versions.

2. Cache everything locally

registry:
primary: ghcr.io/yourorg
fallback: ecr.amazonaws.com/yourorg

When the primary registry is down, you need a mirror. Every production image should exist in at least two registries.

3. Monitor the pipeline

You probably monitor your services. Do you monitor your CI?

pipeline_metrics:
- build_success_rate (target: >99%)
- deploy_duration_p99 (target: <5 min)
- time_to_rollback_p99 (target: <2 min)
- runner_queue_depth (target: <5)

Alert on these the same way you'd alert on a service.

4. Test disaster modes

Can you ship if GitHub Actions is down?
Can you ship if the main registry is unreachable?
Can you ship if ArgoCD is down?

If the answer is "no", you have undocumented SPOFs.

The Rollback Rule

Every deploy must be reversible in under 2 minutes. No exceptions.

time_to_deploy: 15 minutes
time_to_rollback: 90 seconds

If your rollback takes longer than your deploy, your pipeline is backwards.

How to achieve fast rollbacks:

Keep the previous image running in parallel during deploys
Use traffic-shifting deploys (ALB weights, Istio)
Label every image with the git commit
Never deploy untested rollback paths

The Deploy Freeze

Some teams never deploy on Fridays. This is cargo culting.

The real rules:

Don't deploy when the on-call person is asleep
Don't deploy during peak traffic windows
Don't deploy major changes during holidays
DO deploy hotfixes anytime

If Friday at 5pm is the only time you can deploy a fix, you deploy. The alternative is customers suffering all weekend.

A reliable pipeline makes any-time deploys safe. Banning Friday deploys means your pipeline isn't reliable enough.

Multi-Provider Strategy

Big-ticket item: run critical workloads on CI from a different vendor than your code host.

Code: GitHub
CI: CircleCI (not GitHub Actions)

When GitHub Actions is down (it happens twice a year), your builds still run. When CircleCI is down, you can fall back to GitHub Actions.

This doubles your CI bill but removes a major SPOF.

The "Break Glass" Deploy

Every pipeline should have an emergency bypass:

# Normal deploy (takes 15 minutes, runs all tests)
./deploy.sh

# Break-glass deploy (skips tests, full audit log, Slack alert)
./deploy.sh --break-glass --reason "Fixing P1 incident #1234"

The break-glass path:

Requires written justification
Skips long-running tests
Notifies the whole team
Writes to a permanent audit log
Can only be used with incident in progress

Used maybe 3-5 times a year. Without it, your 2-hour deploy pipeline becomes a bottleneck when every minute matters.

The Metric That Matters Most

Mean Time to Deploy a Hotfix (MTTDHF)

From "we need to fix this" to "fix is in production" how long?

Good: under 30 minutes
Great: under 10 minutes
Unicorn: under 5 minutes

Track this. Optimize it. It's the most important reliability metric nobody talks about.

The Takeaway

Your pipeline is production infrastructure. Treat it with the same respect.

Monitor it
Back it up
Test failure modes
Document manual paths
Never let it become a SPOF

When it breaks during an incident, you'll be very glad you did.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Multi-Region Failover: Lessons from Running It Hot

Samson Tanimawo — Thu, 30 Apr 2026 14:47:08 +0000

Why "Hot" Matters

Three multi-region strategies:

Cold: Backup region is off. You start it when primary fails. RTO: hours.

Warm: Backup region runs on minimum capacity. Scale up on failover. RTO: 15-30 minutes.

Hot: Both regions serve live traffic simultaneously. RTO: seconds.

If you need under 15 minutes RTO, you need hot. Everything else is marketing copy.

The Illusion of Warm Failover

Warm sounds great on paper. In practice, on the day you need it:

The warm region has never seen real load
DNS cache propagation takes 5-15 minutes
Autoscaling lags because it's starting cold
Your team has never run on the warm region
Half your connection strings are hardcoded to the primary

Warm failover works in tabletop exercises. It does not work during real incidents under pressure. We learned this the hard way.

Running It Hot: The Architecture

┌─────────────┐
│ DNS / CDN │
└──────┬──────┘
│
┌──────────┴──────────┐
▼ ▼
┌──────────┐ ┌──────────┐
│ Region A │ │ Region B │
│ 50% TX │ │ 50% TX │
└────┬─────┘ └─────┬────┘
│ │
└──────┬──────┬────────┘
▼ ▼
┌─────────────┐
│ Shared DB │
│ (writer + │
│ replicas) │
└─────────────┘

Both regions always serve traffic. Split is usually 50/50 but can shift.

The Hard Parts

1. Database replication

This is where multi-region gets hard. Three options:

Single writer, multi-region readers: simplest, but writes pay cross-region latency
Multi-master: complex, but truly hot requires conflict resolution
Region-sharded: users pinned to a region for writes, simplest if your data model allows it

We use region-sharded for user-scoped data and single-writer for global config.

2. Session stickiness

If a user's session is in Region A, and their next request goes to Region B, things break.

Solutions:

JWT tokens with no server state
Session data in a globally replicated store (DynamoDB Global Tables, CockroachDB)
Cookie routing that pins a user to a region

3. Cache coherence

Region A's cache doesn't know when Region B updates the database. Options:

Short TTLs (1-5 minutes) and accept the inconsistency
Pub/sub cache invalidation across regions (complex)
Read-through caches only, never write-through

The Failover Mechanics

When Region A dies:

Health checks detect failure route53/ALB removes Region A from DNS
Traffic shifts to Region B already warm, already running
Autoscaling kicks in Region B doubles capacity
User sessions degrade gracefully re-authentication, cache warmup
Monitoring reports the shift team gets paged, not customers

Target: customer-facing latency spike of under 30 seconds, full recovery under 5 minutes.

Testing It Monthly

If you don't test failover monthly, you don't have failover. You have hope.

We do this:

First Tuesday of every month, 10 AM
Route100% of traffic to Region B for 30 minutes
Watch dashboards, fix anything that degrades
Route back to 50/50
Document any issues, fix them
Repeat next month with the other region

Cost Reality Check

Running hot doubles your compute cost. For most companies, that's $10K-$100K/month.

The question is: what's your revenue per hour of downtime?

Company A: $100K/hr revenue → 99.99% target → hot is worth it
Company B: $1K/hr revenue → 99.9% target → warm is fine

Do the math. Don't copy FAANG patterns if your revenue doesn't justify them.

The Operational Complexity Tax

Running hot costs more than money. It costs:

More runbooks (one per region)
More monitoring (cross-region latency, replication lag)
Harder debugging ("which region was this request in?")
More compliance surface (data residency, each region)
More deployment pipelines (usually)

Budget 20% more engineering time for multi-region from day one.

Common Mistakes

Single point of failure in DNS config your DNS provider becomes the new SPOF
Testing only with healthy traffic test with 2x normal load during drills
Forgetting about databases DB failover is the hardest part
Using regions as backup, not active never tested until crisis
Not planning for split-brain what if both regions think they're primary?

The Minimum Viable Hot Setup

Two regions, stateless app tier, 50/50 traffic
Database: multi-AZ primary, cross-region async replica
CDN/DNS: health-check-based routing
Session: JWT-based (stateless)
Monthly failover drills
Runbooks tested in last 90 days

Start there. Layer in complexity as you need it.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Multi-Region Failover: Lessons from Running It Hot

Samson Tanimawo — Thu, 30 Apr 2026 14:08:47 +0000

Why "Hot" Matters

Three multi-region strategies:

Cold: Backup region is off. You start it when primary fails. RTO: hours.

Warm: Backup region runs on minimum capacity. Scale up on failover. RTO: 15-30 minutes.

Hot: Both regions serve live traffic simultaneously. RTO: seconds.

If you need under 15 minutes RTO, you need hot. Everything else is marketing copy.

The Illusion of Warm Failover

Warm sounds great on paper. In practice, on the day you need it:

The warm region has never seen real load
DNS cache propagation takes 5-15 minutes
Autoscaling lags because it's starting cold
Your team has never run on the warm region
Half your connection strings are hardcoded to the primary

Warm failover works in tabletop exercises. It does not work during real incidents under pressure. We learned this the hard way.

Running It Hot: The Architecture

┌─────────────┐
│ DNS / CDN │
└──────┬──────┘
│
┌──────────┴──────────┐
▼ ▼
┌──────────┐ ┌──────────┐
│ Region A │ │ Region B │
│ 50% TX │ │ 50% TX │
└────┬─────┘ └─────┬────┘
│ │
└──────┬──────┬────────┘
▼ ▼
┌─────────────┐
│ Shared DB │
│ (writer + │
│ replicas) │
└─────────────┘

Both regions always serve traffic. Split is usually 50/50 but can shift.

The Hard Parts

1. Database replication

This is where multi-region gets hard. Three options:

Single writer, multi-region readers: simplest, but writes pay cross-region latency
Multi-master: complex, but truly hot requires conflict resolution
Region-sharded: users pinned to a region for writes, simplest if your data model allows it

We use region-sharded for user-scoped data and single-writer for global config.

2. Session stickiness

If a user's session is in Region A, and their next request goes to Region B, things break.

Solutions:

JWT tokens with no server state
Session data in a globally replicated store (DynamoDB Global Tables, CockroachDB)
Cookie routing that pins a user to a region

3. Cache coherence

Region A's cache doesn't know when Region B updates the database. Options:

Short TTLs (1-5 minutes) and accept the inconsistency
Pub/sub cache invalidation across regions (complex)
Read-through caches only, never write-through

The Failover Mechanics

When Region A dies:

Health checks detect failure route53/ALB removes Region A from DNS
Traffic shifts to Region B already warm, already running
Autoscaling kicks in Region B doubles capacity
User sessions degrade gracefully re-authentication, cache warmup
Monitoring reports the shift team gets paged, not customers

Target: customer-facing latency spike of under 30 seconds, full recovery under 5 minutes.

Testing It Monthly

If you don't test failover monthly, you don't have failover. You have hope.

We do this:

First Tuesday of every month, 10 AM
Route100% of traffic to Region B for 30 minutes
Watch dashboards, fix anything that degrades
Route back to 50/50
Document any issues, fix them
Repeat next month with the other region

Cost Reality Check

Running hot doubles your compute cost. For most companies, that's $10K-$100K/month.

The question is: what's your revenue per hour of downtime?

Company A: $100K/hr revenue → 99.99% target → hot is worth it
Company B: $1K/hr revenue → 99.9% target → warm is fine

Do the math. Don't copy FAANG patterns if your revenue doesn't justify them.

The Operational Complexity Tax

Running hot costs more than money. It costs:

More runbooks (one per region)
More monitoring (cross-region latency, replication lag)
Harder debugging ("which region was this request in?")
More compliance surface (data residency, each region)
More deployment pipelines (usually)

Budget 20% more engineering time for multi-region from day one.

Common Mistakes

Single point of failure in DNS config your DNS provider becomes the new SPOF
Testing only with healthy traffic test with 2x normal load during drills
Forgetting about databases DB failover is the hardest part
Using regions as backup, not active never tested until crisis
Not planning for split-brain what if both regions think they're primary?

The Minimum Viable Hot Setup

Two regions, stateless app tier, 50/50 traffic
Database: multi-AZ primary, cross-region async replica
CDN/DNS: health-check-based routing
Session: JWT-based (stateless)
Monthly failover drills
Runbooks tested in last 90 days

Start there. Layer in complexity as you need it.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Disaster Recovery Drills That Actually Work

Samson Tanimawo — Wed, 29 Apr 2026 15:45:56 +0000

Most DR Drills Are Theater

Someone schedules a meeting. A few senior engineers walk through a runbook. Everyone agrees "yes, we could do this" and marks it complete.

Then the real disaster hits and nobody remembers the procedure, the runbook is 2 years out of date, and half the backup systems don't work.

Real DR drills test whether your team can actually recover, not whether they can talk about recovery.

The Three Levels of DR Testing

Level 1: Tabletop

Walk through a scenario on paper
Identify missing runbooks
Find ownership gaps
Useful for: New team members, initial gap analysis
Limits: Doesn't prove anything actually works

Level 2: Partial Failure Test

Actually fail one component in staging
Watch recovery happen with real tools
Time the full recovery
Useful for: Validating specific runbooks
Limits: Staging ≠ production

Level 3: Full Production Drill

Actually fail a real production component
Customer-facing (announce a maintenance window if needed)
Full team responds as if it's real
Useful for: Proving you can recover
Limits: Scary, high-coordination

Most teams stop at Level 1. Good teams do Level 2 quarterly. The best teams do Level 3 twice a year.

A Real DR Drill Scenario

Scenario: Primary database becomes unreachable

Setup (48 hours before):

Schedule window with product team
Pick a time when customer impact is minimal
Brief the team: "Something will fail tomorrow, respond as normal"
Pre-position the incident commander

Execution:

At T+0, block network access to primary database via iptables
Start stopwatch
Watch the team respond
Do NOT intervene or give hints
Document every action, every decision, every delay

Metrics to capture:

Time to detection (first alert fire)
Time to engagement (first human acknowledges)
Time to diagnosis ("we know what's wrong")
Time to mitigation (customer impact stops)
Time to recovery (fully restored)

Scoring the Drill

Good scores:

Detection: under 2 minutes
Engagement: under 5 minutes
Diagnosis: under 15 minutes
Mitigation: under 30 minutes (for DR scenarios)
Recovery: depends on scenario

If any of these are 5x longer than target, you have a real problem.

What Always Goes Wrong

In every DR drill we run:

Runbook is out of date. The one that worked 6 months ago has wrong commands now.
Credentials don't work. The service account was rotated, nobody updated the runbook.
Backup is untested. The restore fails because the backup is corrupted.
Escalation paths are stale. The "DBA on-call" has left the company.
Dependencies are missing. The recovery playbook assumes Service X is up, but Service X depends on the failed component.

Every drill uncovers 3-5 of these. Fix them, then drill again.

Chaos Engineering vs. DR Drills

These are different. Chaos engineering is continuous (daily/weekly) and usually automated. DR drills are intentional and large-scale.

Chaos engineering answers: "Can we survive small failures routinely?"

DR drills answer: "Can we survive catastrophic failures at all?"

You need both.

The Blame-Free Rule

DR drills expose weaknesses. Those weaknesses are process problems, not people problems.

The rules:

No firing based on drill performance
No promotions based on "being the hero"
Focus on process gaps, not individual failures
The post-drill retrospective is 90% about fixing systems, 10% about training people

If the team is afraid of the drill, you'll never learn anything real.

Frequency That Actually Works

Level 1 (Tabletop): Monthly, 1 hour
Level 2 (Partial): Quarterly, 4 hours including retro
Level 3 (Full Production): Twice a year, full day

Also: after every major infrastructure change, drill the affected components.

The Hardest Lesson

The drill is the easy part. The hard part is making the fixes from the drill a priority when there's feature pressure.

We track "DR drill remediation items" as a standing OKR. If after two quarters the same items are still open, the SRE team has authority to freeze feature work until they're fixed.

Starting Point

If you've never done a DR drill:

Pick one scenario (database failure, region outage, API gateway down)
Schedule it for a quiet hour
Run a tabletop first find the obvious gaps
Fix those gaps
Run a partial failure test in staging
Measure everything
Run a retro focused on process
Schedule the next one

Do this for three scenarios, and you'll have a DR program. Do it for ten, and you'll have a resilient company.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Disaster Recovery Drills That Actually Work

Samson Tanimawo — Wed, 29 Apr 2026 14:41:48 +0000

Most DR Drills Are Theater

Someone schedules a meeting. A few senior engineers walk through a runbook. Everyone agrees "yes, we could do this" and marks it complete.

Then the real disaster hits and nobody remembers the procedure, the runbook is 2 years out of date, and half the backup systems don't work.

Real DR drills test whether your team can actually recover, not whether they can talk about recovery.

The Three Levels of DR Testing

Level 1: Tabletop

Walk through a scenario on paper
Identify missing runbooks
Find ownership gaps
Useful for: New team members, initial gap analysis
Limits: Doesn't prove anything actually works

Level 2: Partial Failure Test

Actually fail one component in staging
Watch recovery happen with real tools
Time the full recovery
Useful for: Validating specific runbooks
Limits: Staging ≠ production

Level 3: Full Production Drill

Actually fail a real production component
Customer-facing (announce a maintenance window if needed)
Full team responds as if it's real
Useful for: Proving you can recover
Limits: Scary, high-coordination

Most teams stop at Level 1. Good teams do Level 2 quarterly. The best teams do Level 3 twice a year.

A Real DR Drill Scenario

Scenario: Primary database becomes unreachable

Setup (48 hours before):

Schedule window with product team
Pick a time when customer impact is minimal
Brief the team: "Something will fail tomorrow, respond as normal"
Pre-position the incident commander

Execution:

At T+0, block network access to primary database via iptables
Start stopwatch
Watch the team respond
Do NOT intervene or give hints
Document every action, every decision, every delay

Metrics to capture:

Time to detection (first alert fire)
Time to engagement (first human acknowledges)
Time to diagnosis ("we know what's wrong")
Time to mitigation (customer impact stops)
Time to recovery (fully restored)

Scoring the Drill

Good scores:

Detection: under 2 minutes
Engagement: under 5 minutes
Diagnosis: under 15 minutes
Mitigation: under 30 minutes (for DR scenarios)
Recovery: depends on scenario

If any of these are 5x longer than target, you have a real problem.

What Always Goes Wrong

In every DR drill we run:

Runbook is out of date. The one that worked 6 months ago has wrong commands now.
Credentials don't work. The service account was rotated, nobody updated the runbook.
Backup is untested. The restore fails because the backup is corrupted.
Escalation paths are stale. The "DBA on-call" has left the company.
Dependencies are missing. The recovery playbook assumes Service X is up, but Service X depends on the failed component.

Every drill uncovers 3-5 of these. Fix them, then drill again.

Chaos Engineering vs. DR Drills

These are different. Chaos engineering is continuous (daily/weekly) and usually automated. DR drills are intentional and large-scale.

Chaos engineering answers: "Can we survive small failures routinely?"

DR drills answer: "Can we survive catastrophic failures at all?"

You need both.

The Blame-Free Rule

DR drills expose weaknesses. Those weaknesses are process problems, not people problems.

The rules:

No firing based on drill performance
No promotions based on "being the hero"
Focus on process gaps, not individual failures
The post-drill retrospective is 90% about fixing systems, 10% about training people

If the team is afraid of the drill, you'll never learn anything real.

Frequency That Actually Works

Level 1 (Tabletop): Monthly, 1 hour
Level 2 (Partial): Quarterly, 4 hours including retro
Level 3 (Full Production): Twice a year, full day

Also: after every major infrastructure change, drill the affected components.

The Hardest Lesson

The drill is the easy part. The hard part is making the fixes from the drill a priority when there's feature pressure.

We track "DR drill remediation items" as a standing OKR. If after two quarters the same items are still open, the SRE team has authority to freeze feature work until they're fixed.

Starting Point

If you've never done a DR drill:

Pick one scenario (database failure, region outage, API gateway down)
Schedule it for a quiet hour
Run a tabletop first find the obvious gaps
Fix those gaps
Run a partial failure test in staging
Measure everything
Run a retro focused on process
Schedule the next one

Do this for three scenarios, and you'll have a DR program. Do it for ten, and you'll have a resilient company.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Feature Flags as a Reliability Tool, Not Just an A/B Platform

Samson Tanimawo — Tue, 28 Apr 2026 14:11:47 +0000

Most Teams Use Feature Flags Wrong

They wire up LaunchDarkly or Unleash, use it for two A/B tests, then forget about it.

Meanwhile, their production is full of if (isNewCheckoutEnabled) blocks that nobody remembers how to toggle.

Feature flags are not primarily an experimentation tool. They're a reliability tool.

The Real Value

Feature flags let you separate deploy from release. You ship code to production cold, then turn it on gradually for real users.

When things break, you flip the switch back in 10 seconds. No rollback, no redeploy, no PR reverts.

The Four Reliability Patterns

1. Kill Switches

Every risky new feature ships behind a kill switch:

if (featureFlags.isEnabled('new_payment_flow', userId)) {
return newPaymentFlow();
}
return legacyPaymentFlow();

When the new flow has a bug, you don't rollback. You flip the flag.

2. Gradual Rollouts

new_search_algorithm:
rollout_percentage: 1 # Start at 1% of users
rules:
- if: "user.tier == 'internal'"
enabled: true # Internal users always see it

Deploy to 1%, watch metrics, go to 5%, watch, 25%, 50%, 100%. Takes 2-4 hours per rollout instead of a single risky deploy.

3. Circuit Breakers

external_recommendations_service:
enabled: true
automatic_disable_if:
error_rate_above: 5%
for_minutes: 5

If a downstream service starts failing, the flag auto-disables that feature. Your product degrades gracefully instead of crashing.

4. Load Shedding

expensive_realtime_dashboard:
enabled_when:
cpu_utilization_below: 70%
active_users_below: 50000

Under load, disable non-critical features to preserve the critical path.

The Anti-Pattern: Permanent Flags

After a feature is 100% rolled out, the flag should be deleted within 2 weeks. Every flag left in the codebase is technical debt.

Flag hygiene rules:

- Every flag has an expiration date (90 days max)
- Every flag has an owner in CODEOWNERS
- CI fails if a flag is older than 180 days
- Monthly flag cleanup is part of standard operations

We track "flag count" as a reliability metric. If it grows unbounded, we're doing it wrong.

The Architecture

A solid feature flag system has three parts:

1. Definition store

Source of truth for all flags
Versioned in Git or a managed service (LaunchDarkly, Unleash, GrowthBook)
Audit log for every change

2. Client SDK

In-app flag evaluation
Falls back to defaults if the service is unreachable
Caches decisions for 60 seconds
Emits telemetry for flag usage

3. Admin interface

Change flags without deploying code
See current state across environments
Role-based access (not everyone can flip prod flags)
Approval workflow for high-risk flags

Evaluating at the Right Layer

Flags can live at multiple layers:

CDN edge use for marketing experiments
Load balancer use for blue/green deploys
App server use for feature experiments
Database use for schema migrations

The deeper the layer, the faster the rollout. CDN flags flip in seconds. Database flags take minutes to propagate.

The Reliability Metric

Track: mean time to mitigate (MTTM).

If your team can mitigate an incident in under 30 seconds via a feature flag flip, that's a win. If you have to redeploy to mitigate, your reliability is bottlenecked by deploy time.

Good teams: MTTM under 60 seconds
Great teams: MTTM under 15 seconds

Common Gotchas

Stale flags skew A/B results clean them up after experiments
Flags without defaults cause prod outages every flag must have a safe fallback
Flag flips mid-request cause weird bugs evaluate at request start, cache for the request lifetime
Nested flags (flags inside flags) are impossible to reason about avoid

A Reliability-First Flag Strategy

Start simple:

Every new feature ships behind a kill switch
Gradual rollouts for anything touching the critical path
Circuit breakers for external dependencies
Flag cleanup is a monthly ritual
Track MTTM and optimize it

Feature flags are the most underrated reliability tool in modern engineering. Treat them that way.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

eBPF for SREs: Observability Without Agents

Samson Tanimawo — Mon, 27 Apr 2026 14:11:20 +0000

The Agent Problem

Traditional monitoring means shipping an agent with every service. That agent:

Adds memory overhead
Needs to be updated
Gets out of date
Breaks with kernel upgrades
Needs instrumentation code

eBPF says: what if the kernel itself could emit observability data?

What eBPF Actually Is

eBPF (extended Berkeley Packet Filter) lets you run sandboxed programs inside the Linux kernel without recompiling or loading modules. It was originally for packet filtering. Now it powers Cilium, Pixie, Falco, and dozens of other tools.

From an SRE perspective: you get deep visibility into syscalls, network traffic, process behavior, and filesystem operations with zero code changes to your applications.

What You Can Observe

network:
- every TCP connection (src, dst, bytes, duration)
- DNS queries and response times
- TLS handshake failures
- HTTP request/response cycles

application:
- function call latencies (uprobes)
- memory allocations
- lock contention
- GC pauses

security:
- syscall audit trails
- privilege escalations
- suspicious file access
- container escape attempts

performance:
- CPU scheduling delays
- I/O wait time per process
- disk latency histograms
- page fault patterns

All of this without modifying your application code.

A Practical Example: Detecting Slow HTTP Requests

Traditional approach: instrument your HTTP framework with OpenTelemetry, deploy a collector, ship traces.

eBPF approach:

# Install bpftrace
sudo apt install bpftrace

# Trace every HTTP response larger than 1MB
sudo bpftrace -e '
uprobe:/usr/lib/libssl.so:SSL_write {
@http_writes[pid] = count();
@http_bytes[comm] = sum(arg2);
}
'

No code changes. No restarts. Real-time visibility.

Tools Worth Knowing

1. Pixie (now part of New Relic)

Auto-instruments every service in your K8s cluster
No code changes, no sidecars
Full HTTP, MySQL, Postgres, DNS tracing
Open source

2. Cilium

Network observability + security policy enforcement
Replaces kube-proxy
Hubble UI for service-to-service traffic visualization

3. Falco

Runtime security detection
"Alert if a process inside a container spawns a shell"
Writes rules in YAML

4. Parca

Continuous profiling via eBPF
See CPU flame graphs across your entire fleet
Identify the most expensive code paths

5. Tracee

Security-focused eBPF tracing
Detects privilege escalations, cryptojacking, suspicious syscalls

The Tradeoffs

Pros:

Zero app code changes
Near-zero overhead (kernel-level efficiency)
Unified view across languages (Go, Python, Java, Rust, all seen the same way)
No agent lifecycle to manage

Cons:

Requires Linux 4.14+ (5.0+ preferred)
Steep learning curve for custom probes
Limited visibility into in-process logic (you see syscalls, not business logic)
eBPF verifier rejects programs for subtle reasons

When eBPF Shines

Network debugging: "Why is service A slow to reach service B?"
Security auditing: "What containers are making unexpected syscalls?"
Performance profiling: "Where is the cluster CPU time actually going?"
Incident forensics: "Reconstruct the syscall timeline during the outage"

When eBPF Is Wrong

Business logic observability you still need OpenTelemetry for spans
Application errors your logs and exception tracking still matter
Multi-region correlation eBPF is node-local

Use eBPF for infrastructure and network. Use OpenTelemetry for application logic. They complement each other.

Getting Started

Deploy Pixie in a dev cluster (1-line install)
Open the UI, watch real-time HTTP traffic
Try a bpftrace one-liner to trace a specific syscall
Read the Cilium + Hubble docs
Replace one agent-based tool with its eBPF equivalent

The future of observability is kernel-native. Agent-based tools will still exist, but the gap will keep shrinking.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Observability as Code: Managing Dashboards and Alerts with Terraform

Samson Tanimawo — Sun, 26 Apr 2026 14:11:06 +0000

The Problem with Click-Ops Dashboards

Your team has 200 dashboards. You don't know who owns them. Half are broken. The rest show yesterday's reality.

This is click-ops debt, and it compounds faster than code debt.

Observability as Code

Every dashboard, alert, and SLO definition should live in a Git repository alongside your service code.

resource "datadog_dashboard" "api_gateway" {
title = "API Gateway - Golden Signals"
description = "Owner: @platform-team"
layout_type = "ordered"

widget {
timeseries_definition {
title = "Request Rate (per second)"
request {
q = "sum:api.requests{service:gateway}.as_rate()"
}
}
}

widget {
timeseries_definition {
title = "P99 Latency"
request {
q = "max:api.latency{service:gateway}.as_count()"
}
}
}
}

This lives next to main.tf for your service. When you deploy the service, you deploy the observability.

Benefits That Compound

1. Ownership is clear. The file has a CODEOWNERS entry. PRs require review.

2. Dashboards auto-update. Renaming a service? Terraform refactor propagates to all dashboards.

3. Drift detection. Someone clicked "save as" in the UI and now that dashboard is out of sync. terraform plan catches it.

4. Review before production. Alert changes go through PR review. No more "who set this threshold?"

Tooling by Platform

datadog:
provider: DataDog/datadog
resources: datadog_monitor, datadog_dashboard, datadog_slo

grafana:
provider: grafana/grafana
resources: grafana_dashboard, grafana_alert_rule

prometheus:
approach: YAML files in Git, deployed by ArgoCD
resources: alert rules, recording rules

new_relic:
provider: newrelic/newrelic
resources: newrelic_alert_policy, newrelic_dashboard

Pick one source of truth. Don't mix.

A Real Example

We have a module that takes a service name and generates a complete observability stack:

module "service_observability" {
source = "./modules/observability"

service_name = "payment-processor"
team_slack = "#payments"
severity_map = {
error_rate_pct = 1.0
p99_latency_ms = 500
saturation_pct = 80
}

slo_targets = {
availability = 0.9995
latency_p99 = 0.99
}
}

One module call creates: 3 dashboards, 8 alerts, 2 SLOs, a Slack channel binding, and a PagerDuty escalation policy.

The Hardest Part

The code is easy. The hard part is:

Migrating existing click-ops dashboards budget 2 weeks
Getting engineers to edit YAML/HCL instead of the UI budget 3 months of reminders
Blocking UI edits some tools let you set dashboards to read-only
Reviewing alert changes PR reviewers need context

The Anti-Pattern to Avoid

Don't write Terraform for every custom chart an engineer wants. That leads to 500-line dashboard modules nobody understands.

Instead, define standard dashboards (golden signals, RED/USE, SLO burn rate) as modules. Let engineers add their own custom dashboards in the UI if they want, but mark them as "explore-only" (not alert-worthy).

Core observability = code. Experimental exploration = UI.

Migration Strategy

Week 1: Pick 1 service, convert its dashboards to Terraform
Week 2: Add alerts + SLOs to Terraform
Week 3: Delete the UI versions
Week 4: Create a module from the patterns
Month 2: Roll out to 10 more services
Month 3: Require all new services to use the module

Six months in, your click-ops debt is gone and your observability is reproducible.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Service Level Objectives for Complex Microservices

Samson Tanimawo — Sat, 25 Apr 2026 14:11:24 +0000

Why SLOs Break in Microservices

A SLO that works for a monolith often collapses when you distribute the same logic across 30 services. The math of availability is unforgiving.

If your service depends on 5 others, each at 99.9%, your realistic ceiling is 0.999^5 = 99.5%. That 0.4% gap eats your entire error budget before your own code even runs.

The Three Mistakes Teams Make

1. Copying the same SLO to every service

A 99.9% target on a payment service and a batch analytics service are not the same thing. One ruins revenue. One ruins dashboards.

2. Measuring uptime instead of user experience

GET /health returning 200 is not a SLO. Users don't call /health. They check out, log in, view pages. Measure those.

3. Ignoring fan-out

If a user request fans out to 8 downstream calls, and one of them has a 99% SLO, your user-facing reliability is capped at 99% no matter how good your code is.

A Practical SLO Framework for Microservices

user_journey_slos:
critical_path_checkout:
target: 99.95%
window: 30d
metric: "successful_checkouts / total_checkouts"
error_budget: 21.6 minutes / month

user_login:
target: 99.9%
window: 30d
error_budget: 43.2 minutes / month

background_analytics:
target: 99.0%
window: 30d
error_budget: 7.2 hours / month

Notice: we define SLOs on user journeys, not services. This is the biggest mental shift.

Composition Rules

When Service A depends on B and C:

A's SLO must account for B + C availability
If B is 99.9% and C is 99.5%, A's realistic SLO is ~99.4%
Build a dependency graph and calculate compound availability

We use a simple rule: no service promises a SLO tighter than the weakest service it depends on.

The Implementation Pattern

Every service exports three Prometheus metrics:

slo_requests_total{service,journey,status}
slo_budget_remaining{service,journey,window}
slo_burn_rate{service,journey,window}

From these three, you can compute:

Current SLO compliance
Budget remaining (in minutes)
Burn rate (how fast you're consuming budget)

Alert on burn rate, not on individual requests. A 2% error rate for 30 seconds is a blip. A sustained 2% error rate over 10 minutes is an incident.

Budget Policies That Actually Stick

The trick isn't defining SLOs. It's enforcing them. We use a 4-level policy:

budget_exhausted: freeze non-critical deploys, notify product
budget_50_pct: feature freeze, focus on reliability
budget_25_pct: normal operations, monitor carefully
budget_healthy: ship new features, experiment

When the budget is exhausted, product can't ship new features until reliability is restored. This alignment between eng and product is what makes SLOs stick.

Common Anti-Patterns

SLOs nobody looks at if you don't page on budget burn rate, they're dead
SLOs that never fail if you never breach budget, your targets are too loose
SLOs that always fail if you're always in the red, your targets are unrealistic
SLOs without product buy-in engineering-only SLOs get ignored during feature pressure

Final Thoughts

SLOs are a negotiation tool between engineering and product. Without them, every outage becomes a fight. With them, you have a shared contract about what "good enough" means.

Start with one critical journey. Measure it for 30 days. Set a realistic SLO. Enforce budget policies. Then add more journeys.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Building a Culture of Reliability: Beyond the SRE Handbook

Samson Tanimawo — Fri, 24 Apr 2026 22:08:36 +0000

You Can't Hire Your Way to Reliability

I've seen companies hire 5 SREs and expect reliability to magically improve. It doesn't. Reliability is a cultural outcome, not a headcount metric.

The Reliability Maturity Model

Level 1: Reactive
"Things break, we fix them."
No SLOs, no error budgets, post-mortems are optional.

Level 2: Aware
"We know what's breaking and how often."
Basic SLOs defined, post-mortems happen, on-call exists.

Level 3: Proactive
"We prevent most issues before they happen."
Error budgets enforced, chaos engineering started,
automated remediation for common issues.

Level 4: Predictive
"We predict and prevent issues we haven't seen yet."
ML-driven anomaly detection, capacity planning,
reliability is a product feature.

Level 5: Systemic
"Reliability is embedded in everything we do."
Every engineer thinks about reliability, every design
doc includes failure modes, every feature has SLOs.

Most companies are at Level 1-2. Getting to Level 3 is the biggest jump.

The Three Pillars of Reliability Culture

Pillar 1: Ownership

Reliability is not the SRE team's job. It's everyone's job.

ownership_model:
development_teams:
- Write SLOs for their services
- On-call for their services (with SRE backup)
- Fix production issues in their domain
- Include failure modes in design docs

sre_team:
- Build reliability infrastructure (monitoring, alerting, CI/CD)
- Consult on architecture for reliability
- Run chaos engineering program
- Manage cross-cutting reliability projects
- Train development teams on SRE practices

Pillar 2: Learning

Every incident is a learning opportunity. But only if you structure it:

def post_incident_learning(incident):
# 1. Blameless post-mortem (within 48 hours)
postmortem = write_postmortem(incident)

# 2. Share with entire engineering org
post_to_engineering_channel(postmortem.summary)

# 3. Add to searchable incident database
incident_db.insert(postmortem)

# 4. Extract patterns
similar = incident_db.find_similar(postmortem)
if len(similar) >= 3:
create_reliability_project(
title=f"Systemic issue: {postmortem.category}",
evidence=similar,
priority="high"
)

# 5. Update runbooks
if postmortem.new_knowledge:
update_runbook(postmortem.service, postmortem.new_knowledge)

Pillar 3: Investment

Reliability work needs dedicated time:

Engineering time allocation:
Feature development: 60%
Reliability work: 20%
Tech debt reduction: 10%
Learning/experimentation: 10%

The 20% reliability budget includes:
- Alert tuning and noise reduction
- Runbook automation
- Chaos experiments
- SLO reviews and adjustments
- On-call process improvements
- Monitoring and observability improvements

Protect this 20%. When leadership pressures to ship more features, show the correlation between reliability investment and incident reduction.

Measuring Culture

You can't manage what you can't measure. Cultural metrics:

reliability_culture_metrics:
# Engineering engagement
postmortem_attendance_rate: target > 80%
action_item_completion_rate: target > 90%
runbook_update_frequency: target > 2x/month per service

# Design quality
design_docs_with_failure_modes: target > 95%
new_services_with_slos: target 100%
chaos_experiment_frequency: target > 1x/quarter per service

# Team health
oncall_nps: target > 0
developer_survey_reliability_confidence: target > 4/5
sre_team_attrition_rate: target < 10%/year

The Quick Wins

If you're starting from Level 1, here are the highest-impact changes:

Week 1: Define SLOs for your top 3 services. Just availability + latency.
Week 2: Make post-mortems mandatory and blameless. Use a template.
Week 3: Set up on-call rotation with clear escalation paths.
Week 4: Create a reliability Slack channel. Share learnings daily.
Month 2: Start tracking error budgets. Share with product managers.
Month 3: Run your first chaos experiment (kill a pod, see what happens).

Six weeks from chaos to competence. Not perfect, but dramatically better.

If you want to accelerate your reliability culture with AI-powered tools that embed SRE best practices, check out what we're building at Nova AI Ops.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com