DEV Community: Samson Tanimawo

From DevOps to SRE: Making the Transition

Samson Tanimawo — Wed, 22 Jul 2026 22:39:19 +0000

I moved from a DevOps title to an SRE title about 6 years ago. On paper, they look similar. In practice, the mindset is different. Here's what actually changed for me.

DevOps: 'how do we ship faster?'

SRE: 'how do we ship faster without breaking things?'

DevOps is a culture and a set of practices for getting software from code to production quickly. SRE adds a discipline of reliability as a first-class concern.

What changed for me

I stopped being a YES person on deploys. As a DevOps engineer, I was judged on deployment frequency. As an SRE, I was judged on reliability too. Sometimes the right answer to 'can we deploy this now?' is 'no, not until we have a rollback plan.'

I started thinking in error budgets. Not 'is the site up?' but 'have we used too much of our error budget this quarter?' That frame changes everything about how you prioritize work.

I stopped fixing things that weren't my job. SREs are supposed to push work back to product teams when those teams' code is the problem. The first few times you say 'this is not an infrastructure issue, please fix your code,' it feels rude. Then you realize that fixing it for them is the trap.

The skill gap

If you're making this transition, the skills you probably need to build:

Statistics (for SLO math and anomaly detection)
Deep debugging of production systems (not dev)
Writing post-mortems that actually change systems
Saying no to feature work when the reliability budget is spent

The hardest part isn't learning tools. It's accepting that reliability is an ongoing discipline, not a project that ends.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

The SRE Interview: Questions I Actually Ask

Samson Tanimawo — Wed, 22 Jul 2026 13:49:15 +0000

I've interviewed maybe 300 SRE candidates over the years. Most interviews are bad. Leetcode for people who don't write production code. System design questions ripped off from a blog.

Here are the questions I actually ask, and why.

'Walk me through your last bad incident'

The best SREs can tell you about incidents with specificity. They remember the graphs, the timeline, the moment they realized what was wrong, and what they'd do differently. Vague answers mean they weren't really there.

'Your on-call phone rings at 3 AM. Walk me through the first 10 minutes'

Reveals process discipline. Good answers: acknowledge → assess impact → communicate → start diagnosis. Bad answers: jump straight to SSH-ing into a box.

'Tell me about a time you killed an alert'

Not 'added an alert.' Killed one. Shows they understand alert hygiene as a discipline, not a chore.

'What's the difference between an SLI and an SLO?'

Tests fundamentals without being a trick question. You'd be surprised how many senior candidates conflate these.

'What's the most useful tool you've built for yourself?'

Good SREs are tool builders. The answer should be small, specific, and boring — a script that saves 5 minutes a day, not a grand platform.

What I don't ask

'Reverse a linked list on a whiteboard.' SRE is not a coding contest. I care whether you can debug a broken system at 3 AM, not whether you memorized CLRS.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Incident Retrospectives Without Blame

Samson Tanimawo — Tue, 21 Jul 2026 23:52:45 +0000

I've run over 100 post-mortems. The worst ones end with 'Alice will be more careful.' The best ones end with 'we fixed the system.' Here's how you get from the first to the second.

The language rule

Ban these phrases from retros:

'Should have...'
'Alice forgot to...'
'If only...'

Replace with:

'The system let this happen because...'
'The runbook didn't cover...'
'The signal was missing...'

People make mistakes. Systems that allow those mistakes to reach production are the actual bug.

The 5 whys that work

Classic 5 whys often end at human error. Push through that.

Why did the outage happen? Alice deployed broken config.
Why did broken config deploy? Our config validation didn't catch it.
Why didn't validation catch it? It didn't cover this edge case.
Why didn't it cover it? We didn't have a test for this case.
Why didn't we have a test? Nobody owns the config validation pipeline.

Action item: assign an owner to config validation and add the missing test. That's a system fix.

The people thing

Blameless doesn't mean consequence-free. If someone keeps making the same mistake after the system is fixed, that's a management issue, not a retro issue. Handle it privately, not in the retro.

Retros are for learning. Everything else belongs elsewhere.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Alert Fatigue: The Silent Productivity Killer

Samson Tanimawo — Tue, 21 Jul 2026 13:39:56 +0000

Alert fatigue isn't a wellness issue. It's a production risk. And it's cheaper to fix than most people think.

The math

An engineer getting 30 alerts a night sleeps badly. Sleeps badly → cognitive load next day drops 20%. Drops 20% → higher chance of introducing a bug. That bug becomes the next incident. The next incident is another 30 alerts.

You are in a loop.

How it actually happens

Nobody sets out to create alert fatigue. It happens one alert at a time.

'Let's add an alert for this' (reasonable)
'The threshold is too sensitive, let's keep it but note it' (rationalizing)
'It fires sometimes but we know what it means' (tolerating)
'We should probably look at that' (normalizing)

Two years later you have 400 alerts and nobody remembers why half of them exist.

The fix

Delete every alert that hasn't caused action in 30 days. If nobody acted on it, it's not an alert.
Raise the threshold on noisy alerts until they only fire for real problems. Your boss is not going to fire you for missing noise.
Group related alerts. One incident should page once, not 40 times.
Set a max alerts/day per engineer rule. When you hit it, something gets cut.

Teams that aggressively prune alerts sleep better, ship more, and have fewer real incidents. It's not a trade-off. It's just good hygiene.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Why SLIs Matter More Than SLOs

Samson Tanimawo — Tue, 21 Jul 2026 01:54:45 +0000

SLOs get all the attention. I want to argue that your SLIs are more important.

Here's the thing: an SLO is a number you pick. 99.9% uptime. 300ms p95 latency. Whatever. It's a decision.

An SLI is what you're actually measuring. Is it the right signal? Does it reflect user experience? Is it gamed by caching? Those questions matter more than the target.

Bad SLI, good SLO

'99.9% of healthcheck requests return 200.' Looks great on paper. Means nothing to users. Your healthcheck endpoint can be up while your actual API is broken.

Good SLI, any SLO

'99.x% of user-initiated checkout requests complete successfully within 5 seconds.' That signal tells you if the product works. Whatever target you pick, you're measuring the right thing.

The test

When your SLO is missed, does on-call get paged because a real user is suffering? If yes, your SLI is good. If no, you're measuring the wrong thing.

Pick the SLI first. Pick the SLO second. Most teams do it backwards and wonder why their reliability work doesn't move the needle.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

The PagerDuty Migration Playbook

Samson Tanimawo — Mon, 20 Jul 2026 13:28:32 +0000

Migrating from PagerDuty is not a weekend project. I learned this the hard way. Here's the playbook I wish I had.

Before you start

Audit every integration. Alerts, webhooks, Slack hooks, status page syncs. Ours had 47. I found 12 we had forgotten about.
Export every schedule. Including overrides and holiday rules.
Snapshot every escalation policy.
Document your notification rules (who gets SMS vs push vs call).

The parallel period

Do not cutover. Run both systems side-by-side for at least 2 weeks.

Route a small % of alerts to the new system first.
Verify every alert fires in both.
Let on-call engineers use the new app but keep PagerDuty as backup.
Watch for silent failures (missing integrations usually hide here).

The cutover

Pick a Monday morning, not a Friday night. Cutover in stages:

Non-critical services first
Monitor for 24 hours
Critical services next
Keep PagerDuty subscribed but route-away for 1 more week in case of rollback

The cleanup

Don't forget the ghost integrations. Old CI jobs, dead services, third-party tools. They'll keep sending to PagerDuty silently until someone notices the bill.

Total time for a 50-person team migration: 6 weeks, not 6 days. Plan accordingly.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

How We Cut Datadog Bills by 60% Without Losing Observability

Samson Tanimawo — Sun, 19 Jul 2026 23:50:57 +0000

Last year our Datadog bill hit $38k/month. Leadership asked me to cut it in half. Here's how we got to $15k without losing a single useful signal.

The 5 moves that worked

1. Dropped custom metrics with zero dashboard references. We had 2,400 custom metrics. Only 600 were actually graphed or alerted on. We stopped sending the other 1,800. Saved 30%.

2. Aggressive log tier management. Hot logs for 3 days, warm for 7, then cold. Most of our log cost came from full indexing of debug logs nobody queried after 24 hours.

3. Lower-cardinality tags. We were tagging metrics with user_id. That's millions of series. We moved user-level stuff to traces and kept metrics aggregate-only.

4. Dropped synthetic monitors for dev environments. We were running 200+ API checks on dev. We only need them on prod.

5. Consolidated APM sampling. Instead of 100% sampling on everything, we did 10% sampling on healthy traces, 100% on errors and slow requests. Cut APM volume by 85% with no loss in signal.

What didn't work

Trying to negotiate a discount with sales. They'll give you 5-10% if you threaten to leave. Actually cutting volume gets you 60%.

The lesson: your observability bill is almost always a data hygiene problem, not a pricing problem.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Building Your First Runbook: A Template That Actually Works

Samson Tanimawo — Sun, 19 Jul 2026 13:17:20 +0000

Most runbooks are useless. Either they're too abstract ('check the logs') or they're a 40-page Confluence doc that nobody reads at 3 AM.

Here is the template I use. It fits on one page and works.

The template

1. Trigger. The exact alert name and what it means.

2. Impact. Who is affected? What are they seeing? Is this user-facing?

3. First 5 minutes. The single most useful command to run. One. Not five.

4. Common causes. The 3 things that most often cause this alert, in order of likelihood.

5. Fix per cause. For each common cause, the exact fix. Copy-paste-ready.

6. Escalation. Who to page if none of the above works. Include their timezone.

7. Post-incident. What to update after the incident is done (ticket, dashboard, doc).

Why this works

At 3 AM, your brain is running at 60%. You need a runbook that gives you the next action in under 30 seconds. A 40-page doc makes you think. A one-pager tells you what to do.

Start with your noisiest alert. Write the runbook. Test it on a new team member. If they can follow it without you, it works.

Repeat for your top 10 alerts. That's 90% of your on-call load handled.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

AIOps vs Traditional Monitoring: What Actually Changed

Samson Tanimawo — Sun, 19 Jul 2026 00:27:11 +0000

Every vendor now slaps 'AIOps' on the box. Most of them just added a dashboard that says 'anomaly detected' and called it a day.

I want to tell you what AIOps actually changes, and what it doesn't.

What it actually changes

Correlation. Traditional monitoring alerts on a symptom — CPU spike, 500 error, queue depth. AIOps correlates across signals and tells you one story: 'deploy 14a23 on payments-api broke the checkout flow, here are the 7 alerts it triggered.'

Noise reduction. On my old team we got 300 alerts/day. About 40 of them mattered. The rest were duplicates, known-flaky services, or transient spikes. A good AIOps layer suppresses 80% of that before a human sees it.

Root cause suggestions. Not answers — suggestions. 'The top 3 likely causes based on historical incidents are...' Still needs a human to confirm. But it saves you 20 minutes of dashboard-hopping.

What it doesn't change

You still need good instrumentation. You still need runbooks. You still need someone on-call who can make a call.

AIOps is not a replacement for SRE. It is a force multiplier for SREs who already know what they're doing.

If you're drowning in alerts, the answer isn't more dashboards. It's letting AI do the triage so your humans can do the thinking.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Eventual Consistency: Debugging the Hardest Class of Bugs

Samson Tanimawo — Sat, 18 Jul 2026 14:08:10 +0000

The Bug That Only Happens Sometimes

User reports: "I updated my profile but it still shows the old name."
You refresh: new name appears.
You ask them to refresh: still old name.
You debug together: it's back to new name.
The user: "This is clearly broken."
You: "It's working for me..."

Congratulations, you're debugging eventual consistency.

What Eventual Consistency Actually Means

In a distributed system, a write to one node takes time to propagate to others. During that window:

Some reads see the new value
Some reads see the old value
All reads will eventually see the new value

That "eventually" window can be milliseconds or seconds. Sometimes minutes if something is wrong.

Users don't care about CAP theorem. They care that they updated their profile and it shows wrong.

Why It's So Hard to Debug

Three reasons:

1. Non-deterministic reproduction

You can't reliably reproduce it. Sometimes the bug fires, sometimes it doesn't. This makes unit tests useless and traditional debugging painful.

2. Time-dependent

The bug depends on when the read happens relative to the write. Microseconds matter. Your laptop is too fast to see it locally.

3. State propagation is invisible

Logs show you the read and the write. They don't show you "this read hit replica 2 which hadn't received the replication event yet."

The Common Patterns

Pattern 1: Read your own writes

User updates profile → write goes to primary → user reads → read hits a replica that hasn't replicated yet → user sees stale data.

# WRONG
def update_profile(user_id, new_data):
    primary.write(user_id, new_data)
    return read_from_replica(user_id)  # Might be stale!

# RIGHT
def update_profile(user_id, new_data):
    primary.write(user_id, new_data)
    return primary.read(user_id)  # Guaranteed fresh

Pattern 2: Cache staleness

Write goes to database, but the cache still has the old value.

# Fix: invalidate on write
def update_profile(user_id, new_data):
    db.write(user_id, new_data)
    cache.delete(user_id)  # Force next read from DB

Or use a shorter TTL. Or use read-through caching.

Pattern 3: Async propagation

User clicks "upgrade subscription" → API returns 200 → subscription_service emits event → billing_service processes → account_service updates → user's next page load still shows "trial"

# Fix: wait for propagation before returning
async def upgrade(user_id, plan):
    result = await subscription_service.upgrade(user_id, plan)
    # Wait for downstream to process
    await wait_for_account_update(user_id, plan, timeout=5)
    return result

Pattern 4: Eventually consistent views

Using Elasticsearch as a read replica of Postgres. Writes go to Postgres, then replicate to ES. If you write + read, you read the old value.

# Fix 1: read from primary for recent writes
def get_user(user_id, recent_write=False):
    if recent_write:
        return postgres.get(user_id)
    return elasticsearch.get(user_id)

# Fix 2: tracking epoch and waiting
def get_user(user_id, min_epoch=None):
    if min_epoch:
        wait_for_replication_to_reach(min_epoch)
    return elasticsearch.get(user_id)

The Detection Strategy

Since bugs are non-deterministic, detection has to be probabilistic:

# In your logging middleware:
def log_inconsistency(user_id, field, expected, actual):
    if expected != actual:
        metrics.increment('consistency.mismatch', tags=[
            f'field:{field}'
        ])
        logger.warn('Consistency mismatch', user=user_id, 
                    field=field, expected=expected, actual=actual)

Monitor the mismatch rate. If it's > 0.01%, you have a real problem.

The Diagnostic Questions

When debugging, ask:

How many replicas are involved? More replicas = longer propagation delay.
What's the normal replication lag? Check monitoring. Normal is usually < 1 second.
Is the lag elevated right now? A spike from 100ms to 5s is a red flag.
Was the write synchronous or async? Async writes have no guarantee of being visible on reads.
Does the client have a retry? Retries at the wrong layer can produce duplicates or stale reads.
Is there a cache in the path? Caches are the most common source of "phantom staleness."

The Debugging Tools

1. Replication lag metrics

For PostgreSQL:

SELECT now() - pg_last_xact_replay_timestamp() AS replication_lag;

For MongoDB:

rs.printSecondaryReplicationInfo()

For Redis:

INFO replication

Alert on lag > 1 second sustained.

2. Request tracing

Use OpenTelemetry to trace a single user action across services. You'll see where time is spent and which service is talking to which replica.

3. Log all reads with node ID

Every read should log which node/replica served it. When debugging, you can see if the bug correlates with a specific replica.

4. Deterministic test environment

Build a test environment where you can introduce artificial replication delay. Helps reproduce bugs locally:

# In test
replica.delay_ms = 500  # Force 500ms replication lag
# Run the flow, assert consistency

The Fix Strategies

Strategy 1: Strong consistency where it matters

Not everything needs to be eventually consistent. For critical flows:

User-visible state changes → read from primary
Financial operations → strong consistency (transactions)
Security operations (auth, permissions) → strong consistency

Accept the latency hit for correctness.

Strategy 2: Causal consistency

Track "you just wrote X" and ensure subsequent reads see X:

# Client sends "min_epoch" based on last write
def handle_request(request):
    if request.min_epoch:
        wait_for_replication(request.min_epoch)
    return db.read(request.key)

Strategy 3: User-visible time

Sometimes the solution is UX, not infrastructure:

"Your changes are saved. They may take up to 30 seconds to appear."

Set expectations. Users understand "saved but syncing" better than "works for you but broken for me."

Strategy 4: Accept and monitor

Some inconsistency is tolerable. Log it, alert if it exceeds a threshold, and fix the worst offenders.

The Hardest Bugs

The worst eventual consistency bugs happen at failure recovery time. A write that was in-flight during a network partition might:

Succeed on primary but fail to replicate
Fail on primary but succeed on one replica
Be applied in a different order than another write
Be lost entirely

These are fundamental distributed systems problems. Defense: idempotent operations, client-side retries with unique IDs, careful read-after-write semantics.

The Takeaway

Eventual consistency isn't a bug. It's a tradeoff.

The bug is:

Not understanding where consistency matters in your system
Not having visibility into replication lag
Not documenting consistency guarantees to users

Fix these, and "eventually consistent" stops being a dirty phrase and starts being a feature.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

The Economics of Self-Hosting vs. Managed Monitoring

Samson Tanimawo — Sat, 18 Jul 2026 00:06:28 +0000

The "Obvious" Math That's Wrong

Engineer A: "Datadog is $15K/month. Prometheus is free. We should self-host."

Engineer B: "But we'd need to pay an SRE to run it. That's $150K/year."

Engineer A: "Prometheus doesn't need a full SRE. It's easy."

Engineer B: "Famous last words."

This conversation happens at every company. Both sides have points. The real math is more complex.

The Total Cost Breakdown

Managed (Datadog, New Relic, Dynatrace):

Licensing: $X/month (scales with hosts, events, logs)
Integration time: 1-2 weeks per service
Training: 1 day per new hire
Ongoing: minimal

Self-hosted (Prometheus + Grafana + Loki + Alertmanager):

Infrastructure: hosting costs (~$500-$5000/month depending on scale)
Initial setup: 2-4 weeks of engineering time
Ongoing maintenance: 10-20% of 1 FTE
Upgrade costs: quarterly, each upgrade ~1 week
Storage growth: ~20% per year
Expertise: junior → senior SRE hire required

The honest answer: managed is cheaper for teams under 50 engineers. Self-hosted becomes cheaper around 200+ engineers if you can run it well.

The Real Variables

It's not just licensing cost vs. hosting cost. These factors matter more:

1. Data volume growth

Managed tools charge per GB ingested or per metric. If your logs 10x, your bill 10x's.

Self-hosted scales linearly with compute. You control the growth.

2. Retention requirements

Managed tools often charge extra for long retention. Self-hosted you store as much as your disk allows.

3. Cardinality

Prometheus dies at high cardinality. Datadog handles it but charges more. High-cardinality metrics are where self-hosted breaks.

4. Incident rate

Heavy incident load means heavy query load on your monitoring tools. Self-hosted needs bigger compute for this.

5. Team expertise

If your team has never run Prometheus, you'll spend 6 months in the pit learning cardinality mistakes, retention tuning, and HA setups. That's not free.

The Break-Even Calculation

Rough calculation for a 50-engineer startup:

Managed (Datadog):
- Licensing: $10K/month = $120K/year
- Maintenance: ~10 hours/month of engineering time
- Total: ~$125K/year

Self-Hosted (Prometheus + Grafana + Loki):
- Infrastructure: $2K/month = $24K/year
- Maintenance: ~30% of 1 SRE = $45K/year (loaded)
- Initial setup: $30K one-time
- Total year 1: $99K
- Total year 2+: $69K/year

At 50 engineers, self-hosted saves ~$60K/year once you're past the setup phase.

BUT:

If the SRE quits, you're in trouble
If the cardinality explodes, you're in trouble
If an upgrade fails, you're in trouble
If you need 24/7 reliability of monitoring itself, add another $50K/year

Factor these in honestly before you commit.

The Hybrid Approach

Most teams we see end up hybrid:

Metrics: Self-hosted Prometheus (cheap at scale)
Logs: Managed service (complex to self-host at scale)
Traces: Managed service (requires specialized knowledge)
Alerting: Self-hosted Alertmanager (simple, stable)
Dashboards: Self-hosted Grafana

This captures the best of both:

Cheap for commodity metrics
Paid for complex/specialized (logs, traces)
Predictable ongoing costs

When Managed Is Obviously Right

Team has < 20 engineers
No one has ops expertise
Growing fast, need to ship features
Compliance requires certified tools (SOC2, HIPAA specifically)
Need it running in 2 weeks, not 3 months

When Self-Hosted Is Obviously Right

Team has 100+ engineers
Strong ops culture already in place
Budget pressure from managed costs
Very high data volume (10TB+/day logs)
Want full control over data (privacy, residency)

The Gray Zone (50-200 engineers)

Most startups land here. The calculation is close. Three factors break the tie:

Factor 1: Engineering time availability

If your SRE team is underwater with incidents, adding "maintain Prometheus" to their plate is a disaster. Pay for managed.

Factor 2: Growth rate

If you're doubling engineers every 6 months, managed costs explode. Build self-hosted capacity now.

Factor 3: Data sovereignty

Some customers won't accept their data leaving your infrastructure. This forces self-hosted regardless of cost.

The Hidden Costs of Managed

Managed isn't just the invoice. Also consider:

Integration debt: every service you add to Datadog is a service you're locked into Datadog
Custom metric costs: 10M custom metrics in Datadog is expensive
Long-term pricing risk: vendors raise prices once you're locked in
Export restrictions: getting your data out is often painful
Support quality: P1 support response times vary wildly

Factor these into your TCO.

The Hidden Costs of Self-Hosted

On-call for monitoring: when Prometheus dies, someone has to fix it
Upgrade risk: each major version can break dashboards
Scaling anxiety: "will it handle the next 2x growth?" is a question you ask yourself weekly
Knowledge concentration: if the one expert leaves, you're stranded
Integration maintenance: every new service needs exporters/agents

These aren't line items on an invoice. They're real costs.

The Decision Framework

Answer these questions honestly:

How many engineers can dedicate time to monitoring maintenance?
How fast is your data volume growing?
What's your current monthly bill for managed tools?
How much pain is there from vendor lock-in?
What's the opportunity cost of engineering time spent on monitoring?
Can you recruit or train the expertise?

If you can't answer #1-#6 confidently, stay managed until you can.

The Recommendation

Years 0-2: Managed. Ship features, don't fight your monitoring stack.

Years 3-5: Start evaluating. Maybe hybrid. Hire an SRE with deep ops experience.

Years 5+: Likely hybrid or mostly self-hosted. Costs justify the complexity.

The best monitoring stack is the one your team can actually operate. If that's managed, pay the bill. If that's self-hosted, invest in the expertise. Don't try to do both poorly.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Building an Incident Response Playbook Library

Samson Tanimawo — Fri, 17 Jul 2026 13:46:27 +0000

The Folder Full of Stale Runbooks

Every engineering org has a Confluence folder of incident runbooks. Every runbook was written during or after an incident. Each is a snapshot of how to fix one specific thing.

After 2 years, the folder has 400 runbooks. Nobody knows which are current. During the next incident, nobody opens any of them.

A playbook library is supposed to help. Usually it just generates debt.

What Playbooks Should Actually Do

A good playbook library:

Reduces cognitive load during incidents
Provides confidence to junior responders
Captures institutional knowledge
Trains new hires

A bad playbook library:

Is out of date
Is impossible to search
Has no ownership
Contradicts itself across runbooks

The difference is process, not content.

The Structure

Every runbook follows the same template:

# Title: [Specific Problem Being Fixed]

## When to Use This
- Alert that fires: [exact alert name]
- Symptoms: [what the user sees]
- Impact level: [SEV-1/2/3/4]

## Quick Fix (90 seconds)
1. Command to run
2. Command to run
3. Verification step

## Deeper Investigation (if quick fix fails)
- Check this
- Look at this dashboard
- Tail these logs

## Root Cause Category
- Known cause 1: [link to deeper doc]
- Known cause 2: [link to deeper doc]

## Escalation
- Primary: [role]
- Secondary: [role]
- SME: [specific person]

## Related Runbooks
- [Similar problem A]
- [Similar problem B]

## Metadata
- Owner: [team]
- Last Verified: [date]
- Expires: [date + 3 months]

Every field is required. Missing fields fail CI.

The Ownership Rule

Every runbook has an owner. The owner is responsible for:

Keeping it current
Re-verifying it quarterly
Updating when the underlying system changes
Deleting it when no longer relevant

If a runbook has no owner, delete it. A stale runbook is worse than no runbook — it wastes time during incidents and may contain wrong instructions.

The Expiration Date

Every runbook has a 90-day expiration. After 90 days:

CI warns the owner
After 30 more days, CI fails builds that reference it
After 60 more days, the runbook is auto-moved to archive

The owner must re-verify and reset the expiration date. Re-verification means:

Read it end to end
Try the commands in staging
Update anything that's changed
Set a new expiration date

This forces continuous maintenance. It's painful. It also means the runbooks are trustworthy.

The Discovery Problem

A library of 400 runbooks is useless if you can't find the right one during an incident.

Three techniques:

1. Alert-to-runbook mapping

Every alert includes a runbook link:

alert: HighErrorRate
annotations:
  summary: "API error rate above 5%"
  runbook: "https://runbooks.internal/api/high-error-rate"

When the alert fires, the runbook is one click away. No searching.

2. Symptom-based search

We tag runbooks by symptom:

symptoms:
  - "slow response time"
  - "database queries timing out"
  - "connection pool exhausted"

During an incident, you search by symptom, not by service name.

3. AI-assisted search

A Slack bot that takes a natural-language description and returns the top 3 relevant runbooks. Only works if runbooks are well-structured.

The Writing Rule: Before You Have the Outage

The worst time to write a runbook is during an incident. You're stressed, you're in a hurry, you'll write something incomplete.

The best time to write a runbook is:

During a change: new service? write the runbook before launch
During a quiet week: pick a service you're familiar with, write the runbook
During a post-mortem: document what you just learned while it's fresh
During a drill: tabletop exercises expose gaps

Companies that write runbooks proactively have better incident response than those who write reactively.

The Post-Incident Rule

Every incident with a post-mortem generates at least one runbook update. Either:

Create a new runbook if this was a novel problem
Update an existing runbook if the old instructions didn't work
Delete a runbook if it was wrong or misleading

No post-mortem is complete without a runbook change. We enforce this in the template.

The Hidden Cost of Playbooks

Maintaining a library of 100 current runbooks takes real time. Approximately:

Write new runbook: 2 hours
Quarterly verification: 30 min/runbook/quarter
Post-incident update: 1 hour/incident
Search index maintenance: 2 hours/month

Total: ~10 hours/week for a mature library

Budget this time. If you don't, your library will rot.

When NOT to Write a Runbook

Counter-intuitive: sometimes the right answer is not to write a runbook.

If the fix is:

Obvious from the error message → no runbook needed
Already automated → delete the manual path
Only applicable once (one-time migration) → write a ticket, not a runbook
Constantly changing → document the general approach, not specific commands

Runbooks are for recurring, stable, manual procedures. Everything else belongs elsewhere.

The "Death Star" Runbook

The biggest runbook anti-pattern: one giant runbook that covers "incident response" generically. 50 pages long. No specific triggers. No specific steps.

These exist because nobody wanted to write specific ones, so they wrote one massive generic one.

Nobody reads them during incidents. They're doorstops.

Delete them. Replace with specific, focused runbooks tied to specific alerts.

The Starter Kit

If you're starting a runbook library from scratch:

List your top 20 most-fired alerts in the last 6 months
Write a runbook for each of them
Link each alert to its runbook
Set quarterly verification cycles
Assign owners
Put them in version control
Review monthly with your team

After 6 months, you'll have 30-50 high-quality runbooks that are actually used. This is infinitely more valuable than 400 stale ones.

The Ultimate Test

During your next incident, watch the responder:

Did they open a runbook? (good)
Was the runbook current? (better)
Did it contain the right answer? (best)
Did it resolve the incident faster than winging it would have? (mission accomplished)

If the answer to any of these is "no," your library has work to do.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com