DEV Community: Abhishek Singh

Your Disaster Recovery Runbook Is Wrong, and the Auditor Knows It

Abhishek Singh — Wed, 30 Sep 2026 14:23:01 +0000

Every enterprise I have worked with has a disaster recovery runbook. Almost none of them have one that would work at 3 a.m.

I know because I have been on the bridge when a real outage hit a site whose runbook was reviewed and approved eleven months earlier. The IP addresses were wrong. The database had been migrated to a new cluster. The vendor contact had left. The restore order listed a system that had been decommissioned and omitted the identity provider that everything else now depended on. We recovered, but we recovered from memory and from Slack, and the runbook was open on someone's second monitor as a formality.

Then, two months later, the auditor asked for evidence that the DR plan had been tested, and the evidence pack was a folder of screenshots with no timestamps.

This article is about the gap between a runbook that satisfies a review and a runbook that recovers a hospital, and the small set of practices that close it.

Why runbooks rot

A DR runbook is a snapshot of an environment on the day it was written. The environment changes every week: new VMs, a migrated database, a changed replication target, a new dependency introduced by an application upgrade. Nobody updates the runbook for each change because the runbook lives in a Word document on SharePoint and the change lives in a ticket.

The result is that a runbook is most accurate on the day it is approved and steadily less accurate every day after. By the yearly test it is wrong in ways that only show up during the test, and by the real disaster it is wrong in ways that show up at the worst possible moment.

What a runbook actually has to contain

Strip away the boilerplate and a working DR runbook has six parts. If any is missing, the document is a policy, not a runbook.

Declaration criteria and authority. Who decides this is a disaster, on what evidence, and who they call first. Name the role and the person, with a phone number that was verified in the last ninety days.
The dependency order. Identity and network first, then data, then application, then edge. Written as a list that a tired engineer can follow, not a diagram they have to interpret. Every system has a tier, and no tier starts until the previous one is verified.
Per-system restore steps with the exact commands, console paths or recovery-plan names, the expected duration, the owner role, and the verification check that proves it worked. "Fail over the database" is not a step. "Run ASR recovery plan HIS-DB-DR; confirm SQL listener answers on 10.x.x.x; run the smoke query" is.
Contacts — internal, vendor, client — with an escalation path. Verified, dated.
Communication templates. What the client's leadership is told at declaration, hourly, and at resolution. Written in advance because nobody writes well at 3 a.m.
Failback. How you get back to the primary site, which is the half of DR that nobody tests and everyone needs.

The practices that keep it alive

Generate the dependency order from the inventory, not from memory

The single biggest source of runbook error is the restore order being wrong because dependencies changed. The fix is to stop writing the order by hand. Pull the inventory from the source of truth — the cloud provider's resource graph, the CMDB, a maintained CSV for on-prem — and derive the order from the dependency edges. When a new database appears, it appears in the order; when a system is decommissioned, it drops out.

This is not exotic. A spreadsheet with two columns, system and depends on, and a sort is enough for a small site. For a large one it is a query against the CMDB. Either way the order is regenerated, not remembered.

Diff the runbook against reality every week

Once the inventory drives the runbook, checking for drift is cheap: compare this week's inventory to the one the approved runbook was built from. New systems, removed systems, changed replication status. Anything that differs is a flag on the runbook, and the runbook is not "current" until someone has looked at the flag.

In practice this turns the yearly panic into a ten-minute weekly task, and it means the runbook the team opens during a real incident was last reconciled days ago, not months.

Make the test produce evidence as a by-product

The auditor's question is not "did you test DR?" It is "show me". Show the steps that were run, by whom, when, with what result, mapped to the control that requires it — HIPAA §164.308(a)(7)(ii)(D) for testing and revision, ISO 27001 A.17.1.3 for verifying continuity.

The way to have that evidence is to not collect it afterwards. Run the drill from a checklist where each step records the operator, the start and end time, the pass or fail, and a screenshot or log excerpt at the moment of execution. When the drill ends, the evidence pack exists. Nobody spends the following week hunting screenshots.

Two details make the pack audit-proof. Timestamps come from the system, not typed. And the finished pack is hashed — a SHA-256 printed on the cover — so that the auditor, or anyone, can confirm it was not edited after the fact.

Run tabletops between the real tests

A full failover test once a year is expensive and disruptive, which is why it happens once a year. A tabletop — walking the runbook step by step against the current inventory, with the on-call team, without touching anything — takes ninety minutes and can happen quarterly. It finds the wrong IP and the departed vendor contact in a conference room instead of on a bridge.

Close the findings before the next test

Every test produces findings: a step that took twice as long as expected, a system that would not fail over cleanly, an RTO missed. The runbook is only as good as the fixes. Put each finding into the ticketing system with an owner and a due date, and make "open DR findings" a number that appears in the client's quarterly review. Findings with no owner are still open a year later; findings in a review deck get closed.

What the client sees

When this works, the client-side experience changes in three ways. The IT director sees a readiness status that is green, amber or red today, not a PDF dated last spring. The auditor receives a pack that answers the question without a follow-up. And the on-call engineer, at 3 a.m., opens a runbook that was reconciled last Tuesday and follows it.

That last one is the point. Everything else is how you get there.

A checklist

[ ] Every contact in the runbook verified in the last 90 days
[ ] Restore order generated from the inventory, not written by hand
[ ] Weekly drift check between inventory and approved runbook
[ ] Every step has command, owner role, expected duration, verification check
[ ] Drill run from a checklist that captures operator, timestamps, result, evidence
[ ] Evidence pack mapped to controls and hashed
[ ] Quarterly tabletop, annual full test
[ ] Findings tracked with owners and reviewed with the client

If four or more are unticked, the runbook will fail when it is needed. Fix the order and the drift check first; they remove most of the risk for the least effort.

Abhishek Singh leads incident management and monitoring for enterprise healthcare clients on Azure, with seven years in enterprise IT operations including five at Acquia as a Senior Support Engineer.

The RCA Nobody Reads: How to Write a Root Cause Analysis That Changes Something

Abhishek Singh — Wed, 30 Sep 2026 10:37:28 +0000

I have written a great many root cause analyses for enterprise healthcare clients. For the first year, almost none of them changed anything. The document went to the client, the client filed it, and three months later the same class of incident happened again.

The problem was not the analysis. The problem was that I was writing the RCA for the wrong reader, in the wrong order, with the wrong ending. This is what I changed, and the structure I now use for every major incident.

Who actually reads an RCA

Four people read a root cause analysis, and they want different things.

The client's IT director wants to know two things in the first thirty seconds: is it fixed, and will it happen again. They will not read past the first paragraph if the first paragraph does not answer both.

The on-call engineer who will face the next similar incident wants the timeline and the detection gap: how long between the fault and the first alert, and what would have caught it sooner.

The change manager wants to know whether a change caused it and whether the change process failed.

The auditor, months later, wants evidence that an incident was investigated, actions were assigned, and actions were closed.

Most RCAs are written for none of them. They are written for the engineer who did the investigation, in the order the investigation happened, ending with a list of everything that could conceivably be improved. That document serves the writer and no one else.

The structure that works

1. Two-sentence summary, up top

Before the title if you can get away with it. Sentence one: what broke, for whom, for how long. Sentence two: the root cause and whether the permanent fix is in place.

Between 02:14 and 04:51 IST on a weeknight, the interface engine at a hospital site stopped delivering lab results to the EHR, affecting several departments. Root cause was a transaction log that filled the disk after a backup job failed silently; the log has been moved to a monitored volume and the backup job now alerts on failure.

The IT director stops here, satisfied. Everyone else keeps reading.

2. Impact in the client's terms, not yours

"Service degraded for 157 minutes" means nothing to a hospital. "Lab results for the emergency department were delayed by up to two and a half hours; no results were lost; a small number were re-sent manually by the lab" is what they need. Say what happened to the people who use the system. If you do not know, ask the client before you write the RCA, not after.

3. Timeline: fault, detection, response, resolution

A table, not prose. Four columns: time, what happened, who or what noticed, what was done. Every row sourced from a log, a ticket or a chat timestamp, never from memory.

Time	Event	Detected by	Action
01:40	Nightly backup job fails; exits without error code	Nobody	None
02:14	Transaction log fills disk; interface engine stops writing	Nobody	None
02:51	Lab calls the help desk: results not appearing	Human, 37 min later	P1 raised
03:05	On-call engineer joins bridge	Ticket	Investigation
03:40	Disk identified as full; log truncated	Engineer	Service restored 03:52
04:51	Queued messages fully replayed	Engineer	Incident closed

The timeline is where the real findings hide. In this one, the fault happened at 01:40 and the first human knew at 02:51. That seventy-one-minute detection gap is a bigger finding than the disk being full.

4. Root cause: one sentence, then the chain

State the root cause in one sentence a non-engineer can repeat. Then show the chain of contributing causes, because there is always a chain.

Root cause: the backup job failed without raising an alert, so the transaction log grew unchecked until the disk was full.

Contributing causes: (a) the backup job's exit code was not monitored; (b) the disk hosting the transaction log had no growth-rate alert, only a static 90% threshold that fired at 02:11 into an unmonitored mailbox; (c) the interface engine does not raise its own alert when it stops writing.

Use "five whys" if it helps you find the chain, but do not put "Why 1, Why 2, Why 3" in the document. The client does not care about your method.

5. What we did, and what we will do

Two lists, clearly separated. Immediate actions are done and dated. Preventive actions each have an owner, a due date and a way to verify closure. An action without an owner and a date is a wish.

Action	Type	Owner	Due	Verified by
Transaction log moved to dedicated 200 GB volume	Immediate	DBA lead	Done	Change record
Backup job wrapped to alert on non-zero exit	Preventive	Backup owner	Two weeks	Test failure raises P2 in ServiceNow
Disk growth-rate alert added to all DB volumes at the site	Preventive	Monitoring lead	Two weeks	Alert definition attached
Interface engine "no writes for 5 min" alert added	Preventive	Monitoring lead	Two weeks	Alert definition attached

Three to five actions. If you have twelve, you have not decided what matters.

6. Detection gap, called out on its own

I now give this its own heading because it is the finding clients act on most. "The fault was detectable at 01:40 and was detected by a human at 02:51. With the two alerts above in place, detection would have been at 01:41." A number like that gets budget approved.

Three things to stop doing

Stop writing it chronologically. The investigation went: symptom, hypothesis, dead end, second hypothesis, cause. The RCA goes: cause, impact, timeline, actions. Nobody needs your dead ends.

Stop blaming, and stop hiding. "Engineer X truncated the log without approval" belongs in the timeline as a fact, if it happened, with no adjective attached. But "human error" is never a root cause; it is a symptom of a missing control. Ask what would have made the error impossible or harmless, and write that as the action.

Stop sending it a week later. A draft RCA within 24 hours, with "under investigation" where needed, beats a perfect one after seven days. The client's leadership is being asked what happened today, not next Thursday.

The test

Before you send an RCA, read only the first paragraph and the actions table. If a reader who saw nothing else would know what happened, why, and what will stop it recurring, the document works. If they would need to read the middle to find that out, rewrite the top.

A root cause analysis is not a record of your investigation. It is the instrument that makes the next incident not happen. Write it that way and people will read it.

Cutting Alert Noise in Healthcare Operations: How We Took the Pager From Hundreds a Week to Dozens

Abhishek Singh — Wed, 30 Sep 2026 10:35:02 +0000

Every on-call engineer in healthcare IT knows the 3 a.m. page that turns out to be nothing. A disk at 81% on a server that has sat at 81% for six months. A CPU spike on a batch job that spikes every night by design. A "service down" alert for an endpoint that was down for forty seconds during a scheduled restart.

None of these are incidents. All of them wake someone up. And in a hospital environment, where the same team is also carrying pages that matter — an interface engine backing up lab results, a clinical application server refusing logins at shift change — the noise does something worse than cost sleep. It teaches people to ignore alerts. The team that has been paged 400 times in a week for nothing is slower on the one page that is a patient-safety issue.

This is a write-up of how our operations team took a fleet of enterprise healthcare clients from several hundred alerts a week to a few dozen, without turning off a single monitor that mattered. The tools were AppDynamics and LogicMonitor feeding ServiceNow. The principles apply to any stack — OpenTelemetry, Prometheus, Datadog, SigNoz — because the problem was never the tool. It was the rules.

Step 1: Count before you tune

The first thing we did was not touch a threshold. We exported four weeks of alerts from ServiceNow and grouped them three ways: by rule, by configuration item, and by outcome (did a human take an action, or close it as noise?).

The result was uncomfortable and clarifying:

About a dozen rules produced the large majority of all alerts.
Several rules had never once resulted in a human action.
One "disk space warning" rule alone fired more than a thousand times a month across the estate.
The alerts that did lead to real incidents — interface queue depth, database log growth, application login failures — were a small fraction of the total and were buried in the rest.

If you take one thing from this article: do the count. Every ops team believes it knows which alerts are noisy. The export always disagrees with the belief.

Step 2: Every alert must name an action

We adopted one rule for the whole estate: an alert is allowed to exist only if it names the action a human should take when it fires. Not "disk at 80%". Instead: "disk at 80% and growing more than 2% per hour — extend the volume or purge the log directory."

That single test killed a third of the rules outright. Static thresholds on metrics that never change had no action. Informational alerts that duplicated a dashboard had no action. Alerts on components with a healthy redundant pair had no action, because the failover already handled it.

For the alerts that survived, writing the action down did two things. It forced us to set the threshold at the point where the action is actually needed, and it gave us the first line of the runbook for free.

Step 3: Replace static thresholds with rate and duration

Most of the remaining noise came from alerts that fired on a single sample. A server at 95% CPU for five seconds is not a problem; a server at 95% for fifteen minutes usually is.

We changed the rule shape in LogicMonitor from "value above X" to "value above X for N consecutive polls", and where the metric had a natural rate — disk growth, queue depth, error count — we alerted on the rate rather than the level. In AppDynamics the equivalent was moving from static baselines to dynamic baselines with a minimum deviation window, so that the nightly batch job's spike was learned as normal rather than paged as abnormal.

Two rules where this mattered most in a hospital context:

Interface engine queue depth. A queue of 200 messages is normal at 7 a.m. and alarming at 2 a.m. We alerted on queue growth over ten minutes, not on the count.
Database transaction log growth. A log at 60% is fine; a log that has gone from 40% to 60% in an hour will be full before anyone finishes their coffee. Rate, not level.

This step took the estate down by roughly two-thirds.

Step 4: Maintenance windows are not optional

A surprising share of the remaining alerts were self-inflicted: patching windows, planned restarts, backup jobs. Every one of these was known in advance and every one of them paged someone.

We made maintenance windows a hard part of the change process. A change request in ServiceNow that touched a monitored configuration item automatically created a suppression window in the monitoring tool for the change's duration, via a small integration. No window, no change approval.

This is boring engineering and it removed a further meaningful slice of the weekly volume. It also stopped the worst habit on the team, which was manually silencing a whole host "for the patch" and forgetting to turn it back on.

Step 5: Correlate before you page

By this point the remaining alerts were mostly real, but they still arrived one at a time. A network switch failing produced twelve alerts from twelve downstream servers, each opening its own ServiceNow incident, each paging.

We used the event management layer to correlate on topology: if a parent configuration item is down, child alerts are attached to the parent incident rather than opening their own. Where topology was not modelled, we correlated on time and host group — five alerts from the same cluster within two minutes become one incident with five child events.

The on-call engineer now got one page that said "switch X down, 12 services affected" instead of twelve pages that each said "service Y unreachable". Same information, one interruption, and the root cause is in the title.

Step 6: Review the survivors monthly

Alert rules rot. Applications change, capacity gets added, a threshold that was right in March is wrong in September. We put a thirty-minute monthly review on the calendar with one agenda: the ten noisiest rules from the last month, and the question "did any of these lead to an action?"

Rules that did not are tuned or deleted. Rules that did are left alone. The review has caught rules that would otherwise have crept back to the old volume within a quarter.

What changed for the people

The numbers are the easy part: several hundred alerts a week to a few dozen, and the mean time to acknowledge a real incident dropped from over twenty minutes to a handful. The harder-to-measure change is that the on-call engineer now trusts the pager. When it goes off, something is wrong, and they move.

In a hospital that is the outcome that matters. The interface engine backing up at 2 a.m. gets a human on it in minutes instead of being the fourteenth alert in a queue of noise.

A checklist you can run this week

Export four weeks of alerts. Group by rule and by outcome.
Delete every rule that has never led to a human action.
For every rule that survives, write the action in the alert text.
Convert static thresholds to rate-or-duration conditions.
Tie maintenance windows to your change process, not to memory.
Correlate on topology or host group so one cause is one page.
Put a monthly thirty-minute noise review on the calendar.

None of this needs a new tool. It needs someone to do the count and be willing to delete rules that feel safe and are not.

Stop buying courses. Here is the 2026 DevOps Roadmap (Free Resources).

Abhishek Singh — Tue, 03 Feb 2026 13:14:46 +0000

Get the Complete 2026 DevOps Roadmap & Interview Cheat Sheet here

:
👉 Download The Headhunter's Database Here

I used to think I needed to pay $5,000 for a bootcamp to get a remote job in DevOps. I was wrong.

The best resources from Harvard, Google, and IBM are actually free—they are just hard to find.

I spent the last 3 months curating verified resources to build my own "Personal Degree." I wanted to share the best ones here so you can save money.

🚀 Part 1: The AI & Data Science Roadmap

These are the courses used by top engineers at FAANG companies.

Topic	Resource Name	Provider	Status
Foundation	CS50's Introduction to AI	Harvard	✅ Verified
Machine Learning	Machine Learning Crash Course	Google	✅ Verified
Deep Learning	Deep Learning Specialization	Andrew Ng	✅ Verified
LLMs	ChatGPT Prompt Engineering	DeepLearning.AI	✅ Verified
TensorFlow	TensorFlow Developer Cert	Coursera	✅ Verified

☁️ Part 2: The DevOps & Cloud Roadmap

You don't need to learn everything. You just need these core skills.

Topic	Resource Name	Provider	Status
Linux	Intro to Linux	Linux Foundation	✅ Verified
Docker	Docker for Beginners	KodeKloud	✅ Verified
Kubernetes	K8s Fundamentals	CNCF	✅ Verified
AWS	Cloud Practitioner Essentials	AWS	✅ Verified
Terraform	Terraform Associate Prep	HashiCorp	✅ Verified

💡 The "Secret" Interview Cheat Sheet

I also realized that 90% of technical interviews ask the same 5 Linux questions.

Instead of memorizing textbooks, I built a "One-Page Cheat Sheet" for:

File Permissions (chmod 744)
Log Analysis (grep "Error")
Process Management (kill -9)

It saves me about 5 hours of studying every week.

📥 Want the Full Database?

I compiled all 50+ Verified Resources, the Linux/Docker Cheat Sheets, and Ready-to-Deploy Terraform Code into a clean, printable PDF Portfolio Kit.

It costs $19 (less than a pizza), and it supports my work.

👉 Download The Headhunter's Database Here

AWS Disk Full? Why Deleting Logs Might Kill Your Database (And How to Fix It)

Abhishek Singh — Fri, 23 Jan 2026 11:13:47 +0000

Phase 1: Diagnosis (Don’t Panic, Verify)

Before you run any "fix" commands, you need to confirm two things:

Is the disk actually full?
Is InnoDB actually corrupted, or just stopped?

First, check the disk usage. 99% of the time, this is the root cause.

df -h

If you see /dev/xvda1 at 100%, you have your culprit. But here is where most people mess up: Do not just start deleting random files.

You need to find what is taking up space without touching the database files yet. Go to your data directory (usually /var/lib/mysql or /opt/bitnami/mysql/data on AWS images) and check the file sizes:

cd /var/lib/mysql du -h --max-depth=1 | sort -hr

If you see a massive list of mysql-bin.0000X files, those are binary logs. They are safe to purge later, but if you delete them now while the process is hung, you risk breaking the replication chain or crash recovery.

Next, check the logs to see if InnoDB is screaming for help.

tail -n 100 /var/log/mysql/error.log

You are looking for the "Red Flag" error:

[ERROR] InnoDB: Attempted to open a previously opened tablespace. Previous tablespace mysqldb/users uses space ID: 5 at filepath: ./mysqldb/users.ibd

Or the classic:

[ERROR] InnoDB: Page [page id: space=0, page_number=5] log sequence number 123456 is in the future!

If you see these, standard restarts won't work. The database is in a "Crash Loop." It tries to start, reads the corrupted file, panics, and shuts down.

Phase 2: The Surgical Fix (The Scary Part)

If you see those InnoDB corruption errors, standard restarts will fail. You need to force the engine to start in "Read Only" mode so you can extract your data.

Warning: This process involves innodb_force_recovery. It is powerful, but if you use it wrong, you can permanently delete data.

Step 1: Stop everything. Ensure the service is dead.

sudo service mysql stop

Step 2: The Magic Config Open your MySQL configuration file (usually /etc/mysql/my.cnf or /opt/bitnami/mysql/conf/my.cnf).

Add this line under the [mysqld] section:

[mysqld] innodb_force_recovery = 1

How this works:

Start with 1. This tells InnoDB to ignore corrupt pages and let you read the rest.
If it doesn't start, change it to 2, then 3, up to 4.
DANGER ZONE: Levels 5 and 6 are destructive. Do not use them unless you have a snapshot backup and know exactly what you are doing.

Step 3: Start and Dump Start the service again. If it comes up without crashing—congratulations. You are in "Life Support" mode. The database is read-only.

Now, get your data out immediately:

mysqldump -u root -p --all-databases --routines --triggers > /tmp/full_backup.sql

This command exports everything—tables, views, triggers—into a single file. Once this finishes, check the file size to make sure it looks right.

ls -lh /tmp/full_backup.sql

If that file exists and has size, you have successfully saved the company's data.

Phase 3: The Rebuild (Fresh Start)

Now that your data is safe in /tmp/full_backup.sql, it’s time to fix the corrupted engine. We are not going to try to "repair" the files (which rarely works). We are going to nuke them and rebuild.

1. Stop the service again.

sudo service mysql stop

2. Remove the recovery flag. Go back to my.cnf and **remove **or comment out the line we added earlier:

# innodb_force_recovery = 1 <-- Comment this out!

3. Move the corrupted data. Do not rm it yet! Move it to a backup folder just in case.

sudo mv /var/lib/mysql /var/lib/mysql_corrupted_backup

4. Initialize a fresh instance. Create a new, empty data directory.

sudo mkdir /var/lib/mysql sudo chown mysql:mysql /var/lib/mysql sudo mysqld --initialize --user=mysql

5. Start and Import. Start the service. It should come up instantly because it's empty. Now, pour your data back in:

mysql -u root -p < /tmp/full_backup.sql

Conclusion: Don't Let It Happen Again
You saved the database, but you haven't fixed the root cause. If you don't set up Log Rotation, that disk will fill up again in a month.

As a quick fix, set your binary logs to expire automatically. In MySQL 8.0, run this inside the SQL prompt:

SET GLOBAL binlog_expire_logs_seconds = 259200; -- 3 Days

Now, go grab a coffee. You earned it.