DEV Community: DevHelm

Runbooks: Anatomy, Examples, and the AI-Executable Format

DevHelm — Tue, 02 Jun 2026 10:14:30 +0000

The wiki page nobody opens. The Confluence doc that's six months stale. The Notion entry that gets read once during the postmortem and then forgotten. Most "runbooks" fail because they were written for nobody in particular — neither a fresh on-caller at 3 AM, nor a tenured engineer who already knows the system, nor an AI agent that might be the first responder. They serve no one, and they rot quietly.

A useful runbook is a specific, narrow thing: a tightly scoped, executable procedure that turns one known failure into one known recovery. This post pins down what a runbook actually is (and what it isn't), shows the seven sections a good one contains, walks through a worked example you can copy, and ends with the structure that makes a runbook executable by an AI agent — because increasingly that's who reads it first.

What is a runbook (and what it isn't)

A runbook is a document that tells you how to handle one specific operational situation, end-to-end. The trigger that brings you to it, the symptoms you should see, the commands that confirm what's wrong, the steps that fix it, and the checks that prove it's fixed. One runbook covers one failure mode.

It's not the same as some adjacent documents people lump under the term:

Document	Scope	Audience	When you reach for it
Runbook	One specific failure mode (e.g. "API p95 latency above SLO")	On-caller, AI agent, or a teammate paged into an active incident	When that exact alert fires
SOP (standard operating procedure)	Routine, non-incident operations (e.g. "Rotate database credentials quarterly")	Operator on a schedule	On a calendar trigger
Playbook	A class of incidents with branching (e.g. "Customer reports degraded API performance")	Incident commander making routing decisions	At the start of an unknown incident
Dashboard	A live view of system state	Anyone investigating	Continuously, during and outside incidents

The most common mistake is conflating runbooks with playbooks. A playbook is a tree of questions ("Is the database the bottleneck? If yes, go to runbook X. If no, check Y."). A runbook is a leaf of that tree — the actual recovery procedure once you've narrowed down which failure you're looking at. (The PagerDuty incident response guide is a good example of a playbook that links to many runbook-like procedures.) If your "runbook" is more than ~500 lines or covers more than one failure mode, it's a playbook and the runbooks it would link to don't exist yet.

The second common mistake is writing one runbook per service. A service has dozens of failure modes; lumping them all into one document means nobody can find the relevant section under pressure. One runbook, one failure mode, one alert. A slow DNS lookup and an SSL certificate error are two different failure modes — they get two different runbooks, even though they may live on the same load balancer.

The anatomy of a useful runbook

Most runbook templates you'll find on the internet ask for a dozen sections: purpose, scope, owners, dependencies, change history, related links, escalation matrix, last-reviewed date. Almost none of that is useful while an alert is paging. The reader has 30 seconds of working memory and is looking for what to do.

A good runbook contains exactly seven sections:

Trigger — the precise alert or signal that brought the reader here. Not "this is for API issues"; "this runbook is opened when the api-latency-p95-high alert fires."
Symptoms — what the reader can confirm right now. Specific commands, expected output. "The p95 latency panel shows >1s for 5+ minutes; error-rate panel is flat (rules out a 5xx storm)."
Diagnosis — commands to confirm the failure and rule out lookalikes. Each command in a fenced code block; expected output annotated.
Mitigation steps — ordered, idempotent, each with a runnable command. If a step depends on the previous one succeeding, say so.
Verification — how the reader knows it worked. Concrete checks: "the http_request_duration_seconds p95 drops below 500ms for 10 consecutive scrape intervals."
RTO and what data you lose — expected duration of the recovery and any acceptable data loss. The reader needs to know whether this is a 30-second fix or a 30-minute restore so they can communicate up.
Escalation path — when and to whom you escalate if the steps don't work. Real names or rotation references, not "the DBA team."

That's it. Everything else (owners, related links, last-reviewed date) belongs in the file's front-matter or repository metadata, not in the body the on-caller reads while their phone is buzzing. For more on why RTO matters as a success criterion, see MTTR Full Form.

Worked example: API p95 latency runbook

Below is a condensed runbook for a common SaaS failure mode — API latency crossing an SLO threshold while error rates stay flat (often a saturation or dependency slowdown, not a hard outage). The names are illustrative; swap in your service labels and metric names.

Scenario: API p95 latency above SLO

Trigger: the api-latency-p95-high alert (Prometheus rule: p95 > 1s for 5m, error rate < 1%).

Symptoms: Grafana "API latency" panel red; "API errors" panel green. Recent deploy in the last 30 minutes (check CI) OR no deploy (points to dependency or traffic spike).

Diagnosis: (1) kubectl get pods -n api -l app=api — any Not Ready? (2) curl -s http://api.internal/health | jq '.status' — expect "UP". (3) Compare p95 by route in Grafana — one route or all routes?

Mitigation: if post-deploy → roll back to previous revision (kubectl rollout undo deployment/api -n api). If all routes slow and health is UP → check upstream dependency status pages; throttle non-critical traffic if you have a feature flag.

Verification: p95 < 500ms for 10 consecutive scrape intervals; error rate unchanged; no new pages in 15 minutes.

RTO: 5–15 minutes for rollback path; 30–60 minutes for dependency-wait path.

Escalation: if rollback fails twice or p95 still >1s after 30 minutes → page platform lead with dashboard link and deploy SHA.

Notice the shape: each section has a single job. There's no preamble about "the importance of SLOs." The reader who arrived from the alert wants four things in this order — is this the right runbook, what should I see, what should I run, did it work — and the document delivers all four within the first screen.

AI-readable runbooks: structure that an agent can execute

Increasingly the first responder to an incident is not a human. An on-call agent (Cursor, Claude Code, or a dedicated SRE bot) can receive the same alert payload as a human and start triage before anyone is paged — if the alert carries a runbook_url and the runbook body is structured for machines, not just humans.

For that to work, the runbook has to be structured so an agent can extract steps and act on them. The seven sections above are necessary but not sufficient. Five additional properties make a runbook AI-executable:

The trigger is a machine-parseable query, not a description. "Looks slow" can't be matched against telemetry; histogram_quantile(0.95, rate(http_request_duration_seconds_bucket{job="api"}[5m])) > 1.0 can.
Commands live in fenced code blocks with language tags (bash, sql, yaml). The agent (and any markdown parser) needs structural cues to know what's executable.
Expected output is colocated with the command. A step that says "run kubectl get pods" without telling the agent what success looks like is non-executable — there's no way to verify the step worked before moving on.
Failure modes branch explicitly. "If health is UP but p95 is still high, go to Check 3 (dependency status); if pods are Not Ready, go to Check 2 (roll back)" is executable. "If needed, escalate" is not — the agent can't decide what "needed" means.
No prose-only sections in the recovery body. Every step has a runnable artifact or a verifiable check. Background narrative belongs in a separate "Why this happens" section that the agent can skip if it's already remediating.

A human-only version of a step:

"Check whether latency is still elevated. You can look at the metrics in Grafana, or curl the health endpoint. If it's still slow, you'll want to investigate why."

The same step, AI-executable:

Check 1 — is the API still degraded?

curl -s http://api.internal/health | jq -r '.status'

Expected: UP. Then confirm latency in Prometheus:

histogram_quantile(0.95, rate(http_request_duration_seconds_bucket{job="api"}[5m]))

Expected: below 0.5 (500ms). If above 1.0 for two consecutive evaluations, proceed to Check 2 (recent deploy).

Same information, but the agent can run it, parse the output, and decide whether to advance. That's the bar.

Runbook hygiene: where to store them, how to find them at 3 AM

A runbook that exists but can't be found in an incident is worse than no runbook — it costs minutes while the on-caller searches for it. Three rules cover most of the discoverability problem:

Store runbooks in Git, next to the code. Confluence and Notion fail in two ways: they go down during outages of services they themselves depend on (the same DNS provider, the same auth provider), and they have no review workflow that catches stale content. A runbook in runbooks/api-latency-p95-high.md is reviewed every time the surrounding service changes — pull requests force the authors to update the runbook or explain why not.

Link every alert to its runbook. Use the annotation field your alerting system provides. For Prometheus / Grafana, that's runbook_url:

- alert: ApiLatencyP95High
  expr: |
    histogram_quantile(0.95, rate(http_request_duration_seconds_bucket{job="api"}[5m])) > 1
    and rate(http_requests_total{job="api",status=~"5.."}[5m]) / rate(http_requests_total{job="api"}[5m]) < 0.01
  for: 5m
  labels:
    severity: page
  annotations:
    summary: "API p95 latency above 1s for 5m with error rate below 1%."
    runbook_url: "https://docs.your-company.com/runbooks/api-latency-p95-high"

The alert payload that reaches the pager (and the AI agent, if you run one) carries the URL. The on-caller's first click is straight into the right procedure.

One runbook per failure mode, named for the failure. api-latency-p95-high.md, not api.md. When the page fires, the alert name and the file name match — no search needed.

For decay management: review each runbook quarterly, archive any with zero hits in 90 days, and treat a stale runbook found mid-incident as a sev3 of its own — the on-caller files a ticket to fix it; otherwise nobody does.

How DevHelm fits runbooks into your incident flow

DevHelm is built for the moment an alert fires and someone (human or agent) needs context fast. What's shipped today:

Alert channels (PagerDuty, Slack, webhook, email) pass through the payload your upstream system sends. If your Prometheus or Grafana alert includes a runbook_url annotation, that URL can ride along in the notification DevHelm dispatches.
Vendor status context on dependency status pages — when latency looks like an upstream problem, the runbook's "check dependency status" step has a concrete destination instead of a generic Google search.
Resource groups (see MTTR Full Form) collapse multiple monitors that share one failure mode into one incident — so the runbook link in the notification matches one root cause, not three duplicate pages.

What's not yet shipped: a first-class runbook_url field on DevHelm monitors — the kind that would let you set it once on the monitor and have it flow into every notification and MCP tool response automatically. Until then, put the URL in the monitor description and your alert template. The reliability page covers how we operate our own stack; you don't need our internal runbook repo to apply the patterns in this post.

Where to start

Pick your noisiest recurring alert — the one that woke someone up twice last quarter — and write one runbook for it. Seven sections, under 500 lines, stored in Git, linked from the alert annotation. That's the whole commitment.

If you've been troubleshooting slow DNS lookups or SSL certificate errors, you've already done most of the work: those investigations follow exactly the trigger → diagnosis → fix → verify shape described above. Turning them into a runbook is a matter of formatting what you already know so the next person (or agent) doesn't have to rediscover it. And once the runbook exists, measuring whether it actually shortens recovery is what MTTR is for.

Spin up a free account at app.devhelm.io and connect your first dependency status feed in 60 seconds — useful when your runbook's diagnosis step says "check if the vendor is degraded." For AI-native setup, npx devhelm skills install --target cursor installs the skill bundle that can create monitors from your editor.

Originally published on DevHelm.

SLO vs SLA vs SLI: What Each One Means and How to Set Them

DevHelm — Tue, 02 Jun 2026 10:13:44 +0000

Most SLO guides start with the same three-paragraph definitional exercise — SLI is the indicator, SLO is the objective, SLA is the agreement — and then stop. You leave knowing the vocabulary but not how to use it. You can't answer the questions that actually matter: which metric should I measure, what target is realistic for my service, and what happens when I miss it?

This guide starts with the definitions because you need a shared vocabulary, but it spends most of its time on the decisions behind each one: choosing the right SLI for your service, setting an SLO that's strict enough to matter but loose enough to survive, computing and spending an error budget, and knowing when (and when not) to turn an SLO into an SLA.

The three letters, disambiguated

SLI — Service Level Indicator. A quantitative measurement of one dimension of your service's behavior. Latency, availability, throughput, error rate, ticket resolution time. An SLI is always a number with units, derived from real telemetry. "Our API is fast" is not an SLI. "The 95th percentile of API response latency, measured at the load balancer over a 5-minute window" is.

SLO — Service Level Objective. A target you set on an SLI. "p95 latency < 500ms, measured over a rolling 30-day window" is an SLO. It's an internal commitment — your team agrees that the service should meet this bar, and when it doesn't, you treat that as an incident or at least an engineering priority. An SLO is a tool for your team, not a legal document.

SLA — Service Level Agreement. An SLO that's been written into a contract with a customer, usually with financial consequences for missing it. If your SLO says "99.9% availability" and you publish that as an SLA, a customer who experiences more than 43 minutes of downtime in a month has grounds for a credit. SLAs are legal; SLOs are operational. Most internal services should have SLOs and should not have SLAs.

The relationship is directional: you measure an SLI, set an SLO against it, and optionally externalize that SLO as an SLA. Every SLA implies an SLO, but not every SLO should become an SLA.

Choosing the right SLI

The hardest step is the first one: picking what to measure. A service with three SLIs that capture what users actually experience is more useful than one with fifteen SLIs that capture what the infrastructure is doing.

The Google SRE Workbook recommends starting from user journeys:

User journey	SLI category	Example SLI
"The page loads"	Availability	Proportion of HTTP requests returning non-5xx, measured at the edge
"The page loads quickly"	Latency	p95 of response time, measured at the load balancer
"My data is processed"	Freshness	Age of the most recent successful pipeline run, measured in minutes
"My report is accurate"	Correctness	Proportion of API responses returning the expected result (requires a canary or known-answer test)

Two rules of thumb:

Measure at the boundary your user sees, not inside your stack. If you measure latency at the application layer and your CDN adds 200ms, you're lying to yourself. Measure at the load balancer or the edge.
Fewer SLIs, more confidence. Start with availability + latency for any request-serving system. Add freshness only if you run a pipeline. Add correctness only if you have a way to verify it. Three SLIs that are trustworthy beat ten that nobody looks at.

A common mistake: using CPU utilization or memory pressure as SLIs. Those are infrastructure signals, not user-facing indicators. A machine running at 95% CPU but serving all requests under 200ms is fine. A machine running at 30% CPU but dropping 5% of connections is not. SLIs are about the user's experience, not the server's.

Setting a realistic SLO

An SLO has three parts: the SLI, the target, and the measurement window.

Example: "99.9% of HTTP requests return a non-5xx response, measured over a rolling 30-day window."

The target is the part teams argue about. Here's a way to pick it that doesn't require a week of meetings.

Step 1: Measure your current SLI for 30 days. Don't set a target yet — just observe. If your service has been running 99.95% availability without anyone trying, setting 99.9% is reasonable. Setting 99.99% is aspirational. Setting 99% is embarrassing.

Step 2: Set the target slightly below your current baseline. If you've been running at 99.95%, set your SLO at 99.9%. This gives you room to breathe. The point of an SLO is not to describe your best day — it's to define the minimum acceptable. If you set it at your best day, every normal fluctuation is a "violation."

Step 3: Convert the target to an error budget. This is where SLOs get useful. A 30-day window contains 43,200 minutes, so:

SLO target	Error budget	Allowed downtime per 30 days
99.9%	0.1%	43.2 minutes
99.95%	0.05%	21.6 minutes
99.99%	0.01%	4.3 minutes

Those numbers are the entire content of most "what should my SLO be?" debates. A 99.99% SLO on a 30-day window gives you 4.3 minutes of total downtime. If your MTTR is 25 minutes per incident, you can afford zero incidents. That's either an aspirational commitment backed by redundant infrastructure, or it's a lie. Be honest about which one.

The error budget: what it is and how to spend it

The error budget is the gap between 100% and your SLO target. If your SLO is 99.9% availability over 30 days, your error budget is 43.2 minutes. That budget is not "waste allowance" — it's a resource you can spend deliberately.

Useful ways to spend error budget:

Deploy a risky change. If you have 30 minutes left in the budget and the deploy might cause 5 minutes of degradation, that's a calculated risk. If you have 2 minutes left, hold the deploy until the window rolls.
Run a chaos experiment. Kill a database replica, fail over a region, inject latency on a dependency. Each experiment consumes budget. If you can't afford to run experiments, your SLO is probably too tight.
Let a known low-severity issue ride. A p99 latency blip at 3 AM that affects 0.01% of requests is consuming budget, but if the alternative is waking someone up, spending budget is the right call.

The error budget policy is the written agreement about what happens when the budget runs out. Typical policies:

Budget exhausted -> feature freeze. All engineering effort goes to reliability until the budget recovers. This is the Google model and it works if leadership actually enforces it.
Budget below 50% -> deploy gate. Deploys require explicit approval from the on-call engineer. This slows shipping but prevents the "one more deploy" cascade that burns the remaining budget.
Budget healthy -> ship freely. This is the reward for investing in reliability. A team with a full error budget has earned the right to move fast.

The key insight: error budgets turn reliability from a vague mandate ("be more reliable") into a quantitative tradeoff ("we have 20 minutes left this month — is this deploy worth 5 of them?"). Teams that track error budgets make better decisions than teams that track uptime, because uptime has no built-in notion of "how much risk can we take."

When an SLO becomes an SLA

Most internal SLOs should stay internal. An SLA adds legal weight, customer expectations, and credit obligations. Promote an SLO to an SLA only when all three conditions hold:

You've hit the SLO consistently for 3+ months. If you haven't proven you can meet it internally, you definitely can't promise it externally.
You have a remediation path for breaches. What credits do you issue? How are they calculated? Who approves them? If you can't answer these, you don't have an SLA — you have a marketing claim.
The SLA target is looser than your internal SLO. Your SLA should be 99.9% if your SLO is 99.95%. The gap is your operational buffer. If the SLA and SLO are the same number, every SLO breach is also a contract breach, and your team will either burn out or game the measurement.

A public status page (like the ones DevHelm hosts at /status/github) is a middle ground between internal SLOs and contractual SLAs — it shows real uptime data without attaching legal obligations. It builds trust through transparency rather than through contractual obligation.

How DevHelm gives you the data for SLOs

DevHelm doesn't have a first-class SLO resource that you configure with a target and measure against a budget — that's a feature we're building, not one we ship today. What it does give you is the raw material SLOs are made of.

Monitor uptime data. Every monitor computes availability as a weighted daily percentage: (86400 - major_seconds - partial_seconds * 0.3) / 86400 * 100. Major outages count fully against uptime; partial degradations count at 30%. That formula runs across the status page, the dashboard, and the API — all three stay in sync. If your SLI is availability, the monitor's uptime history is the measurement.

Status page uptime bars. The public status page at /status/<service> renders daily uptime per component with a "tracking since" date. An internal team or a customer can see exactly when the service was degraded and for how long — the same data that would feed an error budget computation.

Alert channels for SLO-boundary signals. If your SLI is latency and your monitor checks every 30 seconds, you can set a monitor threshold at the SLO boundary (e.g. p95 > 500ms) and route the alert through DevHelm's notification policies. That's not burn-rate alerting in the formal sense (you'd want a multi-window approach per the Google SRE Workbook), but it catches SLO breaches as they happen rather than at the end of the month.

What we'd tell you honestly: if you need formal error budgets with automated freeze policies, you need a dedicated SLO tool (Nobl9, Sloth, or a Prometheus recording rule setup). DevHelm gives you the uptime data and the alerting layer; the budget math is yours today, ours tomorrow.

Where to start

If you've never set an SLO, start with one. Pick your most important user-facing service, measure its availability SLI for two weeks, then set the SLO 0.05% below the observed baseline. Compute the error budget in minutes. Write it on a whiteboard. The first time someone asks "can we deploy this risky change?" and the answer is "we have 18 minutes of budget left — let's wait until Monday," the SLO has paid for itself.

If your incidents tend to be dependency-driven — AWS degrades, your CDN edge has a regional issue — your SLO's biggest enemy is something outside your stack. A runbook for each known dependency failure mode and a vendor status feed that tells you when the dependency degraded before your monitors notice are the two cheapest investments in protecting your error budget.

Spin up a free account at app.devhelm.io and wire your first monitor in 60 seconds. The uptime data starts accumulating immediately — you'll have your first 30-day SLI baseline before next month's planning meeting.

Originally published on DevHelm.

Incident Severity Levels: Sev1–Sev4 with Triage Matrix

DevHelm — Tue, 02 Jun 2026 10:13:43 +0000

Most teams define their severity levels as a table in a Confluence page, link to it from onboarding docs, and then never reference it during an actual incident. The levels exist, but nobody uses them. Three months later someone opens a sev1 for a broken CSS gradient and the on-call engineer gets paged at 2 AM.

Severity levels only work when three things are true: the scale is simple enough to apply under stress, the response expectations are explicit, and the routing is automated. This guide covers all three — the scale itself, the decision framework for assigning it, and the wiring that turns a severity label into the right alert at the right time.

The four levels

Most incident management systems converge on a four-level scale. The labels vary — sev1/sev2/sev3/sev4, P0/P1/P2/P3, critical/major/minor/info — but the structure is nearly universal.

Level	Also called	Definition	Response expectation
Sev1	P0, Critical	Complete outage of a production system, data loss, or security breach affecting customers	All-hands. Incident commander assigned. Stakeholder updates every 15 minutes.
Sev2	P1, Major	Significant degradation — a core feature is broken or a significant percentage of users are affected. Service is up but materially impaired.	On-call responds immediately. Updates every 30 minutes. Escalation if unresolved in 1 hour.
Sev3	P2, Minor	Limited degradation — a non-critical feature is broken, a workaround exists, or the impact is confined to a small subset of users.	Addressed within business hours. No page. Tracked in the incident backlog.
Sev4	P3, Info	Cosmetic issue, minor inconvenience, or an anomaly that warrants investigation but has no user-facing impact.	Sprint backlog. No incident channel. Closed in the next cycle.

The exact boundaries shift between organizations. A company whose revenue runs through a single API endpoint has a lower threshold for sev1 than a company with redundant payment processors. The table above is a starting point — calibrate it to your blast radius.

What matters more than the exact definitions is that everyone on the team can assign the right level within 60 seconds of seeing the alert. If your engineers argue about severity during an incident, the definitions are too ambiguous.

Severity vs priority — they are not the same

This distinction trips up most teams. Severity describes the impact of the incident — how bad it is right now. Priority describes the urgency of the response — how fast you need to fix it. They usually correlate, but not always:

A sev1 in a staging environment is critical severity, low priority. The environment is completely down, but no customers are affected.
A sev3 that blocks a contractual deadline is minor severity, high priority. The feature works for most users, but the one user who matters is the enterprise customer whose annual renewal depends on it shipping by Friday.
A sev2 that self-resolves in 90 seconds is significant severity, reduced priority after the fact. The incident was real, but by the time an engineer opened the laptop, the system recovered. The retro still matters, but the live response is over.

The Google SRE Workbook formalizes this as "severity is an attribute of the incident; priority is a decision made by the responder." The practical consequence: if your alerting system routes by severity alone, you get the right response most of the time. The rest requires human override — someone promoting a sev3 to high-priority or silencing a sev1 that fired in a non-production context.

A triage matrix that works under stress

When an alert fires, you have roughly 30 seconds of attention before the responder either acts or dismisses. The triage question is: "what severity is this?" The fastest way to answer it is a two-axis matrix of customer impact and scope.

	Single user / account	Significant minority (10-30%)	Majority or all users
Feature broken, no workaround	Sev3	Sev2	Sev1
Feature degraded, workaround exists	Sev4	Sev3	Sev2
Non-functional impact (slow, noisy, ugly)	Sev4	Sev4	Sev3

The matrix is intentionally coarse. Three scope buckets, three impact buckets, nine cells. A responder can place an incident in the right cell in seconds without reading a paragraph of definitions.

Two overrides that bump any cell up by one level:

Data loss or security exposure. A bug that leaks PII to unauthorized users is sev1 regardless of scope — even if it affects one account.
Revenue impact. If the checkout flow is broken and orders are failing, that's sev1 even if the monitoring dashboard reports 95% availability — because the 5% that's failing is the 5% that pays the bills.

What each severity triggers

The scale has no value unless it drives concrete actions. Every severity level should map to four things: who gets notified, how fast they respond, what communication cadence they maintain, and whether a post-incident review is mandatory.

Sev1: page on-call + backup + engineering lead. Acknowledge within 5 minutes. Incident channel created, stakeholder updates every 15 minutes, customer-facing status page updated. Mandatory blameless retro within 48 hours with tracked action items.

Sev2: page on-call. Acknowledge within 15 minutes. Incident channel, updates every 30 minutes. Retro recommended at team discretion.

Sev3: Slack channel or email notification. Response within the next business hour. Ticket created, no incident channel. Retro optional, only if the pattern is recurring.

Sev4: logged but no active notification. Next sprint. No communication, no retro.

If your sev1 and sev2 have the same notification channel, the same response time, and the same retro expectation, you don't have two severity levels — you have one with two names. Merge them or differentiate them.

How severity drives MTTR

Your MTTR target should vary by severity — and if you're tracking the full set of MTTA, MTTR, MTBF, and MTTF, severity determines which metric matters most at each tier. A sev1 with a 4-hour MTTR means your most critical incidents take half a workday to resolve — probably too slow. A sev4 with a 4-hour MTTR means you're spending on-call energy on cosmetic issues — probably too fast.

Level	MTTR target	Rationale
Sev1	< 1 hour	Revenue is actively lost, users are actively blocked
Sev2	< 4 hours	Significant impact but not existential
Sev3	< 1 business day	Limited scope, workaround available
Sev4	Next sprint	Not time-sensitive

These targets feed directly into your SLO error budget. A 99.9% availability SLO on a 30-day window gives you 43 minutes of total downtime. If your sev1 MTTR target is 1 hour, a single sev1 incident blows the budget. That tension is the point — it forces you to invest in the runbooks and automation that keep resolution time below the budget threshold.

How DevHelm routes by severity

DevHelm models incident severity as three operational states: DOWN, DEGRADED, and MAINTENANCE. This is deliberately simpler than a sev1-through-sev4 scale. The numbered scale requires human judgment about scope and blast radius; DevHelm's model is automated from check results. When a monitor's trigger rule fires, the rule specifies whether the incident is DOWN (the service is not responding or failing critically) or DEGRADED (the service is responding but outside acceptable bounds — slow, returning partial errors, or failing specific assertions).

The routing happens in notification policies. Each policy has match rules, and one of those rules is severity_gte — "match when incident severity is greater than or equal to this threshold." Severity is ordered: DOWN > DEGRADED > MAINTENANCE. In practice, this gives you two-track routing:

A policy with severity_gte: DOWN routes to PagerDuty — page the on-call engineer immediately.
A policy with severity_gte: DEGRADED routes to a Slack channel — notify the team, no page.

The first policy fires only for DOWN incidents — your sev1 equivalent. The second fires for both DOWN and DEGRADED, so a DOWN incident sends both a page and a Slack message (the on-call gets paged, the wider team stays informed). A DEGRADED incident reaches Slack but never PagerDuty. You've split your alert routing by severity without writing any code.

For richer routing, combine severity_gte with other match rules. A policy that matches severity_gte: DOWN AND monitor_tag_in: ["payments", "checkout"] pages someone for critical payment failures but not for a down developer docs site. That's severity combined with business context — the same intersection the triage matrix above describes, except it's automated instead of decided in the heat of the moment.

Where to start

If your team doesn't have severity levels, start by writing the four definitions in a shared doc and getting three people to agree on them. That takes 30 minutes and pays for itself the first time someone opens an incident.

Then automate the routing. Set up a monitor in DevHelm, configure a trigger rule that fires as DOWN after two consecutive failures confirmed across regions, and wire a notification policy that pages your on-call for DOWN incidents and sends DEGRADED incidents to Slack. You've just built a severity-routed alerting pipeline that distinguishes between "wake someone up" and "the team should know" — running 24/7 without anyone remembering to check the definitions page.

Originally published on DevHelm.

MTBF Full Form: Mean Time Between Failures — Meaning, Formula, and When It Matters

DevHelm — Tue, 02 Jun 2026 10:12:56 +0000

Most reliability conversations start with uptime percentage and stop there. "We're at 99.95% availability" feels like enough — until you realize that a service with 99.95% availability could be down for 22 minutes once a month, or for 11 seconds every hour. Both hit the same uptime number. MTBF tells you which pattern you actually have.

What MTBF stands for

MTBF — Mean Time Between Failures — measures the average elapsed time from the end of one failure to the start of the next. It's a frequency metric: high MTBF means failures are rare; low MTBF means they're frequent. A service with an MTBF of 720 hours (30 days) averages one failure per month. A service with an MTBF of 24 hours averages one failure per day. Both might have the same uptime percentage if the frequent failures resolve quickly, but the operational burden is completely different.

The term originates in hardware reliability engineering — MIL-HDBK-217, published by the US Department of Defense in 1961, defined MTBF for electronic components. In software, we borrow the concept but adapt it: a "failure" is an incident that crosses a severity threshold (typically sev1 or sev2), not a hardware component burning out.

The formula

For a repairable system (which every software service is), MTBF equals total operating time divided by the number of failures during that period:

Metric	Formula
MTBF	Total uptime / Number of failures
MTTR	Total downtime / Number of failures
MTTF	Time from last recovery to next failure (= MTBF - MTTR)

The three metrics are related: MTBF = MTTF + MTTR. MTTF is the time the system runs without failing; MTTR is the time it takes to recover. Together they span the full cycle from one failure to the next.

Worked example. A payment processing service runs for 30 days (720 hours). During that window, it experiences 3 incidents:

Incident	Started	Resolved	Downtime
#1	Day 4, 14:00	Day 4, 14:45	45 min
#2	Day 12, 03:20	Day 12, 04:10	50 min
#3	Day 25, 09:00	Day 25, 09:30	30 min

Total downtime: 125 minutes (2.08 hours). Total uptime: 720 - 2.08 = 717.92 hours.

Metric	Calculation	Result
MTBF	717.92 / 3	239.3 hours (~10 days)
MTTR	2.08 / 3	41.6 minutes
MTTF	239.3 - 0.69	238.6 hours
Uptime %	717.92 / 720	99.71%

The uptime number (99.71%) tells stakeholders the service was available most of the time. The MTBF (10 days) tells the engineering team that failures happen roughly every week and a half — often enough to warrant investment in prevention, not just faster recovery.

MTBF vs MTTR vs MTTF — side by side

Teams often confuse these three metrics. Here's the clean distinction:

Metric	Full form	Measures	Improves by
MTBF	Mean Time Between Failures	How often failures occur	Preventing failures (better testing, redundancy, dependency isolation)
MTTR	Mean Time To Recovery	How fast you recover	Faster detection, better runbooks, automation
MTTF	Mean Time To Failure	How long the system runs before failing	Same as MTBF — it's the operating-time component

Two less common but useful metrics:

Metric	Full form	Measures
MTTD	Mean Time To Detect	Lag between failure start and the first alert firing
MTTA	Mean Time To Acknowledge	Lag between alert firing and a human acknowledging it

The full incident timeline runs: failure occurs -> MTTD -> alert fires -> MTTA -> responder acknowledges -> works the issue -> MTTR -> recovery. MTBF spans the gap between recoveries.

How to measure MTBF in practice

MTBF requires two inputs: a time window and a count of failures. Both are harder to pin down than they sound.

Defining "failure." Not every alert is a failure. A monitor that flaps for 30 seconds and self-recovers is an anomaly, not an incident. Most teams count only incidents above a severity threshold — sev1 and sev2 — when computing MTBF. If you include sev3 and sev4, your MTBF drops dramatically but the number stops being useful because it mixes service-impacting failures with noise.

Choosing the window. A 7-day window is too noisy — one bad week skews the number. A 365-day window smooths out seasonality but hides recent trends. The sweet spot for most teams is a rolling 30-day or 90-day window, reported weekly. If your MTBF is trending down over 4 consecutive weeks, something systemic is degrading — even if no single week looks alarming.

Handling planned maintenance. Exclude planned maintenance windows from the failure count and from operating time. A team that takes its service down for 2 hours every Sunday for database maintenance should not count those windows as "failures." If you include them, MTBF becomes a meaningless number that punishes disciplined operations.

Per-service, not fleet-wide. A fleet-wide MTBF that averages your payment service (fails once a quarter) with your notification service (fails weekly) tells you nothing actionable. Compute MTBF per service or per component. The payment team needs to know their MTBF, not the company average.

What MTBF tells you that uptime percentage does not

Uptime percentage compresses all failure information into a single number. Two services can have identical uptime (99.9%) with completely different failure patterns:

Service	Uptime	Failures/month	Avg downtime per failure	MTBF
Service A	99.9%	1	43 minutes	720 hours
Service B	99.9%	10	4.3 minutes	72 hours

Service A is stable but slow to recover. Service B is fragile but recovers fast. The SLO error budget treats them identically — both consume the same 43 minutes of downtime per month. But the operational strategies are opposite: Service A needs faster MTTR (better runbooks, automation). Service B needs higher MTBF (better testing, dependency isolation, circuit breakers).

If you only track uptime, you prescribe the same medicine to both patients. MTBF and MTTR together give you the diagnosis.

When MTBF is misleading

MTBF is an average, and averages lie when the underlying distribution is skewed.

Rare catastrophic failures. A service that runs perfectly for 364 days and then suffers a 12-hour outage on day 365 has an MTBF of 8,736 hours. That number sounds excellent — until the one failure costs $2M in lost revenue. MTBF doesn't capture tail risk. For services where the cost of a single failure is extreme, pair MTBF with a worst-case downtime metric (longest incident duration over the window).

Heterogeneous failure modes. If your service fails due to three completely different root causes (DNS resolution, database connection pool exhaustion, and a memory leak), averaging them into a single MTBF obscures the fact that one cause dominates. Compute MTBF per root cause category when possible.

Early-life systems. A new service has no meaningful MTBF. You need at least 3-5 failure cycles to compute a statistically useful average. Reporting MTBF after one incident in two weeks is technically correct ("MTBF = 336 hours") but practically useless — the confidence interval is enormous.

How DevHelm gives you the raw data

DevHelm does not expose MTBF as a dashboard metric today — that's on the roadmap. What it does give you is the incident history that MTBF is computed from.

Every incident records a created_at timestamp (when the incident opened), a resolved_at timestamp (when it closed), and a severity (DOWN or DEGRADED). The dashboard's incident summary already computes MTTR over a rolling 30-day window from these timestamps. MTBF is the complementary calculation: take the resolved_at of incident N and the created_at of incident N+1, average those gaps, and you have MTTF — then add MTTR to get MTBF.

If you need the number today, the API returns the full incident list for any monitor at GET /api/v1/incidents, filtered by severity and date range. A script that walks the list and computes MTBF per monitor is straightforward — and when we ship native MTBF in the analytics view, the underlying calculation will use the same timestamps.

Where to start

If you're tracking uptime but not MTBF, pick your highest-traffic service, pull its sev1+sev2 incident history for the last 90 days, and compute MTBF with the formula above. Compare it to your MTTR. If MTBF is low and MTTR is low, you have a fragile-but-fast-recovering service — invest in prevention. If MTBF is high and MTTR is high, you have a stable-but-slow-recovering service — invest in runbooks and detection speed.

Set up monitoring for the service at app.devhelm.io and let the incident history accumulate. After 30 days you'll have enough data points to compute your first real MTBF — and the trend line from that point forward is worth more than any snapshot.

Originally published on DevHelm.

Jaeger Tracing Explained: How Distributed Tracing Works

DevHelm — Tue, 02 Jun 2026 10:12:56 +0000

Distributed tracing answers the question that uptime monitoring can't: a request failed, but which service in the chain caused it?

When your checkout endpoint returns a 500 and your monitoring dashboard shows the API is degraded, you know that something broke. You don't know where in the chain of payment-service -> inventory-service -> shipping-service the failure originated. Distributed tracing instruments every service in the path and stitches the results into a single timeline — a trace — that shows exactly where latency accumulated or where an error propagated from.

Jaeger is the most widely deployed open-source distributed tracing backend and a CNCF graduated project. This guide covers how it works, how to set it up with OpenTelemetry, and when tracing complements (rather than replaces) uptime monitoring.

What Jaeger does

Jaeger collects, stores, and visualizes traces. A trace is a tree of spans — each span represents one unit of work (an HTTP request handler, a database query, a gRPC call to another service). Spans carry timing data, status codes, and arbitrary tags.

When you instrument your services with OpenTelemetry SDKs, each outgoing request propagates a trace ID via HTTP headers. Jaeger collects spans from every service, groups them by trace ID, and renders the full request path as a timeline.

The architecture has four components:

Component	Role
Agent	Lightweight daemon that receives spans from the SDK and forwards to the collector. Optional with OTLP — the SDK can send directly.
Collector	Receives spans, validates, indexes, and writes to storage.
Query	API + UI for searching and visualizing traces.
Storage	Pluggable backend — Elasticsearch, Cassandra, Kafka, Badger, or in-memory for development.

For development and small deployments, Jaeger ships an all-in-one binary that bundles all four components with in-memory storage:

docker run -d --name jaeger \
  -p 16686:16686 \
  -p 4317:4317 \
  -p 4318:4318 \
  jaegertracing/all-in-one:latest

Port 16686 is the Jaeger UI. Ports 4317 (gRPC) and 4318 (HTTP) receive OTLP spans from OpenTelemetry SDKs.

Instrumenting with OpenTelemetry

Jaeger originally shipped its own client libraries, but the project now officially recommends OpenTelemetry SDKs for instrumentation. The OTel SDK instruments your code; the OTLP exporter sends spans to Jaeger's collector endpoint.

A minimal Python example:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import (
    OTLPSpanExporter,
)

provider = TracerProvider()
exporter = OTLPSpanExporter(
    endpoint="http://localhost:4317", insecure=True
)
provider.add_span_processor(BatchSpanProcessor(exporter))
trace.set_tracer_provider(provider)

tracer = trace.get_tracer("checkout-service")

with tracer.start_as_current_span("process_order") as span:
    span.set_attribute("order.id", "ORD-1234")
    span.set_attribute("order.total", 89.99)

The BatchSpanProcessor buffers spans and flushes them in batches to avoid blocking your request path. For production, add resource attributes (service name, version, environment) so you can filter traces in the Jaeger UI.

The equivalent in TypeScript with auto-instrumentation:

import { NodeSDK } from "@opentelemetry/sdk-node";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-grpc";
import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";

const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter({
    url: "http://localhost:4317",
  }),
  instrumentations: [getNodeAutoInstrumentations()],
  serviceName: "checkout-service",
});

sdk.start();

The auto-instrumentations-node package automatically instruments HTTP, gRPC, Express, database clients, and dozens of other libraries — no manual span creation needed for most frameworks.

Jaeger vs Zipkin vs commercial APMs

	Jaeger	Zipkin	Datadog APM / New Relic
License	Apache 2.0	Apache 2.0	Proprietary
Storage	Elasticsearch, Cassandra, Kafka, Badger	Elasticsearch, Cassandra, MySQL	Vendor-hosted
Protocol	OTLP native, Thrift legacy	Zipkin JSON/Thrift, OTLP via collector	OTLP, proprietary agents
Auto-instrumentation	Via OpenTelemetry SDKs	Via OpenTelemetry or Brave	Proprietary agents with deep integration
Cost	Infrastructure only	Infrastructure only	$15-75/host/month
Trace analytics	Search + compare in UI	Search in UI	ML anomaly detection, dashboards, alerting

Choose Jaeger when you want full control over your tracing data, you're already running Elasticsearch or Cassandra, and you have the capacity to operate the backend.

Choose a commercial APM when you need trace-based alerting, ML anomaly detection, or you don't want to operate storage infrastructure. The cost is real, but so is the operational burden of self-hosted tracing.

Choose Zipkin when you have an existing Zipkin deployment. For new setups, Jaeger has better OTLP support and a more active community.

For larger deployments, the OpenTelemetry Collector sits between your SDKs and Jaeger, handling batching, sampling, and multi-backend fan-out.

When tracing isn't enough

Tracing answers "which service in the chain is slow?" It doesn't answer "is the service reachable from the outside?" A trace only exists when a request is made — if your service is completely down and rejecting connections, there are no spans to collect.

This is where uptime monitoring and tracing complement each other:

Signal	What it tells you	Blind spot
Uptime monitoring	Is the service responding? How fast? Is the SSL cert valid?	Why is it slow? Which internal dependency is the bottleneck?
Distributed tracing	Where in the call chain did latency accumulate? Which downstream service errored?	Is the service reachable at all? Did the DNS resolve? Did the cert expire?

The most useful setup is both: an external monitor that checks your endpoints every 30 seconds from multiple regions, combined with internal tracing that captures the request path when those endpoints are hit. When the monitor fires an alert because p95 latency crossed your SLO threshold, the trace for that time window shows you exactly which downstream call caused the spike.

For dependency-driven outages — a cloud provider degrades, your payment service slows down, your checkout endpoint breaches its latency budget — a vendor status feed tells you the dependency degraded before you start digging through traces. That head start on root cause identification is the difference between a 15-minute MTTR and an hour-long investigation.

Where to start

If you've never used distributed tracing, start with Jaeger all-in-one in Docker (the command above), instrument one service with the OpenTelemetry SDK, and trace a single request end-to-end. The first time you see a 2-second span on a database query that you thought took 50ms, tracing has paid for itself.

Pair it with external monitoring. Set up a monitor at app.devhelm.io for the same endpoint you're tracing — you'll know both that the service is degraded and where in the call chain the problem lives.

Originally published on DevHelm.

MTTA, MTTR, MTBF, MTTF — The Four Incident Metrics, Compared

DevHelm — Tue, 02 Jun 2026 10:12:05 +0000

Four acronyms show up in every incident management conversation: MTTA, MTTR, MTBF, and MTTF. They get jumbled together in slide decks, confused in retro discussions, and mixed up in job interviews. They measure four different things, from four different timestamps, with four different improvement levers.

This guide puts all four side by side, traces them through a single incident timeline, and answers the question that matters: which one should you track, and when?

One incident, four metrics

The cleanest way to understand the four metrics is to walk through one incident and label where each measurement starts and stops.

Time	Event
14:00:00	Service starts returning 500 errors (failure begins)
14:02:30	Monitor fires an alert (detection)
14:05:00	On-call engineer acknowledges the page (acknowledgment)
14:42:00	Service restored, incident resolved (recovery)
—	Next failure occurs 12 days later

From these timestamps:

Metric	Measures	Start	End	This incident
MTTD	Time to detect	Failure begins (14:00)	Alert fires (14:02:30)	2.5 min
MTTA	Time to acknowledge	Alert fires (14:02:30)	Engineer acks (14:05)	2.5 min
MTTR	Time to recovery	Failure begins (14:00)	Recovery (14:42)	42 min
MTTF	Time to next failure	Recovery (14:42)	Next failure start	~12 days
MTBF	Full cycle	This failure start	Next failure start	~12 days + 42 min

MTTD and MTTA are the early-warning metrics — they tell you how fast you detected and responded. MTTR is the incident-duration metric — it measures total impact time. MTTF and MTBF are the reliability metrics — they measure how often failures happen.

MTTA — Mean Time To Acknowledge

MTTA measures the gap between an alert firing and a human confirming they're working on it. It's a measure of your on-call process, not your technical system.

What it captures: pager responsiveness, on-call discipline, alert routing effectiveness.

What it misses: everything after acknowledgment. A team with a 30-second MTTA and a 4-hour resolution time has a fast paging system and a slow debugging process.

Improvement levers: better alert routing (fewer false positives means alerts get trusted and acknowledged faster), escalation policies that page a backup after N minutes, on-call overlap during shift handoffs.

DevHelm tracks confirmedAt (multi-region incident confirmation) but does not yet record a separate human acknowledgment timestamp — the acknowledgment step lives in your PagerDuty, Opsgenie, or Slack integration today.

MTTR — Mean Time To Recovery

MTTR is the most widely tracked incident metric. It measures the total elapsed time from failure start to service recovery. This is the metric your SLO error budget cares about: every minute in MTTR is a minute of downtime consumed.

The deep dive is in the MTTR full form guide, but the key point for comparison: MTTR includes detection time, acknowledgment time, diagnosis, and fix. It's an end-to-end metric, which makes it the most useful for external stakeholders but the hardest to improve because the bottleneck could be anywhere in the chain.

What it captures: total customer-facing impact time.

What it misses: failure frequency. A service with a 5-minute MTTR that fails ten times a month has a fundamentally different problem than one with a 5-minute MTTR that fails once a year.

Improvement levers: faster detection (monitoring with short check intervals), better runbooks (reduce diagnosis time), automated remediation, multi-region failover.

MTBF — Mean Time Between Failures

MTBF measures the average time from the end of one failure to the start of the next. It's the reliability metric: high MTBF means the system is stable; low MTBF means it breaks often.

The deep dive is in the MTBF full form guide. The key point here: MTBF = MTTF + MTTR. It spans the entire failure cycle.

What it captures: system stability, failure frequency, whether your reliability investments are working.

What it misses: failure severity. A service with 100 sev4 flaps per month has a terrible MTBF but no real reliability problem. Filter by severity level (sev1+sev2 only) to keep MTBF meaningful.

Improvement levers: root cause elimination, dependency isolation (circuit breakers, fallbacks), better testing, capacity planning.

MTTF — Mean Time To Failure

MTTF measures the operating time between recovery and the next failure — "how long does the system run before it breaks again?"

In hardware reliability, MTTF is for non-repairable components (light bulbs, hard drives) while MTBF is for repairable systems. In software, everything is repairable, so MTTF is the uptime component of MTBF: MTTF = MTBF - MTTR.

What it captures: the same thing as MTBF minus the recovery time. For services with low MTTR (minutes), MTTF and MTBF are nearly identical.

What it misses: recovery quality. If you "fix" an incident by restarting a pod and the root cause is still there, MTTF will be short because the failure recurs quickly. MTTF rewards durable fixes.

When MTTF matters more than MTBF: when your MTTR is highly variable. If some incidents take 5 minutes and others take 5 hours, MTBF averages the downtime in, masking the variance. MTTF isolates the operating-time question from the recovery-time question.

When you need which one

Not every team needs all four metrics. Here's the decision framework:

Question you're asking	Metric	Why
"How fast do we respond to alerts?"	MTTA	Measures on-call process health
"How long are our customers affected?"	MTTR	Measures total incident duration
"How often do things break?"	MTBF	Measures failure frequency
"Are our fixes durable?"	MTTF	Isolates operating time from recovery
"Is our monitoring fast enough?"	MTTD	Measures detection lag

Start with MTTR. Every team should track it, because it directly maps to customer impact and error budgets. The Google SRE Workbook centers its SLO framework on availability — and availability is the inverse of cumulative MTTR.

Add MTBF when MTTR is stable but incidents are too frequent. If your MTTR is 15 minutes but you're having incidents three times a week, the problem isn't response speed — it's system stability. MTBF makes that visible.

Add MTTA when you suspect paging is the bottleneck. If incidents take 45 minutes to resolve but 20 of those minutes are "waiting for someone to respond," MTTA makes the on-call gap visible.

Track MTTF when you suspect fixes aren't durable. If the same incident recurs within days of being "resolved," MTTF will be conspicuously low while MTBF might still look acceptable (because it averages in the stable periods between recurrence clusters).

Common pitfalls

Averaging across severity levels. A fleet of 10 sev4 flaps and 1 sev1 outage produces an "MTTR of 8 minutes" that hides the 2-hour sev1. Always segment metrics by severity level.

Counting self-healing as incidents. If your system auto-recovers in 30 seconds, is that a "failure" for MTBF purposes? Most teams exclude incidents that resolve within the confirmation window (e.g., DevHelm's multi-region confirmation requires failures across at least 2 regions before opening an incident). If you don't exclude auto-recoveries, MTBF becomes noise.

Comparing MTBF across services. A payment service and a notification service have fundamentally different blast radii. Comparing their MTBF is like comparing a car engine's MTBF to a light switch's. Track each service independently.

Ignoring partial recoveries. An incident where the service is "up but slow" for 2 hours, then fully recovered, has a different MTTR depending on whether you measure to partial recovery or full recovery. Define your measurement convention and stick to it.

Where to start

If you're tracking nothing, start with MTTR. Pull the last 90 days of sev1+sev2 incidents, compute the average duration from start to resolution, and write that number down. Next month, compute it again. The trend matters more than the absolute number.

Once MTTR is stable, add MTBF. Together they tell you whether you're dealing with a fragile-but-fast-recovering system (invest in prevention) or a stable-but-slow-recovering system (invest in runbooks and detection speed). That diagnostic drives your reliability roadmap more than any single metric could.

Set up monitoring at app.devhelm.io — every incident records the timestamps you need for both metrics. The 30-day rolling MTTR is already in your dashboard; MTBF is a script away from the incident API.

Originally published on DevHelm.

OpenTelemetry Collector Explained: What It Does and When You Need One

DevHelm — Tue, 02 Jun 2026 10:12:04 +0000

The first question every team asks when adopting OpenTelemetry is: "Do I need a Collector, or can my SDK export directly to Jaeger / Prometheus / Datadog?" The answer determines whether you run an extra piece of infrastructure or skip it entirely. Most guides explain what the Collector is without answering that question first. This one starts there.

Do you need a Collector?

Three questions decide it:

Do you export to more than one backend? If your traces go to Jaeger and your metrics go to Prometheus and your logs go to Loki, the Collector fans out from a single OTLP stream. Without it, every service needs three exporters configured in its SDK — three sets of credentials, three retry policies, three failure modes.
Do you need to transform telemetry before it lands? Scrubbing PII from span attributes, sampling 10% of low-priority traces, enriching resource labels with Kubernetes metadata — these are processor-layer concerns. The SDK can do basic attribute manipulation, but batch logic, tail-based sampling, and cross-signal correlation live in the Collector.
Do you want to decouple services from backend changes? If you switch from Jaeger to Tempo next quarter, a Collector means you change one exporter config in one place. Without it, you redeploy every instrumented service.

If you answered "no" to all three — you export to one backend, you don't transform telemetry, and your backend choice is stable — skip the Collector. Configure the OTLP exporter in your SDK to point directly at the backend. You can always add a Collector later without changing your instrumentation code, because the SDK speaks OTLP either way.

What a Collector actually is

The OpenTelemetry Collector is a vendor-agnostic proxy that receives telemetry, processes it, and exports it. The pipeline model has three stages:

Stage	Role	Examples
Receivers	Ingest data from sources	OTLP, Prometheus scrape, Jaeger Thrift, Fluent Forward, host metrics
Processors	Transform, filter, enrich, sample	Batch, memory limiter, attributes, tail sampling, k8s attributes
Exporters	Send data to backends	OTLP, Prometheus remote write, Jaeger, Elasticsearch, Datadog, Loki

You wire them together in a YAML config. One Collector can run multiple pipelines — traces, metrics, and logs each get their own receiver-processor-exporter chain.

Agent vs gateway deployment

Two deployment models, not mutually exclusive:

Agent mode. Run one Collector instance per node (DaemonSet in Kubernetes, sidecar in ECS). Each instance receives spans from local services over localhost, batches them, and forwards to the backend. Low latency, no network hop for span delivery, but you run N instances for N nodes.

Gateway mode. Run a small cluster of Collector instances behind a load balancer. All services send OTLP to the gateway endpoint. Centralized processing, easier to manage sampling and routing rules, but introduces a network hop and a single point of failure (mitigate with horizontal scaling and health checks).

Most production setups use both: agents on every node handle local buffering and basic processing, then forward to a gateway that handles tail-based sampling and multi-backend fan-out.

A working configuration

Here's a Collector config for a common stack: receive OTLP from instrumented services, batch spans to reduce export overhead, and send to a Jaeger backend.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
    send_batch_size: 512
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
    spike_limit_mib: 128

exporters:
  otlp/jaeger:
    endpoint: jaeger-collector:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/jaeger]

The memory_limiter processor is not optional in production. Without it, a burst of spans can OOM the Collector. The check_interval: 1s polls memory usage every second; when usage exceeds limit_mib, the Collector starts dropping data (better than crashing).

Run it with the contrib distribution, which includes all community-maintained receivers and exporters:

docker run -d --name otel-collector \
  -v ./otel-config.yaml:/etc/otelcol-contrib/config.yaml \
  -p 4317:4317 -p 4318:4318 \
  otel/opentelemetry-collector-contrib:latest

Common pitfalls

Memory bloat from unbounded batching. The default batch processor has no memory limit. In a spike, it buffers everything in memory until the export succeeds or the process dies. Always pair batch with memory_limiter and set spike_limit_mib to at most 25% of your container's memory limit.

Exporter back-pressure cascading upstream. If Jaeger is slow to ingest, the Collector's export queue fills up, the batch processor blocks, and incoming OTLP requests start timing out in your application SDKs. Set sending_queue.queue_size on the exporter and accept that data will be dropped under sustained back-pressure — dropping spans is better than adding latency to your application's hot path.

Running the core distribution instead of contrib. The core Collector (otel/opentelemetry-collector) ships with a minimal set of components. Most real deployments need contrib receivers (Prometheus, Fluent Forward) or exporters (Datadog, Elasticsearch). Use otel/opentelemetry-collector-contrib unless you're building a custom distribution.

Skipping resource detection. Without the resourcedetection processor (or k8sattributes in Kubernetes), your spans lack metadata like service.namespace, k8s.pod.name, and cloud.region. Debugging a trace without knowing which pod produced it is painful.

Collector vs SDK-direct export

	SDK-direct	Via Collector
Latency	One fewer network hop	Adds ~1-5ms (localhost agent) or ~5-20ms (gateway)
Reliability	SDK retries on failure; spans may be lost if the app crashes	Collector buffers and retries independently of the app
Flexibility	One exporter per backend, configured per service	Fan-out, sampling, enrichment in one place
Operational cost	Zero extra infra	DaemonSet or gateway to run and monitor

For a single-service, single-backend setup (one API exporting traces to Jaeger), SDK-direct is simpler and has fewer moving parts. For anything beyond that — multiple services, multiple backends, or any processing requirement — the Collector is worth the operational cost.

How monitoring complements the Collector pipeline

The Collector is infrastructure, and infrastructure fails. An OOM, a misconfigured exporter, or a network partition between the Collector and your backend means spans are silently dropped. You won't notice until someone asks "why are there no traces for the last 2 hours?"

External uptime monitoring closes that gap. A monitor that checks your Collector's health endpoint (http://collector:13133/) every 30 seconds catches Collector failures before the gap in your trace data becomes an incident. If your MTTR for "Collector is down" is 2 hours because nobody noticed, a 30-second check interval cuts that to minutes.

Set up a monitor at app.devhelm.io for every piece of your observability stack — the Collector, Jaeger, Prometheus, Grafana. The irony of observability infrastructure is that it's the last thing teams monitor, and the first thing that causes blind spots when it fails.

Originally published on DevHelm.

MTTR Full Form: Meaning, Formula, and How to Reduce It

DevHelm — Wed, 27 May 2026 19:50:41 +0000

Ask three SREs what MTTR stands for and you'll get three answers. Mean Time To Recovery. Mean Time To Repair. Mean Time To Respond. Sometimes Resolve. They are not the same metric. They measure different parts of an incident, they imply different ownership boundaries, and conflating them is the single most common reason an engineering team's "MTTR is improving" slide is meaningless. This guide pins down the MTTR full form, walks through the formula with real numbers, distinguishes MTTR from MTBF, MTTF, MTTA, and MTTD, and ends with the six changes that actually reduce the number — including one that most monitoring tools cannot do at all.

What MTTR stands for

MTTR is one of four metrics that share the same letters and very different definitions:

Acronym	Full form	What it measures	Typical owner
MTTR	Mean Time To Recovery	From the moment the service degrades to the moment it is fully back to normal, averaged across incidents	SRE / on-call
MTTR	Mean Time To Repair	From the moment an engineer starts working on a fix to the moment the fix is deployed	Engineering
MTTR	Mean Time To Respond	From alert to first human acknowledgement	On-call rotation
MTTR	Mean Time To Resolve	From incident creation to the incident being closed in the tracker (includes paperwork)	Incident commander

The version you almost always want, and the one used in the SRE literature and the Google SRE Workbook, is Mean Time To Recovery — the customer-visible downtime metric. The other three are sub-stages inside it. When you see a public number that's "MTTR is 8 minutes" without further qualification, assume Recovery. When a vendor pitches you a 30-second MTTR, ask which one they mean — they almost certainly mean Mean Time To Respond, which is the easiest to game.

In this guide, MTTR means Mean Time To Recovery unless explicitly noted.

The MTTR formula

The formula is simple: MTTR = total downtime / number of incidents.

A worked example. In May 2026 a hypothetical SaaS had four customer-impacting incidents:

Incident	Detected	Recovered	Duration
Stripe webhook backlog	02:14	02:51	37 min
Postgres failover	09:22	09:34	12 min
OpenAI 429 spike	14:01	14:48	47 min
CDN cert expiry	18:30	18:34	4 min
Total			100 min

MTTR = 100 min / 4 incidents = 25 minutes.

Two warnings before you start tracking this.

Warning 1: averages hide the long tail. A team with one 4-hour incident and ten 5-minute incidents has the same MTTR as a team with eleven 25-minute incidents. The first team has a tail problem; the second has a baseline problem. Always look at MTTR alongside the p95 incident duration and the count of incidents over 1 hour.

Warning 2: the start time is contested. Customer-visible downtime starts when the first customer experiences a problem, not when your alert fires. If your detection has a 2-minute lag and your MTTR is 8 minutes from alert, the customer-visible MTTR is 10 minutes. Most teams quietly start the clock at alert time because it makes the number smaller. Don't.

MTTR vs MTBF, MTTF, MTTA, and MTTD

The reliability metric family is large and the acronyms overlap enough that even seasoned SREs slip up. Here's the canonical lineup:

Acronym	Full form	What it measures	Typical good value (SaaS)
MTBF	Mean Time Between Failures	Average uptime between two consecutive incidents	Weeks to months
MTTF	Mean Time To Failure	Average uptime of a component before it fails (used for non-repairable parts)	Years (hardware), N/A for services
MTTD	Mean Time To Detect	From the moment a problem starts to the moment it's detected	< 1 minute
MTTA	Mean Time To Acknowledge	From alert fired to first human acknowledgement	< 5 minutes
MTTR	Mean Time To Recovery	From problem start to service restored	< 30 minutes

MTTD + MTTA + MTTR can be added together for a fuller picture of the incident lifecycle. The shorthand is MTTD + MTTA + the time spent on repair = MTTR (Recovery). If you split it that way, you can see which sub-stage is dragging the number — and they almost always need different fixes.

For an internal dashboard, track all five and let each have its own threshold. The teams that ship the fastest improvements treat MTTD as a detection problem, MTTA as an alerting and routing problem, and MTTR-the-repair as an engineering practice problem.

What's a good MTTR?

There is no industry-wide answer. The right MTTR depends on what your service does, how many customers it has, and what you're willing to pay to move the number. A few benchmarks:

DORA elite performers (state of DevOps report, 2023): MTTR under 1 hour. Low performers: more than 1 week.
Google SRE textbook: target depends on the service's error budget. A 99.9% SLO over a 30-day window allows 43 minutes of downtime — your MTTR per incident must fit inside what's left of that budget.
Atlassian incident management benchmark (source): teams at the median run an MTTR around 4 hours. Top quartile is under 30 minutes.
Banking and trading platforms: regulators sometimes mandate MTTR thresholds tied to capital reserves. 5 minutes is not unusual for top-tier financial services.
Internal-only B2B SaaS with no real-time SLA: 1-4 hours is acceptable.

Use these as anchors, not as targets. The right MTTR for your service comes from your SLO and your customer impact model, not from a benchmark.

How to actually reduce MTTR

Six changes consistently move the number. They're listed in order of how quickly they pay off for a small team that's just starting to take this seriously.

1. Write runbooks for the top five recurring incidents

Most incidents repeat. If you've fought the same Postgres failover behavior twice this quarter, you'll fight it again next quarter. Pick the five most common incident types in your tracker and write a runbook for each. The runbook needs five sections: trigger, symptoms, diagnosis steps, mitigation, verification. Aim for 200-400 words per runbook. Store them where on-call can find them in 30 seconds at 3 AM — not in a wiki you have to log into, not in a Notion folder buried three clicks deep. We keep ours in the same git repository as the service code under runbooks/, linked directly from every alert.

2. Tune alert routing to skip people who can't act

If a Stripe webhook backlog wakes up the whole team but only one person knows how to clear it, every other pager-recipient adds noise without adding hands. Route severity-2 alerts to a primary; promote to a wider group only if the primary doesn't acknowledge in 5 minutes. The MTTA drops, the MTTR drops, the team stops resenting on-call.

3. Make rollback the cheap option

If your rollback path takes 30 minutes and your forward-fix path takes 20 minutes, every incident becomes "let's try to fix it forward." Forward-fixes under pressure produce more incidents. Cut your rollback path to under 5 minutes and the first response to most incidents becomes "roll back, then debug calmly." Every CI/CD platform supports this — most teams just don't drill it.

4. Run blameless post-mortems and actually follow the action items

A post-mortem that produces an action-items list which nobody owns is theatre. Assign each action item to a single named person with a due date, track them in your issue tracker, and report on closed-vs-open in your weekly engineering review. The action items from last quarter's incidents are the cheapest way to reduce next quarter's MTTR — they're already pre-prioritized by the fact that the incident hurt enough to investigate.

5. Detect dependency failures before your monitors do

This is the change most monitoring tools cannot make for you — and it's one of the key differences when comparing monitoring platforms. A substantial share of customer-visible SaaS incidents — many teams report something near a third of them — are caused by an upstream dependency degradation: Stripe slows down, OpenAI rate-limits, your CDN edge has a regional issue, your database provider has a partial outage. When that happens, your own monitors fire — but you spend the first 15 minutes deciding whose problem it is. Status pages that watch your vendors and surface their incidents alongside your own check failures collapse that diagnostic window. We'll come back to this in the next section.

6. Schedule game days

A team that has never practiced an incident response will respond slowly. A team that runs a chaos exercise once a quarter — kill the database primary, simulate a CDN outage, page someone who isn't on-call — recovers from real incidents noticeably faster. The investment is one engineering day per quarter. The payback is measured in customer hours.

How DevHelm reduces MTTR

DevHelm is built around one specific MTTR problem: incidents where the root cause is a vendor your service depends on, not your service itself. When Stripe degrades, your checkout monitor fires. Your billing-webhook monitor fires. Your subscription-sync monitor fires. Three pages. Three Slack pings. Twenty minutes of "is anyone seeing the Stripe dashboard?" before someone confirms that Stripe themselves posted a status update.

DevHelm shortens that pattern in two concrete ways today.

1. Vendor status pages, watched continuously. We aggregate 100+ vendor status pages — GitHub, AWS, Cloudflare, Datadog, and every major dependency. Each page has its own incident feed; you can subscribe each one to the same Slack channel your own monitors page to, so an AWS degradation lands in your incident channel within seconds of AWS posting it, alongside your monitor alerts. That collapses the diagnostic loop: instead of fifteen minutes of "is it the dependency or is it us?", the answer is in the same channel.

2. Resource groups, for collapsing self-noise. When several of your monitors share a single failure mode (e.g. all of them call Stripe), you put them in a resource group with a single notification policy. One incident, not three. The recovery clock keeps ticking until Stripe themselves recover, but your on-call isn't paged three times for one root cause. The YAML for the on-call-friendly version looks like:

# devhelm.yml
resourceGroups:
  - name: stripe-fleet
    monitors:
      - checkout-page-uptime
      - billing-webhook-uptime
      - subscription-sync-uptime
notificationPolicies:
  - name: stripe-fleet-sev1
    matchRules:
      - type: monitor_id_in
        monitorNames:
          - checkout-page-uptime
          - billing-webhook-uptime
          - subscription-sync-uptime
    escalation:
      steps:
        - channels: [oncall-slack]

The piece DevHelm doesn't yet do automatically is the cross-correlation step — when Stripe goes down, your in-flight monitor alerts don't get auto-marked as "probably caused by that vendor" without you wiring up the resource group + Slack subscription yourself. That auto-correlation is on the roadmap. Until it ships, the manual wire-up is the work — but the work pays back at every dependency incident, which is a meaningful slice of total incidents for most modern SaaS teams.

The other action items in the previous section — runbooks, alert routing, rollback discipline, post-mortem follow-through, game days — DevHelm can't do for you. But it can give you back the time you currently spend correlating vendor outages by hand. If you're troubleshooting a specific failure mode right now, the DNS resolution guide and the vendor status feeds (GitHub, AWS, Cloudflare, and 100+ more) are useful starting points.

If your last incident was a vendor problem and your team spent the first 20 minutes figuring out whose problem it was, that diagnostic loop is the cheapest fix target. Spin up a free account at app.devhelm.io and connect your first vendor in 60 seconds — no credit card.

Originally published on DevHelm.

Why We Built DevHelm

DevHelm — Sun, 17 May 2026 14:27:44 +0000

The era of monitoring tools built for human engineers is over.

Site reliability is undergoing a profound shift triggered by the agentic AI wave, and to understand why it matters, it helps to look at the pattern that came before. Every major innovation cycle until now was defined by a new environment that software runs in: on-premise to cloud, cloud to mobile, monolith to microservices. Through each of those transitions, humans remained indispensable to the SRE process. Humans debugged. Humans investigated. Humans identified root causes and remediated them, while communicating the full picture to customers and stakeholders along the way.

The next wave is different. AI SRE agents can now automate large parts of that process and free human time for the decisions that actually require judgment. An AI agent can conduct a root cause investigation, understand the blast radius of an incident, classify its priority, and surface that context to on-call engineers — all before a human has finished reading the first alert. In this environment, the speed of iteration on reliability increases dramatically. AI can investigate faster, identify patterns earlier, and be far more proactive about surfacing deep underlying issues by synthesizing information from sources that no single engineer would think to check at once.

In that world, the monitoring infrastructure itself becomes the agent's most critical tool. And for it to be effective, it has to be built in an agent-first, developer-first way. It must provide clear primitives for management, operations, and forensic investigation — alongside the external-facing artifacts that reliability demands, like status pages and incident communications. The old approach to SRE tooling, built around beautiful but unintegrated dashboards designed for human eyes, is fundamentally incompatible with this new paradigm.

That is why we built DevHelm. Our primary focus was to deliver a developer-first, API-first platform that supports operations in this new AI-driven reality. We are launching with uptime monitoring, dependency monitoring, status pages, and developer artifacts purpose-built for agentic operations: a native CLI, Cursor and Claude skills, Python and TypeScript SDKs, an MCP server, and a Terraform provider — all included from the free tier.

Our long-term goal is to build a unified reliability platform that powers the next generation of applications and services built in the agentic AI era.

We are just getting started. Try DevHelm free.

Originally published on DevHelm.

Introducing DevHelm

DevHelm — Sun, 17 May 2026 14:27:39 +0000

Today we are launching DevHelm — a reliability platform built to bring developer-first, agent-first monitoring infrastructure to the teams that need it most.

Seeing a massive shift in how site reliability is practiced, driven by the agentic AI wave, we decided that monitoring needed to be rebuilt around a different premise: something developers define in code, AI agents operate programmatically, and your entire team understands through clear external-facing artifacts. Everything we ship reflects that premise, from the core monitoring infrastructure to the way you interact with it.

Monitoring that understands your stack

At the foundation, DevHelm provides multi-protocol uptime monitoring — HTTP, DNS, TCP, and ICMP checks running from five continents at 30-second intervals with multi-region confirmation before any alert fires. That part is table stakes, and we made sure it works well.

What makes DevHelm different is what sits on top of it. We track over 100 external services — Stripe, AWS, GitHub, Auth0, OpenAI, and dozens more — and correlate their health with your monitors in real time. When a vendor degrades, you don't get a separate alert for every endpoint that happens to depend on it. You get one resource group alert that tells you what's affected and why, with the vendor incident already linked. The goal is signal, not noise: your team should spend time fixing problems, not figuring out whether a problem is even yours to fix.

Built for developers and their agents

The monitoring infrastructure is only useful if it's accessible to the tools and workflows that actually operate your stack. That's why every capability in DevHelm is available through a full developer surface from the free tier: a native CLI, Python and TypeScript SDKs, a Terraform provider, an MCP server, and pre-built skills for Cursor and Claude. Monitors can be defined in a YAML config in your repo and deployed through your CI pipeline — no dashboard clicks required.

In practice, this means an AI agent working in Cursor or Claude can define monitors, configure alert routing, set up a status page, and investigate an incident through the same programmatic interfaces a human developer would use. The platform doesn't distinguish between the two, because in the operating model we're building for, it shouldn't have to.

Status pages and incident communication

Reliability isn't just an internal discipline — it has an external face. Every DevHelm account includes a public status page with custom domain support, real-time monitor status, and subscriber notifications. Status pages are not an upsell; they are part of the reliability infrastructure, and they're included from day one.

What's next

We are launching with uptime monitoring, dependency intelligence, status pages, and the full developer artifact surface. This is the foundation. Our roadmap builds toward a unified reliability platform: deeper forensic investigation tools, richer incident lifecycle management, and tighter integration with the AI agents that increasingly operate alongside engineering teams.

Try DevHelm free — 50 monitors, a status page with custom domain, and the full developer surface. No credit card.

Originally published on DevHelm.

What SSL Error Means and How to Fix It

DevHelm — Sun, 17 May 2026 14:22:13 +0000

An SSL error means your browser or HTTP client could not complete the TLS handshake with the server. The connection was dropped before any data was exchanged. Instead of your page, your users see a full-screen warning — and most of them leave.

The term "SSL error" is a holdover. SSL (Secure Sockets Layer) was deprecated in 2015 when RFC 7568 declared SSL 3.0 obsolete. Every modern HTTPS connection uses TLS (Transport Layer Security) — versions 1.2 or 1.3. Browsers still display "SSL" in error codes because the names stuck, but the protocol under the hood is always TLS. Throughout this article, "SSL error" refers to any TLS handshake failure your browser surfaces.

What happens during a TLS handshake

When a browser connects to an HTTPS server, the TLS handshake negotiates a shared encryption key. The server presents its certificate, the browser verifies the chain of trust back to a root CA, checks the hostname, and confirms the certificate has not expired. If any step fails, the browser aborts and shows an error page.

The three most common failure points:

Certificate validity — the cert is expired, not yet valid, or revoked
Hostname mismatch — the cert was issued for api.example.com but the browser hit www.example.com
Chain of trust — an intermediate certificate is missing, or the cert is self-signed

Understanding which step failed tells you exactly where to look.

Decode the error message — Chrome, Firefox, and Safari

Different browsers surface different error codes for the same underlying TLS failure. This table maps the most common ones:

Problem	Chrome	Firefox	Safari
Expired certificate	`NET::ERR_CERT_DATE_INVALID`	`SEC_ERROR_EXPIRED_CERTIFICATE`	"This certificate has expired"
Wrong hostname	`NET::ERR_CERT_COMMON_NAME_INVALID`	`SSL_ERROR_BAD_CERT_DOMAIN`	"This certificate is not valid for the requested site"
Self-signed cert	`NET::ERR_CERT_AUTHORITY_INVALID`	`SEC_ERROR_UNKNOWN_ISSUER`	"This certificate was signed by an unknown authority"
Incomplete chain	`NET::ERR_CERT_AUTHORITY_INVALID`	`SEC_ERROR_UNKNOWN_ISSUER`	"This certificate is not trusted"
TLS version too old	`ERR_SSL_VERSION_OR_CIPHER_MISMATCH`	`SSL_ERROR_UNSUPPORTED_VERSION`	Connection refused (no specific code)
Revoked certificate	`NET::ERR_CERT_REVOKED`	`SEC_ERROR_REVOKED_CERTIFICATE`	"This certificate has been revoked"

If you see ERR_SSL_PROTOCOL_ERROR in Chrome, the server likely rejected the handshake outright — possibly because it only supports TLS 1.0/1.1 (both deprecated) or has a misconfigured cipher suite.

Fix 1 — Expired or not-yet-valid certificate

An expired certificate is the single most common cause of SSL errors. Certificates have a fixed validity window — typically 90 days for Let's Encrypt and up to 398 days for paid CAs.

Diagnose it with openssl:

openssl s_client -connect yoursite.com:443 -servername yoursite.com 2>/dev/null | openssl x509 -noout -dates

Example output:

notBefore=Feb 15 00:00:00 2026 GMT
notAfter=May 16 23:59:59 2026 GMT

If notAfter is in the past, the cert has expired.

Fix it:

Renew the certificate through your CA or ACME client (certbot renew, for example)
Reload your web server — sudo systemctl reload nginx or sudo systemctl reload apache2
Verify the new cert is live: re-run the openssl command above and confirm the dates

If your cert is not yet valid (notBefore is in the future), either the cert was issued early and installed before activation, or your server clock is wrong. Check with date -u and sync via NTP if needed.

Fix 2 — Wrong hostname or missing SAN

Your certificate must cover the exact hostname the client connects to. A cert issued for example.com does not automatically cover www.example.com — that requires a Subject Alternative Name (SAN) entry.

Diagnose it:

openssl s_client -connect yoursite.com:443 -servername yoursite.com 2>/dev/null | openssl x509 -noout -text | grep -A1 "Subject Alternative Name"

Example output:

X509v3 Subject Alternative Name:
    DNS:example.com, DNS:www.example.com

If the hostname your users hit is not listed, you need to reissue the certificate with the correct SANs — or use a wildcard cert (*.example.com). Wildcard certs cover one level of subdomains only; *.example.com matches api.example.com but not v2.api.example.com.

A common mistake: deploying behind a load balancer or CDN and forgetting that the cert on the edge must match the public hostname, not the origin hostname.

Fix 3 — Incomplete chain or self-signed certificate

Browsers verify certificates by walking the chain from your server cert through intermediate CAs to a trusted root. If an intermediate is missing, the chain breaks and the browser shows ERR_CERT_AUTHORITY_INVALID.

Diagnose the chain:

openssl s_client -connect yoursite.com:443 -servername yoursite.com -showcerts 2>/dev/null | grep "s:" | head -5

A healthy chain shows your cert, then one or two intermediates, ending at the root:

s:CN = yoursite.com
s:CN = R11, O = Let's Encrypt
s:CN = ISRG Root X1, O = Internet Security Research Group

If you see only your cert with no intermediates, your server is not sending the full chain. Fix it by concatenating the intermediate cert(s) with your server cert. For Nginx:

cat server.crt intermediate.crt > bundle.crt

Then reference bundle.crt in your Nginx config's ssl_certificate directive and reload.

Self-signed certificates fail on public-facing sites because they are not issued by a trusted CA. Replace them with a cert from Let's Encrypt (free) or any recognized CA. Self-signed certs are fine for internal services — but add them to your internal trust store explicitly rather than telling users to "click through the warning."

Fix 4 — Mixed content and HSTS issues

Mixed content errors happen when an HTTPS page loads a resource (image, script, stylesheet) over plain HTTP. Modern browsers block mixed active content (scripts, iframes) entirely and show a broken padlock for mixed passive content (images).

Find mixed content using your browser's developer console (F12 → Console tab). The browser logs every blocked resource with its URL.

Fix it by updating hardcoded http:// URLs to https:// or using protocol-relative paths. If you use a CMS, update the site URL in settings.

HSTS (HTTP Strict Transport Security) adds another layer: once a browser has seen an HSTS header, it refuses to connect over HTTP at all — even if the cert is temporarily broken. If you deployed a broken cert and HSTS is active, users cannot click through the warning. The only fix is deploying a valid cert. You can inspect cached HSTS policies in Chrome at chrome://net-internals/#hsts.

Fix 5 — Client-side false positives

Not every SSL error is a server problem. Three common client-side causes:

System clock skew. Certificates are time-sensitive. If a laptop's clock is set to 2024, a cert valid from 2026 appears "not yet valid." Fix: enable automatic time sync in the OS.
Antivirus TLS inspection. Some antivirus software intercepts HTTPS connections by inserting its own root certificate. If the AV root is not trusted by the browser — or if the AV botches the re-encryption — the browser shows an SSL error. Temporarily disabling the AV's "web shield" or "HTTPS scanning" confirms this as the cause.
Corporate proxy. Transparent HTTPS proxies (common in enterprise networks) perform the same kind of TLS interception. The corporate root CA must be installed on the client machine. If it is not, every HTTPS site shows a certificate warning.

These are real scenarios, not edge cases. If users report SSL errors that you cannot reproduce, ask about their local environment first.

Prevent SSL errors with certificate monitoring

You have seen how certificates break: expiry, hostname mismatches, incomplete chains, mixed content, client-side false positives. Every one of these failures is predictable. Certificates do not expire by surprise — they have a fixed lifetime printed right in the X.509 data. The problem is never that the failure was unknowable. The problem is that nobody was watching. And when an SSL failure does hit production, the incident severity depends on how much of your traffic is affected — which comes back to whether you caught it in one region or all of them.

The fix-then-forget cycle is the real trap. You renew the cert, confirm it works, and move on. Ninety days later, the same NET::ERR_CERT_DATE_INVALID reappears because the auto-renewal cron broke silently two weeks ago and nobody noticed until a customer opened a support ticket at 2 AM. That diagnostic loop is exactly what a runbook prevents — and measuring how long it takes is what MTTR is for.

Here is how to build a monitor that catches every failure mode we covered — before your users do.

Set up two expiry thresholds, not one

Most monitoring setups check whether the certificate expires within some number of days and call it done. That is not enough. You need two thresholds: an early warning and a hard deadline.

Let's Encrypt certificates last 90 days. If auto-renewal is working, you will never think about expiry. But if it breaks — a DNS validation failure, a misconfigured certbot hook, a container rebuild that lost the renewal cron — you want to know at 30 days remaining, not the day it expires. A WARN-severity assertion at 30 days gives your team two full weeks to investigate and fix the renewal pipeline without any urgency. A FAIL-severity assertion at 14 days is the hard deadline: drop everything and renew manually, because you are two weeks from a full outage.

Validate the endpoint, not just the certificate

A valid certificate does not mean your site works. The cert could be fine while your origin returns 502s, or while a misconfigured cipher suite causes 15-second handshakes that make the page feel broken. Adding a status code check (expected: 200) and a response time threshold (thresholdMs: 2000) catches the class of problems where TLS technically succeeds but the user experience is degraded. Slow TLS handshakes often point to missing OCSP stapling, oversized certificate chains, or a server negotiating an expensive cipher when a faster one is available.

Monitor from multiple regions

This is the one most teams skip, and it is the one that bites hardest. If you run behind a CDN — Cloudflare, AWS CloudFront, Fastly — your certificates are managed per edge location. A cert that is perfectly valid on the us-east edge node might already be expired on an ap-south node because the edge cert rotation did not propagate uniformly. Checking from a single region gives you a false sense of security. Checking from us-east, eu-west, and ap-south catches regional cert failures before the affected users report them.

Pick the right check frequency

Certificates change slowly. Unlike an API endpoint that might go down and recover in seconds, certificate state transitions happen once every 90 days (or 398 days for paid CAs). Running SSL checks every 30 seconds is wasteful — you are burning check quota on a signal that changes a handful of times per year. A 5-minute interval (frequencySeconds: 300) gives you more than enough visibility. If a cert expires, you will know within 5 minutes. The trade-off is worth it: save the high-frequency checks for your API health endpoints where seconds matter.

The full config

Here is a DevHelm monitor that covers everything above — expiry thresholds, endpoint validation, multi-region checks, and a sensible frequency:

{
  "name": "production-ssl-health",
  "type": "HTTP",
  "frequencySeconds": 300,
  "regions": ["us-east", "eu-west", "ap-south"],
  "config": {
    "url": "https://yourapp.com",
    "method": "GET"
  },
  "assertions": [
    { "config": { "type": "status_code", "operator": "equals", "expected": 200 } },
    { "config": { "type": "response_time", "thresholdMs": 2000 } },
    { "config": { "type": "ssl_expiry", "minDaysRemaining": 14 } },
    { "config": { "type": "ssl_expiry", "minDaysRemaining": 30 }, "severity": "WARN" }
  ]
}

The first two assertions confirm the endpoint is healthy and responsive. The third fires a critical alert at 14 days before expiry — your hard deadline. The fourth fires a warning at 30 days — your early warning that gives you time to fix the renewal pipeline without scrambling.

You can create this monitor from the CLI in one command:

devhelm monitor create --type http

Or configure it through the dashboard. 50 monitors free, no credit card required.

Start monitoring free

Originally published on DevHelm.

How to Fix Slow DNS Lookup: A Complete Troubleshooting Guide

DevHelm — Sun, 17 May 2026 14:22:07 +0000

Every connection your application makes starts with a DNS lookup. When that lookup is slow — or fails entirely — the symptoms range from vague latency increases to hard-down pages that return ERR_NAME_NOT_RESOLVED. This guide walks through how to fix slow DNS lookup issues, diagnose two of the most common DNS errors (DNS_PROBE_FINISHED_NXDOMAIN and "DNS server not responding"), and set up monitoring so these problems never wake you up at 3 AM again.

Why DNS lookups slow down

A DNS lookup traverses multiple layers before returning an IP address. Your stub resolver asks a recursive resolver, which queries root nameservers, then TLD nameservers, then the authoritative nameserver for the domain. Each hop adds latency. In a best case — a warm cache hit on the recursive resolver — resolution takes under 1 ms. In the worst case — a cold cache, long CNAME chains, DNSSEC validation, and an authoritative server on another continent — it can exceed 500 ms.

The most common causes of slow DNS resolution:

Overloaded or distant ISP resolvers. ISP DNS servers are shared infrastructure. During peak hours, query times spike from 20 ms to 200 ms or more.
Low TTL values. A TTL of 60 seconds means every cache expires every minute, forcing full recursive lookups. TTLs under 300 seconds are a common source of unnecessary latency.
CNAME chains. Each CNAME adds an extra lookup. A domain with three CNAME hops requires four total resolutions before returning an A record.
IPv6 fallback. When a system queries for AAAA records first and the authoritative server is slow to respond (or doesn't support IPv6), the client waits for a timeout before falling back to A records — adding 2–5 seconds.
VPN and split-tunnel DNS conflicts. Corporate VPNs often route DNS traffic through a tunnel to an internal resolver, adding 50–150 ms of round-trip latency that doesn't exist when the VPN is off.

Measure first — what "slow" actually means

Before changing anything, measure your current DNS performance. The dig command (Linux/macOS) and nslookup (Windows) are the standard diagnostic tools.

Measure with dig:

dig devhelm.io

The output you care about is at the bottom:

;; ANSWER SECTION:
devhelm.io.          300     IN      A       143.198.168.42

;; Query time: 24 msec
;; SERVER: 1.1.1.1#53(1.1.1.1) (UDP)
;; WHEN: Sun May 11 14:32:07 UTC 2026
;; MSG SIZE  rcvd: 56

The Query time line is what matters. Here is a reference table:

Query time	Rating	Action
< 15 ms	Excellent	No action needed
15–50 ms	Good	Acceptable for most workloads
50–100 ms	Poor	Switch resolver or investigate upstream
100+ ms	Critical	Immediate action required

Compare resolvers directly:

dig @1.1.1.1 devhelm.io | grep "Query time"
dig @8.8.8.8 devhelm.io | grep "Query time"
dig @9.9.9.9 devhelm.io | grep "Query time"

If your default resolver is 3–5x slower than Cloudflare (1.1.1.1) or Google (8.8.8.8), that is the first thing to fix.

Measure with nslookup on Windows:

nslookup devhelm.io
Server:  resolver1.isp.net
Address:  192.168.1.1

Non-authoritative answer:
Name:    devhelm.io
Address:  143.198.168.42

nslookup does not show query time directly. For timing on Windows, use PowerShell:

Measure-Command { Resolve-DnsName devhelm.io } | Select-Object TotalMilliseconds

Fix slow DNS lookup on your machine

These fixes address the most common causes of slow resolution, in order of impact.

1. Flush your local DNS cache

Stale or corrupted cache entries can cause lookups to hang or return wrong results. Flush first, then re-test.

macOS:

sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder

Linux (systemd-resolved):

sudo resolvectl flush-caches
resolvectl statistics | grep "Current Cache Size"

Windows:

ipconfig /flushdns

2. Switch to a faster public resolver

If your ISP resolver is slow, change to Cloudflare (1.1.1.1), Google (8.8.8.8), or Quad9 (9.9.9.9). These resolvers have global anycast networks that consistently resolve in under 15 ms from most locations.

Linux (/etc/resolv.conf or systemd-resolved):

sudo resolvectl dns eth0 1.1.1.1 1.0.0.1
sudo resolvectl dns eth0 # verify

macOS (System Settings > Network > DNS):

networksetup -setdnsservers Wi-Fi 1.1.1.1 1.0.0.1

3. Disable IPv6 DNS if you do not use it

If your network does not have working IPv6 connectivity, AAAA queries add timeout delays to every lookup. Test whether IPv6 is the problem:

dig AAAA devhelm.io @1.1.1.1 | grep "Query time"
dig A devhelm.io @1.1.1.1 | grep "Query time"

If the AAAA query is significantly slower or times out, consider disabling IPv6 resolution on your machine or configuring your resolver to deprioritize AAAA lookups.

4. Check your VPN's DNS configuration

VPNs commonly override DNS settings, routing queries through the tunnel. If DNS is slow only when connected to a VPN:

cat /etc/resolv.conf   # Linux: check which DNS server is active
scutil --dns | head -20 # macOS: check DNS configuration

If the resolver points to a VPN-provided address (e.g., 10.x.x.x), configure split-tunnel DNS so that only internal domains route through the VPN resolver.

How to fix DNS_PROBE_FINISHED_NXDOMAIN

DNS_PROBE_FINISHED_NXDOMAIN means the DNS resolver returned an NXDOMAIN response — the domain does not exist in DNS. Chrome, Edge, and Brave all surface this as an error page. The domain either genuinely does not exist, or something between your machine and the authoritative nameserver is blocking or misconfiguring the lookup.

Diagnosis, in order:

1. Verify the domain is correct. Typos account for most NXDOMAIN errors. Check for swapped letters, missing hyphens, and wrong TLDs (.com vs .io vs .dev).

2. Test from multiple resolvers. If your default resolver returns NXDOMAIN but a public resolver resolves the domain, your resolver has stale or filtered data:

dig example.com @1.1.1.1
dig example.com @8.8.8.8
dig example.com @$(cat /etc/resolv.conf | grep nameserver | head -1 | awk '{print $2}')

3. Check the authoritative nameserver directly. This confirms whether the domain's NS records are configured correctly at the registrar:

dig NS example.com @1.1.1.1
dig example.com @ns1.registrar.com

If the authoritative server itself returns NXDOMAIN, the domain's DNS zone is misconfigured or the domain has expired. Check with your registrar.

4. Flush DNS and restart the DNS client. A cached NXDOMAIN response (negative caching, per RFC 2308) can persist for the SOA minimum TTL, which defaults to hours on some zones.

5. Check your hosts file. A local override in /etc/hosts (Linux/macOS) or C:\\Windows\\System32\\drivers\\etc\\hosts (Windows) can shadow DNS entirely. Remove any stale entries for the domain.

6. Disable Chrome's secure DNS if it conflicts. Chrome aggressively prefetches DNS for links on a page. If prefetch queries go to a different resolver than your system default, you can get spurious NXDOMAIN errors. Navigate to chrome://settings/security and check the "Use secure DNS" setting — ensure it matches your intended resolver.

How to fix DNS server not responding

"DNS server not responding" means your machine sent a DNS query and received no reply at all — not even an error. This is different from NXDOMAIN (which is a valid response saying "this domain does not exist"). No response means the resolver itself is unreachable or unresponsive.

Systematic diagnosis:

Step 1: Confirm basic connectivity

Separate "network is down" from "DNS is down":

ping -c 3 1.1.1.1

If ping fails, the problem is your network connection, not DNS. Check cables, Wi-Fi, and router.

If ping succeeds, your network is fine but DNS is specifically broken. Continue.

Step 2: Test the DNS port directly

DNS uses UDP port 53 (and TCP 53 for large responses). Test whether your resolver is accepting connections:

dig @1.1.1.1 devhelm.io +tcp +timeout=5

If this works but normal queries fail, something is blocking UDP port 53 — commonly a firewall, router ACL, or ISP filter.

Step 3: Check your router

Home and office routers often run a local DNS forwarder. If the router's DNS process crashes or its upstream configuration is wrong, all devices on the network lose DNS.

Access your router admin panel (typically 192.168.1.1)
Check the configured upstream DNS servers
Try setting them to 1.1.1.1 and 8.8.8.8 as primary and secondary
Reboot the router

Step 4: Check for firewall or security software blocking DNS

Firewalls (especially on corporate networks), antivirus software, and parental control tools sometimes intercept or block DNS traffic. Temporarily disable them to isolate:

sudo iptables -L -n | grep 53

Step 5: Try DNS over HTTPS (DoH)

If your ISP is throttling or intercepting standard DNS (UDP/TCP port 53), DNS over HTTPS bypasses the interception by sending queries over HTTPS on port 443:

Firefox: Settings > Privacy & Security > Enable DNS over HTTPS
Chrome: Settings > Security > Use secure DNS > Select Cloudflare or Google
System-wide (Linux): Configure systemd-resolved with DNSOverTLS=yes

When the problem is upstream

Sometimes slow DNS is outside your control. Before blaming your resolver or network, check whether the problem is upstream:

Authoritative nameserver issues. The domain owner's nameserver may be slow or misconfigured. Test with dig +trace example.com to see exactly where in the resolution chain the delay occurs.
CDN misrouting. CDNs like Cloudflare and AWS CloudFront use DNS-based geographic routing. If your resolver's IP geolocation is wrong, you may be routed to a distant edge node. This is common with VPNs and small ISP resolvers.
Registrar glue record problems. If a domain's nameservers are under the same domain (e.g., ns1.example.com for example.com), the registrar must provide glue records — the A records for the nameservers themselves. Missing glue records create a circular dependency that manifests as timeouts.
Enterprise split-horizon DNS. In corporate environments, internal and external DNS zones overlap. A query for api.company.com might resolve to an internal IP on VPN and a public IP off VPN — or fail entirely if the split-horizon configuration has gaps.

Prevent DNS failures with monitoring

Everything you have done so far in this guide — flushing caches, switching resolvers, tracing NXDOMAIN responses, checking firewall rules — is reactive. You noticed a problem, diagnosed it, and fixed it. That reactive investigation is exactly the kind of work a runbook codifies so the next engineer doesn't repeat it from scratch. But the next DNS failure will not look like this one. An A record vanishes because someone fat-fingers a Terraform apply. A TTL gets dropped to 30 seconds during a migration and never gets reverted. Resolution times creep from 20 ms to 150 ms over three weeks because an upstream nameserver is quietly degrading. None of these announce themselves. They just erode your reliability until a user files a ticket or your on-call phone rings at 3 AM — and your MTTR climbs because the failure mode was unfamiliar.

A single "is DNS working?" check does not cover this. What you need is a layered set of assertions that catches the different ways DNS silently breaks.

Layer 1: Does it resolve at all?

The most fundamental check. A dns_resolves assertion confirms that your domain actually returns records — that the A record exists, that the AAAA record exists, that the response is not NXDOMAIN or SERVFAIL. If your A record disappears because of a zone file mistake or a registrar lapse, you find out in five minutes instead of five hours when customers start reporting a blank page.

Check both A and AAAA record types. Even if your application is IPv4-only, a broken AAAA record causes timeout-based fallback delays on clients that try IPv6 first — the exact problem covered in the IPv6 section above. Monitoring both means you catch issues on either path.

Layer 2: Does it resolve fast enough?

DNS that technically resolves but takes 200 ms adds 200 ms to every single page load, every API call, every webhook delivery. This latency is invisible in dashboards that only track HTTP response time because the DNS overhead happens before the connection even opens.

Two thresholds give you the coverage you need. A hard failure assertion (dns_response_time with a maxMs of 100) fires when resolution exceeds a critical ceiling — something is actively broken, whether that is an overloaded resolver, a network path change, or an authoritative server on another continent. A softer warning assertion (dns_response_time_warn with a warnMs of 50) fires at a lower threshold so you catch gradual degradation before it compounds into an outage. The warning gives you time to investigate during business hours. The hard failure pages your on-call immediately.

Layer 3: Are the TTLs healthy?

Low TTLs are a silent performance killer, and they show up constantly in the kinds of issues this guide covers. A TTL of 30 seconds means every visitor's browser, every edge server, and every recursive resolver on the planet discards the cached record every half minute and triggers a full recursive lookup. During a migration, it is common practice to temporarily lower TTLs to speed up propagation — and then forget to raise them back afterward.

A dns_ttl_low assertion with a minTtl of 300 catches exactly this. If someone — or an automated provisioning tool — drops your TTL below five minutes, you get a warning before the extra lookup load starts inflating resolution times across the board.

Layer 4: Check from multiple vantage points

DNS is not globally consistent. A record that resolves correctly from a probe in us-east might be stale, missing, or pointing to the wrong IP in ap-south because of propagation delays, regional resolver differences, or geo-DNS misconfigurations. If you only check from one region, you are testing your DNS health from one perspective and assuming the rest of the world agrees. It often does not.

Running checks from at least three regions — us-east, eu-west, and ap-south — ensures your monitoring reflects what your actual users experience rather than what a single datacenter sees.

Layer 5: Check against specific nameservers

By default, each probe region uses whatever recursive resolver is locally available. That is usually fine, but it means you can miss issues that are specific to a particular public resolver. Explicitly setting your nameservers to 1.1.1.1 and 8.8.8.8 — Cloudflare and Google, the two most widely used public resolvers — lets you test resolution from the same infrastructure your users are most likely hitting. If your domain resolves from Google but not Cloudflare (or vice versa), that points to a propagation issue or a resolver-specific caching problem that would otherwise be invisible until someone on the affected resolver reports it.

Putting it all together

Here is a complete DNS monitor configuration that implements all five layers:

{
  "name": "production-dns-health",
  "type": "DNS",
  "frequencySeconds": 300,
  "regions": ["us-east", "eu-west", "ap-south"],
  "config": {
    "hostname": "yourapp.com",
    "recordTypes": ["A", "AAAA"],
    "nameservers": ["1.1.1.1", "8.8.8.8"]
  },
  "assertions": [
    {"config": {"type": "dns_resolves"}},
    {"config": {"type": "dns_response_time", "maxMs": 100}},
    {"config": {"type": "dns_response_time_warn", "warnMs": 50}, "severity": "WARN"},
    {"config": {"type": "dns_ttl_low", "minTtl": 300}, "severity": "WARN"}
  ]
}

Every five minutes, from three continents, this monitor resolves yourapp.com for both A and AAAA records against Cloudflare's and Google's DNS. It fails hard if the domain does not resolve at all or if resolution takes longer than 100 ms. It warns if resolution exceeds 50 ms or if the TTL drops below 300 seconds.

The severity: "WARN" on the TTL and response time warning assertions is deliberate. These are degradation signals, not outage signals — they belong in a dashboard and a Slack channel, not in your PagerDuty rotation. The resolution check and the hard response time ceiling default to error severity, which is what triggers your incident workflow based on your severity levels. The distinction matters: you want to know about creeping latency during business hours, and you want to be woken up for a missing A record.

If your DNS infrastructure sits behind Cloudflare, you can also track their operational status through their public status feed — useful for distinguishing between your DNS issues and theirs.

You can create this monitor through the DevHelm dashboard, or from the terminal:

devhelm monitor create --type dns

Start monitoring free — DNS, HTTP, TCP, and ICMP checks from five continents, with the full CLI and API surface included.

Originally published on DevHelm.