The pager goes off at 03:12. The alert reads HighErrorRate — checkout-api — 5xx above 2% for 5 minutes. You have been on the team for seven weeks, you have never deployed checkout-api, and you are now the person responsible for it.
That gap between I am responsible and I understand this system is what makes a first rotation frightening. It does not close by studying harder the week before your shift. It closes by having a procedure you can run while confused — one that produces useful outcomes even when your diagnosis is wrong. We put the sequence below together from the parts of on-call practice that hold up across teams and tooling: acknowledge, bound the damage, timebox yourself, escalate on a clock rather than on a feeling.
The first ten minutes are triage, not diagnosis
Your job in the first ten minutes is not to explain the outage. It is to answer three questions: is anything actually broken for users, is it getting worse, and is somebody already on it.
Acknowledge the page first. In PagerDuty, Opsgenie, and Grafana OnCall alike, an unacknowledged alert escalates on a timer to the next person in the chain — usually your lead, then whoever is above them. Acking is not a promise that you can fix it. It says a human has eyes on this, and it stops the escalation clock while you read.
Then check whether an incident is already open. Most teams route alerts into a channel where the same alert has fired before. If two engineers are already in a thread on it, join and post what you are seeing rather than starting a parallel investigation that nobody knows about.
Now bound the blast radius. One customer or all of them? One region or every region? Is the graph a step change or a slow ramp? A step change usually points at something discrete — a deploy, a flag flip, a dependency that fell over at a specific second. A slow ramp points at saturation: a queue filling, a disk filling, connections leaking.
Last, look at what shipped. git log --since="2 hours ago" on the relevant repo, plus your deploy tool's recent history, explains a large share of pages before you understand anything at all about the code. If a deploy landed twenty minutes before the alert fired, that is your first suspect, and rolling it back is usually cheaper than understanding it.
Sort every action into reversible and irreversible before you take it. Restarting a pod, rolling back a deploy, scaling a replica count up, failing over to a standby — reversible, and fine to do at 3am on a hunch. Dropping a table, purging a queue, deleting a volume, running a manual data fix, force-pushing a branch — irreversible, and never a solo decision on your first rotation. If you cannot name the undo, wake somebody up instead of running the command.
Four questions that narrow most pages
Is the system broken, or is the alert broken? Load the user path yourself. Open the checkout page. Curl the health endpoint. If the product works and the dashboard is red, you may be looking at a broken exporter, an expired certificate on a probe, or a threshold nobody retuned after traffic patterns changed. That is still a real problem, but it is a business-hours problem.
What changed? Deploys, feature flags, config pushes, infrastructure changes, certificate expiries, and other teams' incidents upstream of you. Flags are the ones new engineers forget: a flag flipped from an admin UI leaves no commit and no deploy record, but it changes runtime behavior exactly like code does.
Is it one thing or everything? A single service degraded while its dependencies look healthy means you should look inside that service. Several unrelated services degraded at once means you should look below them — the shared database, the cluster, DNS, the cloud provider's status page. A lot of first-rotation hours get spent reading one service's logs when the answer was one layer down.
Is there a runbook, and does it still describe reality? Search the alert name verbatim, both in your docs and in chat history. Chat history is often better than the docs, because the previous three times this alert fired, somebody typed the actual fix into a thread and never wrote it up. Search the exact alert string, not your paraphrase of it.
Escalate on a clock, not on a feeling
The most common failure mode of a first rotation is not breaking production. It is a new engineer quietly struggling for ninety minutes because escalating felt like admitting they did not belong there.
Set the timebox before your shift starts, so you are not making the judgment call at the moment you are least equipped to make it. Fifteen minutes for something actively breaking a user-facing path, thirty for a degradation customers have not hit yet. When the timer runs out and you do not have a working theory, escalate.
Skip the apology opener — it invites the other person to reassure you instead of helping you. Send four lines:
- What fired: the alert name, the time, and what it measures.
- What users see: your own check of the product, not the dashboard's opinion of it.
- What you ruled out: deploy log clean back to 22:00, dependencies green, no flag changes in the audit log.
- What you want: can you confirm whether this warrants rolling back release 4.２.1 beats help.
The person you page will care more about the third line than about whether you solved it. Ruling things out is real progress, and it is the part a half-asleep senior engineer would otherwise have to redo from scratch.
Write the timeline before you go back to sleep
The incident review happens two days later, and by then nobody remembers whether the restart came before or after the error rate dropped. Ten minutes of notes now saves an hour of reconstruction later, and it is the highest-leverage thing a junior on-call engineer does all shift.
Capture five things: timestamps for when the alert fired and when you acked, what you observed, what you tried and what effect each attempt had, who you escalated to and when, and the state of the system when you handed off. Record the attempts that changed nothing — they are the most valuable entries and the least often written down, because they tell the next person which paths are dead ends.
Then make exactly one improvement to the runbook while it is fresh: the query you wish had been there, the dashboard link you had to hunt for, the note that this alert has fired three times and twice it was the same upstream dependency. Rotations compound if each person leaves the docs slightly better than they found them.
Your first rotation is not a test of whether you can debug an unfamiliar system alone at 3am. It is a test of whether you can follow a procedure while scared, keep the damage bounded, and hand a clear picture to the next person. Those are all learnable in one shift.
Originally published at pickuma.com. Subscribe to the RSS or follow @pickuma.bsky.social for new reviews.
Top comments (0)