Escalation Policies as Code: On-Call Rotations in Git

#oncall #escalationpolicies #gitops #devops

Ask most teams who's allowed to change an escalation policy, and the honest answer is "whoever has admin access and remembers to." The policy that decides who gets woken up at 3 a.m., and how fast, typically lives entirely inside a vendor's web UI: click into a policy, adjust a timeout, save. No review, no diff, no record of why the timeout went from five minutes to ten, and no way to see what the policy looked like six months ago short of asking whoever made the change, if they still remember.

This is the same problem infrastructure had before Terraform: configuration that only exists as the current state of a system, changeable by anyone with access, with history that lives in people's memory instead of a repository. The fix there was treating infrastructure as code. The same fix applies to on-call config, and for the same reason: escalation policies and schedules are operational logic, not administrative settings, and operational logic that can silently change without review is a real risk during an actual incident, and a quiet contributor to alert fatigue months later when nobody remembers why a timeout is set the way it is.

The pattern

The shape is the same as any other infrastructure-as-code setup: a directory of declarative files, a tool that reconciles them against the live system, and a CI pipeline that runs that tool on merge.

oncall/
  schedules/
    payments-primary.yaml
    payments-secondary.yaml
    infra-oncall.yaml
  escalation-policies/
    payments-outage.yaml
    infra-warning.yaml
  integrations/
    prometheus-payments.yaml
    cloudwatch-infra.yaml

A schedule file might look like:

# schedules/payments-primary.yaml
name: payments-primary
timezone: America/New_York
rotation:
  type: weekly
  handoff: monday 09:00
participants:
  - alice@company.com
  - bob@company.com
  - carla@company.com
overrides:
  - date: 2026-12-24
    user: dave@company.com
    reason: holiday coverage swap

And an escalation policy:

# escalation-policies/payments-outage.yaml
name: payments-outage
severity: critical
steps:
  - after: 0m
    notify: schedule:payments-primary
  - after: 5m
    notify: schedule:payments-secondary
  - after: 15m
    notify: user:eng-manager@company.com
    channel: phone

Nothing here is exotic. It's the same declarative-config idea applied to a domain that mostly hasn't gotten it yet.

What you actually gain

Review before it's live: A change to a critical escalation policy goes through a pull request like any other production change. A reviewer can catch "this removes the secondary on-call from the payments policy" before it ships, not after an incident reveals it.

A real audit trail: git blame on an escalation policy answers "who changed this and why" in one command, with the linked PR and its discussion attached.

Rollback: Revert the commit, re-apply, done.

Disaster recovery for the on-call config itself: If the on-call platform loses data or an account gets locked out, the entire schedule and escalation setup can be rebuilt from a git checkout.

What's still missing

The gap in this pattern today isn't tooling to declare the config, it's tooling to validate it before it's live. Terraform can plan a diff and show you what will change, but it can't tell you that a proposed escalation policy would have caused an on-call engineer to get skipped during last month's actual incident. Policy simulation, replaying a set of historical alerts against a proposed policy change, is close to nonexistent across the ecosystem right now.

The other open problem is drift. If someone edits a schedule directly in the vendor UI during an incident (which will happen, and should be allowed to happen), the git state and the live state disagree until someone reconciles them. Few setups have automated drift detection for on-call config the way they do for infrastructure.

Existing building blocks

Terraform has official or community providers for several on-call and incident platforms, letting you declare schedules and escalation policies as Terraform resources and manage them through a normal plan/apply workflow. Where a provider doesn't exist, the fallback is a sync script: read the YAML, diff it against the platform's REST or GraphQL API, apply the difference, run it in CI on merge to main.

If you're evaluating on-call tools with this pattern in mind, check whether the platform exposes full API coverage over schedules, escalation policies, and integrations, and whether you can inspect the underlying data model directly if you're self-hosting it. That's part of why we designed FluidifyAI Regen as open-source and API-first, with the source and API reference on GitHub.

If you haven't settled on a platform yet, our rundown of open-source on-call tools is worth a read before you pick one to build this pattern on top of.

FAQ

Do we need a custom sync script if a Terraform provider already exists? No, use the provider. Reach for a sync script only when no provider covers what you need.

What about changes made during an actual incident? Let them happen directly in the tool. Reconcile afterward: either update the git source to match, or revert the live change back once the incident is over.

Is this worth it for a five-person team? The audit trail and rollback value scale with team size, but even a small team benefits from "why does this escalation policy skip Tuesdays" having an answer in git history.

Where this is heading

Policy simulation is the missing piece that would make this pattern complete: replay real alert history against a proposed change and see the difference before merging, the way terraform plan shows an infrastructure diff before apply. Nobody has shipped this well yet across the ecosystem. It's the natural next step for whichever platform builds it first.

DEV Community