Most teams do not wake up one morning excited to build an on-call system.
It usually starts with a smaller problem. Alerts arrive in a chat channel. Someone is probably responsible. The runbook is probably somewhere. Then the team grows, the alerts multiply, vacations happen, and "just ping whoever is online" quietly becomes an operational process.
IncidentRelay is an MIT-licensed, self-hosted project for that process: on-call scheduling, alert routing, escalation and incident response.
The project is actively developed and still beta. It is not trying to claim that self-hosting magically removes cost. It replaces a vendor subscription with a different kind of ownership: infrastructure, backups, upgrades and the slightly recursive task of monitoring the system that tells you monitoring is broken.
The core workflow
IncidentRelay models alert delivery as an explicit chain:
Monitoring source
-> Route token and matchers
-> Service and team
-> Rotation or escalation policy
-> Current on-call user
-> Notification channels
This matters because alert ownership is often hidden inside a collection of webhook URLs, chat channels and monitoring configuration.
In IncidentRelay, an intake token belongs to a route. The route decides:
- which source is allowed to send the alert;
- which team owns it;
- how labels and payload fields are matched;
- which service is affected;
- which schedule or escalation policy should be used;
- which notification channels receive updates.
The goal is not to make routing clever. The goal is to make it explainable.
Scheduling is more than a list of names
A rotation can contain multiple schedule layers. A team might use:
- a weekday business-hours layer;
- a night layer;
- a weekend layer;
- separate primary and backup rotations;
- different timezones for regional coverage.
Each layer has its own members, handoff rules, timezone, restrictions and priority. IncidentRelay calculates one final schedule:
temporary override
> highest-priority active layer
> no assignment
Temporary overrides cover vacations, shift swaps and one-off absences without rewriting the normal schedule.
Rotation membership is stored as periods with start and end timestamps. If an engineer leaves a rotation and returns later, past shifts remain intact while future shifts use the new membership period. Historical calendars should not change because someone edited tomorrow's schedule.
Alerts are connected to services
An alert is not only a message. It is usually evidence that a service or dependency is unhealthy.
IncidentRelay services can contain:
- ownership and team information;
- links to dashboards, logs, traces and repositories;
- runbooks;
- dependencies;
- default rotation or escalation policy;
- default stakeholders;
- service impact and analytics.
Routes can assign a default service or select one through label and payload match rules. This keeps the incident connected to the system that is actually affected.
Escalation without mystery
A route can point directly to a rotation or use an escalation policy.
An escalation policy is an ordered set of rules. A rule can target the current user from a rotation or a specific team member:
Primary rotation
-> wait 5 minutes
Backup rotation
-> wait 10 minutes
Team lead
Policies can repeat a configured number of times. Alert details retain the current policy state so responders can see who owns the incident and when the next escalation is expected.
Explaining why an alert went there
The latest IncidentRelay releases include Alert Explain Trace.
Each incoming alert can produce a trace ID and a sequence of processing steps. The trace records decisions such as:
- alert received and normalized;
- route matched or not matched;
- service and team selected;
- existing alert group reused or a new group created;
- maintenance or silence matched;
- notification scheduled or suppressed;
- processing completed or stopped.
This is useful for successful alerts, but it is even more useful for alerts that never created an incident. A routing failure can still return a trace ID, which an administrator can open from the Alerts page.
The question changes from:
Why did nothing happen?
to:
Which processing step stopped, and what data did it see?
That is a much better question during an incident.
Integrations
IncidentRelay currently receives alerts from:
- Prometheus Alertmanager;
- Grafana;
- Zabbix;
- Sentry;
- RMON;
- LibreNMS;
- generic webhooks.
Notifications can be delivered through:
- Mattermost;
- Telegram;
- Slack-compatible webhooks;
- Discord;
- Microsoft Teams;
- email;
- browser/PWA push;
- generic outbound webhooks;
- pluggable voice-call providers.
Responder workflows include acknowledge, resolve, silence, comments,
priorities, responder requests and stakeholder lifecycle notifications.
Trying IncidentRelay
The repository includes Docker Compose, Helm, RPM and manual systemd
installation paths.
For a local evaluation, start with one real alert flow:
- Create a team and rotation.
- Add a route and intake token.
- Connect Alertmanager or a generic webhook.
- Attach one notification channel.
- Send a firing event.
- Confirm assignment, notification and acknowledgement.
- Send the resolved event.
- Open Explain Trace and inspect the routing decisions.
Do this before importing every team and every alert rule. A small end-to-end test reveals much more than a long feature checklist.
IncidentRelay is open source, MIT licensed and actively developed.
Website: https://incidentrelay.io/
GitHub: https://github.com/roxy-wi/IncidentRelay
Documentation: https://incidentrelay.io/docs/
Changelog: https://incidentrelay.io/changelog
Feedback on routing, schedules, integrations and production-readiness
requirements is very welcome.
Top comments (0)