Alertmanager to Telegram without a bridge: severity routing, inhibit rules and runbook links that cut alert noise

Written with AI assistance; the alert pack itself was built and tested by the author's AI assistant.

If Prometheus pings your phone fifty times when one server reboots, you will soon mute the chat, and then the one real alert gets missed too. Most of that noise can be removed in Alertmanager itself, with no extra containers.

This article walks through a small config: native Telegram delivery, routing by severity, inhibit rules, and a runbook link in every message. Everything here works with plain Alertmanager and nothing has to be bought.

No bridge needed

Older guides run a webhook bridge that converts Alertmanager JSON into Telegram messages. Recent Alertmanager versions have a built-in telegram_configs receiver, so that is one less container to keep alive. Check your version's documentation to confirm it supports telegram_configs and bot_token_file.

The bot token: a file, not inline

Create a bot with @botfather and keep the token out of alertmanager.yml, which tends to end up in git, backups and screenshots. Put it in a file and use bot_token_file:

sudo install -d -m 0750 /etc/alertmanager/secrets
printf '%s' '<TOKEN_FROM_BOTFATHER>' | sudo tee /etc/alertmanager/secrets/telegram_bot_token >/dev/null
sudo chown alertmanager: /etc/alertmanager/secrets/telegram_bot_token
sudo chmod 0600 /etc/alertmanager/secrets/telegram_bot_token

Adjust the user name to whatever account runs Alertmanager. I use printf so the file has no trailing newline.

Finding the chat_id

There is no menu that shows it. The generic method:

1. Add the bot to your group (or open a private chat with it).
2. Send any message in that chat. In a group the bot may not see ordinary messages because of privacy mode, so send a /command or mention the bot.
3. Open https://api.telegram.org/bot<TOKEN>/getUpdates in a browser or with curl and look for "chat":{"id":...}.

Private chats have positive ids. Groups are negative. Supergroups, which is what most groups become, start with -100. This article uses the placeholder -1001234567890; use the number you actually see, including the minus sign. If getUpdates returns an empty list, send another message and retry.

Routing by severity

Not every alert deserves the same urgency. The routing tree sets different timings for critical and warning:

route:
  receiver: telegram-warning
  group_by: [alertname, instance, namespace]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 12h
  routes:
    - matchers: ['severity="critical"']
      receiver: telegram-critical
      group_wait: 10s
      repeat_interval: 1h
    - matchers: ['severity="warning"']
      receiver: telegram-warning
      repeat_interval: 12h
receivers:
  - name: telegram-critical
  - name: telegram-warning

(Receivers are stubs so the snippet stands alone; real ones are below.)

What each setting does:

• group_by [alertname, instance, namespace]: alerts with the same name, host and namespace are merged into one message. Without grouping, ten failing pods become ten messages.
• group_wait: how long Alertmanager waits after the first alert of a new group before sending, so related alerts can join it. 30s for warnings gives a burst time to gather. Critical uses 10s because a few seconds of batching is worth less than a fast page.
• group_interval: how long to wait before sending notifications about new alerts added to a group that was already notified. 5m keeps a flapping group from producing a message every minute.
• repeat_interval: how often an unresolved alert is re-sent. Critical repeats hourly, so it can't be forgotten. A warning repeats every 12 hours, which is a reminder and not a nag.

Treat these numbers as a starting point and tune them to how quickly you can actually respond.

The receiver: HTML, runbook link, escaping

receivers:
  - name: telegram-critical
    telegram_configs:
      - bot_token_file: /etc/alertmanager/secrets/telegram_bot_token
        chat_id: -1001234567890
        parse_mode: HTML
        send_resolved: true
        message: |-
          🔥 <b>[{{ .Status | toUpper }}] {{ .CommonLabels.alertname }}</b>
          {{ range .Alerts -}}
          {{ .Annotations.summary }}
          {{ .Annotations.description }}
          <a href="{{ .Annotations.runbook_url }}">Runbook</a>
          {{ end }}

The warning receiver is identical apart from the ⚠️ emoji; it could point at a different chat.

parse_mode: HTML gives bold text and clickable links. The pitfalls:

• Telegram accepts only a small subset of HTML tags. Anything else, or an unbalanced tag, makes Telegram reject the message with a "can't parse entities" error, and you get no alert at all.
• A literal <, > or & in text must be written as <, >, &. A description like latency > 500ms or a label with & in a URL is enough to break the message. As far as I know Alertmanager's notification templates are plain text templates and do not auto-escape, so this is your job.
• Escape the free-text fields you don't control. Template functions such as reReplaceAll can do it, and the order matters because & must be replaced first:

message: |-
  {{ .Annotations.description | reReplaceAll "&" "&amp;" | reReplaceAll "<" "&lt;" | reReplaceAll ">" "&gt;" }}

• If an alert has no runbook_url, <a href=""> is an empty link and may be rejected. Wrap it in {{ if .Annotations.runbook_url }}…{{ end }} if some of your rules lack the annotation.
• Telegram limits a message to 4096 characters, so keep annotations short and groups small.

Always verify with a real test alert (below) after changing a template.

Runbook links in the alert itself

The link has to come from the rule, so that the person reading the message at 3 a.m. gets a single tap to "what do I do". In the rules the annotation looks like this:

- alert: DockerContainerRestartLoop
  expr: changes(container_start_time_seconds{name!=""}[15m]) >= 3
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: "Container {{ $labels.name }} keeps restarting ({{ $labels.instance }})"
    runbook_url: "https://devkit-shop.surge.sh/runbook/en/#dockercontainerrestartloop"

The convention is simple: the anchor is the alert name in lower case, and the runbook has a section with that name. Each section follows the same order: what it means, common causes, diagnostics, how to fix, and when it is OK to silence. For example, the section for NodeDown begins by saying that it is not necessarily a server outage, because only the exporter or the network path may be down, and then lists ping, curl against port 9100 and systemctl status node_exporter.

You can write the same for your own alerts; even three lines per alert beat none.

Inhibit rules: where the noise actually goes

Inhibition mutes a target alert while a source alert is firing, as long as the labels listed in equal match. Here are the first two rules:

inhibit_rules:
  # Same alert on the same host as critical and warning: keep critical.
  - source_matchers: ['severity="critical"']
    target_matchers: ['severity="warning"']
    equal: [alertname, instance]

  # Host is unreachable: other node-exporter alerts from it are meaningless.
  - source_matchers: ['alertname="NodeDown"']
    target_matchers: ['alertname=~"Node.+"', 'alertname!="NodeDown"']
    equal: [instance]

1. Critical beats warning. If a rule exists at two thresholds under the same alertname, you see only the critical one. This only matters if you use the same alert name for both levels; otherwise it does nothing, and the specific rules below cover the pairs.
2. A dead host silences its own symptoms. When NodeDown fires, disk, CPU and memory alerts for that instance would be stale or misleading. Note that the target pattern is Node.+, so alerts with other prefixes (for example Docker…) are not covered by this rule.

The remaining three follow the same shape:

inhibit_rules:
  # Whole site is down: skip slow-response and bad-status alerts.
  - source_matchers: ['alertname="BlackboxProbeFailed"']
    target_matchers: ['alertname=~"BlackboxSlowResponse|BlackboxHttpStatusNotOk"']
    equal: [instance]

  # PVC critically full: the "almost full" warning is redundant.
  - source_matchers: ['alertname="KubePvcCritical"']
    target_matchers: ['alertname="KubePvcAlmostFull"']
    equal: [namespace, persistentvolumeclaim]

  # API certificate expires within a day: drop the one-week warning.
  - source_matchers: ['alertname="KubeApiCertExpiresCritical"']
    target_matchers: ['alertname="KubeApiCertExpiresSoon"']
    equal: [instance]

Each time, a more severe or more fundamental alert makes a weaker one about the same thing redundant. These alert names come from the pack's rules; replace them with yours, and make sure the equal labels exist on both alerts, otherwise the rule never matches.

Testing before you trust it

Validate syntax first:

amtool check-config alertmanager.yml

Then send a fake alert straight to the Alertmanager API (default port 9093):

curl -sS -X POST http://localhost:9093/api/v2/alerts \
  -H 'Content-Type: application/json' \
  -d '[{"labels":{"alertname":"TestAlert","severity":"critical","instance":"test:9100"},
        "annotations":{"summary":"Test","description":"a < b & c","runbook_url":"https://example.com/"}}]'

The < and & in the description are deliberate: they show whether your escaping works. Expect the message after group_wait (10s for critical). Repeat with "severity":"warning" to check the other route.

Silences: with a reason and an expiry

For planned maintenance, silence instead of ignoring:

amtool silence add alertname=NodeDown instance=web1:9100 \
  --comment="kernel update, planned reboot" --duration=2h

A silence without a comment tells the next person nothing, and one without an expiry is just switched-off monitoring.

Limits

• telegram_configs and bot_token_file need a reasonably recent Alertmanager. Check your version's documentation instead of trusting this article.
• Telegram can be unreachable from your network, down, or rate-limiting you. If your alerts are bursty, rate limiting shows up as delayed or dropped messages. Grouping reduces volume but doesn't remove the risk.
• Don't make Telegram the only channel for critical alerts. Add a second receiver for them, such as email or a pager service, and ideally something that doesn't share a failure with the rest of your setup. If the server running Alertmanager is the one that died, nothing gets sent at all, so an external heartbeat check is worth having.
• Inhibition hides alerts. A badly chosen equal can hide something real, so re-read every rule once.

Where to get the rest

The full pack has rules for Linux, Docker, Kubernetes and websites/TLS, this Alertmanager config, a runbook in English and Russian, and promtool tests. It is sold for 9 USDT via the Telegram shop. The runbook itself is free at https://devkit-shop.surge.sh/runbook/en/ , and there is a free monitoring kit for a single VPS.