React faster: Forward Prometheus Alerts to Teams

Many companies use Microsoft Teams. Receiving alerts in Teams allows you to react faster. It makes working on an incident visible, as you can answer the message. Let's add it to our monitoring stack!

Head over to GitHub to get the full code.

Teams configuration

Alertmanager does not support Teams out of the box. You must use webhooks to achieve this. The webhook is used by prom2teams to write messages.

Add a channel to the team which should receive the notifications. Click on the three dots beside the team name and choose Add channel.

To add the webhook, click on the dots beside the channel and click on Connectors.

Configure an Incoming Webhook. Teams shows the webhook URL once you click Create. Copy this URL in the next step.

Add prom2teams to the stack

prom2teams will use the webhook to send messages to Teams.

Add prom2teams to your docker-compose.yml:

  prom2teams:
    image: idealista/prom2teams:2.7.0
    restart: unless-stopped
    environment:
      PROM2TEAMS_CONNECTOR: "url from the webhook"
    ports:
      - 8089:8089

Update your alerts

The Alertmanager uses labels to decide which alert goes to which notification channel. Change the prometheus/alerts.yml to contain the following:

groups:
  - name: DemoAlerts
    rules:
      - alert: InstanceDown 
        expr: up{job="services"} < 1 
        for: 1m
        labels: # labels and annotations are new
          severity: low
        annotations:
          summary: 'Alert with low severity.'

      - alert: InstanceDownCritical
        expr: up{job="services"} < 1 
        for: 1m 
        labels:
          severity: high
        annotations:
          summary: 'Alert with high severity.'

Now you have two alerts with different labels.

Configure the Alertmanager

Change the alertmanager/alertmanager.yml so that the Alertmanager is aware of prom2teams.

route:
  group_by: [ alertname ]
  receiver: 'mail' # default receiver
  repeat_interval: 24h
  routes:
    - receiver: 'teams'
      repeat_interval: 12h
      matchers:
        - severity="medium"

    - receiver: 'teams'
      repeat_interval: 4h
      matchers:
        - severity="high"


receivers:
  - name: 'mail'
    email_configs:
      - smarthost: 'yourmailhost.com:465'
        auth_username: '[e-mail]'
        auth_password: "your mail password"
        from: '[e-mail]'
        to: '[e-mail]'
        require_tls: false

  - name: 'teams'
    webhook_configs:
      - url: "http://prom2teams:8089"
        send_resolved: true

Now, the Alertmanager can publish to two channels.
matchers tell the Alertmanager which channel to use.

The Alertmanager sends alerts with medium severity to Teams. This is repeated every four hours - until you fix it!

The Alertmanager uses email if no route matcher matches.

You can use the routing tree editor to check if the configuration is what you need. Add labels and values of your alert to see which route they will take.

Check the result

Start the stack with docker-compose up and wait until the alerts fire.

You will get a notification from Teams for the InstanceDownCritical alert:

And an email for the InstanceDown alert:

Which notification channels do you want to see next? Leave me a message!
Head over to GitHub to get the full code.

If this article was helpful for your, please consider to buy me a coffee :-)