Mattermost Receiver For Alertmanager in Prometheus-Stack in K8S

Overview

Since Prometheus-Stack Alertmanager doesn’t include Mattermost as a built-in receiver, we will integrate it using a webhook via creating a Python relay application that accepts Alertmanager webhook notifications, formats them into a Mattermost-compatible payload, and sends them to a Mattermost incoming webhook URL.

Then we will deploy this relay inside Kubernetes using a Deployment and expose it with a Service so Alertmanager can reliably reach it.

Finally, we will configure Alertmanager using an AlertmanagerConfig resource to define a new receiver and route, so alerts matching our rules (severity/namespace/etc.) get delivered to Mattermost through the relay.

How

• Prometheus Alert Rules trigger alerts.
• Alertmanager processes and routes the alerts.
• Alertmanager sends webhook notifications to the Python Relay Service.
• The relay formats the alert payload.
• The relay forwards the formatted message to Mattermost Incoming Webhook.

Steps

• Create an Incoming Webhook in Mattermost. This webhook URL will be used by the Python relay to deliver alert messages to the desired channel.

  Ensure you securely store the generated webhook URL, as it will be injected into the relay application via environment variables or Kubernetes secrets.

• Python Relay Application

  A lightweight webhook server that receives Alertmanager alerts and forwards them to Mattermost.

• Build the Docker Image Dockerfile

  Once the relay application is complete, build the image and push it to a container registry accessible by your Kubernetes cluster.

• Deploy in Kubernetes: K8S Manifests

• Configure Alertmanager with AlertmanagerConfig

  Ensure having a label in AlertmanagerConfig to select via prometheus operator

• Add the Alertmanagerconfig to prometheus values.yaml
  

  ⋮
  alertmanager:
    alertmanagerSpec:
      alertmanagerConfigSelector:
        matchLabels:
          alertmanagerConfig: alertmanager-mattermost
      alertmanagerConfigMatcherStrategy:
        type: None
  ⋮
  

  And then do helm upgrade to prometheus operator selects the alertmanagerConfig and route alerts.
  

  helm upgrade --install prometheus prometheus-community/kube-prometheus-stack -f values.yaml