DEV Community

loading...

Prometheus: Alertmanager – send alerts to a “/dev/null”

Arseny Zinchenko
DevOps, cloud and infrastructure engineer. Love Linux, OpenSource, and AWS.
Originally published at rtfm.co.ua on ・2 min read

In addition to the Prometheus: Alertmanager’s alerts receivers and routing based on severity level and tags post post.

Have an Alertmanager config with routes.

The task is – send all alerts from a Dev-environment into a “/dev/null”.

To do this – create an empty receiver:

...
receivers:
  - name: 'blackhole'
  - name: 'default'
    slack_configs:
      - send_resolved: true
        title_link: 'http://dev.monitor.example.world/prometheus/alerts'
        title: '{{ if eq .Status "firing" }}:confused:{{ else }}:dancing_panda:{{ end }} [{{ .Status | toUpper }}] {{ .CommonAnnotations.summary }}'
        text: "{{ range .Alerts }}*Priority*: `{{ .Labels.severity | toUpper }}`\nMonitoring host: {{ .Labels.monitor }}\n{{ .Annotations.description }}\n{{ end }}"
...

And in routes add new rules – check the env tag used in our monitoring with the match_re and send all matches to the receiver: blackhole:

...
  receiver: 'default'
  routes:
    # capture All INFO
    - match:
        severity: info
      receiver: default
      routes:
      # forward Dev INFO to the 'blackhole'
      - match_re:
          env: .*(-dev).*
        receiver: blackhole
    # capture All WARN to the 'warning' with P3
    - match:
        severity: warning
      receiver: warning
      routes:
      # forward Dev WARNING to the 'blackhole'
      - match_re:
          env: .*(-dev).*
        receiver: blackhole
      # forward Stage WARN to the 'default'
      - match_re:
          env: .*(-stage).*
        receiver: default
    # capture All CRIT to the 'critical' with P1
    - match:
        severity: critical
      receiver: critical
      routes:
      # forward Dev CRIT to the 'blackhole'
      - match_re:
          env: .*(-dev).*
        receiver: blackhole
      # forward Stage CRIT to the 'default'
      - match_re:
          env: .*(-stage).*
        receiver: default
...

Done.

Similar posts

Discussion (0)

Forem Open with the Forem app