You can use the Alertmanager to silence and group alerts as well.
You can get all source code from GitHub. Check out the tag
part-2-grafana if you want to follow along.
First of all, add the Alertmanager and a volume to
alertmanager: image: prom/alertmanager:v0.23.0 restart: unless-stopped ports: - "9093:9093" volumes: - "./alertmanager:/config" - alertmanager-data:/data command: --config.file=/config/alertmanager.yml --log.level=debug volumes: alertmanager-data:
Alertmanager will persist silence configurations to the volume.
This configuration contains information about which channels to send to. For simplicity, we use E-Mail. Refer to the Alertmanager docs to learn about other channels.
Create a folder
alertmanager and add a file
alertmanager.yml to it:
route: receiver: 'mail' repeat_interval: 4h group_by: [ alertname ] receivers: - name: 'mail' email_configs: - smarthost: 'smtp.gmail.com:465' auth_username: 'firstname.lastname@example.org' auth_password: "" from: 'email@example.com' to: 'firstname.lastname@example.org'
route section configures which alerts will be sent. In our case, we sent all alerts. You could add more routes and filter, for example, on alert tags (see the example in the docs.).
receivers configures our target channels. Note how
route refers to the receiver
Finally, we need to tell Prometheus about the Alertmanager.
prometheus/prometheus.yml and add the following:
alerting: alertmanagers: - scheme: http static_configs: - targets: [ 'alertmanager:9093' ]
docker-compose up. Open http://localhost:9093 in your browser to see the Alertmanager UI.
After a couple of minutes, the test alert fires. You can check this in your Prometheus instance.
Now, you can see the alert in the http://localhost:9093 as well:
This series shows how to set up a basic Prometheus stack. Keep in mind that this is not production-ready! But it is enough to get your hands dirty.