What my nightmares were like trying to use multiple configuration files with Alertmanager 💀👀
I recently started working on monitoring some of the internal tools and services at We Provide. I decided to tinker with the "Prometheus, Grafana and Alertmanager" stack as a first attempt.
Obviously, there we some hiccups... (literally and figuratively) 💩
Originally I started thinking about a tool I could write, something like
am-stitcher (ssooo original, not like
amtool, at all...), but... I realized that I was over-engineering something that was basically a matter of merging multiple YAML files into one.
Google, here I come, gimme something useful or copy-pastable please! ❤️
I didn't really find anything sufficient for this use-case in terms of literally merging.
Some of the tools I came across did what I wanted them to do, but didn't give me control over in what order to merge them. Some did, but they required me to specify every file individually instead of being able to recursively search a directory.
PyYAML with pyyaml-include 💌 💕
I found an article about
!include in YAML, and as naive as I am, I thought it was something that might work in YAML natively... But, of course, it didn't!
!include is added by the
pyyaml-include extension, which adds this "directive" to PyYAML.
So... I tried installing PyYAML and the
pyyaml-include extension using PIP, hoping that the parsing with PyYAML would work with only the CLI, and it'd automatically use the
pyyaml-include extension. But I couldn't find a way to use PyYAML only in the CLI...
I should really stop being so naive... Don't you think so too? 😬😅
What I ended up writing was a small Python script that imports PyYAML and uses
pyyaml-include to parse
Don't be fooled, this was a lot of copy-paste in combination with trial and error! 🔥
import sys import os import yaml from yamlinclude import YamlIncludeConstructor YamlIncludeConstructor.add_to_loader_class(loader_class=yaml.FullLoader, base_dir=os.getcwd()) data = '' for line in sys.stdin: data += line data = yaml.load(data, Loader=yaml.FullLoader) yaml.dump(data, sys.stdout, default_flow_style=False)
I decided to use the standard input (
sys.stdin) and output (
sys.stdout) streams to read and write. I didn't want to bother with using arguments when I didn't even know the Python syntax 😅
python parse-yaml.py < alertmanager.yml > merged.yml
Shall we add a tiny bit of automation?! 🙌
At We Provide we mostly use Bitbucket, for open-source we use GitHub (obviously...), but for most of the stuff we use Bitbucket.
It consists out of three steps, "Parse & Merge", "Validate" and "Deploy". I think those are kinda self-explanatory, don't you think? 😊
pipelines: default: - step: name: Parse & Merge image: python:3.5.1 script: - pip install pyyaml pyyaml-include - python parse-yaml.py < alertmanager.yml > merged.yml artifacts: - merged.yml - step: name: Validate image: serializator/amtool script: - amtool check-config merged.yml - step: name: Deploy image: eeacms/rsync script: - rsync merged.yml $USER@$HOST:$ALERTMANAGER_YAML - curl -L -X POST $HOST$RELOAD_PATH
As you can see, one of the images is
serializator/amtool. Originally I tried using the
amtool-docker image from Cisco Metacloud (Anthony Rogliano). But the fact that it was built on top of
golang:1.9-alpine3.6 caused issues with the usage of
/bin/bash instead of
/bin/sh in the Bitbucket pipeline.
^ If you know a solution for this, or if I overlooked something in the Bitbucket pipeline that I could've configured, please let me know! ❤️
What I ended up doing was forking the repository and build the image on top of
golang:1.13-stretch instead of
^ I thought about the increase in size by not using the alpine tag, but it's negligible in my opinion, looking at the use-case.
I'm in love with the "/-/reload" endpoint 🙈🙊
I thought that I'd had to use SSH and use Docker Compose to restart the Alertmanager service, but it luckily didn't end up being that ugly.
Somewhere in 2016 the
/-/reload endpoint was introduced by ZhenyangZhao, and I'm glad he made that pull request! I don't know how I'd live with myself knowing that one of my deployments didn't have cURL in it...
So, without much hesitation, I quickly wrote one line more to send a POST request to the
/-/reload endpoint, configured some stuff using environment variables, and pushed it to master!
I didn't know that this kinda happiness existed! 😊🔥
I finally laid down on bed, closed my eyes and began dreaming about the many files, folders and
!include's I was going to write with a happy smile on my face! 😆
I hope you got something useful out of this, and if not, I hope you at least enjoyed reading it!