Microservices architecture is going to be one of the essential features in software development in the coming years. Packing a large monolithic application into small containers poses various advantages. One of the key advantages I can think of is that if certain things fail, then a part of the application will be down and it can auto-heal rather than crashing the whole application. This is one of the reasons that when Instagram, Facebook, and Whatsapp crashes, only certain functionalities stop working. The application, as a whole, is still working. This serves a great advantage. Also, when the demand for a particular service or component increases, the number of components can be increased or decreased easily. The small containers which I am referring here are the Docker containers and the application is deployed on Kubernetes. Kubernetes is a container orchestration software which manages the containers at various levels and enables the management of connections and endpoints of these containers.
Kubernetes or K8s is an open-source self-healing application which manages the deployment, scaling, and operation of these containers. It was originally developed by Google but later on, donated to CNCF. Since Kubernetes provides tons of services, there needs to be an easier way to monitor the activities in the Kubernetes cluster. This is possible by Prometheus. It is an open-source solution to monitor the metrics and manage the alerts in the system. It was developed by SoundCloud, but later joined CNCF as the second hosted project after Kubernetes. Prometheus provided a rich set of monitoring metrics and alert management system which helps the developers to monitor and get notified about any unusual activity or consumption.
Prometheus Operator
CoreOs launched Prometheus operator to ease the process of integrating K8s with Prometheus. It preserves the configuration of both the K8s and Prometheus while installing and configuring the cluster. It provides easy monitoring for K8s services and deployments, along with managing Prometheus, Grafana and Alertmanager configuration.
When a new version of the application is deployed, K8s manages the creation of new pod and deletes the older one. Prometheus, on the other hand, constantly watches the K8s API and creates a new Prometheus configuration whenever it detects a change, based on the services/pods changes. It uses a ServiceMonitor, a CRD(Custom Resource Definition), to abstract the configuration to target.
Installation
Prerequisites
- Kubernetes
- Helm (Package installer for K8s)
Steps
Install Prometheus operator in a different namespace. It is preferable to keep your monitoring containers in a separate namespace.
$ helm install stable/prometheus-operator --name prometheus-operator --namespace monitor
If everything got installed perfectly, you can see these pods available:
$ kubectl get pods -n monitor
NAME READY STATUS RESTARTS AGE
alertmanager-prometheus-operator-alertmanager-0 2/2 Running 0 13d
prometheus-operator-grafana-749b598b6c-t4r48 2/2 Running 0 13d
prometheus-operator-kube-state-metrics-d7b8b7666-zfqg5 1/1 Running 0 13d
prometheus-operator-operator-667dd7cbb7-hjbl6 1/1 Running 0 13d
prometheus-operator-prometheus-node-exporter-mgsqb 1/1 Running 0 13d
prometheus-prometheus-operator-prometheus-0 3/3 Running 1 13d
To run the dashboard, enter the following command and go to http://localhost:9000.
$ kubectl port-forward -n monitor prometheus-prometheus-operator-prometheus-0 9090
Prometheus Dashboard
You can enter your query to get the results about any particular instance or even a graph of it as shown in the figure above.
To see the visual representation at each level, we use Grafana. It provides some great visual insights regarding the usage, health and other metrics. We can also add more custom metrics. We will get real-time analysis of the data.
$ kubectl port-forward $(kubectl get pods --selector=app=grafana -n monitor --output=jsonpath="{.items..metadata.name}") -n monitor 3000
Go to http://localhost:3000 and enter ‘admin’ as username and ‘prom-operator’ as password. These are the available options:
You can get visual graphs by selecting any one of the options. Node-level metrics are shown here:
We can configure alerts in many ways. We can access the dashboard to configure the AlertManager by going to http://localhost:9093 after executing this command:
$ kubectl port-forward -n monitor alertmanager-prometheus-operator-alertmanager-0 9093
It’ll look like this. Here you can add more alerts and can see the Slack API URL under the status tab. We can set the notifications on Slack, HipChat or even email. Some of the templates are available here.
Conclusion
The use of microservices is going to increase soon and monitoring the metrics and alert notifications are going to be an essential part of it. Prometheus provides optimal monitoring services with easy installation services.
Top comments (3)
Great to see a DevOps tooling post. If you like helm check out helmfile it lets you have one yaml file that describes many helm chart applications in one declarative file.
I wrote a generic helm chart that can deploys any web application in any language. We configure the one generic chart to install many different webapps via separate helmfiles in a GitHub repo. There is a webhook on the GitHub repo that will fire when any yaml changes. It runs helmfile to sync the updated app into kubenetes. So all devs need to do is push some YAML into GitHub that describes the docker image and env vars of their new app and it appears up on kubernetes on AWS. I open sourced all of the charts and scripts as ocd-scm/ocd-meta on GitHub. It’s essential built out of helmfile and some other good tools as simple glue scripts with some strong conventions.
Great write-up, but a lot of people ask the question.. how do we change the adminPassword from the default prom-operator?
I believe this will work:
helm install stable/prometheus-operator --name prometheus-operator --namespace monitor --set adminPassword=yourpassword
on first install. If you're deploying using a values.yaml file using the
argument, you can upgrade your helm chart install to receive the new password:
helm upgrade prometheus-operator stable/prometheus-operator -f values.yaml
You can modify the password by going to
prometheus-operator-grafanasecrets in themonitornamespace and editing the admin-password and admin-user. Alternatively, you can also edit it by executing this command:This will open up the secrets file and you can enter the base64 encoded secret values in the data field.