Whenever you’re implementing a monitoring and observability solution for your environment, a few questions occur like what you’re going to monitor and the specific production environments that need to be monitored.
Another large and important question is how you’re going to monitor the environment.
With Kubernetes and monitoring/observability platforms that go with it, there are a ton of options. It can definitely get a bit overwhelming.
With Kube-Prometheus, the feeling of being overwhelmed should hopefully subside.
In this blog post, you’ll learn all about what Kube-Prometheus is and how to get it up and running.
What Is Kube-Prometheus?
It’s the “easy button”.
The Prometheus Operator itself gives you the ability to install all Prometheus and Grafana features in a declarative way. For example, below is a Kubernetes Manifest to install Alertmanager, which can be installed in a declarative fashion because the Prometheus Operator exists.
apiVersion: monitoring.coreos.com/v1
kind: Alertmanager
metadata:
labels:
app.kubernetes.io/component: alert-router
app.kubernetes.io/instance: main
app.kubernetes.io/name: alertmanager
app.kubernetes.io/part-of: kube-prometheus
app.kubernetes.io/version: 0.24.0
name: main
namespace: monitoring
spec:
image: quay.io/prometheus/alertmanager:v0.24.0
nodeSelector:
kubernetes.io/os: linux
podMetadata:
labels:
app.kubernetes.io/component: alert-router
app.kubernetes.io/instance: main
app.kubernetes.io/name: alertmanager
app.kubernetes.io/part-of: kube-prometheus
app.kubernetes.io/version: 0.24.0
replicas: 3
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 4m
memory: 100Mi
securityContext:
fsGroup: 2000
runAsNonRoot: true
runAsUser: 1000
serviceAccountName:
Kube-Prometheus on the other hand installs Alertmanager for you.
Essentially, Kube-Prometheus contains a bunch of Kubernetes Manifests for installing components like Grafana, Alertmanager, Node Exporter, Kube State Metrics, and several other pieces including all of the RBAC permissions and Service Accounts needed to get it up and running.
Instead of installing all of the components manually, you can just use Kube-Prometheus.
Here’s a list of what’s installed: https://github.com/prometheus-operator/kube-prometheus/tree/main/manifests
Methods Of Installation
Now that you know what Kube-Prometheus is, which is essentially a collection of Kubernetes Manifests to get up and running fast with the Prometheus/Grafana stack, let’s learn a few different installation options.
The first is Helm and the second is a traditional Kubernetes Manifest deployment.
Please note that you don’t have to run both of these. You can run one or the other, or run one, delete one, and then run the next. That way there are no conflicts and you see both processes.
Helm
First, add the prometheus-community repo.
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
Next, ensure that it’s up to date.
helm repo update
When installing Kube-Prometheus, you’ll use the prometheus-community Helm Chart. However, the Kube-Prometheus stack isn’t the only option available when using the prometheus-community Chart. Because of that, ensure that you specify the kube-prometheus-stack option.
helm install kube-prometheus prometheus-community/kube-prometheus-stack
You can find the Helm Chart here: https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack
Kubernetes Manifests
First, clone the prometheus-operator Git repository.
git clone https://github.com/prometheus-operator/kube-prometheus.git
Next, run the configuration for the Kubernetes Manifests.
The following command is used with sleep to avoid any race conditions, which ensures that the proper Service Accounts and RBAC permissions are set up prior to any of the Manifests getting deployed.
kubectl create -f manifests/setup
until kubectl get servicemonitors --all-namespaces ; do date; sleep 1; echo ""; done
kubectl create -f manifests/
Retrieve all Namespaces and you should see one called monitoring.
kubectl get namespaces
After a minute or two, ensure that all of the resources in the monitoring Namespace are up and running.
kubectl get all -n monitoring
Viewing Grafana
Now that you’ve installed the Kube-Prometheus stack, let’s log into Grafana.
First, retrieve the Grafana service. For the Helm installation, the service name will be kube-prometheus-grafana. For the Manifest installation, the service name will be grafana.
Next, forward the port so you can access the Grafana UI.
For Helm installation:
kubectl port-forward svc/kube-prometheus-grafana :80
For Manifest installation:
kubectl port-forward svc/grafana -n monitoring :3000
The default username/password for Grafana with the Helm Installation is:
- admin
- prom-operator.
The default username/password for Grafana with the Manifest installation is:
- admin
- admin
Click the Browse button under dashboards.
Under general (or default), you’ll see a bunch of pre-populated dashboards for you.
For example, click on the Kubernetes / Computer Resources / Cluster option. You’ll see a Grafana dashboard with information around your cluster.
What makes Kube-Prometheus so great is that all of the dashboards are populated for you. You don’t have to do much of anything to get the environment up and running out of the box.
You now have a great place to start with Prometheus and Grafana for monitoring/observability of your environment.
Top comments (4)
Thanks a lot mate.
How do I persist configurations done from the UI ? If I create alerts, contact points, alerts rule, notification policies...... etc the moment i restart the pod or the grafana deployment I lose those Have been trying to use PVCs
manifest setup is breaking my
kubectl top nodesand making my metrics server dead, thanks for helm chart guide it really fixed my problems (which are those damned manifests who do the RBAC wrongly)The tutorial is very interesting, but how can I add another configuration in Prometheus, I want to add other devices and show them in the grafana ?