DEV Community

Michael Levan
Michael Levan

Posted on

How To Configure Kube-Prometheus

Whenever you’re implementing a monitoring and observability solution for your environment, a few questions occur like what you’re going to monitor and the specific production environments that need to be monitored.

Another large and important question is how you’re going to monitor the environment.

With Kubernetes and monitoring/observability platforms that go with it, there are a ton of options. It can definitely get a bit overwhelming.

With Kube-Prometheus, the feeling of being overwhelmed should hopefully subside.

In this blog post, you’ll learn all about what Kube-Prometheus is and how to get it up and running.

What Is Kube-Prometheus?

It’s the “easy button”.

The Prometheus Operator itself gives you the ability to install all Prometheus and Grafana features in a declarative way. For example, below is a Kubernetes Manifest to install Alertmanager, which can be installed in a declarative fashion because the Prometheus Operator exists.

apiVersion: monitoring.coreos.com/v1
kind: Alertmanager
metadata:
  labels:
    app.kubernetes.io/component: alert-router
    app.kubernetes.io/instance: main
    app.kubernetes.io/name: alertmanager
    app.kubernetes.io/part-of: kube-prometheus
    app.kubernetes.io/version: 0.24.0
  name: main
  namespace: monitoring
spec:
  image: quay.io/prometheus/alertmanager:v0.24.0
  nodeSelector:
    kubernetes.io/os: linux
  podMetadata:
    labels:
      app.kubernetes.io/component: alert-router
      app.kubernetes.io/instance: main
      app.kubernetes.io/name: alertmanager
      app.kubernetes.io/part-of: kube-prometheus
      app.kubernetes.io/version: 0.24.0
  replicas: 3
  resources:
    limits:
      cpu: 100m
      memory: 100Mi
    requests:
      cpu: 4m
      memory: 100Mi
  securityContext:
    fsGroup: 2000
    runAsNonRoot: true
    runAsUser: 1000
  serviceAccountName:
Enter fullscreen mode Exit fullscreen mode

Kube-Prometheus on the other hand installs Alertmanager for you.

Essentially, Kube-Prometheus contains a bunch of Kubernetes Manifests for installing components like Grafana, Alertmanager, Node Exporter, Kube State Metrics, and several other pieces including all of the RBAC permissions and Service Accounts needed to get it up and running.

Instead of installing all of the components manually, you can just use Kube-Prometheus.

Here’s a list of what’s installed: https://github.com/prometheus-operator/kube-prometheus/tree/main/manifests

Image description

Methods Of Installation

Now that you know what Kube-Prometheus is, which is essentially a collection of Kubernetes Manifests to get up and running fast with the Prometheus/Grafana stack, let’s learn a few different installation options.

The first is Helm and the second is a traditional Kubernetes Manifest deployment.

Please note that you don’t have to run both of these. You can run one or the other, or run one, delete one, and then run the next. That way there are no conflicts and you see both processes.

Helm

First, add the prometheus-community repo.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

Enter fullscreen mode Exit fullscreen mode

Next, ensure that it’s up to date.

helm repo update
Enter fullscreen mode Exit fullscreen mode

When installing Kube-Prometheus, you’ll use the prometheus-community Helm Chart. However, the Kube-Prometheus stack isn’t the only option available when using the prometheus-community Chart. Because of that, ensure that you specify the kube-prometheus-stack option.

helm install kube-prometheus prometheus-community/kube-prometheus-stack
Enter fullscreen mode Exit fullscreen mode

You can find the Helm Chart here: https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack

Image description

Image description

Kubernetes Manifests

First, clone the prometheus-operator Git repository.

git clone https://github.com/prometheus-operator/kube-prometheus.git
Enter fullscreen mode Exit fullscreen mode

Next, run the configuration for the Kubernetes Manifests.

The following command is used with sleep to avoid any race conditions, which ensures that the proper Service Accounts and RBAC permissions are set up prior to any of the Manifests getting deployed.

kubectl create -f manifests/setup

until kubectl get servicemonitors --all-namespaces ; do date; sleep 1; echo ""; done

kubectl create -f manifests/
Enter fullscreen mode Exit fullscreen mode

Retrieve all Namespaces and you should see one called monitoring.

kubectl get namespaces
Enter fullscreen mode Exit fullscreen mode

After a minute or two, ensure that all of the resources in the monitoring Namespace are up and running.

kubectl get all -n monitoring
Enter fullscreen mode Exit fullscreen mode

Image description

Viewing Grafana

Now that you’ve installed the Kube-Prometheus stack, let’s log into Grafana.

First, retrieve the Grafana service. For the Helm installation, the service name will be kube-prometheus-grafana. For the Manifest installation, the service name will be grafana.

Next, forward the port so you can access the Grafana UI.

For Helm installation:

kubectl port-forward svc/kube-prometheus-grafana :80
Enter fullscreen mode Exit fullscreen mode

For Manifest installation:

kubectl port-forward svc/grafana -n monitoring :3000
Enter fullscreen mode Exit fullscreen mode

The default username/password for Grafana with the Helm Installation is:

  • admin
  • prom-operator.

The default username/password for Grafana with the Manifest installation is:

  • admin
  • admin

Click the Browse button under dashboards.

Image description

Under general (or default), you’ll see a bunch of pre-populated dashboards for you.

Image description

For example, click on the Kubernetes / Computer Resources / Cluster option. You’ll see a Grafana dashboard with information around your cluster.

Image description

What makes Kube-Prometheus so great is that all of the dashboards are populated for you. You don’t have to do much of anything to get the environment up and running out of the box.

You now have a great place to start with Prometheus and Grafana for monitoring/observability of your environment.

Top comments (4)

Collapse
 
786gagan profile image
gagan

Thanks a lot mate.

Collapse
 
cdetaka profile image
Ta Ka

How do I persist configurations done from the UI ? If I create alerts, contact points, alerts rule, notification policies...... etc the moment i restart the pod or the grafana deployment I lose those Have been trying to use PVCs

Collapse
 
mrwormhole profile image
Talha Altınel

manifest setup is breaking my kubectl top nodes and making my metrics server dead, thanks for helm chart guide it really fixed my problems (which are those damned manifests who do the RBAC wrongly)

Collapse
 
vaskata84 profile image
Vasil Dobchev

The tutorial is very interesting, but how can I add another configuration in Prometheus, I want to add other devices and show them in the grafana ?