Prometheus is an open-source monitoring and alerting system that helps you collect and store metrics about your software systems and infrastructure and analyze that data to gain insights into their health and performance. It provides a powerful query language, a flexible data model, and a range of integrations with other tools and systems. With Prometheus, you can easily monitor metrics such as CPU usage, memory usage, network traffic, and application-specific metrics, and use that data to troubleshoot issues, optimize performance, and create alerts to notify you when things go wrong.
Install Prometheus via 2 ways.
- Helm Charts
- Operators
Using operators, using Operator life cycle management (OLM) one can automatically have latest updates without worrying to update manually and therefore securing cluster with zero-day vulnerabilities.
Install using Helm.
1. Add helm repo
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
2. Update helm repo
helm repo update
3. Install helm
helm install prometheus prometheus-community/prometheus
PS C:\WINDOWS\system32> helm install prometheus prometheus-community/prometheus
NAME: prometheus
LAST DEPLOYED: Mon Jan 22 12:55:19 2024
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The Prometheus server can be accessed via port 80 on the following DNS name from within your cluster:
prometheus-server.default.svc.cluster.local
Get the Prometheus server URL by running these commands in the same shell:
export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=prometheus,app.kubernetes.io/instance=prometheus" -o jsonpath="{.items[0].metadata.name}")
kubectl --namespace default port-forward $POD_NAME 9090
The Prometheus alertmanager can be accessed via port 9093 on the following DNS name from within your cluster:
prometheus-alertmanager.default.svc.cluster.local
Get the Alertmanager URL by running these commands in the same shell:
export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=alertmanager,app.kubernetes.io/instance=prometheus" -o jsonpath="{.items[0].metadata.name}")
kubectl --namespace default port-forward $POD_NAME 9093
#################################################################################
###### WARNING: Pod Security Policy has been disabled by default since #####
###### it deprecated after k8s 1.25+. use #####
###### (index .Values "prometheus-node-exporter" "rbac" #####
###### . "pspEnabled") with (index .Values #####
###### "prometheus-node-exporter" "rbac" "pspAnnotations") #####
###### in case you still need it. #####
#################################################################################
The Prometheus PushGateway can be accessed via port 9091 on the following DNS name from within your cluster:
prometheus-prometheus-pushgateway.default.svc.cluster.local
Get the PushGateway URL by running these commands in the same shell:
export POD_NAME=$(kubectl get pods --namespace default -l "app=prometheus-pushgateway,component=pushgateway" -o jsonpath="{.items[0].metadata.name}")
kubectl --namespace default port-forward $POD_NAME 9091
For more information on running Prometheus, visit:
https://prometheus.io/
- Verify Prometheus Pods Running
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
prometheus-alertmanager-0 1/1 Running 0 3m8s
prometheus-kube-state-metrics-745b475957-ngqm7 1/1 Running 0 3m8s
prometheus-prometheus-node-exporter-698nc 1/1 Running 0 3m8s
prometheus-prometheus-pushgateway-6ccd698d79-ld86d 1/1 Running 0 3m8s
prometheus-server-bc7ccb595-jpl7p 2/2 Running 0 3m8s
sample-python-deployment-99fcf4b6f-rb59w 1/1 Running 2 (5m40s ago) 8d
sample-python-deployment-99fcf4b6f-vcnwt 1/1 Running 2 (5m40s ago) 8d
- Verify Prometheus services via Kubectl get svc
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 8d
prometheus-alertmanager ClusterIP 10.100.6.213 <none> 9093/TCP 7m38s
prometheus-alertmanager-headless ClusterIP None <none> 9093/TCP 7m38s
prometheus-kube-state-metrics ClusterIP 10.103.166.42 <none> 8080/TCP 7m38s
prometheus-prometheus-node-exporter ClusterIP 10.104.211.162 <none> 9100/TCP 7m38s
prometheus-prometheus-pushgateway ClusterIP 10.100.179.109 <none> 9091/TCP 7m38s
prometheus-server ClusterIP 10.105.170.132 <none> 80/TCP 7m38s
sample-python-service LoadBalancer 10.101.160.8 <pending> 80:30007/TCP 8d
4. Expose Prometheus Service
This is required to access prometheus-server using your browser.
kubectl expose service prometheus-server --type=NodePort --target-port=9090 --name=prometheus-server-ext
Notice below is created
prometheus-server-ext NodePort 10.99.142.160 <none> 80:31401/TCP 3sGet svc
kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 8d
prometheus-alertmanager ClusterIP 10.100.6.213 <none> 9093/TCP 10m
prometheus-alertmanager-headless ClusterIP None <none> 9093/TCP 10m
prometheus-kube-state-metrics ClusterIP 10.103.166.42 <none> 8080/TCP 10m
prometheus-prometheus-node-exporter ClusterIP 10.104.211.162 <none> 9100/TCP 10m
prometheus-prometheus-pushgateway ClusterIP 10.100.179.109 <none> 9091/TCP 10m
prometheus-server ClusterIP 10.105.170.132 <none> 80/TCP 10m
prometheus-server-ext NodePort 10.99.142.160 <none> 80:31401/TCP 3s
sample-python-service LoadBalancer 10.101.160.8 <pending> 80:30007/TCP 8d
Output:
$ kubectl expose service prometheus-server --type=NodePort --target-port=9090 --name=prometheus-server-ext
service/prometheus-server-ext exposed
Get Minikube IP
minikube ip
192.168.59.107
Open Browser and access Prometheus
- Go to http://192.168.59.107:31401/
Install Grafana
1. Install using Helm
- Add helm repo
- _Ignore is you already have help and grafana repo added _
helm repo add grafana https://grafana.github.io/helm-charts
$helm repo add grafana https://grafana.github.io/helm-charts
"grafana" has been added to your repositories
2. Update helm repo
-
helm repo update
$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "eks" chart repository
...Successfully got an update from the "istio" chart repository
...Successfully got an update from the "grafana" chart repository
...Successfully got an update from the "prometheus-community" chart repository
Update Complete. ⎈Happy Helming!⎈
3. Install Grafana via helm
-
helm install grafana grafana/grafana
$ helm install grafana grafana/grafana
NAME: grafana
LAST DEPLOYED: Mon Jan 22 13:20:52 2024
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
1. Get your 'admin' user password by running:
kubectl get secret --namespace default grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
2. The Grafana server can be accessed via port 80 on the following DNS name from within your cluster:
grafana.default.svc.cluster.local
Get the Grafana URL to visit by running these commands in the same shell:
export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=grafana" -o jsonpath="{.items[0].metadata.name}")
kubectl --namespace default port-forward $POD_NAME 3000
3. Login with the password from step 1 and the username: admin
#################################################################################
###### WARNING: Persistence is disabled!!! You will lose your data when #####
###### the Grafana pod is terminated. #####
#################################################################################
4. Get Grafana Login Details
- Get password:
$ kubectl get secret --namespace default grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
$ kubectl get secret --namespace default grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
XPhzARnKv93uhRGt8Nu9cimhuAHZoj64ZtCHDeQd
- Get Grafana services
kubectl get svcNotice its running on Cluster IP
kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
grafana ClusterIP 10.109.8.93 <none> 80/TCP 8m12s
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 8d
prometheus-alertmanager ClusterIP 10.100.6.213 <none> 9093/TCP 33m
prometheus-alertmanager-headless ClusterIP None <none> 9093/TCP 33m
prometheus-kube-state-metrics ClusterIP 10.103.166.42 <none> 8080/TCP 33m
prometheus-prometheus-node-exporter ClusterIP 10.104.211.162 <none> 9100/TCP 33m
prometheus-prometheus-pushgateway ClusterIP 10.100.179.109 <none> 9091/TCP 33m
prometheus-server ClusterIP 10.105.170.132 <none> 80/TCP 33m
prometheus-server-ext NodePort 10.99.142.160 <none> 80:31401/TCP 23m
sample-python-service LoadBalancer 10.101.160.8 <pending> 80:30007/TCP 8d
5. Expose Grafana Service to Access IT
kubectl expose service grafana --type=NodePort --target-port=3000 --name=grafana-ext
- Verify exposed Grafana service
kubectl expose service grafana --type=NodePort --target-port=3000 --name=grafana-ext
service/grafana-ext exposed
- Get Grafana-Ext Service at port
30197
kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
grafana ClusterIP 10.109.8.93 <none> 80/TCP 11m
grafana-ext NodePort 10.111.66.82 <none> 80:30197/TCP 51s
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 8d
prometheus-alertmanager ClusterIP 10.100.6.213 <none> 9093/TCP 37m
prometheus-alertmanager-headless ClusterIP None <none> 9093/TCP 37m
prometheus-kube-state-metrics ClusterIP 10.103.166.42 <none> 8080/TCP 37m
prometheus-prometheus-node-exporter ClusterIP 10.104.211.162 <none> 9100/TCP 37m
prometheus-prometheus-pushgateway ClusterIP 10.100.179.109 <none> 9091/TCP 37m
prometheus-server ClusterIP 10.105.170.132 <none> 80/TCP 37m
prometheus-server-ext NodePort 10.99.142.160 <none> 80:31401/TCP 26m
sample-python-service LoadBalancer 10.101.160.8 <pending> 80:30007/TCP 8d
- Access via minikube IP
- http://192.168.59.107:30197/
- Login via user=admin and password=XPhzARnKv93uhRGt8Nu9cimhuAHZoj64ZtCHDeQd
- Grafana Dashboard
- Configure Prometheus as Datasource to Grafana
- Go to Data sources --> Add Prometheus --> Provide IP Address - Save and Test
Import an existing dashboard
Dashboard --> Import --> Enter ID
3662--> Load
- Dashboard
6. Expose prometheus-kube-state-metrics
- How do we do it by following the steps below.
kubectl expose service prometheus-kube-state-metrics --type=NodePort --target-port=8080 --name=prometheus-kube-state-metrics-ext
- Verify kube-state-metrics exposed
kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
grafana ClusterIP 10.109.8.93 <none> 80/TCP 54m
grafana-ext NodePort 10.111.66.82 <none> 80:30197/TCP 43m
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 8d
prometheus-alertmanager ClusterIP 10.100.6.213 <none> 9093/TCP 79m
prometheus-alertmanager-headless ClusterIP None <none> 9093/TCP 79m
prometheus-kube-state-metrics ClusterIP 10.103.166.42 <none> 8080/TCP 79m
prometheus-kube-state-metrics-ext NodePort 10.100.12.19 <none> 8080:30229/TCP 15s
prometheus-prometheus-node-exporter ClusterIP 10.104.211.162 <none> 9100/TCP 79m
prometheus-prometheus-pushgateway ClusterIP 10.100.179.109 <none> 9091/TCP 79m
prometheus-server ClusterIP 10.105.170.132 <none> 80/TCP 79m
prometheus-server-ext NodePort 10.99.142.160 <none> 80:31401/TCP 69m
sample-python-service LoadBalancer 10.101.160.8 <pending> 80:30007/TCP 8d
- Open browser and access via port 30229
- http://192.168.59.107:30229/
- Kubectl get cm or configmap
kubectl get cm
NAME DATA AGE
grafana 1 61m
kube-root-ca.crt 1 8d
prometheus-alertmanager 1 86m
prometheus-server 6 86m
- Edit config map of
prometheus-server
kubectl edit cm prometheus-server
change scrpe confi from local-host to prometheus-kube-state-metrics-ext service
- BEFORE:
scrape_configs:
- job_name: prometheus
static_configs:
- targets:
- localhost:9090
- AFTER: Add config to
http://192.168.59.107:30229/state-metrics-ext
scrape_configs:
- job_name: prometheus
static_configs:
- targets:
- localhost:9090
- job_name: prometheus
static_configs:
- targets:
- http://192.168.59.107:30229
Credits:-
Thanks to Abhishek Veeramalla
Top comments (0)