Prometheus is an excellent monitoring tool developed for Kubernetes monitoring.
Many of the helm charts like Nginx, RabbitMq provides inbuilt provision for metrics exposing for Prometheus monitoring.
These metrics are generally exposed on an endpoint, say, /metrics and prometheus pulls down, process them on certain interval.
But the question arrises how does Prometheus knows from where to scrap metrics?
The answer to this is via Service Monitor.
Note: This articles assumes the reader has basic knowledge of Prometheus, Grafana, Kubernetes, Helm Charts
What is Service Monitor?
Service Monitor is a CRD provided by Prometheus Operator, which provides configuration on how provided services should be monitored. Or how we wish to collect metrics from different services.
Prometheus operator uses this service monitor and configures itself internally.
Requirements
- All services which are wished to be monitored must have following annotations in their manifest configuration:
annotations:
prometheus.io/port: "metrics"
prometheus.io/scrape: "true"
Here,
First line represents PORT number(We used a PORT name instead of actual port number here).
Second line represents a flag whether to scrap metrics or not.
- A service monitor which monitors which services' metrics to scrap.
Example
Let's say we have a simple k8s deployment(service) having following manifest:
#########################################################
# Deployment
#########################################################
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-service-deployment
namespace: production
spec:
replicas: 1
selector:
matchLabels:
app: my-service
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
minReadySeconds: 5
template:
metadata:
namespace: production
labels:
app: my-service
spec:
nodeSelector:
kubernetes.io/os: linux
containers:
- name: my-service-pod
image: myacr.azurecr.io/my_service:latest
envFrom:
- secretRef:
name: prod-secrets
ports:
- containerPort: 4050
resources:
requests:
cpu: 250m
memory: 250Mi
limits:
cpu: 500m
memory: 500Mi
---
#########################################################
# Service
#########################################################
apiVersion: v1
kind: Service
metadata:
name: my-service-service
namespace: production
spec:
type: ClusterIP
selector:
app: my-service
ports:
- port: 4050
protocol: TCP
targetPort: 4050
name: "metrics"
---
Note: the config may look daunting, but it just has few extra optional parameters
The above is a simple K8s Service and Deployement having type ClusterIP (not exposed as we have Nginx gateway)
If we assume, we have already setup our prom-client in our NodeJs image to scrap metrics, we would have a endpoint /metrics where all prometheus metrics are exposed.
Now, we just need to connect this service to prometheus.
STEP 1:
Add required annotations in service spec template:
annotations:
prometheus.io/port: "metrics"
prometheus.io/scrape: "true"
and add labels to identify service:
labels:
app.kubernetes.io/part-of: dms
Note- Put lables according to new standard as defined in K8s Docs
Our final service manifest shall look like:
Note: no changes to deployment maifest needed
#########################################################
# Service
#########################################################
apiVersion: v1
kind: Service
metadata:
name: my-service-service
namespace: production
annotations:
prometheus.io/port: "metrics"
prometheus.io/scrape: "true"
labels:
app.kubernetes.io/part-of: dms
spec:
type: ClusterIP
selector:
app: my-service
ports:
- port: 4050
protocol: TCP
targetPort: 4050
name: "metrics"
---
Deploy this configuration.
For eg kubectl apply -f my-service.yml
STEP 2:
After deployment new service manifest, create a new Service monitor as follows:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: dms-service-monitor
namespace: monitoring
spec:
endpoints:
- interval: 15s
port: metrics
scrapeTimeout: 14s
namespaceSelector:
matchNames:
- production
selector:
matchLabels:
app.kubernetes.io/part-of: dms
Note:
- The selector: config should match with that deined in service as above.
- Namespace can be different. You can include specifc namespaces to monitor using
namespaceSelector:.
With this our setup is done.
STEP 3:
Check if Prometheus is collecting your metrics.
Head over to your prometheus server (using port-forward if not exposed). On top menu select Status>Targets.
You can see Prometheus has discovered the new target my-service-monitor and working successfully.
You can even see your scrape data on home page using query.
Troubleshoot
If you have deployed Prometheus using kube-prometheus-stack helm chart you might need to upgrade chart using following chart values:
values.yml
prometheus:
prometheusSpec:
podMonitorSelectorNilUsesHelmValues: false
serviceMonitorSelectorNilUsesHelmValues: false
For eg:
helm upgrade prometheus-operator prometheus-community/kube-prometheus-stack --namespace monitoring --namespace monitoring --values values.yml
This way we remove pre configured selectors constraints from prometheus deployement.
Also there is an excellent answer on stackoverflow which describes the whole monitoring flow to troubleshoot.
Conclusion
So we see how easily we can connect our Prometheus and Nodejs metrics on our k8s cluster.
You dont need a separate Service Monitor everytime you create a new service so long your service correctly passes constraints inside selector in service monitor.
*Whola! Both you and I learnt something new today. Congrats
👏 👏 👏 *
Top comments (1)
Nice, thank you.