DEV Community

Cover image for Monitoring on Kubernetes
Omar Ahmed
Omar Ahmed

Posted on

Monitoring on Kubernetes

#kubernetes #grafana #prometheus

Prometheus Architecture in Kubernetes

  • Prometheus is a monitoring system that collects metrics from applications and infrastructure, stores them as time-series data, and evaluates queries and alerting rules.
  • Prometheus is primarily a Pull-Based Monitoring System so No agent credentials in targets, No push client libraries needed for most services.

  • Pull model disadvantages:

    • Short-lived jobs: Batch jobs that run and exit in under one scrape interval can't be scraped. Solution: Pushgateway
    • Firewall/NAT traversal: Prometheus needs network access to all targets. In complex multi-datacenter setups this can be complex. Solution: remote_write + federation

  • Its architecture consists of several components:

    1. Prometheus Server - Scrapes and stores metrics.
    2. TSDB - Stores time-series data.
    3. Service Discovery - Finds monitoring targets automatically.
    4. Exporters / Applications - Expose metrics.
    5. PromQL / Rule Engine - Queries metrics and evaluates rules.
    6. Alertmanager - Handles alert notifications.
    7. Grafana - Visualizes metrics in dashboards.
Component Responsibility
Service Discovery Manager Discovers monitoring targets from Kubernetes, DNS, cloud providers, etc.
Target / Scrape Manager Maintains scrape targets and schedules metric collection
Relabeling Filters targets, changes addresses, and adjusts labels
Ingestion Pipeline Processes collected samples before storing them
TSDB Stores timestamped metrics and label indexes
WAL Logs recent writes for crash recovery
Head Block Holds recent time-series samples
Persistent Blocks Stores compacted historical time-series data
PromQL Query Engine Executes queries against TSDB data
Rule Manager Evaluates recording and alerting rules
HTTP API / Web UI Provides access to queries, targets, rules, and metrics
Notifier Sends firing and resolved alert updates to Alertmanager

Pull model (Prometheus):

          +--------------------+
          |     Prometheus     |
          |                    |
          |   Scrape Manager   |
          +--------------------+
                    |
                    | HTTP GET /metrics
                    | Every 15 seconds
                    v
          +--------------------+
          |    Application     |
          |                    |
          |   Metrics :8080    |
          +--------------------+
                    |
                    | HTTP Response
                    | Metrics data
                    v
          +--------------------+
          |     Prometheus     |
          |        TSDB        |
          +--------------------+
Enter fullscreen mode Exit fullscreen mode

Kubernetes deployment architecture:

monitoring namespace
|
+-- prometheus              Deployment / StatefulSet
|   +-- prometheus-config   ConfigMap
|   +-- prometheus-rules    ConfigMap
|   +-- prometheus-data     PersistentVolumeClaim
|
+-- node-exporter           DaemonSet
|   +-- one Pod per node
|
+-- kube-state-metrics      Deployment
|
+-- alertmanager            Deployment / StatefulSet
|
+-- grafana                 Deployment
    +-- grafana-datasources ConfigMap
    +-- grafana-dashboards  ConfigMap
Enter fullscreen mode Exit fullscreen mode

What a Metric Is: Reading a Pod's /metrics

The shop app is running. Is it busy? Is it failing? Instead of guessing from logs, the app counts what it does and shows the numbers on a web page, usually at /metrics. Each of those numbers is a metric. Prometheus reads that page every few seconds and stores the numbers over time.

Reading a Service's page with kubectl

kubectl get --raw can fetch any Service's URL through the API server:

$ kubectl get --raw /api/v1/namespaces/monitoring/services/shop:8080/proxy/metrics
# HELP http_requests_total Requests handled by the shop, by path and status code.
# TYPE http_requests_total counter
http_requests_total{path="/",code="200"} 1284
http_requests_total{path="/cart",code="200"} 311
http_requests_total{path="/cart",code="404"} 9
http_requests_total{path="/checkout",code="200"} 87
http_requests_total{path="/checkout",code="500"} 4
http_requests_total{path="/checkout",code="502"} 2
http_requests_total{path="/pay",code="503"} 1
# HELP shop_orders_in_progress Orders being processed right now.
# TYPE shop_orders_in_progress gauge
shop_orders_in_progress 3
Enter fullscreen mode Exit fullscreen mode

The path follows one pattern: /api/v1/namespaces/NAMESPACE/services/NAME:PORT/proxy/PATH.

How to read it

Every line is one of three things:

Line Means
# HELP name text What the metric measures, in words
# TYPE name type Its type (below)
name{labels} value One series: the metric for one combination of labels, and its current value

So http_requests_total{path="/cart",code="404"} 9 , reads: requests to /cart that got a 404= 9 so far.

Counter or gauge

The # TYPE line tells you how a number behaves:

Type Behaves like Example
counter Only goes up (it resets to 0 when the app restarts) http_requests_total
gauge Goes up and down shop_orders_in_progress, node_memory_MemFree_bytes
histogram Counts values into buckets, such as request durations http_request_duration_seconds_bucket
summary Like a histogram, with percentiles computed by the app rare in new apps

Labels make series

Series (time series) is a metric with a unique combination of labels, whose value is tracked over time.
Imagine your application exposes these metrics:

http_requests_total{path="/cart",code="200"} 311
http_requests_total{path="/cart",code="404"} 9
http_requests_total{path="/checkout",code="200"} 87
Enter fullscreen mode Exit fullscreen mode

All three belong to the same metric: http_requests_total
But they represent 3 different series because their label combinations are different.

Series Path Code Value
1 /cart 200 311
2 /cart 404 9
3 /checkout 200 87

Why is it called a time series?
Prometheus scrapes your application's /metrics endpoint repeatedly, collecting a new value for each series.
For example, this is one series: http_requests_total{path="/cart",code="200"}

http_requests_total is one metric but 7 is a series: one per path and code combination. To count them, count the series lines and skip the comments:

$ kubectl get --raw /api/v1/namespaces/monitoring/services/shop:8080/proxy/metrics | grep -c '^http_requests_total{'
7
# ^ means the line must start with http_requests_total{.
# -c counts matching lines.
Enter fullscreen mode Exit fullscreen mode

To see a slice, filter on a label:

$ kubectl get --raw /api/v1/namespaces/monitoring/services/shop:8080/proxy/metrics | grep 'path="/cart"'
http_requests_total{path="/cart",code="200"} 311
http_requests_total{path="/cart",code="404"} 9
Enter fullscreen mode Exit fullscreen mode

Common mistakes

  • Counting the # HELP and # TYPE lines as series. They're comments. Count only lines that start with the metric name and {.
  • A typo in the proxy path. You get Error from server (NotFound): the server could not find the requested resource. Check the namespace, the Service name and the port against kubectl get svc -n monitoring.
  • Reading a counter's value as "per second". It's a running total since the app started.

Deploying Prometheus: ConfigMap, Deployment, Service

Prometheus: every few seconds it scrapes (fetches) each app's /metrics page and stores every value with a timestamp. Then you can ask how many errors in the last hour?

On Kubernetes, Prometheus is just three objects: a ConfigMap with its settings, a Deployment that runs it, and a Service to reach it.

The configuration file

Prometheus reads one file, prometheus.yml:

global:
  scrape_interval: 5s
  evaluation_interval: 5s
rule_files:
- /etc/rules/*.yml
Enter fullscreen mode Exit fullscreen mode
  • scrape_interval: Controls how frequently Prometheus collects metrics from your applications. Real setups often use 15–30 seconds.

  • evaluation_interval: Controls how frequently Prometheus evaluates its configured rules.

    • There are two main types of rules:
      1. Recording rules: Precalculate PromQL expressions and store the results as new metrics.
      2. Alerting rules: Evaluate conditions that can trigger alerts.

  • rule_files: Tells Prometheus where to find the recording and alerting rule definitions. In this course they come from a ConfigMap called prometheus-rules, mounted at /etc/rules.

Turn the file into a ConfigMap:

kubectl create configmap prometheus-config -n monitoring --from-file=prometheus.yml
Enter fullscreen mode Exit fullscreen mode

The Deployment and the Service 

apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus
  namespace: monitoring

spec:
  replicas: 1

  selector:
    matchLabels:
      app: prometheus

  template:
    metadata:
      labels:
        app: prometheus

    spec:
      containers:
        - name: prometheus
          image: prom/prometheus:v3.13.4

          args:
            - --config.file=/etc/prometheus/prometheus.yml
            - --storage.tsdb.path=/prometheus

          ports:
            - containerPort: 9090

          volumeMounts:
            - name: config
              mountPath: /etc/prometheus

            - name: rules
              mountPath: /etc/rules

      volumes:
        - name: config
          configMap:
            name: prometheus-config

        - name: rules
          configMap:
            name: prometheus-rules

---
apiVersion: v1
kind: Service
metadata:
  name: prometheus
  namespace: monitoring

spec:
  selector:
    app: prometheus

  ports:
    - port: 9090
      targetPort: 9090
Enter fullscreen mode Exit fullscreen mode

The ConfigMap is mounted where the image looks for its config, /etc/prometheus. Apply it and wait:

$ kubectl get pods -n monitoring
NAME                          READY   STATUS    RESTARTS   AGE
prometheus-6689c68d6b-2lf6b   1/1     Running   0          12s
shop-76cb9f89b9-sgwgg         1/1     Running   0          28s

$ kubectl get --raw /api/v1/namespaces/monitoring/services/prometheus:9090/proxy/-/ready
Prometheus Server is Ready.
Enter fullscreen mode Exit fullscreen mode

Ready, but scraping nothing

Ask Prometheus for its targets. Nothing comes back:

kubectl get --raw /api/v1/namespaces/monitoring/services/prometheus:9090/proxy/api/v1/targets | jq -c '.data.activeTargets[] | {job: .labels.job, health, scrapeUrl, lastError}'
Enter fullscreen mode Exit fullscreen mode

Nothing in the file says what to scrape. A scrape job does, under scrape_configs in prometheus.yml :

global:
  scrape_interval: 5s
  evaluation_interval: 5s

rule_files:
  - /etc/rules/*.yml

scrape_configs:
  - job_name: shop
    static_configs:
      - targets:
          - shop.monitoring.svc:8080
Enter fullscreen mode Exit fullscreen mode

The target is the shop Service's DNS name and port. After changing the ConfigMap, restart Prometheus: it reads its file only when it starts.

kubectl create configmap prometheus-config -n monitoring --from-file=prometheus.yml --dry-run=client -o yaml | kubectl replace -f -
kubectl rollout restart deployment/prometheus -n monitoring
Enter fullscreen mode Exit fullscreen mode

Now the target is there, and healthy:

$ kubectl get --raw /api/v1/namespaces/monitoring/services/prometheus:9090/proxy/api/v1/targets | jq -c '.data.activeTargets[] | {job: .labels.job, health, scrapeUrl, lastError}'
{"job":"shop","health":"up","scrapeUrl":"http://shop.monitoring.svc:8080/metrics","lastError":""}
Enter fullscreen mode Exit fullscreen mode

In Prometheus, up is a built-in metric that tells you whether Prometheus successfully scraped a target's /metrics endpoint.
It has only two values:

  • up = 1 → Prometheus successfully collected metrics from the target.
  • up = 0 → Prometheus tried to collect metrics, but the scrape failed. 
kubectl get --raw \
  '/api/v1/namespaces/monitoring/services/prometheus:9090/proxy/api/v1/query?query=up'

up{job="shop",instance="shop.monitoring.svc:8080"} 1
# job="shop"                          => The scrape job configured in prometheus.yml
# instance="shop.monitoring.svc:8080" => The target Prometheus scrapes
# 1                                   => Last scrape succeeded
Enter fullscreen mode Exit fullscreen mode

Common mistakes

  • Changing the ConfigMap and stopping there. The running Prometheus still uses the old file. Restart the Deployment.
  • Mounting the config at the wrong path. Prometheus then starts with its built-in default file. Check the loaded config at /api/v1/status/config.
  • kubectl create configmap on a ConfigMap that exists fails with "already exists". Pipe --dry-run=client -o yaml into kubectl replace -f - instead.

A Target That Is DOWN

Empty graphs, running Prometheus

Monday, 9 a.m. The shop dashboard is blank. Prometheus is up, so the problem is somewhere between Prometheus and the app. Prometheus already knows what went wrong: it records the result of every scrape. You only have to ask.

Ask the targets page

/api/v1/targets lists every target with its health and the last error. jq trims it to what matters:

kubectl get --raw \
  /api/v1/namespaces/monitoring/services/prometheus:9090/proxy/api/v1/targets \
  | jq -c '.data.activeTargets[] | {job: .labels.job, health, scrapeUrl, lastError}'

# kubectl get --raw, This sends a request through the Kubernetes API server to the Prometheus Service.
# jq filters the JSON response.
# .data.activeTargets[] selects each active target.
# job: .labels.job gets its job name.
# health gets its scrape status.
# scrapeUrl gets the URL Prometheus is scraping.
# lastError gets the latest scrape error.
# -c prints each JSON result on one line.

{"job":"prometheus","health":"up","scrapeUrl":"http://localhost:9090/metrics","lastError":""}
{"job":"shop","health":"down","scrapeUrl":"http://shop.monitoring.svc:8081/metrics","lastError":"Get \"http://shop.monitoring.svc:8081/metrics\": context deadline exceeded"}

# http://localhost:9090/metrics            =>  Scraping its own metrics successfully
# http://shop.monitoring.svc:8081/metrics  =>  Prometheus did not complete the scrape before its timeout.
Enter fullscreen mode Exit fullscreen mode

Two fields tell the story:

  • scrapeUrl: the exact address Prometheus is calling, here port 8081.
  • lastError: what happened. context deadline exceeded means nothing answered in time.

The up metric says the same thing as a number, 0 for the shop:

$ kubectl exec deploy/prometheus -n monitoring -- wget -qO- --post-data 'query=up' localhost:9090/api/v1/query | jq -c '.data.result[] | {metric, value}'
{"metric":{"__name__":"up","instance":"localhost:9090","job":"prometheus"},"value":[1792188858.578,"1"]}
{"metric":{"__name__":"up","instance":"shop.monitoring.svc:8081","job":"shop"},"value":[1792188858.578,"0"]}
Enter fullscreen mode Exit fullscreen mode

Compare with the truth

What port does the shop actually listen on? The Service says:

$ kubectl get svc shop -n monitoring
NAME   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
shop   ClusterIP   10.109.73.210   <none>        8080/TCP   15s
Enter fullscreen mode Exit fullscreen mode

The Service is on 8080, and Prometheus calls 8081. The app is fine, and the scrape configuration is wrong. So the fix belongs in prometheus-config, not in the app.

Fix, reload, confirm

Correct the target in the ConfigMap, then restart Prometheus so it reads the file again (lesson 2). Ask the targets page once more:

$ kubectl get --raw /api/v1/namespaces/monitoring/services/prometheus:9090/proxy/api/v1/targets | jq -c '.data.activeTargets[] | {job: .labels.job, health, scrapeUrl, lastError}'
{"job":"prometheus","health":"up","scrapeUrl":"http://localhost:9090/metrics","lastError":""}
{"job":"shop","health":"up","scrapeUrl":"http://shop.monitoring.svc:8080/metrics","lastError":""}
Enter fullscreen mode Exit fullscreen mode

This is the loop for every scrape problem: read lastError, compare the address with the Service, fix the config, reload, then check again.

Common mistakes

  • "Fixing" the Service to match the wrong port. The app was healthy. Changing it breaks everything else that uses port 8080.
  • Editing the ConfigMap and checking right away. Without a restart the old address stays, and the target stays DOWN.
  • Reading only health. down says that it failed. lastError says why: a timeout, connection refused, or an HTTP error such as 404 each point somewhere different.

In production

  • An alert on up == 0 is often the first one a team writes. A target that silently stops reporting is worse than one that reports errors, because every graph that depends on it just goes flat.
  • up = 0 means scraping failed, not necessarily that the application itself is down

Node Metrics: node-exporter as a DaemonSet

The app is fine. Is the machine?

  • The shop looks healthy, but checkout is slow. Maybe a node is out of memory, or its CPU is pegged. Apps don't report that; the machine does. node-exporter reads the node's own counters (CPU, memory, disk, network) and serves them as metrics on port 9100.
  • You need exactly one per node, so the right object is a DaemonSet: Kubernetes runs one pod on every node, including nodes added later

Measuring the node, not the pod

A normal pod sees its own little world: its own network, its own /proc. To report on the machine, node-exporter needs the machine's view:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter
  namespace: monitoring

spec:
  selector:
    matchLabels:
      app: node-exporter

  template:
    metadata:
      labels:
        app: node-exporter

    spec:
      hostNetwork: true

      tolerations:
        - operator: Exists # you haven't specified a key or effect, it matches all taint keys, values, and effects.

      containers:
        - name: node-exporter
          image: prom/node-exporter:v1.12.1

          args:
            - --path.procfs=/host/proc
            - --path.sysfs=/host/sys

          ports:
            - containerPort: 9100

          volumeMounts:
            - name: proc
              mountPath: /host/proc
              readOnly: true

            - name: sys
              mountPath: /host/sys
              readOnly: true

      volumes:
        - name: proc
          hostPath:
            path: /proc

        - name: sys
          hostPath:
            path: /sys

---
apiVersion: v1
kind: Service
metadata:
  name: node-exporter
  namespace: monitoring

spec:
  clusterIP: None

  selector:
    app: node-exporter

  ports:
    - port: 9100
      targetPort: 9100
Enter fullscreen mode Exit fullscreen mode
Setting Why
hostNetwork: true The pod uses the node's network, so it reports the node's traffic and listens on the node's IP
hostPath for /proc and /sys, plus --path.procfs and --path.sysfs It reads the host's CPU and memory files instead of the container's
tolerations with operator: Exists It also runs on the control-plane node, which normally refuses ordinary pods
clusterIP: None A headless Service: its DNS name returns one address per pod instead of a single virtual IP

Note

1. Understand Taints and Tolerations

  • A Taint is applied to a Node to restrict which Pods can run on it.
  • For example, Kubernetes control-plane nodes commonly have this taint: Taints: node-role.kubernetes.io/control-plane:NoSchedule
  • This means ordinary Pods cannot be scheduled onto this node unless they have a matching toleration.

What does operator: Exists mean?
Normally, you could write a specific toleration:

tolerations:
  - key: node-role.kubernetes.io/control-plane
    operator: Exists
    effect: NoSchedule
Enter fullscreen mode Exit fullscreen mode

This means:

  • key: Match the control-plane taint.
  • operator: Exists: Match the taint based on its key, regardless of its value.
  • effect: NoSchedule: Match only taints with the NoSchedule effect.

But your configuration is:

tolerations:
  - operator: Exists
Enter fullscreen mode Exit fullscreen mode

Since you haven't specified a key or effect, it matches all taint keys, values, and effects.

2. Normal Service (ClusterIP) and a Headless Service
Normal Service (ClusterIP):

apiVersion: v1
kind: Service
metadata:
  name: shop
  namespace: monitoring
spec:
  type: ClusterIP
  selector:
    app: shop
  ports:
    - port: 8080
      targetPort: 8080
Enter fullscreen mode Exit fullscreen mode

Imagine you have three Shop Pods, When a client resolves shop.monitoring.svc.cluster.local, DNS returns the Service's ClusterIP.
shop.monitoring.svc.cluster.local → 10.96.0.100

                 +----------------------+
                 |        Client        |
                 |      shop:8080       |
                 +----------------------+
                            |
                            v
                 +----------------------+
                 |  ClusterIP Service   |
                 |     10.96.0.100      |
                 +----------------------+
                            |
              +-------------+-------------+
              |             |             |
              v             v             v
    +---------------+ +---------------+ +---------------+
    |  Shop Pod 1   | |  Shop Pod 2   | |  Shop Pod 3   |
    |  10.244.1.2   | |  10.244.1.3   | |  10.244.1.4   |
    +---------------+ +---------------+ +---------------+
Enter fullscreen mode Exit fullscreen mode

Kubernetes Service routing then directs connections to one of the available Pods.

Headless Service:

apiVersion: v1
kind: Service
metadata:
  name: node-exporter
  namespace: monitoring
spec:
  clusterIP: None
  selector:
    app: node-exporter
  ports:
    - port: 9100
      targetPort: 9100
Enter fullscreen mode Exit fullscreen mode 
                    +------------------------+
                    |       Prometheus       |
                    |       DNS lookup       |
                    +------------------------+
                                |
                                v
                    +------------------------+
                    |    Headless Service    |
                    |  Returns endpoint IPs  |
                    +------------------------+
                                |
                +---------------+---------------+
                |               |               |
                v               v               v
       +----------------+ +----------------+ +----------------+
       |   Exporter 1   | |   Exporter 2   | |   Exporter 3   |
       |  192.168.1.11  | |  192.168.1.12  | |  192.168.1.13  |
       +----------------+ +----------------+ +----------------+
Enter fullscreen mode Exit fullscreen mode

Imagine Node Exporter is running on three nodes, DNS can return multiple IP addresses:

node-exporter.monitoring.svc.cluster.local
    → 192.168.1.11
    → 192.168.1.12
    → 192.168.1.13
Enter fullscreen mode Exit fullscreen mode

Because your Node Exporter uses hostNetwork: true, these endpoint IPs can be the nodes' IP addresses.

One pod per node 

$ kubectl get ds -n monitoring
NAME            DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
node-exporter   2         2         2       2            2           <none>          26s

$ kubectl get pods -n monitoring -o wide
NAME                          READY   STATUS    RESTARTS   AGE   IP             NODE         NOMINATED NODE   READINESS GATES
node-exporter-5jpsp           1/1     Running   0          27s   192.168.67.3   cb-mon-m02   <none>           <none>
node-exporter-gksbh           1/1     Running   0          27s   192.168.67.2   cb-mon       <none>           <none>
prometheus-78c96876b5-h98jj   1/1     Running   0          44s   10.244.1.12    cb-mon-m02   <none>           <none>
Enter fullscreen mode Exit fullscreen mode

DESIRED 2 matches the 2 nodes. The node-exporter pods have the nodes' IPs (192.168.67.x), while Prometheus has a pod IP (10.244.x.x). That's hostNetwork at work.

Why headless

  • Normal Service: Provides one virtual IP and load-balances traffic across backend Pods.
  • Headless Service: Does not provide a virtual ClusterIP. DNS can return the IP addresses of individual Pods instead.
  • Prometheus must scrape every node, not whichever pod a normal Service picks. The job in this lab's config asks DNS for the headless Service's name and gets every pod's address back, one target each: 
- job_name: node-exporter
  dns_sd_configs:
  - names: ['node-exporter.monitoring.svc.cluster.local']
    type: A
    port: 9100
    refresh_interval: 5s
Enter fullscreen mode Exit fullscreen mode

What you get

Hundreds of metrics per node:

$ kubectl exec deploy/prometheus -n monitoring -- wget -qO- node-exporter.monitoring.svc:9100/metrics | grep -E '^node_(load1|memory_MemAvailable_bytes|network_receive_bytes_total)' | head -6
node_load1 7.14
node_load15 8.94
node_memory_MemAvailable_bytes 3.824058368e+09
node_network_receive_bytes_total{device="docker0"} 0
node_network_receive_bytes_total{device="eth0"} 5.8914394e+07
node_network_receive_bytes_total{device="lo"} 2.9718904e+07
Enter fullscreen mode Exit fullscreen mode

Common mistakes

  • A Deployment instead of a DaemonSet. You get some number of pods, placed anywhere, not one per node.
  • No tolerations. The control-plane node gets no pod, and its target is missing.
  • A normal (non-headless) Service. DNS returns one virtual IP, so Prometheus sees one target and scrapes a random node each time.
  • Forgetting the hostPath mounts. The numbers come from the container, not the machine, and look plausible but are wrong.

In production

node-exporter is the standard way to watch Linux machines, inside Kubernetes or not. On real nodes its memory and disk metrics feed some of the most important alerts, such as a disk filling up.

PromQL Selectors and a Recording Rule

PromQL, Prometheus's query language, picks the right series and adds them up.

Asking Prometheus a question

Queries go to /api/v1/query. In the labs, run them from inside the Prometheus pod with wget:

$ kubectl exec deploy/prometheus -n monitoring -- wget -qO- --post-data 'query=http_requests_total' localhost:9090/api/v1/query | jq -c '.data.result[] | {metric, value}'

{"metric":{"__name__":"http_requests_total","code":"200","instance":"shop.monitoring.svc:8080","job":"shop","path":"/"},"value":[1792188952.116,"1284"]}
# 1792188952.116 =>  Timestamp - Unix timestamp

{"metric":{"__name__":"http_requests_total","code":"200","instance":"shop.monitoring.svc:8080","job":"shop","path":"/cart"},"value":[1792188952.116,"311"]}

{"metric":{"__name__":"http_requests_total","code":"404","instance":"shop.monitoring.svc:8080","job":"shop","path":"/cart"},"value":[1792188952.116,"9"]}

{"metric":{"__name__":"http_requests_total","code":"200","instance":"shop.monitoring.svc:8080","job":"shop","path":"/checkout"},"value":[1792188952.116,"87"]}

{"metric":{"__name__":"http_requests_total","code":"500","instance":"shop.monitoring.svc:8080","job":"shop","path":"/checkout"},"value":[1792188952.116,"4"]}

{"metric":{"__name__":"http_requests_total","code":"502","instance":"shop.monitoring.svc:8080","job":"shop","path":"/checkout"},"value":[1792188952.116,"2"]}

{"metric":{"__name__":"http_requests_total","code":"503","instance":"shop.monitoring.svc:8080","job":"shop","path":"/pay"},"value":[1792188952.116,"1"]}
Enter fullscreen mode Exit fullscreen mode

Notice two labels the app never wrote: job and instance. Prometheus adds them to every series it scrapes, so you always know where a number came from.

Selectors: picking series by label

Put conditions in braces after the name:

Selector Matches
{path="/checkout"} label equals the value
{code!="200"} label is not the value
{code=~"5.."} label matches a regular expression
{code!~"5.."} label does not match it

5.. means "a 5 and then any two characters", so every 5xx code:

$ kubectl exec deploy/prometheus -n monitoring -- wget -qO- --post-data 'query=http_requests_total{code=~"5.."}' localhost:9090/api/v1/query | jq -c '.data.result[] | {metric, value}'

{"metric":{"__name__":"http_requests_total","code":"500","instance":"shop.monitoring.svc:8080","job":"shop","path":"/checkout"},"value":[1792188952.445,"4"]}

{"metric":{"__name__":"http_requests_total","code":"502","instance":"shop.monitoring.svc:8080","job":"shop","path":"/checkout"},"value":[1792188952.445,"2"]}

{"metric":{"__name__":"http_requests_total","code":"503","instance":"shop.monitoring.svc:8080","job":"shop","path":"/pay"},"value":[1792188952.445,"1"]}
Enter fullscreen mode Exit fullscreen mode

sum: many series into one

sum(...) adds the selected series into a single number with no labels:

$ kubectl exec deploy/prometheus -n monitoring -- wget -qO- --post-data 'query=sum(http_requests_total{code=~"5.."})' localhost:9090/api/v1/query | jq -c '.data.result[] | {metric, value}'

{"metric":{},"value":[1792188968.418,"7"]}
# metric{} => No labels remain after aggregation
Enter fullscreen mode Exit fullscreen mode

4 + 2 + 1 = 7 server errors

Recording rules: save the answer as a metric

  • Typing that query into every dashboard is slow and error-prone. A recording rule makes Prometheus compute it every evaluation_interval and store the result as a new metric. Rules live in a file in the prometheus-rules ConfigMap:
  • A recording rule tells Prometheus to execute a PromQL query periodically and save its result as a new metric. 
groups:
  - name: shop # Name of the rule group
    rules:
      - record: errors:http_requests:total # Name of the new metric Prometheus will create
        expr: sum(http_requests_total{code=~"5.."}) # PromQL expression used to calculate the metric
Enter fullscreen mode Exit fullscreen mode

The name follows the community convention level:metric:operation. Load it the same way as any config change: replace the ConfigMap and restart Prometheus. Then check that Prometheus sees the rule and that it's healthy:

$ kubectl get --raw /api/v1/namespaces/monitoring/services/prometheus:9090/proxy/api/v1/rules | jq -c '.data.groups[] | {name, rules: [.rules[] | {name, query, type, health}]}'
{"name":"shop","rules":[{"name":"errors:http_requests:total","query":"sum(http_requests_total{code=~\"5..\"})","type":"recording","health":"ok"}]}

# Instead of writing this full query every time: sum(http_requests_total{code=~"5.."})
# You can query the recorded metric directly: errors:http_requests:total
$ kubectl exec deploy/prometheus -n monitoring -- wget -qO- --post-data 'query=errors:http_requests:total' localhost:9090/api/v1/query | jq -c '.data.result[] | {metric, value}'
{"metric":{"__name__":"errors:http_requests:total"},"value":[1792188967.563,"7"]}
Enter fullscreen mode Exit fullscreen mode

Explain PromQL with examples

  • Instant vector selector: Returns current values for a metric http_requests_total{job="api", status="500"}
  • Range vector selector: Returns values over a time range (needed for rate/increase functions) http_requests_total[5m]
  • rate(): Per-second rate of a counter over a window rate(http_requests_total[5m])
  • Aggregation with sum: sum(rate(http_requests_total[5m])) by (job)
  • Error rate percentage: sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) * 100
  • P99 latency from histogram: histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

Common mistakes: Using rate() on a Gauge (only valid on Counters). Forgetting that rate() requires a range vector, not an instant vector.

Common mistakes

  • code="5.." with = instead of =~. That looks for the literal text 5.., and matches nothing.
  • Forgetting sum. You get three series, not one number.
  • Writing the number instead of computing it, such as vector(7). It's right today and wrong after the next error.
  • No restart after changing the rules. /api/v1/rules still shows the old groups.

In production

Recording rules keep dashboards fast. A query over thousands of series runs once per interval inside Prometheus, instead of every time anyone opens a graph.

rate(), sum by and Histograms

A counter that only grows tells you nothing

The traffic app counts its requests. Look twice, a minute and a half apart:

$ kubectl exec deploy/prometheus -n monitoring -- wget -qO- --post-data 'query=http_requests_total' localhost:9090/api/v1/query | jq -c '.data.result[] | {metric, value}'

{"metric":{"__name__":"http_requests_total","instance":"traffic.monitoring.svc:8080","job":"traffic","path":"/a"},"value":[1792189002.417,"30"]}

{"metric":{"__name__":"http_requests_total","instance":"traffic.monitoring.svc:8080","job":"traffic","path":"/b"},"value":[1792189002.417,"75"]}

$ kubectl exec deploy/prometheus -n monitoring -- wget -qO- --post-data 'query=http_requests_total' localhost:9090/api/v1/query | jq -c '.data.result[] | {metric, value}'

{"metric":{"__name__":"http_requests_total","instance":"traffic.monitoring.svc:8080","job":"traffic","path":"/a"},"value":[1792189088.298,"202"]}

{"metric":{"__name__":"http_requests_total","instance":"traffic.monitoring.svc:8080","job":"traffic","path":"/b"},"value":[1792189088.298,"505"]}
Enter fullscreen mode Exit fullscreen mode

Bigger numbers, as always. A graph of a counter is a line going up forever. What the on-call engineer actually wants is how fast it's growing.

rate(): growth per second

rate(counter[1m]) looks at the last minute of samples and returns the average increase per second:

$ kubectl exec deploy/prometheus -n monitoring -- wget -qO- --post-data 'query=rate(http_requests_total[1m])' localhost:9090/api/v1/query | jq -c '.data.result[] | {metric, value}'

{"metric":{"instance":"traffic.monitoring.svc:8080","job":"traffic","path":"/a"},"value":[1792189091.637,"1.9591771455988483"]}

{"metric":{"instance":"traffic.monitoring.svc:8080","job":"traffic","path":"/b"},"value":[1792189091.637,"4.8979428639971205"]}
Enter fullscreen mode Exit fullscreen mode

About 2 requests per second on /a and 5 on /b.
The window [1m] must cover several scrapes. A shorter window reacts faster but is noisier.

sum by: choosing which labels survive

sum alone adds everything into one number and drops every label:

$ kubectl exec deploy/prometheus -n monitoring -- wget -qO- --post-data 'query=sum(rate(http_requests_total[1m]))' localhost:9090/api/v1/query | jq -c '.data.result[] | {metric, value}'

{"metric":{},"value":[1792189092.410,"6.872727272727273"]}
Enter fullscreen mode Exit fullscreen mode

sum by (path) adds up per path and keeps only the path label:

$ kubectl exec deploy/prometheus -n monitoring -- wget -qO- --post-data 'query=sum by (path) (rate(http_requests_total[1m]))' localhost:9090/api/v1/query | jq -c '.data.result[] | {metric, value}'

{"metric":{"path":"/a"},"value":[1792189093.737,"1.9636363636363636"]}

{"metric":{"path":"/b"},"value":[1792189093.737,"4.909090909090909"]}
Enter fullscreen mode Exit fullscreen mode

The order matters: rate first, on each raw counter, then sum. Saved as a recording rule (lesson 5), it becomes a ready-made metric:

groups:
  - name: traffic
    rules:
      - record: path:http_requests:rate1m
        expr: sum by (path) (rate(http_requests_total[1m]))
Enter fullscreen mode Exit fullscreen mode

Histograms: how long requests take

  • Histogram, used to measure the distribution of values, such as how long HTTP requests take to complete.
  • Unlike a counter that counts requests, a histogram helps you understand whether your requests are fast or slow.
  • An average hides the slow ones. A histogram counts requests into buckets by duration. Each bucket counts everything less than or equal to (le means Less Than or Equal To) its limit: 
# TYPE http_request_duration_seconds histogram

http_request_duration_seconds_bucket{le="0.1"} 50
http_request_duration_seconds_bucket{le="0.5"} 90
http_request_duration_seconds_bucket{le="1"} 100
http_request_duration_seconds_bucket{le="+Inf"} 100

http_request_duration_seconds_sum 31.5
http_request_duration_seconds_count 100
Enter fullscreen mode Exit fullscreen mode

Imagine your application has processed 100 HTTP requests, each with a different response time. The histogram groups those requests by duration:

  • http_request_duration_seconds_bucket{le="0.1"} 50: Means 50 requests completed in 0.1 seconds (100 milliseconds) or less.
  • http_request_duration_seconds_bucket{le="0.5"} 90: Meaning 90 requests completed in 500 ms or less.
  • http_request_duration_seconds_bucket{le="1"} 100: Meaning 100 requests completed in 1 second or less.
  • http_request_duration_seconds_bucket{le="+Inf"} 100: Meaning The histogram has recorded 100 requests regardless of their duration.
  • http_request_duration_seconds_sum 31.5: This means the total combined duration of all 100 requests is 31.5 seconds - means the sum of the durations of all 100 requests, as if their execution times sequentially.
  • http_request_duration_seconds_count 100: Meaning Your application has recorded 100 completed request durations. 
$ kubectl exec deploy/prometheus -n monitoring -- wget -qO- --post-data 'query=histogram_quantile(0.95,sum(http_request_duration_seconds_bucket)by(le))' localhost:9090/api/v1/query | jq -c '.data.result[] | {metric, value}'

{"metric":{},"value":[1792189097.933,"0.75"]}
# metric{} => No labels remain after aggregation
Enter fullscreen mode Exit fullscreen mode

In real use you'd wrap the buckets in rate() to get the percentile of recent traffic.

$ kubectl exec deploy/prometheus -n monitoring -- wget -qO- --post-data 'query=histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[1m])))' localhost:9090/api/v1/query | jq -c '.data.result[] | {metric, value}'

{"metric":{},"value":[1792189101.138,"0.35"]}
# P95 latency = 350 milliseconds.

{"metric":{},"value":[1792189101.138,"NaN"]}
# No new requests during the last minute
Enter fullscreen mode Exit fullscreen mode
  • This command calculates the 95th percentile (P95) of HTTP request durations over the last 1 minute using a Prometheus histogram.
  • rate() determines which time period you're analyzing, sum by (le) combines histogram data while preserving bucket boundaries, and histogram_quantile(0.95, ...) estimates the P95 latency.

Common mistakes

  • rate on a gauge. Rates are for counters only. A gauge already says "how much right now".
  • sum before rate. rate has to see each raw counter, so it can spot when one of them resets to 0 after a restart. Add them up first and the reset is hidden, and the rate comes out wrong.
  • sum by (instance) or no by at all when the task asks per path. Check which labels the result carries.
  • Reading NaN as an error. It means there was nothing to divide, such as a histogram with no new requests in the window.

In production

Teams set their promises on percentiles, not averages: "95% of checkout requests under 300 ms". A rate per path plus a 95th percentile from a histogram is the core of most service dashboards.

Service Discovery Without RBAC

  • Service Discovery in Prometheus is the process of automatically finding applications, Pods, Nodes, or Services that Prometheus should monitor, instead of manually configuring their IP addresses.
  • So far every target was typed into the config: shop.monitoring.svc:8080. Fine for one app. With fifty apps and pods that move between nodes, nobody can keep that list right. Service discovery lets Prometheus ask the Kubernetes API which pods exist right now? and scrape them as they come and go.
  1. Without Service Discovery (Static Configuration): Imagine you have three applications running in Kubernetes: 
Kubernetes Cluster
│
├── Shop Pod       10.244.1.10:8080
├── Payment Pod    10.244.1.11:8080
└── Orders Pod     10.244.1.12:8080
Enter fullscreen mode Exit fullscreen mode

Without service discovery, you manually define each target in prometheus.yml

scrape_configs:
  - job_name: applications
    static_configs:
      - targets:
          - 10.244.1.10:8080
          - 10.244.1.11:8080
          - 10.244.1.12:8080
Enter fullscreen mode Exit fullscreen mode
  1. With Service Discovery: Instead of manually providing Pod IPs, you configure Prometheus to communicate with the Kubernetes API Server. 
                  Kubernetes Cluster
                           |
                           v
              +-------------------------+
              |  Kubernetes API Server  |
              |                         |
              |  Knows existing Pods,   |
              |  Nodes, and Services    |
              +-------------------------+
                           ^
                           |
                    List / Watch
                           |
              +-------------------------+
              |       Prometheus        |
              |    Service Discovery    |
              +-------------------------+
                           |
                Discover target IPs
                           |
            +--------------+--------------+
            |              |              |
            v              v              v
       +----------+   +----------+   +----------+
       | Shop Pod |   | Pay Pod  |   | Order Pod|
       | /metrics |   | /metrics |   | /metrics |
       +----------+   +----------+   +----------+
Enter fullscreen mode Exit fullscreen mode
  • Prometheus automatically discovers the targets and scrapes their metrics.
  • When a Pod is replaced, Prometheus can discover the replacement without manually changing prometheus.yml.

Discovering annotated pods

This job, already in the lab's config, finds every pod and keeps only the ones that ask to be scraped:

- job_name: kubernetes-pods
  kubernetes_sd_configs:
    - role: pod # This tells Prometheus: Use the Kubernetes API to discover Pods.

  relabel_configs:
  # Rule 1: Keep only Pods enabled for scraping
  - source_labels: # reads a label discovered from Kubernetes.
      - __meta_kubernetes_pod_annotation_prometheus_io_scrape # represents the Pod annotation prometheus.io/scrape.
    action: keep # keeps targets only if they match the regex.
    regex: "true" # requires the annotation value to equal "true".
    # Pods with annotations like the below will be scraped by Prometheus.
    # annotations:
    #   prometheus.io/scrape: "true"

  # Rule 2: Build the scrape address
  - source_labels:
      - __meta_kubernetes_pod_ip
      - __meta_kubernetes_pod_annotation_prometheus_io_port
    regex: (.+);(.+) # (.+) captures 10.244.1.10. , and Prometheus joins multiple source labels using ; by default.
    replacement: $1:$2 # will be 10.244.1.10:8080
    target_label: __address__ # Store the result in __address__ , __address__ is the internal Prometheus label that specifies the address to scrape.
    # 10.244.1.10 + 8080  =>>  __address__ = 10.244.1.10:8080 then Prometheus scrapes the target http://10.244.1.10:8080/metrics

  # Rule 3: Add namespace label
  - source_labels:
      - __meta_kubernetes_namespace
    target_label: namespace

  # Rule 4: Add Pod name label
  - source_labels:
      - __meta_kubernetes_pod_name
    target_label: pod
Enter fullscreen mode Exit fullscreen mode
  • The first rule keeps only pods annotated prometheus.io/scrape: "true".
  • The second builds the address from the pod IP and port.
  • The last two copy the namespace and pod name onto every series, so you can tell pods apart.

Everything configured, nothing found

After the switch, the only target left is Prometheus scraping itself:

$ kubectl get --raw /api/v1/namespaces/monitoring/services/prometheus:9090/proxy/api/v1/targets | jq -c '.data.activeTargets[] | {job: .labels.job, health, scrapeUrl, lastError}'
{"job":"prometheus","health":"up","scrapeUrl":"http://localhost:9090/metrics","lastError":""}
Enter fullscreen mode Exit fullscreen mode

No down target to read. The discovery itself is failing, and that shows up in the logs:

$ kubectl logs deploy/prometheus -n monitoring | grep forbidden | head -3
time=2026-10-16T22:18:54.070Z level=ERROR source=reflector.go:204 msg="Failed to watch" component=k8s_client_runtime logger=UnhandledError err="failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:monitoring:prometheus\" cannot list resource
Enter fullscreen mode Exit fullscreen mode

forbidden is the Kubernetes API refusing. Prometheus runs as the ServiceAccount prometheus, and nobody gave it the right to list pods. Confirm it without guessing:

$ kubectl auth can-i list pods --all-namespaces --as=system:serviceaccount:monitoring:prometheus
no
Enter fullscreen mode Exit fullscreen mode

The minimum access

Discovery needs to read, never to change. A ClusterRole lists what's allowed across the whole cluster, and a ClusterRoleBinding gives it to the ServiceAccount:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus
rules:
  - apiGroups: [""]
    resources: [nodes, nodes/metrics, services, endpoints, pods]
    verbs: [get, list, watch]
  - nonResourceURLs: [/metrics]
    verbs: [get]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
  - kind: ServiceAccount
    name: prometheus
    namespace: monitoring
Enter fullscreen mode Exit fullscreen mode

Apply it, restart Prometheus, and check both what it can do and what it can't:

$ kubectl auth can-i list pods --all-namespaces --as=system:serviceaccount:monitoring:prometheus
yes

$ kubectl auth can-i get secrets --all-namespaces --as=system:serviceaccount:monitoring:prometheus
no
Enter fullscreen mode Exit fullscreen mode

Now the targets appear by themselves:

$ kubectl get --raw /api/v1/namespaces/monitoring/services/prometheus:9090/proxy/api/v1/targets | jq -c '.data.activeTargets[] | {job: .labels.job, health, scrapeUrl, lastError}'
{"job":"kubernetes-nodes","health":"up","scrapeUrl":"https://192.168.67.2:10250/metrics","lastError":""}
{"job":"kubernetes-nodes","health":"up","scrapeUrl":"https://192.168.67.3:10250/metrics","lastError":""}
{"job":"kubernetes-pods","health":"up","scrapeUrl":"http://10.244.1.19:8080/metrics","lastError":""}
{"job":"prometheus","health":"up","scrapeUrl":"http://localhost:9090/metrics","lastError":""}
Enter fullscreen mode Exit fullscreen mode

Common mistakes

  • Binding cluster-admin. It works, and it lets anyone who takes over the Prometheus pod read every Secret in the cluster. Give exactly the verbs on the resources it needs.
  • A Role and RoleBinding instead of the Cluster versions. They only cover one namespace, and nodes don't belong to any namespace.
  • Binding the wrong ServiceAccount, such as default. Check with kubectl auth can-i --as=system:serviceaccount:NAMESPACE:NAME.
  • Looking only at the targets page. A discovery failure produces no target at all. The reason is in the logs.

In production

Every monitoring stack on Kubernetes, including the Helm charts, ships this same read-only ClusterRole. Knowing its shape lets you spot when a chart asks for more than it should.

kube-state-metrics: What Kubernetes Thinks

  • kube-state-metrics (KSM) is a monitoring component that reads the state of Kubernetes objects from the Kubernetes API Server and exposes that information as Prometheus metrics.
  • Service Discovery: Finds which Pods, Nodes, or Services Prometheus should scrape.
  • kube-state-metrics: Exposes the state of Kubernetes objects as metrics that Prometheus can collect. ## Something is not fully up
  • A colleague says one of the apps "isn't fully up". You could run kubectl get in every namespace and read the columns. Or you could ask Prometheus one question that covers the whole cluster at once. For that, Prometheus needs numbers about Kubernetes objects: how many replicas a Deployment wants, how many are available, which pods are stuck.
  • Node-exporter measures machines. Apps measure themselves. kube-state-metrics measures what the Kubernetes API knows. It watches Deployments, pods, nodes and more, and turns their status into metrics.

kube-state-metrics vs node-exporter vs application metrics

Component What it monitors Example metric
node-exporter Node CPU, memory, disk, network node_memory_MemAvailable_bytes
kube-state-metrics Kubernetes object state kube_deployment_spec_replicas
Application metrics Application requests, errors, latency http_requests_total
Kubelet / cAdvisor Container CPU and memory usage container_cpu_usage_seconds_total

For example, suppose your Shop application is slow:

  • node-exporter: Is the node running out of memory?
  • kube-state-metrics: Are all three Shop replicas available?
  • Application metrics: Are HTTP requests taking too long?
  • Kubelet/cAdvisor: Is the Shop container using too much CPU?

How kube-state-metrics works 

          +-----------------------+
          | Kubernetes API Server |
          |                       |
          | Deployments           |
          | Pods                  |
          | Nodes                 |
          | StatefulSets          |
          +-----------------------+
                      |
                      | List / Watch
                      v
          +-----------------------+
          |   kube-state-metrics  |
          |                       |
          | Converts object state |
          | into Prometheus       |
          | metrics               |
          +-----------------------+
                      |
                      | Exposes /metrics
                      | Port 8080
                      v
          +-----------------------+
          |      Prometheus       |
          |                       |
          | Scrapes and stores    |
          | metrics over time     |
          +-----------------------+
                      |
                      v
          +-----------------------+
          |        Grafana        |
          |  Dashboards / Alerts  |
          +-----------------------+
Enter fullscreen mode Exit fullscreen mode

KSM does not directly monitor the application or measure CPU usage. It reports information Kubernetes already knows.
Imagine you have a Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: shop
  namespace: apps
spec:
  replicas: 3
  selector:
    matchLabels:
      app: shop
  template:
    metadata:
      labels:
        app: shop
    spec:
      containers:
        - name: shop
          image: nginx:stable
Enter fullscreen mode Exit fullscreen mode

You requested 3 replicas, but only 2 are available.
KSM exposes metrics such as:

kube_deployment_spec_replicas{deployment="shop",namespace="apps"} 3
# Desired replicas

kube_deployment_status_replicas_available{deployment="shop",namespace="apps"} 2
# Available replicas

kube_deployment_status_replicas_unavailable{deployment="shop",namespace="apps"} 1
# Unavailable replicas
Enter fullscreen mode Exit fullscreen mode

Where the numbers come from

Usually, KSM runs as a Deployment with a Service exposing port 8080:

kubectl get --raw \
  /api/v1/namespaces/monitoring/services/kube-state-metrics:8080/proxy/metrics

# Or query Prometheus: 
kube_pod_status_phase{phase="Pending"} == 1
Enter fullscreen mode Exit fullscreen mode

The labels say which object a value belongs to. Here's how many replicas each Deployment wants:

$ kubectl exec deploy/prometheus -n monitoring -- wget -qO- --post-data 'query=kube_deployment_spec_replicas' localhost:9090/api/v1/query | jq -c '.data.result[] | {metric, value}'

{"metric":{"__name__":"kube_deployment_spec_replicas","deployment":"orders","instance":"kube-state-metrics.monitoring.svc:8080","job":"kube-state-metrics","namespace":"apps"},"value":[1792189275.257,"2"]}

{"metric":{"__name__":"kube_deployment_spec_replicas","deployment":"search","instance":"kube-state-metrics.monitoring.svc:8080","job":"kube-state-metrics","namespace":"apps"},"value":[1792189275.257,"1"]}

{"metric":{"__name__":"kube_deployment_spec_replicas","deployment":"coredns","instance":"kube-state-metrics.monitoring.svc:8080","job":"kube-state-metrics","namespace":"kube-system"},"value":[1792189275.257,"1"]}

{"metric":{"__name__":"kube_deployment_spec_replicas","deployment":"kube-state-metrics","instance":"kube-state-metrics.monitoring.svc:8080","job":"kube-state-metrics","namespace":"monitoring"},"value":[1792189275.257,"1"]}

{"metric":{"__name__":"kube_deployment_spec_replicas","deployment":"prometheus","instance":"kube-state-metrics.monitoring.svc:8080","job":"kube-state-metrics","namespace":"monitoring"},"value":[1792189275.257,"1"]}

{"metric":{"__name__":"kube_deployment_spec_replicas","deployment":"catalog","instance":"kube-state-metrics.monitoring.svc:8080","job":"kube-state-metrics","namespace":"apps"},"value":[1792189275.257,"2"]}
Enter fullscreen mode Exit fullscreen mode

Every Deployment in every namespace, in one query.

A few metrics worth knowing

Metric Tells you
kube_deployment_spec_replicas Replicas the Deployment asks for
kube_deployment_status_replicas_unavailable Replicas that should exist but aren't ready
kube_pod_status_phase One series per pod and phase: 1 for the pod's current phase
kube_node_info One series per node, with its versions in the labels

The labels carry information too, not only the value:

$ kubectl exec deploy/prometheus -n monitoring -- wget -qO- --post-data 'query=kube_node_info' localhost:9090/api/v1/query | jq -c '.data.result[] | .metric | {node, kernel_version, kubelet_version}'

{"node":"cb-mon","kernel_version":"6.18.54.1-microsoft-standard-WSL2","kubelet_version":"v1.29.6"}
{"node":"cb-mon-m02","kernel_version":"6.18.54.1-microsoft-standard-WSL2","kubelet_version":"v1.29.6"}
Enter fullscreen mode Exit fullscreen mode

Filtering by value

A comparison in PromQL keeps only the series where it's true. kube_deployment_status_replicas_unavailable > 0 returns just the Deployments with a problem, out of the whole cluster. You can narrow it further with a selector, such as {namespace="apps"}.

Once Prometheus points at the Deployment, the investigation goes back to plain Kubernetes: kubectl get pods for the status, and kubectl describe pod for the Events at the bottom, which say why a pod can't start.

Common mistakes

  • Expecting kube-state-metrics to report CPU or memory. It reports object state only. Usage comes from the kubelet and node-exporter.
  • Forgetting the namespace label. Two namespaces can each have a Deployment called web.
  • Fixing the metric instead of the cause. The number drops to 0 only when the pods are really available. Fix the Deployment, then watch the metric follow.

In production

Alerts such as "a Deployment has had unavailable replicas for 10 minutes" or "a pod keeps restarting" are built on kube-state-metrics. It turns kubectl get into history you can graph.

A Grafana Data Source That Cannot Connect

Graphs for humans

Prometheus answers questions in JSON. People want pictures. Grafana draws graphs and dashboards, and it doesn't store any numbers itself: it asks a data source, here Prometheus, every time a panel loads. If the data source is wrong, every panel says "No data".

Grafana is up

In the lab, Grafana runs as a Deployment in monitoring, on port 3000. Its API answers from inside the pod:

$ kubectl exec deploy/grafana -n monitoring -- wget -qO- localhost:3000/api/health
{
  "database": "ok",
  "version": "12.4.13",
  "commit": "6367d911d57309d1da370872b8fd76237b5a0ffa"
}
Enter fullscreen mode Exit fullscreen mode

/api/health only says Grafana itself is running. Whether it can reach Prometheus is a different question.

Provisioned data sources

You can add a data source by clicking in Grafana's UI, but then it lives only in Grafana's database. If the pod is recreated, or a colleague builds a second Grafana, it's gone. Provisioning puts it in a file instead. Grafana reads every file in /etc/grafana/provisioning/datasources when it starts. In this course that folder is a ConfigMap, grafana-datasources, with a key datasources.yaml:

apiVersion: 1
datasources:
  - name: Prometheus
    uid: prometheus
    type: prometheus
    access: proxy
    url: http://prometheus.monitoring.svc:9090
    isDefault: true
Enter fullscreen mode Exit fullscreen mode
  • url is where Grafana's server sends queries. With access: proxy, it's Grafana's pod that connects, so it must be a name that resolves inside the cluster: SERVICE.NAMESPACE.svc:PORT.
  • uid is the stable ID you use in the API and in dashboards.

Asking whether it works

List what Grafana loaded, and test the connection:

$ kubectl exec deploy/grafana -n monitoring -- wget -qO- localhost:3000/api/datasources | jq -c '.[] | {name, type, url, uid, isDefault, readOnly}'
{"name":"Prometheus","type":"prometheus","url":"http://prometheus.monitoring.svc:9090","uid":"prometheus","isDefault":true,"readOnly":true}

$ kubectl exec deploy/grafana -n monitoring -- wget -qO- localhost:3000/api/datasources/uid/prometheus/health
{"details":{"application":"Prometheus","features":{"rulerApiEnabled":false}},"message":"Successfully queried the Prometheus API.","status":"OK"}
Enter fullscreen mode Exit fullscreen mode

That's a healthy data source. readOnly: true is how you know it was provisioned and not clicked in. When the connection is broken, the health check fails instead:

$ kubectl exec deploy/grafana -n monitoring -- wget -qO- localhost:3000/api/datasources/uid/prometheus/health
wget: server returned error: HTTP/1.1 400 Bad Request
command terminated with exit code 1
Enter fullscreen mode Exit fullscreen mode

Then compare the url Grafana uses with what really exists: kubectl get svc -n monitoring shows each Service's name and port, and the namespace is part of the DNS name. The fix goes in the ConfigMap, and Grafana reads it only when it starts, so restart the Deployment afterwards.

Common mistakes

  • Adding a second data source by hand in the UI. The broken provisioned one is still there, dashboards may still use it, and the fix disappears with the pod.
  • The wrong namespace in the DNS name. api.default.svc and api.monitoring.svc are two different Services, and maybe only one of them exists.
  • Using localhost:9090. Inside Grafana's pod, localhost is Grafana, not Prometheus.
  • Editing the ConfigMap without a restart. Grafana keeps the data source it loaded at startup.

In production

Teams keep data sources and dashboards in git and provision them, so a new Grafana comes up already connected. Clicking things into a production Grafana is how setups drift and get lost.

Dashboards That Live in a ConfigMap

The dashboard that disappeared

Last month someone spent an afternoon building the Shop Overview dashboard in Grafana's UI. Then the Grafana pod restarted, and the dashboard was gone. It lived only in the pod's database. The fix is the same idea as lesson 9: keep the dashboard as a file, and let Grafana load it at startup.

A dashboard is a JSON file

Every Grafana dashboard can be exported as JSON. The lab's copy is already in your home folder:

$ head -12 ~/shop-dashboard.json
{
  "uid": "shop-overview",
  "title": "Shop Overview",
  "schemaVersion": 39,
  "version": 1,
  "time": {"from": "now-15m", "to": "now"},
  "panels": [
    {"id": 1, "type": "timeseries", "title": "Requests per second by path",
     "datasource": {"type": "prometheus", "uid": "prometheus"},
     "gridPos": {"h": 8, "w": 12, "x": 0, "y": 0},
     "targets": [{"refId": "A", "expr": "sum by (path) (rate(http_requests_total[1m]))"}]},
    {"id": 2, "type": "stat", "title": "Server errors (5xx)",
Enter fullscreen mode Exit fullscreen mode

You know every part of that already: each panel's expr is PromQL (lesson 6), and the datasource points at the data source by its uid (lesson 9).

How Grafana finds dashboard files

A dashboard provider tells Grafana which folder to load dashboards from. In this lab Grafana already has one:

apiVersion: 1
providers:
- name: hire-spec
  type: file
  disableDeletion: true
  updateIntervalSeconds: 5
  options:
    path: /var/lib/grafana/dashboards
Enter fullscreen mode Exit fullscreen mode

That folder is a ConfigMap called grafana-dashboards, mounted into the Grafana pod. It's marked optional, so Grafana starts even before the ConfigMap exists. Right now it doesn't exist, and Grafana has no dashboards:

$ kubectl exec deploy/grafana -n monitoring -- wget -qO- 'localhost:3000/api/search?type=dash-db'
[]
Enter fullscreen mode Exit fullscreen mode

Shipping the file

Create the ConfigMap straight from the file, so the JSON goes in untouched. The key is the file name, and it must end in .json. Then restart Grafana so the pod gets a fresh mount and loads it:

kubectl create configmap grafana-dashboards -n monitoring --from-file=shop-dashboard.json
kubectl rollout restart deployment/grafana -n monitoring
Enter fullscreen mode Exit fullscreen mode

The dashboard is back, and Grafana knows where it came from:

$ kubectl exec deploy/grafana -n monitoring -- wget -qO- 'localhost:3000/api/search?type=dash-db' | jq -c '.[] | {uid, title, type}'
{"uid":"shop-overview","title":"Shop Overview","type":"dash-db"}

$ kubectl exec deploy/grafana -n monitoring -- wget -qO- localhost:3000/api/dashboards/uid/shop-overview | jq -c '.meta | {provisioned, provisionedExternalId, folderTitle}'
{"provisioned":true,"provisionedExternalId":"shop-dashboard.json","folderTitle":"General"}
Enter fullscreen mode Exit fullscreen mode

provisioned: true is the proof that it came from a file. It survives any restart, and Grafana refuses to save UI edits over it, which is what you want.

Common mistakes

  • Importing the JSON through the UI or the API. The dashboard shows up, but provisioned is false, and it's gone with the next pod.
  • A key without .json, such as --from-file=dashboard=shop-dashboard.json. The provider only loads .json files.
  • Editing the JSON while copying it in. Keep the file as it is. To change a dashboard, change the file in git and ship it again.
  • No restart. Grafana is still running with the mount it had before the ConfigMap existed.

In production

"Dashboards as code" is standard practice. The JSON sits in git next to the app, goes through review, and is deployed by the same pipeline. The Grafana Helm chart can even pick up dashboard ConfigMaps by label, without a restart.

Alertmanager: Routing an Alert to the Right Receiver

Alertmanager receives alerts from Prometheus and decides where to send notifications based on their labels, such as severity, team, or service.

             Kubernetes Cluster
                     |
                     v
        +-------------------------+
        |       Prometheus        |
        |                         |
        | Evaluates alert rules   |
        |                         |
        | up{job="shop"} == 0      |
        +-------------------------+
                     |
                     | Alert becomes FIRING
                     | HTTP POST
                     v
        +-------------------------+
        |       Alertmanager      |
        |                         |
        |  Group alerts           |
        |  Match severity labels  |
        |  Route notifications    |
        +-------------------------+
                     |
             +-------+-------+
             |               |
             v               v
       +-----------+   +-----------+
       | Critical  |   | Warning   |
       | Receiver  |   | Receiver  |
       +-----------+   +-----------+
             |               |
             v               v
         PagerDuty          Slack
         (example)         (example)
Enter fullscreen mode Exit fullscreen mode

We'll configure the following resources:

  • alertmanager.yml - Routing rules and notification receivers.
  • alertmanager-config - Kubernetes ConfigMap containing the configuration.
  • alertmanager - Kubernetes Deployment and Service.
  • prometheus.yml - Connection from Prometheus to Alertmanager.
  • prometheus-rules - Alerting rules defining when alerts fire.

Create alertmanager.yml:

global:
  resolve_timeout: 5m

route:
  receiver: default-receiver

  group_by:
    - alertname
    - namespace

  group_wait: 10s
  group_interval: 5m
  repeat_interval: 1h

  routes:
    - receiver: critical-receiver
      matchers:
        - severity="critical"

    - receiver: warning-receiver
      matchers:
        - severity="warning"

receivers:
  - name: default-receiver
    webhook_configs:
      - url: http://default-webhook.monitoring.svc:8080/alerts

  - name: critical-receiver
    webhook_configs:
      - url: http://critical-webhook.monitoring.svc:8080/alerts

  - name: warning-receiver
    webhook_configs:
      - url: http://warning-webhook.monitoring.svc:8080/alerts
Enter fullscreen mode Exit fullscreen mode

Create the Alertmanager ConfigMap:

kubectl create configmap alertmanager-config \
  -n monitoring \
  --from-file=alertmanager.yml
Enter fullscreen mode Exit fullscreen mode

Create alertmanager-deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: alertmanager
  namespace: monitoring

spec:
  replicas: 1

  selector:
    matchLabels:
      app: alertmanager

  template:
    metadata:
      labels:
        app: alertmanager

    spec:
      containers:
        - name: alertmanager
          image: prom/alertmanager:latest

          args:
            - --config.file=/etc/alertmanager/alertmanager.yml
            - --storage.path=/alertmanager

          ports:
            - containerPort: 9093

          volumeMounts:
            - name: config
              mountPath: /etc/alertmanager
              readOnly: true

            - name: data
              mountPath: /alertmanager

      volumes:
        - name: config
          configMap:
            name: alertmanager-config

        - name: data
          emptyDir: {}

---
apiVersion: v1
kind: Service
metadata:
  name: alertmanager
  namespace: monitoring

spec:
  type: ClusterIP

  selector:
    app: alertmanager

  ports:
    - port: 9093
      targetPort: 9093
Enter fullscreen mode Exit fullscreen mode 
kubectl apply -f alertmanager-deployment.yaml
Enter fullscreen mode Exit fullscreen mode

Connect Prometheus to Alertmanager: Edit your existing prometheus.yml and add the alerting section

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - /etc/rules/*.yml

alerting: # This tells Prometheus where to send firing and resolved alerts
  alertmanagers:
    - static_configs:
        - targets:
            - alertmanager.monitoring.svc:9093

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets:
          - localhost:9090

  - job_name: shop
    static_configs:
      - targets:
          - shop.monitoring.svc:8080
Enter fullscreen mode Exit fullscreen mode 
# Update the existing ConfigMap:
kubectl create configmap prometheus-config \
  -n monitoring \
  --from-file=prometheus.yml \
  --dry-run=client -o yaml | kubectl apply -f -

# Restart Prometheus to load the configuration:
kubectl rollout restart deployment/prometheus -n monitoring
Enter fullscreen mode Exit fullscreen mode

Create a Prometheus alerting rule: Create shop-alerts.yml

groups:
  - name: shop-alerts
    rules:
      - alert: ShopDown
        expr: up{job="shop"} == 0
        for: 1m

        labels:
          severity: critical
          team: backend

        annotations:
          summary: "Shop application target is down"
          description: "Prometheus cannot scrape the Shop target."
Enter fullscreen mode Exit fullscreen mode

This rule means:

  1. Prometheus evaluates up{job="shop"} == 0
  2. If the condition stays true for one minute, the alert becomes firing.
  3. Prometheus sends the alert to Alertmanager.
  4. Alertmanager detects severity="critical"
  5. Alertmanager routes the notification to critical-receiver 
# Update the rules ConfigMap:
kubectl create configmap prometheus-rules \
  -n monitoring \
  --from-file=shop-alerts.yml \
  --dry-run=client -o yaml | kubectl apply -f -

# Restart Prometheus:
kubectl rollout restart deployment/prometheus -n monitoring

# Check Alertmanager's readiness:
kubectl get --raw \
  /api/v1/namespaces/monitoring/services/alertmanager:9093/proxy/-/ready

# Check that Prometheus can see Alertmanager:
kubectl get --raw \
  /api/v1/namespaces/monitoring/services/prometheus:9090/proxy/api/v1/alertmanagers \
  | jq '.data.activeAlertmanagers'

# Check your Prometheus alerting rules:
kubectl get --raw \
  /api/v1/namespaces/monitoring/services/prometheus:9090/proxy/api/v1/rules \
  | jq '.data.groups[].rules[] | select(.type=="alerting")'

# You can also open Alertmanager locally:
kubectl port-forward -n monitoring svc/alertmanager 9093:9093
# http://localhost:9093

# Test alert routing
curl -X POST http://localhost:9093/api/v2/alerts \
  -H "Content-Type: application/json" \
  -d '[
    {
      "labels": {
        "alertname": "TestCriticalAlert",
        "severity": "critical",
        "team": "backend"
      },
      "annotations": {
        "summary": "Testing critical alert routing"
      },
      "startsAt": "2026-11-06T12:00:00Z",
      "endsAt": "2026-11-07T12:00:00Z"
    }
  ]'
Enter fullscreen mode Exit fullscreen mode

Top comments (0)