DEV Community

loading...

Discover pods by label in Prometheus

ahf90 profile image Alex Friedman ・2 min read

What are we doing?

Kubernetes SD configurations allow retrieving scrape targets from Kubernetes' REST API in order to stay synchronized with the cluster state.

I created an image of a Python program that exports metrics using the the Prometheus Python client. Setting the environment variable EXPORTER_PORT will publish metrics to that port.

I deployed this container in pods with labels foo and bar and sent metrics to port 9000 and 8000 respectively. You can see in the below Prometheus config that I target them separately by their label name.

Why would you need to do this?

I found a case where I wanted different scrape configurations per pod in my cluster. Usually, I can set the kubernetes_sd_configs role to service, which discovers a target for each service port for each service (see the docs). However, in this case I had different collection requirements for the pods inside my k8s abstractions.

Example config files

Foo Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: foo-deployment
  labels:
    app: foo
spec:
  selector:
    matchLabels:
      app: foo
  replicas: 2
  template:
    metadata:
      labels:
        app: foo
    spec:
      containers:
      - name: python-prometheus-exporter-example
        image: ahf90/python-prometheus-exporter-example
        imagePullPolicy: Always
        ports:
          - containerPort: 9000
        env:
          - name: EXPORTER_PORT
            value: '9000'

Bar Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: bar-deployment
  labels:
    app: bar
spec:
  selector:
    matchLabels:
      app: bar
  replicas: 2
  template:
    metadata:
      labels:
        app: bar
    spec:
      containers:
      - name: python-prometheus-exporter-example
        image: ahf90/python-prometheus-exporter-example
        imagePullPolicy: Always
        ports:
          - containerPort: 8000
        env:
          - name: EXPORTER_PORT
            value: '8000'

Prometheus config.yml

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
    - targets: ['localhost:9090']
  - job_name: 'foo-targets'
    scrape_interval: 10s
    metrics_path: '/'
    kubernetes_sd_configs:
    - role: pod
      namespaces:
        names:
        - default
      selectors:
      - role: "pod"
        label: "app=foo"
  - job_name: 'bar-targets'
    scrape_interval: 5s
    metrics_path: '/'
    kubernetes_sd_configs:
      - role: pod
        namespaces:
          names:
            - default
        selectors:
          - role: "pod"
            label: "app=bar"

Prometheus targets page

Discussion (0)

Forem Open with the Forem app