Discover pods by label in Prometheus

What are we doing?

Kubernetes SD configurations allow retrieving scrape targets from Kubernetes' REST API in order to stay synchronized with the cluster state.

I created an image of a Python program that exports metrics using the the Prometheus Python client. Setting the environment variable EXPORTER_PORT will publish metrics to that port.

I deployed this container in pods with labels foo and bar and sent metrics to port 9000 and 8000 respectively. You can see in the below Prometheus config that I target them separately by their label name.

Why would you need to do this?

I found a case where I wanted different scrape configurations per pod in my cluster. Usually, I can set the kubernetes_sd_configs role to service, which discovers a target for each service port for each service (see the docs). However, in this case I had different collection requirements for the pods inside my k8s abstractions.

Example config files

Foo Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: foo-deployment
  labels:
    app: foo
spec:
  selector:
    matchLabels:
      app: foo
  replicas: 2
  template:
    metadata:
      labels:
        app: foo
    spec:
      containers:
      - name: python-prometheus-exporter-example
        image: ahf90/python-prometheus-exporter-example
        imagePullPolicy: Always
        ports:
          - containerPort: 9000
        env:
          - name: EXPORTER_PORT
            value: '9000'

Bar Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: bar-deployment
  labels:
    app: bar
spec:
  selector:
    matchLabels:
      app: bar
  replicas: 2
  template:
    metadata:
      labels:
        app: bar
    spec:
      containers:
      - name: python-prometheus-exporter-example
        image: ahf90/python-prometheus-exporter-example
        imagePullPolicy: Always
        ports:
          - containerPort: 8000
        env:
          - name: EXPORTER_PORT
            value: '8000'

Prometheus config.yml

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
    - targets: ['localhost:9090']
  - job_name: 'foo-targets'
    scrape_interval: 10s
    metrics_path: '/'
    kubernetes_sd_configs:
    - role: pod
      namespaces:
        names:
        - default
      selectors:
      - role: "pod"
        label: "app=foo"
  - job_name: 'bar-targets'
    scrape_interval: 5s
    metrics_path: '/'
    kubernetes_sd_configs:
      - role: pod
        namespaces:
          names:
            - default
        selectors:
          - role: "pod"
            label: "app=bar"