DEV Community

Alex Friedman
Alex Friedman

Posted on

8 2

Discover pods by label in Prometheus

What are we doing?

Kubernetes SD configurations allow retrieving scrape targets from Kubernetes' REST API in order to stay synchronized with the cluster state.

I created an image of a Python program that exports metrics using the the Prometheus Python client. Setting the environment variable EXPORTER_PORT will publish metrics to that port.

I deployed this container in pods with labels foo and bar and sent metrics to port 9000 and 8000 respectively. You can see in the below Prometheus config that I target them separately by their label name.

Why would you need to do this?

I found a case where I wanted different scrape configurations per pod in my cluster. Usually, I can set the kubernetes_sd_configs role to service, which discovers a target for each service port for each service (see the docs). However, in this case I had different collection requirements for the pods inside my k8s abstractions.

Example config files

Foo Deployment



apiVersion: apps/v1
kind: Deployment
metadata:
  name: foo-deployment
  labels:
    app: foo
spec:
  selector:
    matchLabels:
      app: foo
  replicas: 2
  template:
    metadata:
      labels:
        app: foo
    spec:
      containers:
      - name: python-prometheus-exporter-example
        image: ahf90/python-prometheus-exporter-example
        imagePullPolicy: Always
        ports:
          - containerPort: 9000
        env:
          - name: EXPORTER_PORT
            value: '9000'


Enter fullscreen mode Exit fullscreen mode

Bar Deployment



apiVersion: apps/v1
kind: Deployment
metadata:
  name: bar-deployment
  labels:
    app: bar
spec:
  selector:
    matchLabels:
      app: bar
  replicas: 2
  template:
    metadata:
      labels:
        app: bar
    spec:
      containers:
      - name: python-prometheus-exporter-example
        image: ahf90/python-prometheus-exporter-example
        imagePullPolicy: Always
        ports:
          - containerPort: 8000
        env:
          - name: EXPORTER_PORT
            value: '8000'


Enter fullscreen mode Exit fullscreen mode

Prometheus config.yml



scrape_configs:
  - job_name: 'prometheus'
    static_configs:
    - targets: ['localhost:9090']
  - job_name: 'foo-targets'
    scrape_interval: 10s
    metrics_path: '/'
    kubernetes_sd_configs:
    - role: pod
      namespaces:
        names:
        - default
      selectors:
      - role: "pod"
        label: "app=foo"
  - job_name: 'bar-targets'
    scrape_interval: 5s
    metrics_path: '/'
    kubernetes_sd_configs:
      - role: pod
        namespaces:
          names:
            - default
        selectors:
          - role: "pod"
            label: "app=bar"


Enter fullscreen mode Exit fullscreen mode

Prometheus targets page

Image of AssemblyAI tool

Transforming Interviews into Publishable Stories with AssemblyAI

Insightview is a modern web application that streamlines the interview workflow for journalists. By leveraging AssemblyAI's LeMUR and Universal-2 technology, it transforms raw interview recordings into structured, actionable content, dramatically reducing the time from recording to publication.

Key Features:
🎥 Audio/video file upload with real-time preview
🗣️ Advanced transcription with speaker identification
⭐ Automatic highlight extraction of key moments
✍️ AI-powered article draft generation
📤 Export interview's subtitles in VTT format

Read full post

Top comments (0)

Billboard image

The Next Generation Developer Platform

Coherence is the first Platform-as-a-Service you can control. Unlike "black-box" platforms that are opinionated about the infra you can deploy, Coherence is powered by CNC, the open-source IaC framework, which offers limitless customization.

Learn more

👋 Kindness is contagious

Discover a treasure trove of wisdom within this insightful piece, highly respected in the nurturing DEV Community enviroment. Developers, whether novice or expert, are encouraged to participate and add to our shared knowledge basin.

A simple "thank you" can illuminate someone's day. Express your appreciation in the comments section!

On DEV, sharing ideas smoothens our journey and strengthens our community ties. Learn something useful? Offering a quick thanks to the author is deeply appreciated.

Okay