DEV Community: Nishant Bhardwaj

Observability Stack: Prometheus, Node Exporter & Grafana

Nishant Bhardwaj — Wed, 26 Aug 2026 12:46:03 +0000

A solid observability setup usually comes down to three pieces working together: something that collects metrics, something that exposes system-level metrics, and something that visualizes it all. Here's what each one does and how to install them.

The Theory: How This All Fits Together

Before installing anything, it helps to understand the model, because it's a bit different from how logging or alerting tools usually work.

Pull, not push. Most people's first instinct is "the app should send its metrics somewhere." Prometheus flips that around — it pulls metrics on a timer instead. Every target (a machine, a service, an app) exposes a simple HTTP endpoint, usually /metrics, that just returns plain text numbers. Prometheus visits that endpoint every N seconds (the "scrape interval") and saves whatever it finds, with a timestamp attached. Nothing gets pushed to Prometheus — Prometheus goes and asks.

This means for anything to show up in Prometheus, it has to satisfy one requirement: something has to expose a /metrics endpoint Prometheus can reach. That's the whole game. Everything else in this stack exists to satisfy that one requirement or to make the data useful afterward.

Why Node Exporter exists. Your operating system doesn't naturally speak Prometheus's language — it doesn't expose CPU/memory/disk stats as a /metrics endpoint by default. Node Exporter's only job is to read stats the OS already tracks (via /proc and /sys on Linux) and republish them in the text format Prometheus expects, on port 9100. It's a translator, not a monitoring tool by itself — it collects nothing, decides nothing, alerts on nothing. It just answers "what does this machine look like right now?" whenever asked.

Why Prometheus itself is separate. Prometheus doesn't know anything about CPUs or memory — it has no idea what it's scraping. It just knows: "go hit this list of URLs on a schedule, and remember what comes back." The intelligence is in the config (which targets to scrape, how often) and in queries (PromQL) you write later to make sense of the numbers. This separation is deliberate — the same Prometheus can scrape a server, a database, a Kubernetes pod, or your own app, as long as each one exposes metrics in the right format.

Why Grafana is a separate tool entirely. Prometheus stores numbers and can run queries, but its built-in UI is bare-bones — good for debugging, bad for a dashboard someone glances at every morning. Grafana's only job is to ask Prometheus questions (via those same PromQL queries) and draw the answers as graphs. Grafana stores no metrics itself — if Prometheus goes down, Grafana has nothing to show. They're two separate concerns: Prometheus is the database, Grafana is the window into it.

Putting it together, the flow for a single metric is always:

OS stat → Node Exporter formats it → Prometheus scrapes it on a timer → Grafana queries Prometheus → you see a graph

Once that clicks, installing the three pieces is mostly mechanical — you're just standing up each link in that chain.

1. Prometheus

Prometheus is a time-series database and monitoring system. It periodically scrapes metrics from configured targets, stores them, and lets you query them with PromQL.

Once it's installed and running, it's also a scrape target for itself — Prometheus exposes its own internal metrics on port 9090, which is a handy sanity check: if you can query Prometheus about its own scrape performance, you know the whole pull mechanism is working.

Official download: https://prometheus.io/download/

Install (Linux binary)

wget <paste-official-tarball-link-here>
tar xvfz prometheus-*.tar.gz
cd prometheus-*/
sudo mv prometheus promtool /usr/local/bin/

Run

prometheus --config.file=prometheus.yml

2. Node Exporter

Node Exporter runs on a machine and exposes hardware and OS-level metrics — CPU, memory, disk, network — in a format Prometheus can scrape.

It doesn't talk to Prometheus, doesn't know Prometheus exists, and doesn't store any history — it just sits there and answers HTTP requests on port 9100 with whatever the machine's stats are right now. You install one Node Exporter per machine you want visibility into. If you have five servers, that's five Node Exporters, each scraped independently.

Official download: https://prometheus.io/download/

Install (Linux binary)

wget <paste-official-tarball-link-here>
tar xvfz node_exporter-*.tar.gz
sudo mv node_exporter-*/node_exporter /usr/local/bin/

Run

node_exporter

3. Grafana

Grafana connects to Prometheus (or other data sources) and turns the raw metrics into dashboards, graphs, and alerts.

Grafana is completely independent of the other two — you could uninstall it entirely and Prometheus would keep scraping and storing data without noticing. Its only job is to be a nicer way to look at what's already there.

Official download: https://grafana.com/docs/grafana/latest/setup-grafana/installation/

Install (Debian/Ubuntu)

sudo apt-get install -y apt-transport-https software-properties-common
sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"
sudo apt-get update
sudo apt-get install grafana
sudo systemctl enable --now grafana-server

4. Connecting Everything: exporter-config.yaml

This is Prometheus's own config file — it tells Prometheus what to scrape. Save as prometheus.yml (or exporter-config.yaml) and point Prometheus at it with --config.file.

A quick walkthrough of the fields, since the names aren't obvious the first time you see them:

scrape_interval — how often Prometheus visits every target. 15s is a common default; lower means more granular data but more storage and load.
job_name — just a label you choose, so when you later query the data you can tell "this metric came from the node_exporter job" vs "this one came from the prometheus job." It has no special meaning to Prometheus beyond that.
static_configs / targets — the actual list of host:port addresses to scrape. "Static" just means you're typing the addresses by hand here, as opposed to Prometheus discovering them automatically (which is how it works in more dynamic environments like Kubernetes).

Add one job_name block per thing you want to monitor — one machine's Node Exporter, another machine's Node Exporter, your own app once it exposes metrics, and so on.

global:
  scrape_interval: 15s

scrape_configs:
  # Prometheus scraping itself
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # Node Exporter — system metrics
  - job_name: 'node_exporter'
    static_configs:
      - targets: ['localhost:9100']

5. Grafana Setup

Once Grafana is running, open it in a browser and add Prometheus as a data source. This step is the one thing that trips people up as a junior — you're not "installing" the connection, you're just telling Grafana one URL to send its queries to:

Go to Connections → Data sources → Add data source
Choose Prometheus
Set the URL to http://localhost:9090
Click Save & Test

From there, import a dashboard (e.g. the community Node Exporter Full dashboard) or build your own panels. A dashboard is really just a saved collection of PromQL queries with a chart type attached to each one — nothing magic. Importing one from the community just saves you from writing those queries yourself on day one; you can always open any panel later and see (and edit) the exact query behind it.

Default Ports

Component	Default Port
Prometheus	`9090`
Node Exporter	`9100`
Grafana	`3000`

Prometheus's web UI and API live on 9090, Node Exporter exposes its /metrics endpoint on 9100 (not a UI — just raw metrics text), and Grafana's login and dashboards live on 3000.

Observability Workflow

                    OS / Machine Stats
                           │
                           ▼
                     Node Exporter
                (exposes :9100/metrics)
                           │
                           ▼
                      Prometheus
                (scrapes targets on a timer,
                    stores as time series)
                           │
                           ▼
                    PromQL Queries
                           │
                           ▼
                        Grafana
                (renders dashboards & graphs)

Node Exporter turns raw machine stats into something Prometheus can read, Prometheus pulls and stores that data on a schedule, and Grafana queries Prometheus to turn the numbers into dashboards. Each layer only does one job — that's what makes the pieces easy to swap or reason about individually.

Key Learnings

Prometheus pulls metrics on a schedule — nothing pushes data to it.
Node Exporter doesn't monitor anything itself; it only translates OS stats into a format Prometheus can scrape.
Prometheus has no idea what CPU or memory even are — all the meaning comes from config and PromQL queries.
Grafana stores no data of its own; it's just a window into whatever Prometheus already has.
Default ports to remember: Prometheus 9090, Node Exporter 9100, Grafana 3000.

Conclusion

Prometheus, Node Exporter, and Grafana each do exactly one job, and understanding that separation is most of the battle — once it's clear that Node Exporter just exposes data, Prometheus just pulls and stores it, and Grafana just visualizes it, setting up the stack becomes a mechanical checklist rather than a mystery. From here, natural next steps are adding more scrape targets, writing custom PromQL queries, and eventually setting up Alertmanager to get notified when something crosses a threshold.

Kubernetes HPA: Guide with Apache on KIND

Nishant Bhardwaj — Thu, 23 Jul 2026 17:28:28 +0000

Horizontal Pod Autoscaler (HPA) automatically increases or decreases the number of Pods based on CPU or memory utilization. In this project, I configured HPA on a local KIND cluster and verified autoscaling using CPU-based metrics.

1. Namespace

A Namespace logically separates Kubernetes resources, making it easier to organize and manage applications independently.

namespace.yml

kind: Namespace
apiVersion: v1
metadata:
  name: apache

2. Deployment

The Deployment manages the desired number of application Pods. I configured CPU requests and limits because HPA uses these values to calculate resource utilization and make scaling decisions.

deployment.yml

apiVersion: apps/v1
kind: Deployment

metadata:
  name: apache-deployment
  namespace: apache

spec:
  replicas: 3

  selector:
    matchLabels:
      app: apache-app

  template:
    metadata:
      labels:
        app: apache-app

    spec:
      containers:
        - name: apache

          image: httpd:latest

          imagePullPolicy: Always

          ports:
            - containerPort: 80

          resources:
            requests:
              cpu: "100m"
              memory: "100Mi"

            limits:
              cpu: "200m"
              memory: "250Mi"

3. Service

A Service provides a stable endpoint for accessing the application and distributes incoming traffic across all available Pods.

service.yml

apiVersion: v1
kind: Service
metadata:
  name: apache-sv
  namespace: apache
spec:
  selector:
    app: apache-app
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: ClusterIP

4. Metrics Server

Metrics Server collects CPU and memory usage from each Pod and exposes these metrics to Kubernetes. Without Metrics Server, HPA cannot monitor resource utilization.

Verify Metrics Server

@root-IdeaPad-Gaming-3-15IHU6:~/Code/K8s/django_hpa$ kubectl top nodes -n apache
NAME                              CPU(cores)   CPU(%)   MEMORY(bytes)   MEMORY(%)   
demo-kind-cluster-control-plane   126m         1%       612Mi           7%          
demo-kind-cluster-worker          29m          0%       487Mi           6%          
demo-kind-cluster-worker2         27m          0%       471Mi           6%          
demo-kind-cluster-worker3         34m          0%       520Mi           6%          
@root-IdeaPad-Gaming-3-15IHU6:~/Code/K8s/django_hpa$ kubectl top pods -n apache
NAME                                 CPU(cores)   MEMORY(bytes)   
apache-deployment-67856f954c-b4n2x   1m           13Mi            
apache-deployment-67856f954c-xkmwb   1m           12Mi

If CPU and memory metrics are displayed, the Metrics Server is working correctly.

5. Horizontal Pod Autoscaler (HPA)

The Horizontal Pod Autoscaler continuously monitors CPU utilization and automatically increases or decreases the number of Pod replicas based on the configured target utilization.

hpa.yml

# Horizontal Pod Autoscaler for Django Todo Application
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: apache-hpa
  namespace: apache
spec:
  # Target the Apache deployment
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: apache-deployment  

  # Scaling limits
  minReplicas: 2
  maxReplicas: 10

  # Metrics to scale on
  metrics:
  # CPU-based scaling
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 50  

  # Memory-based scaling
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 70  

  # Scaling behavior configuration
  behavior:
    # Scale down policies
    scaleDown:
      stabilizationWindowSeconds: 30 
      policies:
      - type: Percent
        value: 50 
        periodSeconds: 15
      - type: Pods
        value: 2  
        periodSeconds: 15
      selectPolicy: Min  

    # Scale up policies
    scaleUp:
      stabilizationWindowSeconds: 0  
      policies:
      - type: Percent
        value: 100  
        periodSeconds: 15
      - type: Pods
        value: 4  
        periodSeconds: 15
      selectPolicy: Max

6. Generate Load

Generate continuous HTTP requests to simulate client traffic and trigger autoscaling.

while true; do curl -s http://localhost:8000 > /dev/null; done

7. Monitor Autoscaling

Watch the HPA status in real time.

kubectl get hpa -w

Watch Pods being created or terminated.

kubectl get pods -w

Check CPU utilization of each Pod.

kubectl top pods

HPA Workflow

                 Client Requests
                        │
                        ▼
                    Kubernetes Service
                        │
                        ▼
                    Deployment
                        │
                        ▼
                Application Pods
                        │
                        ▼
                 Metrics Server
                        │
                        ▼
        Horizontal Pod Autoscaler
                        │
                        ▼
        Scale Pods Up or Scale Pods Down

Key Learnings

HPA automatically scales Pods based on CPU or Memory utilization.
Metrics Server is mandatory for CPU-based autoscaling.
CPU requests must be defined in the Deployment.
kubectl top pods helps monitor real-time resource usage.
HPA improves application scalability without manual intervention.

Conclusion

Horizontal Pod Autoscaler makes Kubernetes applications more resilient by automatically adjusting the number of running Pods according to workload demand. This hands-on implementation on a KIND cluster helped me understand the complete autoscaling workflow—from collecting metrics to dynamically scaling application replicas.