DEV Community

Nishant Bhardwaj
Nishant Bhardwaj

Posted on

Observability Stack: Prometheus, Node Exporter & Grafana

#observability #prometheus #grafana #devops

A solid observability setup usually comes down to three pieces working together: something that collects metrics, something that exposes system-level metrics, and something that visualizes it all. Here's what each one does and how to install them.

The Theory: How This All Fits Together

Before installing anything, it helps to understand the model, because it's a bit different from how logging or alerting tools usually work.

Pull, not push. Most people's first instinct is "the app should send its metrics somewhere." Prometheus flips that around — it pulls metrics on a timer instead. Every target (a machine, a service, an app) exposes a simple HTTP endpoint, usually /metrics, that just returns plain text numbers. Prometheus visits that endpoint every N seconds (the "scrape interval") and saves whatever it finds, with a timestamp attached. Nothing gets pushed to Prometheus — Prometheus goes and asks.

This means for anything to show up in Prometheus, it has to satisfy one requirement: something has to expose a /metrics endpoint Prometheus can reach. That's the whole game. Everything else in this stack exists to satisfy that one requirement or to make the data useful afterward.

Why Node Exporter exists. Your operating system doesn't naturally speak Prometheus's language — it doesn't expose CPU/memory/disk stats as a /metrics endpoint by default. Node Exporter's only job is to read stats the OS already tracks (via /proc and /sys on Linux) and republish them in the text format Prometheus expects, on port 9100. It's a translator, not a monitoring tool by itself — it collects nothing, decides nothing, alerts on nothing. It just answers "what does this machine look like right now?" whenever asked.

Why Prometheus itself is separate. Prometheus doesn't know anything about CPUs or memory — it has no idea what it's scraping. It just knows: "go hit this list of URLs on a schedule, and remember what comes back." The intelligence is in the config (which targets to scrape, how often) and in queries (PromQL) you write later to make sense of the numbers. This separation is deliberate — the same Prometheus can scrape a server, a database, a Kubernetes pod, or your own app, as long as each one exposes metrics in the right format.

Why Grafana is a separate tool entirely. Prometheus stores numbers and can run queries, but its built-in UI is bare-bones — good for debugging, bad for a dashboard someone glances at every morning. Grafana's only job is to ask Prometheus questions (via those same PromQL queries) and draw the answers as graphs. Grafana stores no metrics itself — if Prometheus goes down, Grafana has nothing to show. They're two separate concerns: Prometheus is the database, Grafana is the window into it.

Putting it together, the flow for a single metric is always:

OS stat → Node Exporter formats it → Prometheus scrapes it on a timer → Grafana queries Prometheus → you see a graph
Enter fullscreen mode Exit fullscreen mode

Once that clicks, installing the three pieces is mostly mechanical — you're just standing up each link in that chain.

1. Prometheus

Prometheus is a time-series database and monitoring system. It periodically scrapes metrics from configured targets, stores them, and lets you query them with PromQL.

Once it's installed and running, it's also a scrape target for itself — Prometheus exposes its own internal metrics on port 9090, which is a handy sanity check: if you can query Prometheus about its own scrape performance, you know the whole pull mechanism is working.

Official download: https://prometheus.io/download/

Install (Linux binary)

wget <paste-official-tarball-link-here>
tar xvfz prometheus-*.tar.gz
cd prometheus-*/
sudo mv prometheus promtool /usr/local/bin/
Enter fullscreen mode Exit fullscreen mode

Run

prometheus --config.file=prometheus.yml
Enter fullscreen mode Exit fullscreen mode

2. Node Exporter

Node Exporter runs on a machine and exposes hardware and OS-level metrics — CPU, memory, disk, network — in a format Prometheus can scrape.

It doesn't talk to Prometheus, doesn't know Prometheus exists, and doesn't store any history — it just sits there and answers HTTP requests on port 9100 with whatever the machine's stats are right now. You install one Node Exporter per machine you want visibility into. If you have five servers, that's five Node Exporters, each scraped independently.

Official download: https://prometheus.io/download/

Install (Linux binary)

wget <paste-official-tarball-link-here>
tar xvfz node_exporter-*.tar.gz
sudo mv node_exporter-*/node_exporter /usr/local/bin/
Enter fullscreen mode Exit fullscreen mode

Run

node_exporter
Enter fullscreen mode Exit fullscreen mode

3. Grafana

Grafana connects to Prometheus (or other data sources) and turns the raw metrics into dashboards, graphs, and alerts.

Grafana is completely independent of the other two — you could uninstall it entirely and Prometheus would keep scraping and storing data without noticing. Its only job is to be a nicer way to look at what's already there.

Official download: https://grafana.com/docs/grafana/latest/setup-grafana/installation/

Install (Debian/Ubuntu)

sudo apt-get install -y apt-transport-https software-properties-common
sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"
sudo apt-get update
sudo apt-get install grafana
sudo systemctl enable --now grafana-server
Enter fullscreen mode Exit fullscreen mode

4. Connecting Everything: exporter-config.yaml

This is Prometheus's own config file — it tells Prometheus what to scrape. Save as prometheus.yml (or exporter-config.yaml) and point Prometheus at it with --config.file.

A quick walkthrough of the fields, since the names aren't obvious the first time you see them:

  • scrape_interval — how often Prometheus visits every target. 15s is a common default; lower means more granular data but more storage and load.
  • job_name — just a label you choose, so when you later query the data you can tell "this metric came from the node_exporter job" vs "this one came from the prometheus job." It has no special meaning to Prometheus beyond that.
  • static_configs / targets — the actual list of host:port addresses to scrape. "Static" just means you're typing the addresses by hand here, as opposed to Prometheus discovering them automatically (which is how it works in more dynamic environments like Kubernetes).

Add one job_name block per thing you want to monitor — one machine's Node Exporter, another machine's Node Exporter, your own app once it exposes metrics, and so on.

global:
  scrape_interval: 15s

scrape_configs:
  # Prometheus scraping itself
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # Node Exporter — system metrics
  - job_name: 'node_exporter'
    static_configs:
      - targets: ['localhost:9100']
Enter fullscreen mode Exit fullscreen mode

5. Grafana Setup

Once Grafana is running, open it in a browser and add Prometheus as a data source. This step is the one thing that trips people up as a junior — you're not "installing" the connection, you're just telling Grafana one URL to send its queries to:

  1. Go to Connections → Data sources → Add data source
  2. Choose Prometheus
  3. Set the URL to http://localhost:9090
  4. Click Save & Test

From there, import a dashboard (e.g. the community Node Exporter Full dashboard) or build your own panels. A dashboard is really just a saved collection of PromQL queries with a chart type attached to each one — nothing magic. Importing one from the community just saves you from writing those queries yourself on day one; you can always open any panel later and see (and edit) the exact query behind it.

Default Ports

Component Default Port
Prometheus 9090
Node Exporter 9100
Grafana 3000

Prometheus's web UI and API live on 9090, Node Exporter exposes its /metrics endpoint on 9100 (not a UI — just raw metrics text), and Grafana's login and dashboards live on 3000.

Observability Workflow 

                    OS / Machine Stats
                           │
                           ▼
                     Node Exporter
                (exposes :9100/metrics)
                           │
                           ▼
                      Prometheus
                (scrapes targets on a timer,
                    stores as time series)
                           │
                           ▼
                    PromQL Queries
                           │
                           ▼
                        Grafana
                (renders dashboards & graphs)
Enter fullscreen mode Exit fullscreen mode

Node Exporter turns raw machine stats into something Prometheus can read, Prometheus pulls and stores that data on a schedule, and Grafana queries Prometheus to turn the numbers into dashboards. Each layer only does one job — that's what makes the pieces easy to swap or reason about individually.

Key Learnings

  • Prometheus pulls metrics on a schedule — nothing pushes data to it.
  • Node Exporter doesn't monitor anything itself; it only translates OS stats into a format Prometheus can scrape.
  • Prometheus has no idea what CPU or memory even are — all the meaning comes from config and PromQL queries.
  • Grafana stores no data of its own; it's just a window into whatever Prometheus already has.
  • Default ports to remember: Prometheus 9090, Node Exporter 9100, Grafana 3000.

Conclusion

Prometheus, Node Exporter, and Grafana each do exactly one job, and understanding that separation is most of the battle — once it's clear that Node Exporter just exposes data, Prometheus just pulls and stores it, and Grafana just visualizes it, setting up the stack becomes a mechanical checklist rather than a mystery. From here, natural next steps are adding more scrape targets, writing custom PromQL queries, and eventually setting up Alertmanager to get notified when something crosses a threshold.

Top comments (0)