DEV Community

Cover image for Prometheus and Grafana: Metrics Monitoring and Dashboards
Rhuturaj Takle
Rhuturaj Takle

Posted on

Prometheus and Grafana: Metrics Monitoring and Dashboards

#prometheusandgrafana #dotnet #programming #learning

Prometheus and Grafana: Metrics Monitoring and Dashboards

A practical guide to Prometheus and Grafana — the de facto open-source standard for collecting, querying, alerting on, and visualizing metrics — covering Prometheus's pull-based model, metric types, PromQL, alerting rules, Grafana dashboards, and .NET integration, completing this series' observability trio alongside Structured Logging and Distributed Tracing.

Table of Contents

  1. Introduction
  2. Why Prometheus's Pull Model Is Different
  3. Metric Types
  4. Exposing Metrics from .NET
  5. Service Discovery and Scrape Configuration
  6. PromQL: Querying Metrics
  7. Recording Rules
  8. Alerting
  9. Grafana Dashboards
  10. Cardinality: The Silent Cost Multiplier
  11. Long-Term Storage and Federation
  12. Prometheus/Grafana Within the Broader Observability Stack
  13. Common Pitfalls
  14. Quick Reference Table
  15. Conclusion

Introduction

Prometheus is an open-source metrics collection and alerting system built around a distinctive pull-based model, and Grafana is the open-source visualization layer most commonly paired with it — together they form the de facto standard, vendor-neutral stack for metrics monitoring in cloud-native environments, especially Kubernetes (per this series' Kubernetes/Helm guide), where both originated from and remain most deeply integrated. This guide completes this series' observability trio: Structured Logging covers the logs pillar, Distributed Tracing (built on OpenTelemetry) covers the traces pillar, and this guide covers the metrics pillar in depth.

histogram_quantile(0.99, sum(rate(http_server_request_duration_seconds_bucket[5m])) by (le, route))
Enter fullscreen mode Exit fullscreen mode

That single PromQL expression computes the p99 latency per API route over a rolling 5-minute window — the kind of question this guide builds toward answering fluently, along with how to get the underlying data into Prometheus in the first place and turn it into dashboards and alerts that actually help.

1. Why Prometheus's Pull Model Is Different

Pull, not push 

Prometheus server → periodically SCRAPES → /metrics endpoint on each target
Enter fullscreen mode Exit fullscreen mode 
app.MapPrometheusScrapingEndpoint(); // exposes GET /metrics for Prometheus to scrape
Enter fullscreen mode Exit fullscreen mode

Unlike many metrics systems (and unlike the OpenTelemetry Collector's typical push-based OTLP export, covered in this series' OpenTelemetry guide), Prometheus works by pulling — the Prometheus server itself periodically makes an HTTP request to a /metrics endpoint exposed by each monitored application, rather than applications pushing their metrics out to a central collector.

Why this design choice matters practically

  • Prometheus can tell if a target is down — a failed scrape (connection refused, timeout) is itself a meaningful signal ("this target isn't reachable"), distinct from a target that's simply not emitting metrics; a push-based system generally can't distinguish "not sending data" from "not running" as easily.
  • No agent needed on the application side for basic exposition — the application just needs to expose an HTTP endpoint; it doesn't need to know Prometheus's address, handle export retries, or manage a connection to a remote collector.
  • Centralized control over scrape frequency and target list — operators configure what to scrape and how often from the Prometheus server's own configuration, rather than every application independently deciding its own export cadence.

The trade-off

Pull-based scraping requires Prometheus to have network access to every target it monitors — for genuinely short-lived jobs (a batch job that completes in seconds, potentially before a scrape interval elapses) or targets behind restrictive network boundaries, this model needs a workaround (Section 4's discussion of the Pushgateway) rather than working naturally out of the box the way a push-based system would for those specific scenarios.

2. Metric Types

Prometheus defines four core metric types, and choosing the right one for a given measurement is what makes later querying (Section 5) actually work correctly.

Counter: a value that only ever increases 

var ordersPlacedCounter = meter.CreateCounter<int>("orders_placed_total");
ordersPlacedCounter.Add(1);
Enter fullscreen mode Exit fullscreen mode

A counter represents a cumulative count that only goes up (or resets to zero on a restart) — total requests served, total orders placed, total errors encountered. Counters are never queried for their raw value directly in practice; they're almost always queried via rate() (Section 5) to get a meaningful per-second rate over time, since the raw cumulative total by itself ("14,382,910 total requests since the process started") is rarely the interesting number.

Gauge: a value that can go up or down 

var activeConnectionsGauge = meter.CreateObservableGauge("active_connections", () => GetCurrentConnectionCount());
Enter fullscreen mode Exit fullscreen mode

A gauge represents a value that can increase or decrease freely — current memory usage, active connection count, queue depth right now. Unlike a counter, a gauge's raw current value is directly meaningful and commonly graphed as-is.

Histogram: a distribution of observed values, bucketed 

var requestDurationHistogram = meter.CreateHistogram<double>("http_server_request_duration_seconds");
requestDurationHistogram.Record(elapsedSeconds, new KeyValuePair<string, object?>("route", "/orders"));
Enter fullscreen mode Exit fullscreen mode

A histogram samples observations (request durations, response sizes) into a configured set of buckets, and exposes both a total count and a running sum, alongside per-bucket cumulative counts — this is what enables the percentile calculations (histogram_quantile, Section 5) central to latency analysis, directly connecting to this series' Distributed Tracing guide's percentile-based latency discussion, but computed from aggregated metric data rather than derived from individual traces.

Summary: client-side-calculated percentiles (generally the less-preferred option)

A summary calculates percentiles directly on the client (application) side before exposing them, rather than exposing raw bucket counts for the server to calculate from — this avoids histogram's bucket-configuration considerations but has a significant limitation: summary percentiles cannot be meaningfully aggregated across multiple instances (you can't average or combine pre-calculated p99s from ten different pod replicas into a genuine overall p99), which is precisely why histograms are generally the recommended choice for anything that will run as multiple replicas — nearly every real production service.

3. Exposing Metrics from .NET

Via OpenTelemetry's Prometheus exporter (the recommended modern path) 

builder.Services.AddOpenTelemetry()
    .WithMetrics(metrics => metrics
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddRuntimeInstrumentation()
        .AddMeter("OrderApi.Metrics")
        .AddPrometheusExporter());

var app = builder.Build();
app.MapPrometheusScrapingEndpoint(); // exposes GET /metrics
Enter fullscreen mode Exit fullscreen mode

As covered in this series' OpenTelemetry guide, the same Meter/Counter/Histogram API used for OTLP export can also be exposed in Prometheus's native text format via the OpenTelemetry Prometheus exporter — meaning a .NET application instrumented once with OpenTelemetry's metrics API can serve both an OTLP-based pipeline and a traditional Prometheus scrape target simultaneously, without maintaining two separate instrumentation approaches.

What a scraped /metrics endpoint actually looks like 

# HELP http_server_request_duration_seconds Duration of HTTP requests
# TYPE http_server_request_duration_seconds histogram
http_server_request_duration_seconds_bucket{route="/orders",le="0.1"} 245
http_server_request_duration_seconds_bucket{route="/orders",le="0.5"} 480
http_server_request_duration_seconds_bucket{route="/orders",le="1.0"} 495
http_server_request_duration_seconds_bucket{route="/orders",le="+Inf"} 500
http_server_request_duration_seconds_sum{route="/orders"} 62.4
http_server_request_duration_seconds_count{route="/orders"} 500
Enter fullscreen mode Exit fullscreen mode

This plain-text exposition format is genuinely simple — human-readable, easy to curl and inspect directly, and straightforward enough that writing a custom exporter for something not already covered by an existing instrumentation library is a modest undertaking, not a significant engineering project.

Custom application metrics 

private static readonly Meter OrderMeter = new("OrderApi.Metrics");
private static readonly Counter<int> OrdersPlacedCounter = OrderMeter.CreateCounter<int>("orders_placed_total");
private static readonly Histogram<double> OrderProcessingDuration = OrderMeter.CreateHistogram<double>("order_processing_duration_seconds");

public async Task<Order> PlaceOrderAsync(CreateOrderRequest request)
{
    var stopwatch = Stopwatch.StartNew();
    var order = await _repository.CreateAsync(request);
    OrdersPlacedCounter.Add(1, new KeyValuePair<string, object?>("customer_tier", request.CustomerTier));
    OrderProcessingDuration.Record(stopwatch.Elapsed.TotalSeconds);
    return order;
}
Enter fullscreen mode Exit fullscreen mode

Business-specific metrics (orders placed by tier, processing duration) follow exactly the same pattern covered in this series' OpenTelemetry guide — registered once via AddMeter, they flow through the same pipeline as framework-level metrics and become queryable in Prometheus alongside them.

4. Service Discovery and Scrape Configuration

Static configuration for a small, fixed set of targets 

scrape_configs:
  - job_name: 'order-api'
    scrape_interval: 15s
    static_configs:
      - targets: ['order-api-1:8080', 'order-api-2:8080']
Enter fullscreen mode Exit fullscreen mode

For a small number of known, stable targets, static configuration is simple and sufficient — but this doesn't scale to environments where instances come and go dynamically (autoscaling, rolling deployments), which is the normal case for anything covered in this series' Kubernetes/Helm and cloud compute guides.

Kubernetes service discovery 

scrape_configs:
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
Enter fullscreen mode Exit fullscreen mode 
# On the pod itself, in the Deployment manifest
metadata:
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "8080"
Enter fullscreen mode Exit fullscreen mode

Prometheus's Kubernetes service discovery integration automatically discovers pods (or services, endpoints, nodes) matching configured criteria — commonly, annotations on the pod itself (as shown above) opt it into scraping — meaning as pods are created and destroyed by deployments, rollouts, and autoscaling (per this series' Kubernetes/Helm guide), Prometheus's scrape target list stays automatically current without manual configuration updates for every new instance.

The Pushgateway: the workaround for short-lived jobs 

echo "batch_job_duration_seconds 45.2" | curl --data-binary @- http://pushgateway:9091/metrics/job/nightly-cleanup
Enter fullscreen mode Exit fullscreen mode

For genuinely short-lived batch jobs (per this series' Background Services guide's discussion of scheduled jobs) that might complete and exit before Prometheus's next scheduled scrape, the Pushgateway provides an intermediary a job can push its final metrics to, which Prometheus then scrapes from instead of the job itself — an explicit, deliberate exception to the pull model, reserved specifically for this narrow use case rather than a general-purpose push mechanism for anything that finds pull inconvenient.

5. PromQL: Querying Metrics

Instant vectors and range vectors 

http_server_request_duration_seconds_count                    # instant vector: current value of every matching series
http_server_request_duration_seconds_count[5m]                 # range vector: every sample over the last 5 minutes
Enter fullscreen mode Exit fullscreen mode

An instant vector returns the most recent value of a metric (per unique label combination); a range vector returns every sample within a specified time window — most useful PromQL functions (like rate()) operate on range vectors to compute something meaningful over time, rather than working with a single instant snapshot.

rate(): turning a counter into a meaningful per-second value 

rate(http_requests_total[5m])
Enter fullscreen mode Exit fullscreen mode

Since a counter (Section 2) only ever increases, rate() computes the per-second average rate of increase over the specified window — this is almost always how a counter is actually queried in practice; the raw cumulative counter value on its own is rarely the interesting number.

Aggregation: sum, avg, grouping with by 

sum(rate(http_requests_total[5m])) by (route, method)
Enter fullscreen mode Exit fullscreen mode

Aggregation operators combine multiple time series (one per unique label combination) into fewer, more meaningful series — sum(...) by (route, method) computes total request rate per unique route/method combination, collapsing away other labels (like the specific pod instance) that aren't relevant to this particular question.

histogram_quantile(): computing percentiles from histogram buckets 

histogram_quantile(0.99, sum(rate(http_server_request_duration_seconds_bucket[5m])) by (le, route))
Enter fullscreen mode Exit fullscreen mode

This is the PromQL idiom for the percentile-based latency analysis covered in this series' Distributed Tracing guide, but computed from aggregated histogram data across every instance of a service rather than from individual sampled traces — histogram_quantile interpolates a percentile value from the cumulative bucket counts, and grouping by (le, route) (keeping the histogram's bucket boundary label while aggregating away instance-specific labels) gives a genuine, combined p99 across every replica of a service for a specific route.

Alert-style boolean expressions 

rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) > 0.05
Enter fullscreen mode Exit fullscreen mode

This expression computes the error rate (5xx responses divided by total requests) over a 5-minute window and evaluates whether it exceeds 5% — exactly the kind of expression that becomes an alerting rule (Section 7), turning a metric query into an actionable, automatically-evaluated condition.

increase() for counting events over a window 

increase(orders_placed_total[1h])
Enter fullscreen mode Exit fullscreen mode

increase() computes the total increase in a counter over the specified window (accounting for counter resets, e.g., from a process restart) — useful for "how many orders were placed in the last hour," as distinct from rate()'s per-second average.

6. Recording Rules

Pre-computing expensive queries 

groups:
  - name: order-api-recording-rules
    interval: 30s
    rules:
      - record: order_api:request_duration_p99
        expr: histogram_quantile(0.99, sum(rate(http_server_request_duration_seconds_bucket[5m])) by (le, route))
Enter fullscreen mode Exit fullscreen mode

A recording rule pre-computes a PromQL expression on a schedule and saves the result as a new, permanently-stored time series — rather than recalculating an expensive aggregation (like the p99 example from Section 5) every time a dashboard panel or alert needs it, the recording rule computes it once, on Prometheus's own schedule, and every downstream consumer just reads the pre-computed result.

Why this matters at scale

For a genuinely high-cardinality metric (many distinct label combinations, Section 9) queried frequently by multiple dashboards and alerts, repeatedly recalculating the same expensive aggregation on every dashboard refresh and every alert evaluation cycle is wasteful — recording rules compute it once and let everything else read the cheap, pre-aggregated result, a meaningful performance and cost optimization once a Prometheus deployment reaches real production scale.

7. Alerting

Alertmanager: Prometheus's companion alerting component 

groups:
  - name: order-api-alerts
    rules:
      - alert: HighErrorRate
        expr: rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Error rate above 5% for {{ $labels.route }}"
          description: "Current error rate: {{ $value | humanizePercentage }}"
Enter fullscreen mode Exit fullscreen mode

Prometheus itself evaluates alerting rules (PromQL expressions with a threshold) on a schedule, and when a rule's condition is true continuously for the duration specified by for (avoiding alerting on a single, momentary blip), it fires an alert to Alertmanager — a separate component responsible for deduplication, grouping related alerts together, silencing, and routing to the actual notification channels (PagerDuty, Slack, email).

Why for matters: avoiding alert flapping 

for: 5m  # the condition must be true continuously for 5 minutes before actually firing
Enter fullscreen mode Exit fullscreen mode

Without a for duration, a metric that briefly crosses a threshold for a single evaluation cycle (a momentary spike that resolves itself immediately) would fire and immediately resolve an alert — for requires the condition to hold continuously across multiple evaluation cycles before actually notifying anyone, filtering out exactly the kind of noisy, self-resolving blips that erode trust in alerting (the same "flaky test" trust-erosion problem covered in this series' CI/CD Pipelines guide, applied to alerts instead of test results).

Alertmanager routing and grouping 

route:
  group_by: ['alertname', 'route']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'slack-oncall'

receivers:
  - name: 'slack-oncall'
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/...'
        channel: '#oncall-alerts'
Enter fullscreen mode Exit fullscreen mode

Grouping related alerts (multiple routes all showing elevated error rates simultaneously, likely from the same underlying cause) into a single notification, rather than paging on-call separately for each one, is what keeps alerting genuinely actionable at scale — an on-call engineer receiving 40 separate notifications for what's actually one incident is a well-documented path to alert fatigue and, eventually, ignored pages.

Designing alerts around symptoms, not causes

The general, widely-adopted guidance: alert on symptoms users would actually notice (elevated error rate, high latency, a failed health check) rather than on every possible underlying cause independently (CPU usage, memory usage, a specific internal queue depth) — a single well-designed symptom-based alert, investigated using the distributed tracing and structured logging techniques covered in this series' companion guides, is generally more actionable and less noisy than dozens of narrowly-scoped, cause-based alerts that may or may not actually correspond to a real user-facing problem.

8. Grafana Dashboards

Connecting Grafana to Prometheus 

# Grafana data source configuration
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    url: http://prometheus:9090
    isDefault: true
Enter fullscreen mode Exit fullscreen mode

Grafana connects to one or more data sources (Prometheus being the most common for metrics, but also Loki for logs and Tempo for traces, forming the LGTM stack referenced in this series' OpenTelemetry guide) and builds dashboards from queries against them — Grafana itself stores no metric data; it's purely a query and visualization layer.

Building a dashboard panel 

{
  "title": "P99 Request Latency by Route",
  "targets": [
    { "expr": "histogram_quantile(0.99, sum(rate(http_server_request_duration_seconds_bucket[5m])) by (le, route))" }
  ],
  "type": "timeseries"
}
Enter fullscreen mode Exit fullscreen mode

A Grafana panel is, at its core, a PromQL query (or several) paired with a visualization type (time series graph, gauge, heatmap, table) — the PromQL expertise from Section 5 transfers directly into building genuinely useful dashboard panels, rather than being a separate skill.

Dashboards as code 

// A dashboard JSON model, version-controlled and provisioned automatically
Enter fullscreen mode Exit fullscreen mode 
apiVersion: 1
providers:
  - name: 'default'
    folder: 'Order API'
    type: file
    options:
      path: /etc/grafana/provisioning/dashboards
Enter fullscreen mode Exit fullscreen mode

Storing dashboard definitions as version-controlled JSON files (exported from Grafana's UI, or authored directly) and provisioning them automatically on Grafana startup — rather than manually clicking together dashboards through the UI, which tends to drift and doesn't survive a Grafana redeployment — extends the same "infrastructure and configuration as code" discipline covered throughout this series (Terraform/Bicep, GitOps, CI/CD Pipelines) to dashboards themselves.

Variables for reusable, parameterized dashboards 

$environment  → dropdown: production, staging, development
$service       → dropdown: dynamically populated from label_values(up, job)
Enter fullscreen mode Exit fullscreen mode 
histogram_quantile(0.99, sum(rate(http_server_request_duration_seconds_bucket{environment="$environment", job="$service"}[5m])) by (le))
Enter fullscreen mode Exit fullscreen mode

Grafana template variables let one dashboard definition serve many contexts — the same latency panel, filterable by a dropdown to any service or environment — avoiding the need to maintain nearly-identical, hand-duplicated dashboards per service, which (like the copy-pasted pipeline YAML problem covered in this series' GitHub Actions and Azure DevOps guides) tends to drift out of sync as one copy gets updated and others don't.

9. Cardinality: The Silent Cost Multiplier

What cardinality actually means here 

http_requests_total{route="/orders", method="POST", status="200", customer_id="42"}
Enter fullscreen mode Exit fullscreen mode

Every unique combination of a metric name and its label values is a distinct time series that Prometheus must store and index independently — including customer_id as a label above means Prometheus is now storing a separate time series per individual customer, not one aggregated series for the /orders route.

Why high-cardinality labels are a genuine operational risk, not just a style preference 

10 routes × 5 methods × 10 status codes = 500 time series      ← entirely manageable
10 routes × 5 methods × 10 status codes × 100,000 customers = 50,000,000 time series  ← a serious problem
Enter fullscreen mode Exit fullscreen mode

Adding a high-cardinality label (a customer ID, a user ID, a raw request ID, anything with effectively unbounded distinct values) to a metric multiplies the number of stored time series by that label's cardinality — this is one of the most common, most damaging Prometheus operational mistakes, capable of degrading query performance and dramatically increasing memory/storage usage for the entire Prometheus deployment, not just for the one metric that introduced it.

Where high-cardinality data actually belongs instead 

Metrics (Prometheus): aggregate counts and rates — "how many orders per minute," not "which specific customer"
Traces (per this series' Distributed Tracing guide): individual request detail, including customer ID as a SPAN attribute
Logs (per this series' Structured Logging guide): individual event detail, including customer ID as a structured property
Enter fullscreen mode Exit fullscreen mode

This is a direct, practical consequence of the "three pillars answer different questions" principle from this series' OpenTelemetry guide — genuinely per-entity, high-cardinality detail (which specific customer, which specific order ID) belongs in traces and logs, which are architecturally designed to handle high-cardinality, per-event data; metrics and Prometheus specifically are optimized for aggregate, bounded-cardinality dimensions, and forcing high-cardinality data into a metric label is using the wrong pillar for the job.

10. Long-Term Storage and Federation

Prometheus's default local storage limitation

By default, a single Prometheus server stores data locally on disk with a configured retention period (commonly 15 days to a few months) — appropriate for recent operational monitoring and alerting, but not designed as a long-term historical data warehouse, and a single Prometheus instance doesn't natively scale horizontally for very high metric volume on its own.

Remote write to long-term storage backends 

remote_write:
  - url: "http://mimir:9009/api/v1/push"
Enter fullscreen mode Exit fullscreen mode

Prometheus supports remote write — continuously streaming scraped samples to an external, horizontally-scalable long-term storage system (Grafana Mimir, Thanos, Cortex, or a managed cloud equivalent) — decoupling "how long can we retain and efficiently query this data" from Prometheus's own local storage and retention configuration.

Federation and Thanos/Mimir for genuinely large-scale deployments

For organizations running many Prometheus instances (one per Kubernetes cluster, per region, per team), Thanos or Grafana Mimir provide a global query layer aggregating data across all of them, plus long-term, cost-efficient object storage (S3/Azure Blob-backed) for historical retention — the practical solution once a single Prometheus server's local storage and single-instance query scope genuinely becomes a limiting factor, mirroring the "start simple, add the distributed/scaled version once genuine scale demands it" pattern covered throughout this series' database and infrastructure guides.

11. Prometheus/Grafana Within the Broader Observability Stack

Completing this series' observability picture 

Metrics (Prometheus)  → aggregate, dashboards, alerting — THIS GUIDE
Traces (OpenTelemetry + a tracing backend) → per-request, causal chain across services — Distributed Tracing guide
Logs (structured, centralized) → per-event detail, correlated with traces → Structured Logging guide
Enter fullscreen mode Exit fullscreen mode

With this guide, this series' observability trio is complete — metrics for the aggregate "how is the system behaving" question and alerting, traces for the per-request "what actually happened, across which services" question, and logs for the detailed "what exactly occurred at this specific point" question, all correlated together via the shared trace/span IDs and consistent labeling conventions covered across these three guides.

Prometheus and OpenTelemetry: complementary, not competing

As covered in Section 3, a modern .NET application can instrument once with OpenTelemetry's metrics API and export to both an OTLP pipeline and a native Prometheus scrape endpoint simultaneously — Prometheus's pull-based scraping and PromQL query language remain genuinely valuable and widely adopted specifically for the metrics pillar, even as OpenTelemetry has become the standard for the instrumentation layer sitting above it.

Grafana as the unifying visualization layer

Because Grafana can query Prometheus (metrics), Loki (logs), and Tempo (traces) from within the same dashboard — and increasingly supports jumping directly from a metric panel to a correlated trace or log query — it's commonly the single pane of glass tying together every guide in this series' observability trio into one coherent operational view, rather than three separate tools an engineer has to manually context-switch between during an investigation.

12. Common Pitfalls

Pitfall Why it hurts Better approach
Adding a high-cardinality label (customer ID, request ID) to a metric Multiplies stored time series, degrades query performance cluster-wide Keep high-cardinality detail in traces/logs; keep metric labels bounded
Querying a counter's raw value instead of using rate() The raw cumulative total is rarely the meaningful number Always wrap counters in rate() or increase() for meaningful queries
Using summaries instead of histograms for anything running as multiple replicas Summary percentiles can't be meaningfully aggregated across instances Use histograms with histogram_quantile() for any multi-replica service
Alerting with no for duration Fires and resolves on momentary, self-correcting blips, eroding trust Require the condition to hold for a meaningful duration before firing
Alerting on internal causes instead of user-facing symptoms Noisy, less actionable; doesn't reliably correspond to real user impact Alert on symptoms (error rate, latency, health checks); investigate causes via traces/logs
Manually clicking together dashboards through the Grafana UI Drifts out of sync, doesn't survive a redeployment, not reviewable Provision dashboards as version-controlled JSON, per this series' IaC-as-code principles
Assuming a single Prometheus instance scales indefinitely Local storage and single-instance query scope hit real limits at genuine scale Use remote write to Thanos/Mimir once retention or multi-cluster query needs grow

Quick Reference Table

Concept Purpose
Pull-based scraping Prometheus fetches /metrics from targets; enables target-down detection
Counter Cumulative, always-increasing value; query via rate()/increase()
Gauge A value that can go up or down; queried directly
Histogram Bucketed observations enabling histogram_quantile() percentile calculation
PromQL Prometheus's query language for aggregation, rates, and alert conditions
Recording rule Pre-computes an expensive expression on a schedule for reuse
Alerting rule + for A PromQL condition that must hold continuously before firing
Alertmanager Deduplicates, groups, and routes fired alerts to notification channels
Grafana panel A visualization built from one or more PromQL (or other data source) queries
Cardinality The number of distinct label-value combinations for a metric; keep bounded
Remote write Streams scraped data to long-term/horizontally-scalable storage (Thanos, Mimir)

Conclusion

Prometheus and Grafana together form the metrics half of the observability picture this series has built out across its Structured Logging, Distributed Tracing, and OpenTelemetry guides — Prometheus's pull-based model and purpose-built metric types (especially histograms, and the histogram_quantile percentile analysis they enable) give the aggregate, "how is the system behaving" view that traces and logs individually can't provide efficiently, while Grafana turns that data into dashboards and, via Alertmanager, into genuinely actionable alerts.

The disciplines that make this stack valuable rather than noisy echo the same themes across this series' observability guides: keep high-cardinality detail out of metrics and in traces/logs where it belongs, alert on user-facing symptoms rather than every possible internal cause, and treat dashboards and alerting rules as version-controlled configuration rather than manually-maintained UI state. Done well, Prometheus and Grafana complete a genuinely coherent observability stack — one where a metric-driven alert leads naturally into a specific trace, which leads naturally into the exact correlated log lines that explain precisely what happened.

Found this useful? Feel free to star the repo, open an issue with corrections, or share the cardinality explosion that taught you to keep customer IDs out of metric labels.

Top comments (0)