DEV Community

Devanshu Biswas
Devanshu Biswas

Posted on

OrderHub Day 38: a Grafana dashboard provisioned from git — rate() and histogram_quantile() over the Day-37 meters, 131 tests

#springboot #java #observability #grafana

Day 37 taught OrderHub to emit three meters and let Prometheus scrape and store them — but nobody watches a text endpoint. Prometheus is the database and query engine; Grafana is the view: it runs a PromQL query on an interval and paints the result. Day 38 puts a Grafana dashboard on top, and the discipline that matters is that the whole thing is provisioned as code — a datasource, a dashboard provider, and one orderhub.json, all checked into git. No clicking, reproducible on every boot. Here's the whole step.

Provisioning as code — three files, no clicks

Grafana starts empty but reads its provisioning folder on boot. Three files wire it up: a datasource pointing at Prometheus, a file provider that auto-loads dashboards, and the dashboard JSON itself.

grafana/
├─ provisioning/
│  ├─ datasources/prometheus.yml   # wire Grafana -> Prometheus
│  └─ dashboards/dashboards.yml    # a FILE provider (loads *.json)
└─ dashboards/orderhub.json        # the dashboard itself
Enter fullscreen mode Exit fullscreen mode

The datasource carries a stable uid so the dashboard JSON can reference it portably, and points at the compose service name — not localhost, which inside the container would be Grafana itself:

# datasources/prometheus.yml
datasources:
  - name: Prometheus
    uid: orderhub-prometheus       # stable id the JSON targets
    type: prometheus
    url: http://prometheus:9090    # the COMPOSE service name
    access: proxy                  # the Grafana server queries, not the browser
Enter fullscreen mode Exit fullscreen mode

The grafana service in docker-compose.yml bind-mounts those folders read-only, so I don't even need a persistent Grafana volume — recreate the container and it rebuilds itself from git.

A dashboard is just saved queries with a layout

orderhub.json is a valid Grafana dashboard — a panels[] array where each panel has a type, a gridPos, and targets carrying PromQL on the provisioned datasource.

{ "type": "timeseries", "title": "Orders placed — rate by outcome",
  "gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 },
  "datasource": { "type": "prometheus", "uid": "orderhub-prometheus" },
  "targets": [{ "expr": "sum by (outcome) (rate(orders_placed_total[$__rate_interval]))",
                "legendFormat": "{{outcome}}" }] }
Enter fullscreen mode Exit fullscreen mode

The panels answer the RED method

They aren't arbitrary — they map onto the exact meters Day 37 wired. The counter gives Rate and, split by outcome, Errors; the timer gives Duration; the gauge gives Saturation.

  • Rate + Errors: sum by (outcome) (rate(orders_placed_total[$__rate_interval])) — one query, three series (success, rejected, reservation_failed).
  • Duration: histogram_quantile(0.95, sum by (le) (rate(order_processing_seconds_bucket[$__rate_interval]))) for p95 (and 0.50 / 0.99).
  • Saturation: orders_open as a stat — no rate(), it's a live level.
  • Error SLO: sum(rate(orders_placed_total{outcome=~"rejected|reservation_failed"}[...])) / sum(rate(orders_placed_total[...])), shown as a percent.

That p95 query only works because the Day-37 Timer called publishPercentileHistogram(), so Prometheus stores _bucket series keyed by le ("less-or-equal"). histogram_quantile() interpolates the percentile from those buckets server-side — you never average latency (averages hide tail pain) and never recompute percentiles in the browser. Note the rate() inside: it makes the quantile reflect the window, not since-startup.

Guard the JSON with a test

The danger with dashboards is silent drift — rename a meter in the app and the panel goes blank at 3am. So a plain JUnit GrafanaDashboardTest (no Spring context, so the shared context and every prior test are untouched) parses orderhub.json with Jackson and asserts the shape and the queries.

assertThat(q).contains("orders_placed_total")
             .contains("order_processing_seconds")
             .contains("orders_open");
assertThat(q).contains("rate(orders_placed_total")
             .contains("histogram_quantile");
Enter fullscreen mode Exit fullscreen mode

A renamed meter now fails the build, not the panel. The reactor goes from 126 to 131 tests, BUILD SUCCESS.

Prometheus remembers; Grafana shows — and both come up from git, reviewable as a JSON diff in a PR instead of a screenshot of someone's clicks. Day 37 made the app observable; Day 38 makes it legible. Next, Day 39 adds distributed tracing, so a slow p99 on this dashboard can be followed down to the exact span.

Boot the stack and watch Grafana provision itself from files:
https://dev48v.infy.uk/orderhub/day38-grafana-dashboards.html
Repo: https://github.com/dev48v/order-hub-from-zero

Top comments (0)