Hermes-Memory-Installer: SLO Rollup and Grafana Dashboard

The latest hermes-memory-installer update brings a focused capability for teams already running memory-critical workloads: native SLO rollup and an accompanying Grafana dashboard. This isn’t about generic monitoring—it’s about moving from raw metrics to actionable compliance data, directly from the installer’s resource management layer. The feature targets engineers who need to track memory SLIs, define SLOs, and react to burn rates without cobbling together custom PromQL rules and dashboard panels.

The core addition is slo.rollup, a sub-system that precomputes key SLO metrics from memory usage, latency, and error signals exposed by the Hermes runtime. Instead of polling raw time series for every dashboard load, the rollup aggregates into compliance windows (7d, 30d, custom) and stores the results in a dedicated Prometheus SLO metric namespace. This drastically reduces query overhead for both dashboards and alerting pipelines.

Under the hood, the rollup uses Prometheus recording rules defined in a configuration file auto-deployed by the installer. Each SLO window creates a recording rule that calculates the ratio of good events (e.g., successful memory allocation under 100ms) over total events. For instance, a 30-day SLO for p99 latency is computed as:

- record: hermes:slo:memory_latency_sli_30d
  expr: |
    sum(rate(hermes_memory_request_duration_seconds_bucket{le="0.1"}[30d]))
    / 
    sum(rate(hermes_memory_request_duration_seconds_count[30d]))

This gives a direct SLI value for the SLO target. The installer also exposes the error budget (1 - SLI) for real-time burn tracking. All rollup metrics are labeled with slo_name and slo_window to simplify dashboard queries.

On the visualization side, the installer ships a hermes-memory-slo-dashboard.json that targets Grafana 10.x. It includes three core panels:

• SLO Compliance Over Time: A time series comparing weekly SLI against the SLO target line. Rolling windows update automatically based on the stored rollup data.
• Error Budget Consumption: A gauge showing remaining budget percentage, with thresholds for warning (50% left) and critical (10% left). This uses the hermes:slo:error_budget metric.
• Burn Rate Heatmap: A table showing short-term (1h) vs. long-term (24h) burn rates, helping you detect rapid consumption before a breach.

The dashboard is designed for “import and run”. Once the installer is configured and running, import the JSON via Grafana’s UI or API. No additional data source plugins are required—it assumes a single Prometheus data source named Prometheus.

To enable the rollup, set slo.rollup.enabled: true in the installer’s config. Optionally, define custom SLO windows under slo.rollup.windows. The installer will then deploy the recording rules and the dashboard file to a designated directory (/etc/hermes/slo/ by default). Alerting is left to external tools, but the hermes:slo:error_budget metric can be used in Prometheus alert rules immediately.

This feature is opinionated: it works best if your memory SLIs follow the standard format of ratio of successful events to total events over a window. If your SLIs are non-binary (e.g., value aggregates), you can still use the rollup but need to adapt the recording rules manually. The pre-deployed rules target the common hermes_memory_* metrics emitted by the Hermes core, but the config file documents how to override them.

For teams that previously maintained ad-hoc dashboards and PromQL snippets for memory SLOs, this update cuts that effort to nearly zero. The rollup reduces Prometheus query time for SLO panels by an order of magnitude, and the dashboard provides enough context to detect budget depletion early. It’s a direct addition to the installer’s resource management pipeline—no extra agents or sidecars needed.

The implementation is deliberately minimal. There are no soft-errors or missing data fill strategies in the rollup; if a window has fewer than 1% of expected data points, the SLI is marked as null. The dashboard handles this with null as zero for gauges, but for time series, missing points are hidden. This means you should ensure your Hermes metrics are complete, or adjust the recording rules to handle data gaps if needed.

In summary, the slo rollup and grafana dashboard feature is a practical addition for anyone using hermes-memory-installer in a production memory-critical stack. It brings SLO visibility directly into the installer’s lifecycle, reduces the cognitive load of building monitoring from scratch, and enforces a consistent pattern for SLO tracking. Import the dashboard, enable the rollup, and shift from metric hunting to budget management.