Krishan Thisera

Posted on Aug 13

Monitoring Mixins, Observability Libraries, and the Signal Pattern: A Practitioner's Guide

#kubernetes #grafana #observability #devops

A few months ago, I completed a significant revamp of our metrics and monitoring implementation. That work meant taking a hard look at how we package and distribute Prometheus alerts and Grafana dashboards across services, and led me down a rabbit hole most DevOps and SRE folks eventually encounter: monitoring mixins, observability libraries, and Grafana's newer signal pattern.

This article is a write-up of what I found, centred on Grafana's own jsonnet-libs repository. If you're comfortable with Prometheus and Grafana but haven't yet explored how the ecosystem handles reusable, composable monitoring configuration, this is meant as a practical starting point.

Chapter 1: The Dashboard That Wouldn't Load

You've done this before. You find a highly-rated Kubernetes dashboard on the Grafana community site, import it into your own Grafana instance, and wait for the panels to populate. Instead, half of them show "No data." The PromQL queries underneath assume a job label or selector that doesn't match how you name things. The dashboard isn't broken. It's written for someone else's Prometheus.

This kind of failure is common enough to be a recognised pattern in the observability community. A dashboard built for one environment carries hidden assumptions about label names, selectors, and job structures that rarely survive contact with a different one.

The root cause is structural, not accidental. The engineers building a piece of infrastructure software are best positioned to know which metrics matter and how to interpret them. But they are rarely the ones running that software in someone else's production environment. It's like a car built by one team and serviced by another: the people who know the engine best rarely turn the wrench.

For years, the default response was to route around this gap manually: build dashboards from scratch, or copy one from the internet and hope. Neither scales. What changed the equation was a specific piece of tooling built to close that gap directly, and that's where the story of monitoring mixins begins.

Chapter 2: How Mixins Fixed Monitoring Distribution

A monitoring mixin is a Jsonnet program. When you evaluate it with your own configuration values, it produces three things at once: a Grafana dashboard, a set of Prometheus alerting rules, and a set of Prometheus recording rules, all built from the same label and selector logic. Think of it as a recipe rather than a finished meal: the mixin ships with instructions and defaults, and you supply the ingredients specific to your kitchen.

Tom Wilkie and Frederic Branczyk proposed this format in 2018, packaging the idea as a way to distribute Grafana dashboards and Prometheus rules together rather than as separate, disconnected artifacts. The premise, as Grafana Labs later described it on its own blog, was simple: system authors should not only emit metrics, they should also tell you how to use them. That shifts the burden of building good observability artifacts onto the open-source community, instead of leaving every team to solve the same problem alone.

What a mixin actually produces follows a community-maintained style guide. Alert names use camelCase with no whitespace (NodeFilesystemAlmostOutOfFiles is one example), and every alert carries a severity label: critical, warning, or info. Every alert also carries two mandatory annotations, summary and description, so the person paged at 2 a.m. knows what fired and why.

The other core capability is customisation: you inject or override configuration values, such as job selectors and thresholds, at build time, without touching the mixin's source. That's what lets an upstream author ship something useful without hard-coding your specific configurations.

But that flexibility has a ceiling. A generic dashboard stays generic no matter how many configuration knobs it exposes; deciding which panels matter to your team requires local knowledge no upstream author has. Understanding why requires understanding the language mixins are built on.

Chapter 3: The Jsonnet Engine Underneath

Every mixin is written in Jsonnet, a data-templating language developed at Google. Grafana Labs describes it as a powerful extension of JSON, one that adds variables, functions, conditionals, imports, and error handling on top of the plain JSON data model. Tom Wilkie has pointed out that this lineage isn't accidental: Kubernetes borrowed from Google's internal Borg, Prometheus borrowed from Borgmon, and Jsonnet followed the same pattern for monitoring configuration.

Before settling on Jsonnet, the mixin's authors ruled out several alternatives. YAML was judged insufficiently dynamic for injecting labels into templates. JSON, a strict subset of YAML, offered even less. String substitution was rejected because it doesn't understand the structure it's editing; a bad substitution can silently produce broken syntax. Templating engines like Jinja only allow structural changes at points the template author explicitly anticipated. And general-purpose languages such as Go or Python were set aside because, in Wilkie's words, writing full programs to produce configuration "feels too much like writing programs, and it's generally a bit of a pain."

A handful of Jsonnet features do the actual structural work. The + operator merges two objects, and when fields collide, the right-hand side wins. Fields declared with :: are hidden: they exist during evaluation but never appear in the final rendered JSON, which is how internal objects like _config stay invisible until a build step extracts them. The self keyword is a late-bound reference to the current object, letting a mixin refer to configuration you haven't supplied yet. super reaches into a base object during a merge, so you can extend rather than replace inherited behavior.

Jsonnet isn't the only answer to this problem. CUE, created by former Google engineer Marcel van Lohuizen, takes a validation-first approach instead of a templating-first one.

We'll return to that trade-off, and to the teams who've made the switch, in a later chapter. For now, these operators are what turned a single mixin into something composable, which is exactly what observability libraries build on next.

Chapter 4: From Mixins to Observability Libraries

A plain mixin (see legacy istio-mixin) has a ceiling: it's a self-contained bundle built for one piece of software. Two independent mixins generally can't merge cleanly into a single dashboard set; you end up reconciling conflicts by hand. Observability libraries, or observ-libs, exist to remove that ceiling. According to Grafana's own jsonnet-libs repository, an observ-lib is a flexible format for describing dashboards and alerts in a modular way, built specifically so libraries can import into one another, or into a mixin.

You'll recognise an observ-lib by its folder name: kafka-observ-lib, jvm-observ-lib, windows-observ-lib, snmp-observ-lib, process-observ-lib, golang-observ-lib. A shared common library applies consistent style across all of them.

The directory structure tells you what changed. A legacy plain mixin ships dashboards.libsonnet, alerts.libsonnet, and config.libsonnet. An observ-lib adds three things: a signals/ directory, a panels.libsonnet file for reusable panel definitions, and a rows.libsonnet file for reusable dashboard row groupings. It also splits its entry point in two: main.libsonnet for the full modern feature set, and mixin.libsonnet retained purely so older tooling built around the plain-mixin standard keeps working.

This is composition, not just organisation. A team building observability for a Java application can import the JVM library for garbage-collection and heap metrics, then layer application-specific dashboards on top, with no need to reimplement JVM monitoring from scratch. It's the difference between building with prefabricated modules and pouring a new foundation every time.

The payoff shows up in the dashboards themselves, which tend to follow a consistent shape across the ecosystem: an Overview section for health indicators, a Performance section for throughput and latency, a Resources section for CPU and memory, and a Details section for diagnostics.

That signals/ directory isn't decorative. It houses the ecosystem's most consequential idea yet, and it's where we're headed to see next.

Chapter 5: Signals: Define Once, Render Everywhere

Here's a failure mode every SRE has lived through: a dashboard panel and its corresponding alert quietly drift apart. Someone edits the panel's query to fix a bug. Nobody updates the alert. Months later, what you see on the dashboard during an incident no longer matches the condition that actually paged you. The signal pattern exists to make that drift structurally impossible.

A signal lets you declare a metric once and render it as a time series panel, a stat panel, a table column, or a Prometheus alert, all from that single declaration. The core abstraction lives in common-lib/common/signal, inside files like base.libsonnet, signal.libsonnet, and one file per supported type: counter.libsonnet, gauge.libsonnet, histogram.libsonnet, info.libsonnet, raw.libsonnet.

The type you choose does real work. Each signal type carries an automatic PromQL transformation, so you don't have to remember to apply it by hand:

Type	Transformation
`gauge`	Rendered as-is
`counter`	Wrapped in `rate(...[interval])`
`histogram`	Wrapped in `histogram_quantile(q, sum(rate(...[interval])) by (le, ...))`
`info`	Rendered as static values, not a time series
`raw`	No transformation, used verbatim

This table is arguably the single most consequential design choice in the whole pattern. Graphing a raw counter instead of its rate, or averaging a histogram's _bucket (i.e., request_duration_seconds_bucket, http_client_request_body_size_bytes_bucket) series without histogram_quantile, is one of the most common Prometheus mistakes there is. Tying the transform to the declared type, instead of to each place the metric gets used, removes that error class entirely.

Once declared, a signal exposes render functions: asTimeSeries(), asStat(), asGauge(), asTable() for panels, and asRuleExpression() for a Prometheus alert. A separate family of modification functions, including withTopK(), withQuantile(), withLegendFormat(), and withInterval(), lets you reshape the output without redefining the underlying query. You can declare signals in bulk, JSON-style, for simple cases, or through a lower-level imperative API when you need finer control over an individual metric.

Think of a signal as a single master recording that gets remixed for different formats: same source material, different output, guaranteed to stay in sync. Because asTimeSeries() and asRuleExpression() pull from the same underlying expression, the panel and the alert can never quietly disagree.

That guarantee comes with a caveat. The signal pattern is explicitly marked experimental by its own maintainers at the time of writing. Its render-function surface is large, with more than a dozen panel functions alone, and the interface isn't stabilised. Adopting it today means accepting some risk of backward-incompatible change in exchange for the correctness it buys you.

Understanding what a signal is only gets you so far. Next, you'll want to know how to actually run one: which tools you need installed, and what the workflow looks like from a fresh clone to a deployed dashboard.

Chapter 6: Getting Hands-On: The Toolchain and Workflow

Three tools cover most of what you need.

Jsonnet is the language runtime itself
jb (jsonnet-bundler) is the package manager that fetches mixins and libraries.
mixtool validates and scaffolds mixins against the standard schema.

Trying an existing mixin takes three commands:

git clone <mixin-repo-url>
cd <mixin-repo>/<mixin-dir>
jb install

Customising a mixin follows a different path. You don't edit the upstream source; you start a fresh project instead:

jb init
jb install github.com/kubernetes-monitoring/kubernetes-mixin

Then write a local mixin.libsonnet that imports the upstream mixin and overrides its _config:

local kubernetes = import "kubernetes-mixin/mixin.libsonnet";
kubernetes {
  _config+:: {
    kubeStateMetricsSelector: 'job="kube-state-metrics"',
  },
}

You can use the jsonnet CLI, or the Jsonnet Language Server extension for VS Code for inline evaluation, linting, and navigation.

From here, deployment branches based on your preference, for instance, through your own CI/CD, bundled with kube-prometheus for a full Kubernetes-native stack, or through Tanka.

This is the entire mixin/observ-lib/signal pattern in working form. It's also not the only way to solve this problem, and the next chapter looks squarely at the alternatives.

Chapter 7: The Other Side of the Fence: Alternatives Worth Knowing

The strongest critique of this entire ecosystem comes from inside Grafana Labs itself. Grafana's own documentation states plainly that Grafonnet, while useful for generating dashboard JSON, "requires knowing Jsonnet, so that can be unappealing to some users." That's not outside commentary. It's the tool's primary vendor naming its own barrier to adoption.

The Grafana Terraform provider is the most direct answer to that barrier. Its grafana_dashboard resource holds the dashboard model in a config_json field, populated inline with jsonencode(), loaded from a JSON file, or templated with Terraform's templatefile() function. Grafana documentation lists three more non-Jsonnet paths:

Grafana Ansible collection, for teams already standardised on Ansible playbooks.
Grafana Operator, a Kubernetes-native controller that fits naturally into ArgoCD-style GitOps.
Grafana Crossplane provider (experimental), exposing the same resource surface as native Kubernetes manifests.

Rule management has a parallel alternative, distinct from dashboards. A large share of the Kubernetes-native monitoring community manages Prometheus alerting and recording rules through the Prometheus Operator's PrometheusRule custom resource: native Kubernetes manifests reconciled by a controller, rather than YAML generated ahead of time from Jsonnet. The two approaches are often complementary rather than competing. Many teams use a mixin, or kube-prometheus (which bundles several mixins together), specifically to generate the PrometheusRule manifests that the Operator then deploys: Jsonnet handles authoring, and Kubernetes-native reconciliation handles runtime.

Terraform and the Operator both change how configuration gets deployed. CUE makes a different argument entirely. CUE is validation-first where Jsonnet is templating-first: its merge operation is associative, commutative, and idempotent, a guarantee that the same configuration fragments always produce the same result regardless of combination order. Jsonnet's simpler right-hand-side-wins merge doesn't make that promise. Mercari's observability platform team adopted CUE for parts of their workflow specifically to cut down on the number of configuration languages engineers needed to know.

None of this makes Jsonnet's approach wrong. It has the deepest community track record here: the mixin idea dates to 2018 and has years of production use. Nothing else discussed offers an equivalent to the signal pattern's single-definition, multi-render guarantee. The honest answer is contextual: teams mainly consuming community mixins get the most value staying in Jsonnet, since that's where the maintained content lives.

That answer holds for the tools available right now. Where Grafana Labs is taking the platform next is a separate question, and it's the one Chapter 8 takes on.

Chapter 8: Where It's Heading, And Why It's Worth Exploring

Grafana Labs' own roadmap tells you where this is going. Grafana 12 introduced the Foundation SDK, entering public preview with libraries for defining Grafana resources programmatically in Go, Java, PHP, Python, and TypeScript: mainstream languages, not Jsonnet. Alongside it came the App Platform APIs and Dashboard schema v2, both experimental, and grafanactl for validating and pushing resources. By 2026, the Grafana Terraform provider had already split its documentation between a legacy HTTP API for Grafana 12 and earlier, and a newer Kubernetes-style API for Grafana 13 and beyond. The resource model itself is mid-migration.

Don't read this as Jsonnet losing. The Foundation SDK solves a different problem than mixins do. It answers how do I define a Grafana resource in a language my team already knows. Mixins and observ-libs answer how do I package and distribute best-practice monitoring so other organisations can use and improve it. Grafonnet sits at the bridge between the two: it's a Jsonnet library, but it's generated straight from the Foundation SDK's OpenAPI specs. These are complementary tracks, not a fork in the road.

What stays constant is the idea, not the tooling: define your signal once, render it everywhere, and let configuration, not a fork, absorb the differences between environments. That's the part worth carrying forward regardless of which SDK wins.

If you're running Prometheus and Grafana already, you have a fast way to test this yourself. Clone the upstream mixin, run jb install, and generate its dashboards against your own cluster. You'll see the pattern this whole article has been describing, in about ten minutes.