DEV Community

Anand Rathnas
Anand Rathnas

Posted on Originally published at jo4.io

Grafana 11 Provisions Alerts Before Dashboards — and That Breaks folderUid

#grafana #monitoring #debugging #devops

This article was originally published on Jo4 Blog.

We rolled a tidy little change into our Grafana provisioning: pin the dashboard folder's UID so deep links don't break across redeploys. Restarted the container. Watched it exit 1 with this:

failed to create folder for provisioned dashboards" folder=jo4 ... err="a folder with the same name already exists in the current location"
Enter fullscreen mode Exit fullscreen mode

That error makes no sense. We hadn't created any folder called jo4 anywhere else. The provisioning files in /etc/grafana/provisioning/dashboards/ are the only place the name appears. So who created the first one?

The answer turned out to be: Grafana did. Before our dashboard provider ran. And it's not what the docs imply.

What We Expected

The Grafana provisioning docs are organized in roughly the order we'd expect a fresh install to come up:

  1. Datasources (so dashboards have something to query)
  2. Dashboards (the things humans look at)
  3. Alerting (rules that fire against dashboards/queries)

Our providers.yaml set the folder name and the folder UID, on the reasonable theory that pinning the UID protects external deep links from rotating after a rebuild:

apiVersion: 1
providers:
  - name: jo4
    folder: jo4
    folderUid: jo4-folder           # <-- our innocent addition
    type: file
    options:
      path: /etc/grafana/provisioning/dashboards/jo4
Enter fullscreen mode Exit fullscreen mode

If dashboards are provisioned before alerting, this is fine. The dashboard provider creates a folder named jo4 with UID jo4-folder. Alerting provisioning runs later, references folder: jo4 by name, matches the existing one. Everyone goes home.

That is not what Grafana 11.2.0 actually does.

What Actually Happens

We spun up a sandbox container, dropped in the exact same provisioning tree, and tailed the logs. Here's the relevant window, milliseconds and all:

06:26:57.106  msg="Initialising datasources"
06:26:57.204  msg="starting to provision alerting"
06:26:57.322  msg="finished to provision alerting"
06:26:57.344  msg="starting to provision dashboards"
06:26:57.346  level=error msg="failed to create folder for provisioned dashboards"
              folder=jo4 ... err="a folder with the same name already exists in the current location"
06:26:57.346  msg="Stopped background service" service=*provisioning.ProvisioningServiceImpl
              reason="failed to provision dashboards"
06:26:57.346  level=fatal msg="Server shutdown" reason="..."
Enter fullscreen mode Exit fullscreen mode

Three things to read off this:

  1. Alerting provisions before dashboards. It's not even close — alerting starts 140ms before dashboards and finishes 22ms before dashboards even start. This is the opposite of how the docs are organized.
  2. The alerting provisioner created the folder. Two milliseconds into dashboard provisioning, the error is already firing.
  3. The whole provisioning service stops. Not "skip this dashboard provider and continue." It stops everything. The Grafana process then exits 1 because provisioning is a required background service. The container restart-loops.

Why

Once you accept the ordering, the rest falls out cleanly.

The alerting provisioner takes a folder field — a name, not a UID. There is no folderUid field accepted in alerting rule provisioning files. When alerting runs first and sees folder: jo4, it needs that folder to exist. It doesn't. So Grafana creates it, with a UID it generates on the spot (something like ffn7ba7vmp3i8a — looks random, is deterministic for this install).

Now the dashboard provisioner runs. Our providers.yaml says: "make sure there's a folder named jo4 with UID jo4-folder." Grafana does the name lookup first, finds an existing folder named jo4 — but its UID is ffn7ba7vmp3i8a, not jo4-folder. The provisioner's reconciliation logic decides this is a different folder and tries to create one with the pinned UID. The unique-name constraint on folders fires. Provisioning service stops. Container exits 1.

So folderUid in the dashboard provider is fundamentally incompatible with also having alerting rules in the same folder — at least under Grafana 11.2.0's current ordering — because alerting will have created the folder first, with its own auto-generated UID, and your pinned UID can never win the race.

The Fix

One removed line. The dashboard provider keeps folder: jo4 (the human-readable name) and drops folderUid entirely:

Before:

apiVersion: 1
providers:
  - name: jo4
    folder: jo4
    folderUid: jo4-folder
    type: file
    disableDeletion: false
    updateIntervalSeconds: 30
    allowUiUpdates: true
    options:
      path: /etc/grafana/provisioning/dashboards/jo4
      foldersFromFilesStructure: false
Enter fullscreen mode Exit fullscreen mode

After:

apiVersion: 1
providers:
  - name: jo4
    folder: jo4
    type: file
    disableDeletion: false
    updateIntervalSeconds: 30
    allowUiUpdates: true
    options:
      path: /etc/grafana/provisioning/dashboards/jo4
      foldersFromFilesStructure: false
Enter fullscreen mode Exit fullscreen mode

That's it. With no folderUid, the dashboard provider does a name-only lookup, finds the folder alerting already created, reuses it, and provisions all the dashboards into it. The container comes up clean.

We also left a load-bearing comment in providers.yaml so the next person who sees a dangling UID in deep links and thinks "I'll just pin this" gets the full story before they break the deploy:

# IMPORTANT: do NOT set `folderUid`. Grafana 11.2.0 provisioning order is
# datasources → alerting → dashboards (verified empirically). Alerting
# provisioning auto-creates the folder by name (only field accepted) with a
# Grafana-generated UID. If we then pin a different folderUid here, Grafana
# tries to create a SECOND folder with the same name, errors, stops the
# provisioning service, and the container exits 1.
Enter fullscreen mode Exit fullscreen mode

Verifying It Worked

Re-running the sandbox after removing folderUid, here's the timeline:

msg="starting to provision alerting"
msg="finished to provision alerting"
msg="starting to provision dashboards"
msg="finished to provision dashboards"
Enter fullscreen mode Exit fullscreen mode

No error. Process stays up. Then we hit the Grafana API to see who owns the folder:

curl -s -u admin:admin http://localhost:3000/api/folders | jq '.[] | {title, uid}'
Enter fullscreen mode Exit fullscreen mode

Output:

{ "title": "jo4", "uid": "ffn7ba7vmp3i8a" }
Enter fullscreen mode Exit fullscreen mode

And the dashboards inside that folder all report folderUid: ffn7ba7vmp3i8a. So do the alerting rule groups. One folder, one UID, both provisioners pointing at it. Exactly what we wanted — we just don't get to choose the UID.

If you genuinely need a stable folder UID for deep links (we don't, but you might), the only path we've found that works is: let alerting auto-create it, read the resulting UID back via the API once after the first install, and codify it as an external constant. Don't try to pin it through providers.yaml.

Lessons Learned

  • Documentation ordering is not runtime ordering. The Grafana docs describe provisioning subsystems in the order you'd intuitively initialize them, but the actual startup sequence in 11.2.0 is datasources → alerting → dashboards. If something cross-references between subsystems, find out who runs first the empirical way — read the logs.
  • A unique-name constraint plus two creators is always a race. Whenever two independent provisioners can both produce the "same" named resource and only one of them lets you specify the UID, the one without UID control wins by going first. Your pinned UID has no path to victory.
  • Provisioning service failure = container failure. Grafana doesn't gracefully degrade when one provisioning file is bad. The whole service stops and the process exits. Treat your provisioning YAML with the same rigor as a database migration: it can take down the system.
  • folderUid in a dashboard provider is a foot-gun the moment you also have alerting in the same folder. Drop it. Let the folder be created by whoever runs first and look it up by name everywhere else.
  • Leave the comment. Future-you, or the next operator, will look at the removed folderUid and wonder why. A five-line comment in the YAML is cheaper than another two-hour debugging session.

Got bit by Grafana provisioning ordering? What was your symptom? Drop it in the comments.

Building jo4.io — a URL shortener with analytics for developers who ship.

Top comments (0)