DEV Community

Wesley Skeen
Wesley Skeen

Posted on

Add deployment annotations to grafana dashboards

When noticing spikes in exceptions on our dashboards, one of the first things we do is check if there has been a recent deployment. Checking to see if a deployment caused the spikes.

This can be straightforward, but if you have many teams deploying many apps all throughout the day, it may be a little trickier tracking down a related deployment.

I am going to talk you through how to add a grafana annotation to visualize when a deployment has taken place. This will be a simple annotation with some tags.

Here is what we are gonna create

grafana annotation

First of all, I am going to be running this using my grafana playground which you can find here

Once you have that set up, there are a few steps we need to do to enable annotations to display on a dashboard.

Create a service account

Browse here and click Add service account. If you have the grafana playground running, you should be able to access the link.

add service account

Give it a name and the Editor role

add name and editor role

Create a service account token

Click Add service account token

Add service account token button

Then click Generate token

generate token

Once you have this token, copy it and keep it save.

Create a dashboard

For this example, I am just gonna create a really simple dashboard here with a random metric. Just to have something to show.

dashboard

Create annotations

Next we will create some annotations to display. We are going to send them as a http post request. The URL we create annotations on is http://localhost:3000/api/annotations.

We next have to add the Authorization header. The value for this will be Bearer {your service account token we created}.

The last thing to do is send a request. The payload will look like

{
  "created": EPOCH with milliseconds, // eg 1684502867000
  "updated": EPOCH with milliseconds,
  "time": EPOCH with milliseconds,
  "timeEnd": EPOCH with milliseconds,
  "text": "Deployment", // text to display on the annotation pop up
  "tags": [ // tags to display on the annotation pop up
    "kind:deployment",
    "project:your-project-or-app-name",
    "env:prod",
    "region:eu-west-1"
  ]
}
Enter fullscreen mode Exit fullscreen mode

The response should be

{
    "id": {annotation_id},
    "message": "Annotation added"
}
Enter fullscreen mode Exit fullscreen mode

Now that we have created an annotation, it is time to set up our dashboard to view them.

Set up dashboard annotations

Go to the dashboard you created previously and go to it's settings.

Next click on Annotations and then Add annotation query.

add annotation query

Chose Grafana as the data source.

Grafana datasource

Next under Filter by, choose Tags and then turn on Match any.

tags filter

You can then filter by tags you sent when sending the http request. I am just gonna filter on one tag kind:deployment.

tags

Click Apply and Save dashboard.

Browse back to your dashboard and you now should see an annotation being displayed

grafana annotation

If you don't see any annotations check the following things

  • You are using a recent timestamp for your EPOCH time when creating your annotations
  • You are filtering by the correct tags
  • Your dashboard has annotations turned on

annotations switch

Adding to Azure DevOps pipeline

I have created a gist that can be added to your Azure DevOps pipeline.

Top comments (0)