Who did what?!? Keep an eye on changes with PagerDuty Change Events and Buildkite

Have you ever gotten an alert on a service that’s normally pretty quiet and just wanted to yell “Who changed this service?!?” across the office, or in the group channel in ALL CAPS?

Change is everywhere. It comes from different sources, which access different parts of the stack. Sources like your CI/CD tools, your infrastructure management, your security and compliance reporting. Each team might have their own tools for making changes and adjustments to their part of your resources.

For teams to be effective, they need to be empowered to make the changes they need when they need them. What happens when something unexpected happens after a change? How do teams know what changed on which systems at what time when there might be dozens of places for change to originate?

Fortunately, you’ve got PagerDuty Change Events!

Change Events are non-alerting datapoints that your team and your tools can write to your PagerDuty services to mark the time a change was made and annotate the change with useful information. You’ll find them in your PagerDuty account under the Incidents menu as “Recent Changes”.

All Change Events in your PagerDuty account will show up in a single list that you can sort and search from. Events are also listed as “Recent Changes” in each service page.

What’s In a Change Event?

Change events have a two-part structure. The first section consists of several required fields and the second section allows you to add any number of fields as “custom details”.

The required fields tell PagerDuty where to associate the event, what timestamp to attach to it, a summary of the change, and the source. Change Events are associated with a specific service in your PagerDuty account, and to create the events you’ll need a routing key for that service. You can find or create these in the “Integrations” tab for the specific service:

Additionally, you can add links to the source of the change event, if there are job details or other information in the sending application. This is helpful for tracking changes back to their source if you need to during an incident.

The rest of the event is up to you. The API provides a custom_details object that you can pile information into, formatted as JSON. CI/CD tools might add things like build_state and build_number. Your infrastructure as code tools could add all the objects that changed when a package was updated. Other automation could report their status, success, errors, and other important information.

Services can receive updates from multiple sources as well. Any process that creates a change to the environment where a service is running can send information to PagerDuty so you know exactly what has been going on in that environment, no matter where it originates.

Change Events from Buildkite

One of the integrations that sends Change Events to PagerDuty is Buildkite. Buildkite is a continuous integration pipeline that allows you to notify PagerDuty with just the information you want to know. Conditional rules allow you to send only what matters.

You can see some of the custom details in this screenshot. Buildkite sends information about who created a build, which commit it is derived from in your source code repository, and other pertinent information. The box on the left features a link back to your Buildkite account to see the full details of the build.

Responders to incidents in your ecosystem will have the most up-to-date information about changes that are flowing into your production environment so your team stays informed!

Check out the Buildkite documentation for more on integrating with PagerDuty. Buildkite also joined us on our Twitch channel to talk about their product and sending Change Events to PagerDuty. You can see the recording of that stream on our YouTube Channel or view their session from PagerDuty Summit.

You can also join us for a workshop at (UnblockConf in November)[https://unblockconf.dev/21]!

Create Change Events via the API

The flexibility of the Change Events API means you can send just about anything to your PagerDuty account to augment your knowledge about your services and their status. Calling the PagerDuty Change Events API requires a couple of authorization pieces and some setup, but you can do it with a tool as simple as cURL.

You’ll need a service integration key like the one above. To call the API from cURL, create a new integration on your service, and select “Events API V2” from the “Most popular integrations” list and click the “Add” button. You’ll see your new integration listed for the service. You can change the name to something meaningful if you’d like. The “Integration Key” listed will be part of your Change Event request.

Then the Custom Details are up to you. If you can pack it into JSON, you can send it. Let’s look at a simple example using cURL:

curl --request POST \
 --url https://events.pagerduty.com/v2/change/enqueue \
 --header 'Content-Type: application/json' \
 --data '{
 "routing_key": "737ea619db564d41bd9824063e1f6b08",
 "payload": {
   "summary": "Build Success: Increase snapshot create timeout to 30 seconds",
   "timestamp": "2015-07-17T08:42:58.315+0000",
   "source": "prod-build-agent-i-0b148d1040d565540",
   "custom_details": {
     "build_state": "passed",
     "build_number": "220",
     "run_time": "1236s"
   }
 },
 "links": [
   {
     "href": "https://buildpipeline.com/pagerduty/deployment/builds/220",
     "text": "View in Build Pipeline"
   }
 ]
}'

Currently, the URL for Events is different from other parts of the API. So you’ll use the URL endpoint at https://events.pagerduty.com/v2/change/enqueue to send change events. Make sure to set the Content-Type header for JSON. The rest of the interesting information goes into the data.

The Integration Key you created above will be set using the routing_key in the data structure. The Integration Key is authentication and authorization to the API for Change Events; you won’t need a global API key to send new events to PagerDuty. This keeps the scope of the Integration Keys narrow and specific to the integration you are connecting, whether it is your own code or another piece of software.

Summary

Aggregating change information into your PagerDuty account gives you a reliable way to attribute change to potential impacts across your services. Leverage the rest of PagerDuty’s features to correlate changes to other events across your ecosystem.

To learn more about setting up Change Events in your PagerDuty account, check out our knowledge base article. To see all of the current integrations that leverage Change Events, visit our integrations page and select “Change” in the Category list.

More integrations are being added all the time!

You can also join us on our Community and follow us on Twitch.

Header Photo by Ross Findon on Unsplash