Configuring Chef Automate to Trigger PagerDuty Alerts

Wouldn’t it be great if we could generate an incident when our systems fell out of compliance? By combining Chef Automate and PagerDuty, through simple webhooks, we can absolutely do this.

Prerequisites

I make the following assumptions:

1. You already have Chef Automate installed with at least one node configured to send Compliance information to Automate (if you need help configuring this, I recommend this excellent post by Nick Rycar).
2. You have a PagerDuty account.

Create Service in PagerDuty for Chef Compliance

We begin by creating a service in PagerDuty. A “service” represents an application, component, or team — in this case, we consider Compliance as an overall state. Currently, Chef Automate doesn’t provide us the ability to filter on which nodes get reported, but in a future post, I will dig into further configuration in PagerDuty to separate these into different service. For now, we’ll consider all of Compliance across our fleet as something we care about as a whole.

To create a service, click on Configuration | Services from the PagerDuty website, and then Add New Service.

1. Name the service Chef Compliance.
2. Provide an optional description.
3. The integration type should be Custom Event Transformer.
4. Integration name should be Chef Automate Compliance Webhook
5. Configure the remaining settings as appropriate for your organization.

After you save this new service, you’ll see it listed. Click on the integration name( Chef Automate Compliance Webhook ) so we can configure it:

Click on Edit Integration

Replace the JavaScript with the following:

var webhook = PD.inputRequest.body;

var normalized_event = {
  event_type: PD.Trigger,
  incident_key: webhook.node_uuid,
  description: "InSpec found a critical control failure on "+webhook.node_name,
    "details": {
    "Number of failed critical tests": webhook.number_of_failed_critical_tests,
    "Total number of critical tests": webhook.number_of_critical_tests
  },
  client: "Chef Automate",
  client_url: "https://automate.mattstratton.io/viz/#/compliance/reporting/nodes/"+webhook.node_uuid
};

PD.emitGenericEvents([normalized_event]);

You will eventually want to disable the setting Debug Mode, but it’s okay to leave it on for testing. Also be sure to replace automate.mattstratton.io with the FQDN of your own Automate server!

Configure the Chef Automate Notification

You will need the Integration URL from the Chef Automate Compliance Webhook integration. It should be something like https://events.pagerduty.com/integration/XXXXXXXXXXXX/enqueue

Open your Chef Automate console, and switch to the Nodes  tab:

Click on Notifications | Create Notification :

We want to add a Webhook notification, so select that one.

Choose InSpec scan failures, and paste in your integration URL. Call the notification PagerDuty InSpec Scan Failures, and click Save.

Testing the Notification

Assuming that we have a node that will fail the configured compliance profile, all we have to do is run chef-client on that node, and we should see it come up in PagerDuty like this:

Here’s what the generated incident looks like in PagerDuty:

And to be even fancier, we can see it in the PagerDuty mobile app:

Reporting on Chef Client Errors in PagerDuty

Similar to Compliance failures, we can also generate alerts and incidents on a failed chef-client run. It’s a very similar process:

1. Create a new service in PagerDuty, but this time, call it Chef Client .
2. Use the Custom Event Transformer just as before, but name it Chef Automate Chef Client Webhook (or something less verbose, if you prefer).
3. Use the following JavaScript for the integration:

var webhook = PD.inputRequest.body;

var normalized_event = {
  event_type: PD.Trigger,
  incident_key: webhook.node_uuid,
  description: "Chef client failed on "+webhook.node_name+" with error: "+webhook.exception_message,
  details: webhook.exception_backtrace,
  client: "Chef Automate",
  client_url: webhook.automate_failure_url
};

PD.emitGenericEvents([normalized_event]);

In Automate, you will create a notification similar to the Compliance one, but instead of reporting on InSpec failures, select the option for Chef client run failures

The reporting in PagerDuty is slightly different for this one — it will give the information on the error from chef-client, as well as the backtrace:

In Summary

This is a pretty basic integration, but hopefully, it illustrates how easy it is to tie these two together. In a future post, I’ll dig into methods for sending specific Compliance failures to particular teams. Let me know what questions I can answer for you!