Automatically transition Jira issues using a Github webhook

Kro — Sat, 22 Feb 2020 02:08:43 +0000

What's this all about eh?

One of the perks of my job is that on the last Friday of every month we get to work on our hack of choice, as long as it is in some way work related - come join us 👋.

We use Github for our repos and Jira for project management. This pairing offers some nice functionality through the use of third-party addons such as GitHub for Jira which allow us to add pull request context to Jira issues.

Jira supports many workflow transitions out-of-the-box including the following:

Pull Request created
Branch created
Commit created
Review rejected

See the full list at confluence.atlassian.com

Our Work Setup 💻

Our Jira Workflows are quite involved but for us developers our primary focus is on the following Jira workflow transition states:

In Development
Ready for UI review
Ready for review (Dev)
Ready for QA

As part of a busy team there’s the inevitable context switching and we have many responsibilities including:

Performing due diligence on new functionality
PR reviews within our team
Feature planning and development
Bug fixing
Mentoring
etc.

Once our PRs have passed UI approval we then assign two Dev reviewers from within our team and transition the Jira ticket to Ready for review.
Reviews can take time for many reasons including the size of the feature, the amount of feedback, changes requested, and the number of PRs we have on the go at a given point in time; it can be difficult to notice when one has been granted the required number of approving Dev reviews and to manually then move the associated Jira ticket to the next stage in the workflow (Pull Panda can be hugely beneficial in this area, it's a must-have for any large team).

For this reason tickets don’t always get moved on to Ready for QA in a timely manner and this can lead to completed pieces of functionality not becoming visible to our QA team, having the knock on effect of delaying its ultimate release.

As a developer my instinct was to look at somehow automating this process. I initially investigated if Jira offered support for a PR Approved transition, but unfortunately this led me down a black hole as the following public Jira request hilights - https://jira.atlassian.com/browse/JRACLOUD-71798

After giving this more thought it also became apparent that should this be supported by Jira in the future, it wouldn’t suffice as I would also need to apply some additional business-logic once a PR approval was received.

So What Now? 🕵️

I set about coming up with a loose plan of action which would enable me to create a proof of concept. At this stage I had a rough idea of the pieces of the puzzle which I would need to combine to achieve my end goal, these included:

Github’s Webhook support
Github’s REST API
JIRA’s REST API
Serverless backend

Github provides an exhaustive list of supported webhook events but the one that appeared to best suit my needs was

Pull request reviews - review submitted, edited or dismissed.

The payload from this event provides us with useful details including:

Action - submitted, edited, dismissed
Review and reviewer details
Details of the pull request including link, creator, etc

Further details can be found in the Github docs along with sample payloads.

Are We Missing Anything? 🔍

It quickly became apparent to me that I would need to make subsequent requests against the Github REST API to retrieve all other associated reviews; this additional info would allow me to do the processing required to consider the Jira ticket ready for transition.

At this point I had the following mental model in my head:

Github review webhook fires
Parse review and if approval then proceed
Fetch additional PR reviews (Github REST API)
Parse reviews only counting latest per user and those that are approvals
If two or more approving Dev reviews then transition associated issue (Jira REST API)
Notify the pull request creator via Slack (also surface any errors which might occur)

Now that I had a plan in place the next step was to choose how I would process the Webhook request and make the necessary requests to Github, Jira and Slack.
I’ve worked with Amazon’s Lambda functions in the past and knew that if I combined this with an Amazon API Gateway it could be a good fit for this project.

To avoid overcomplicating this article I’m going to generate the Lambda function inline using the provided editor, relying on the following natively supported packages for request processing - crypto, https and url.

I've also decided not to cover setting up the Slack API and the associated webhook in the interest of brevity (if this is something you would like me to cover then please let me know).

As this was to be run on the companies Github, Jira, AWS accounts I also wanted to focus on ensuring the project was secure despite the fact that it would be a POC.

For this purpose I added the following steps:

Included an optional secret with the Github webhook payload which would be validated
Configured the API Gateway to validate the presence of the following headers
- X-GitHub-Delivery
- X-Hub-Signature
Store all required tokens (API tokens, etc.) in Amazon Parameters Store
Set up an IAM role providing the Lambda with access to only the parameters it required by leveraging ssm:GetParameters

Okay, enough talk, let's get down to business!

I’ve organised the technical overview into the following sections:

Setting up the Lambda function
Creating the API Gateway - configuring the endpoint - forwarding the payload to the Lambda function - verifying the request headers
Setting up the Github Webhook
Storing the sensitive parameters using the Parameter Store
Generating the Lambda function role - configuring this role to only have access to the specific parameters required
Adding the control logic to our Lambda function - ~~testing our Lambda function~~
Areas where we could improve the functionality

1) Setting up the Lambda function 👨‍💻

Log into AWS and choose the Lambda service.

from here choose 'Create function'

Next choose ‘Author from scratch’ and give your function a meaningful name. I went with the default runtime and chose ‘Create a new role with basic Lambda permissions’.

Once created we now have a placeholder Lambda function that we can refer to when setting up our API Gateway.

2) Setting up the API Gateway 🛣️

Choose API gateway from the Services menu.

I initially tried creating a HTTP API (Beta) Gateway but ran into some issues so switched to an externally accessible REST API.

Fill in a name and description and continue with the defaults to generate a new REST API.

Once the gateway is created we can add a resource and further configure our endpoint.
Choose the ‘Actions’ dropdown and select ‘Create Resource’, giving the new endpoint a meaningful name and path.

With our resource selected we then choose ‘Create method’ from the dropdown and select POST. This then allows us to choose our Lambda function as the integration for our endpoint.

Next click on the ‘Method Request’ header to further configure the endpoint.

We can then expand the ‘HTTP Request Headers’ section and add the following as required headers:

There’s one final step required which can easily be overlooked. Before we can access our endpoint we first need to activate it.
We do this by choosing ‘Deploy’ from the ‘Actions’ drop down menu. Give the deployment a name and hit deploy!

Finally grab the ‘invoke url’ which we will use when setting up our webhook.

With our Lambda function and API Gateway in place we can move on to setting up our Github Webhook.

3) Setting up the Github Webhook 🕸️

Create a new Webhook for your repo at the following url:
https://github.com/{your organisation}/{your app}/settings/hooks

We will configure it as follows:
Payload URL: the invoke url from the API Gateway we created in the previous section.
Secret: choose to include a secret and give it a value (store this value as we will use it later).
Content type: application/json
SSL verification: enabled
Events: choose ‘let me select the individual events’ > ‘Pull request reviews’
Finally choose to activate the webhook.

ℹ️ Github provides the handy option of sending a test payload if we want to validate that our endpoint is working as expected. It also logs all webhook requests, even allowing us to replay previous payloads which can be very helpful.

4) Storing the sensitive parameters using the Parameter Store 🤫

Amazon's blurb

AWS Systems Manager Parameter Store provides secure, hierarchical
storage for configuration data management and secrets management. You
can store data such as passwords, database strings, and license codes
as parameter values. You can store values as plaintext (unencrypted
data) or ciphertext (encrypted data). You can then reference values by
using the unique name that you specified when you created the
parameter. Highly scalable, available, and durable, Parameter Store is
backed by the AWS Cloud.

I'm not going to go into any detail on setting up the parameters as it's pretty self-explanatory.
My advice would be to ensure you name them sensibly and I added a tag value to each defining the common project.
I've utilised the Store to securely maintain the various API tokens required for this project:

your-github-rest-api-token-param-name - token required for REST requests
your-github-webhook-secret-param-name - webhook has been configured to include a secret in the Headers - 'X-Hub-Signature'
your-jira-api-user-param-name - REST API creds
your-jira-api-token-param-name - REST API creds
your-slack-api-token-param-name - App webhook token for messaging
your-github-dev-reviewer-ids-param-name - list of comma separated Github Dev user ids (I could have simply declared this as a const in the Lambda)

Utilising these in the Lambda function wasn't as straight forward as I had hoped and admittedly my final solution just felt a little contrived (if you know a better way please let me know!). That said, it feels a lot more secure than defining them inline in the code or setting then in the functions env variables. The fact that they can be shared between projects and we have a single source of truth is also an added bonus.

The following code hilights how I retrieved these in the Lambda function:

Initial setup:

let state = {}
// require the sdk
const AWS = require('aws-sdk')
AWS.config.update({
  region: 'eu-west-1'
})

// create a new SSM instance
const parameterStore = new AWS.SSM()
// List of the names of the parameters defined in the Parameter Store

const REQUIRED_PARAMS  = [
'your-github-rest-api-token-param-name',
'your-github-webhook-secret-param-name',
'your-github-dev-reviewer-ids-param-name',
'your-jira-api-user-param-name',
'your-jira-api-token-param-name',
'your-slack-api-token-param-name',
]

Helper function to make the getParameters request

const getParams = async(params) => {
    return parameterStore.getParameters({
        Names: params,
        WithDecryption: true,
    }).promise()
 }

Fetch the parameters and store them in State

// Pull the required values from the Parameter Store and store them in State
const retrievedParams = await getParams(REQUIRED_PARAMS)

// Push these into State for later use
REQUIRED_PARAMS.forEach((param) => {
    state[param] = retrievedParams.Parameters.find((p) => p.Name === param).Value
})

5) Generating the Lambda function role 🔐

We created our Lambda with a default role. In order for it to be able to access the parameters defined in the previous section we need to update this role to support getParameters and we will also individually define the parameters that it will be able to access.

To do this we begin by choosing to view the existing role via the Lambda function Configuration view.

From the role view choose + add inline policy
Use the UI to add the following configuration:

Individually add ARN parameters (or generate the first and then *copy paste and tweak under list ARNs manually*)

its JSON representation should look like this when you're done:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "ssm:GetParameters",
            "Resource": [
                "arn:aws:ssm:your-region-1:your-account-id:parameter/your-github-webhook-secret-param-name",
                "arn:aws:ssm:your-region-1:your-account-id:parameter/your-github-rest-api-token-param-name",
                "arn:aws:ssm:your-region-1:your-account-id:parameter/your-github-dev-reviewer-ids-param-name",
                "arn:aws:ssm:your-region-1:your-account-id:parameter/your-jira-api-user-param-name",
                "arn:aws:ssm:your-region-1:your-account-id:parameter/your-jira-api-token-param-name",
                "arn:aws:ssm:your-region-1:your-account-id:parameter/your-slack-api-token-param-name"
            ]
        }
    ]
}

Once the role has been configured and updated your Lambda function will have access to the parameters 👍

6) Lambda function control logic 🧠

The logic can be found here 🚀
🍄If you've made it this far please show your appreciation by ⭐'n the repo.

For the most part I hope the code is pretty self explanatory and I've added plenty of comments. The file is rather large at over 300 LOC but for simplicity I've kept everything in the lambda-handler.js file.
It's laid out with requires at the top, then helper methods and the main function that receives the request payload from the Gateway is exports.handler.

🎉...and now for the end result... 🎉

This is what it looks like when the notification comes into
the Slack channel 👀

7) Room for Improvement 🧽✨

As this was a proof of concept and I haven't had a lot of time to revise the code don't judge me! 🙈😃

There's certainly room for improvement here.
It would be nice to split the logic out, e.g.

Lambda which handles Github requests
Lambda which handles sending Slack messages
Lambda which handles Jira requests
Query Jira to ensure ticket can be transitioned before making the transition request
etc...

Some of the caveats with the current implementation include:

Fixed to 2 dev reviews for transition (this could potentially be configured per team as we can grab the team prefix from the Jira issue)
It will use the first Jira ticket id it finds in the body. This may not always work as one or more additional Jira tickets could also be listed in the PR body. We could mitigate against this by having a predefined character sequence in our PR template which would denote the beginning of the ticket id
We could potentially support transitioning multiple tickets as some PRs could contain multiple tickets - again we would need to define a way of denoting this in the body, e.g. support an Array of ids - [AT-123, AT-456]

Tools used to create this write-up:
stackedit.io invaluable in-browser Markdown editor ✅
pixlr.com fantastic in-browser image editor (used to create cover image) ✅

DEV Community: Kro