Automatic "Ready for Review" Github Action

TLDR: We wanted a GitHub Action to automatically assign reviewers and mark a draft pull request as "Ready for review" after our test suite passes. The final code can be found in this gist here.

At Potloc, our continuous integration process involves, among other things, a GitHub workflow running on each push that tests the code against our full test suite. This check must pass for a pull request to be merged.

Our test suite has gotten to a size where it is difficult to run on a personal computer in a reasonable amount of time, hence our developers usually rely on this GitHub workflow to run the full test suite.

The process looks something like this:

1. Push code for a new feature
2. Create a new Pull Request in "Draft" mode
3. Wait for all the tests to pass
4. Mark the Pull Request as "Ready for review" and assign reviewers

Note that we consider it a good practice to wait until tests pass before assigning reviewers in order to prevent notifying them only to realize that some more changes are necessary.

In practice, we have an in-house tool to help us automate most of these tasks through the GitHub CLI, but for a long time we didn't have a way to automatically mark a pull request as "Ready for review" when the all tests passed, meaning we had to wait and periodically check the status of each of our PR.

Inspired by Artur Dryomov's excellent post on Autonomous GitHub Pull Requests, we set out to create a GitHub Action to help us automate this.

Solution:

At the moment of creating the draft pull request, we want to be able to specify what to do in the event that all tests pass.

To achieve this, we will use a tag named autoready that we can put on our pull requests to signify that this PR should be automatically marked as "Ready for review" when all tests pass.

In addition, we want to be able to automatically assign reviewers when that happens. For that, we will be using a specific comment format that looks like this:

autoready-reviewers: reviewer1,reviewer2,organization/team1

Our workflow will automatically detect comments like this and assign each of the listed individual or team reviewers.

Implementation

GitHub Workflow configuration

Our workflow should run after each run of our Test workflow and use its output status to determine whether or not to mark the pull request as "Ready for review".
.github/workflows/ready_for_review.yml:

name: Ready For Review
on:
  workflow_run:
    workflows: ["Test"]
    branches-ignore: [main]
    types:
      - completed
jobs:
  mark_as_ready_for_review:
    runs-on: self-hosted
    if: ${{ github.event.workflow_run.conclusion == 'success' }}
    steps:
      - name: Checkout Code
        uses: actions/checkout@v3
      - name: Mark as Ready for Review
        run: bash .github/workflows/mark_as_ready_for_review.sh "${{ secrets.ACCESS_TOKEN }}" "${{ join(github.event.workflow_run.pull_requests.*.number) }}"

This will run our custom script mark_as_ready_for_review.sh after each successful run of the Test workflow.

Some noteworthy points:

• We need the Checkout Code action to get the latest version of this mark_as_ready_for_review.sh script.
• Our script takes a couple of arguments as input:
  1. A GitHub access token of the "user" on behalf of whom we will be performing these automatic actions. In our case, we have a dedicated bot account for this. We store this value in a GitHub secret secrets.ACCESS_TOKEN.
  2. A comma-separated list of all pull request IDs associated with this workflow run. Since a workflow run is attached to a particular commit hash, it is possible that multiple PRs have that same commit hash as HEAD.

Bash Script

Here is the script dissected and explained (scroll to the bottom for the full script):

#!/bin/bash
set -eou pipefail # Make sure we get useful error messages on failure

Our inputs and constants:

TOKEN="${1}"
PR_NUMBERS="${2}"
LABEL="autoready" # the name of the 'label' on the PR used to detect whether or not this script should run
REPO="your-repository" # the name of your repository on GitHub
ORGANIZATION="potloc" # the name of your GitHub organization or user to which the repository belongs

Then, we want to repeat the whole thing for as many pull requests as have been passed as input:

# Split the numbers string (comma-delimited)
for pr_number in $(echo $PR_NUMBERS | tr "," "\n"); do

Fetch the labels from the pull request. We will also need the Node ID of the PR to use GitHub's GraphQL API in a later step, so we also grab this at the same time.

# Get the node_id (and labels) from the PR number
# - https://docs.github.com/en/graphql/guides/using-global-node-ids
# - https://docs.github.com/en/rest/reference/pulls#get-a-pull-request
out=$(curl \
        --fail \
        --silent \
        --show-error \
        --header "Accept: application/vnd.github.v3+json" \
        --header "Authorization: token ${TOKEN}" \
        --request "GET" \
        --url "https://api.github.com/repos/${ORGANIZATION}/${REPO}/pulls/${pr_number}"
      )
node_id=$(jq -r '.node_id' <<< $out)
contains_label=$(jq "any(.labels[].name == \"${LABEL}\"; .)" <<< $out)
comments_url=$(jq -r ".comments_url" <<< $out)

# Check if the PR contains the label we want
if [ "$contains_label" == "true" ]; then
  # Continued below

Note that we use jq to simplify parsing of the JSON body returned by the GitHub API. This needs to be installed on the workers that will run this Workflow.

If the label exists on the PR, then we can mark is as "Ready for review". This API only exists in GitHub's GraphQL API, hence the different request. This is where we make use of the previously-retrieved node_id:

# Mark the PR as ready for review
curl \
  --fail \
  --silent \
  --show-error \
  --header "Content-Type: application/json" \
  --header "Authorization: token ${TOKEN}" \
  --request "POST" \
  --data "{ \"query\": \"mutation { markPullRequestReadyForReview(input: { pullRequestId: \\\"${node_id}\\\" }) { pullRequest { id } } }\" }" \
  --url https://api.github.com/graphql

Delete the label to prevent running this script for this PR:

# Remove the label
curl \
  --request "DELETE" \
  --header "Accept: application/vnd.github.v3+json" \
  --header "Authorization: token ${TOKEN}" \
  --url "https://api.github.com/repos/${ORGANIZATION}/${REPO}/issues/${pr_number}/labels/${LABEL}"

Finally, we want to find which reviewers to assign to this PR. To do this, we fetch all comments on the PR and use a regex to find a comment matching our autoready-reviewers: format we defined:

# Get the comments on the PR
comments_out=$(curl \
                --fail \
                --silent \
                --show-error \
                --header "Content-Type: application/vnd.github.v3+json" \
                --header "Authorization: token ${TOKEN}" \
                --request "GET" \
                --url $comments_url)

# Look for a comment matching the 'autoready-reviewers: ' pattern
# If found, assign the mentionned reviewers to review this PR
jq -r ".[].body" <<< $comments_out | while IFS='' read comment; do
  if [[ $comment =~ autoready-reviewers:[[:space:]]([a-zA-Z0-9,\-\/]+) ]]; then
    all_reviewers=${BASH_REMATCH[1]} # Get the first matching group of the regex (the comma-separated list of reviewers)

Using this list of reviewers, we differentiate between teams (e.g. potloc/devs) and individuals to assign by looking for the / character:

# Split the reviewers between teams and individuals
reviewers_array=()
team_reviewers_array=()
for reviewer in $(echo $all_reviewers | tr "," "\n"); do
  if [[ $reviewer =~ [a-zA-Z0-9,\-]+\/[a-zA-Z0-9,\-]+ ]]; then
    # In the case of a team reviewer, only take the part of the username after the '/':
    slug_array=(${reviewer//\// })
    team_slug=${slug_array[1]}
    team_reviewers_array+=("\"$team_slug\"")
  else
    reviewers_array+=("\"$reviewer\"")
  fi
done

# Join the array elements into a single comma-separated string:
reviewers=$(IFS=, ; echo "${reviewers_array[*]}")
team_reviewers=$(IFS=, ; echo "${team_reviewers_array[*]}")

The very last step is to make the API call to assign these individual and teams as reviewers to the PR:

# Assign reviewers
curl \
  --fail \
  --silent \
  --show-error \
  --output /dev/null \
  --header "Accept: application/vnd.github.v3+json" \
  --header "Authorization: token ${TOKEN}" \
  --request "POST" \
  --url "https://api.github.com/repos/${ORGANIZATION}/${REPO}/pulls/${pr_number}/requested_reviewers" \
  --data "{\"reviewers\":[${reviewers}], \"team_reviewers\":[${team_reviewers}]}"

Conclusion

And that is it! Now, to use this tool we can put the autoready label on a draft pull request and write a comment in the form autoready-reviewers: reviewer1,reviewer2,organization/team1.

In practice, at Potloc, we have a little helper in-house tool do these steps for us using the GitHub CLI and tty-prompt to
ease the selection of reviewers/teams and the formatting of this comment.

And this is what it looks like on GitHub's interface!

Interested in what we do at Potloc? Come join us! We are hiring 🚀

Full code:
https://gist.github.com/jeromepl/02e70f3ea4a4e8103da6f96f14eb213c