GSoC with CHAOSS - Final three weeks!

This is a detailed work report of my last three weeks of GSoC with the CHAOSS Badging project.

During these three weeks, I worked on writing and bringing Probot bots together for D&I Badging.

Matt Snell, one of my mentors, took up the task of putting together a frontend of Event Badging which can be accessed through the CHAOSS website. Check it out here!

Counter bot

Throughout building different components for this review process, one of the more confusing things has been settling on a method for Badge assignments. Some of the things we have tried have been as follows:

Deliverable based badge assignment - Assigning Badges based on the number of deliverables. This was the initial approach, and for Events, it was tweaked a bit to include additional criteria for Events. During the first pilot testing, we discovered that this approach was clumsy, and also, not scalable. Meaning adding new metric information to the checklist or the template would mean editing and decide on a new Badge assignment criteria.
Implementation complexity - This approach divided up the checklist items into basic, intermediate, and advanced based on the assumed complexity behind implementing the associated D&I metrics. This approach seemed slightly more scalable than the first one.
It was cumbersome to divide the metrics based on implementation, especially project metrics that were a mixed bag. Like the issue tracker inclusivity metric could include basic checks for assigning labels, but something which is more advanced would be doing relative color coding for similar kinds of labels.
Percentage based badge assignment - This is the current approach as of the end of Pilot testing phase two, and this has proved to be simpler and more scalable than the above two approaches. Here, the Badge calculation is left up to a workflow bot, which:
- Calculates the overall number of markdown checkmarks in a checklist.
- Calculates the number of checks marked positively by the reviewer, excluding the number of initial checks, since they are assumed to be completely positive before badge calculation.
- Comments appropriate badge one invoking a certain slash command in a GitHub issue.

Here is the implementation of the bot-section that calculates and displays Badges - https://github.com/bistaastha/badging-bot/blob/master/src/calculateBadge.js

Reviewer Assignment

Automating reviewer assignment allowed for getting a review set up for starting as soon as the reviewers were available to mark their observations. Right now, reviewers can work on a review if they meet the following two technical requirements:

They are a part of the badging/Reviewers team, which gives them access to edit bot created checklists.
Their GitHub username is added to the .github/reviewers.md in the respective Event or Project repositories.

Reviewer assignment works if both these conditions are met. When an Applicant opens an issue, two usernames would be picked at random from reviewers.md and will be successfully assigned an issue if they are a part of the Reviewers team.

Here is the implementation of the bot-section for assignment - https://github.com/bistaastha/badging-bot/blob/master/src/assignReviewers.js

Second pilot testing

From August 17 to August 21, the second pilot testing for D&I Badging took place. The improved front-end was a major contributor to speeding up the process. Also, it meant that the review-flow shifted to issues rather than pull requests, so applicants are no longer required to deal with creating commits, which also reduced the overhead the process had during the first pilot testing.