DEV Community: Aastha Bist

GSoC with CHAOSS - Final three weeks!

Aastha Bist — Mon, 24 Aug 2020 19:35:38 +0000

This is a detailed work report of my last three weeks of GSoC with the CHAOSS Badging project.

During these three weeks, I worked on writing and bringing Probot bots together for D&I Badging.

Matt Snell, one of my mentors, took up the task of putting together a frontend of Event Badging which can be accessed through the CHAOSS website. Check it out here!

Counter bot

Throughout building different components for this review process, one of the more confusing things has been settling on a method for Badge assignments. Some of the things we have tried have been as follows:

Deliverable based badge assignment - Assigning Badges based on the number of deliverables. This was the initial approach, and for Events, it was tweaked a bit to include additional criteria for Events. During the first pilot testing, we discovered that this approach was clumsy, and also, not scalable. Meaning adding new metric information to the checklist or the template would mean editing and decide on a new Badge assignment criteria.
Implementation complexity - This approach divided up the checklist items into basic, intermediate, and advanced based on the assumed complexity behind implementing the associated D&I metrics. This approach seemed slightly more scalable than the first one.
It was cumbersome to divide the metrics based on implementation, especially project metrics that were a mixed bag. Like the issue tracker inclusivity metric could include basic checks for assigning labels, but something which is more advanced would be doing relative color coding for similar kinds of labels.
Percentage based badge assignment - This is the current approach as of the end of Pilot testing phase two, and this has proved to be simpler and more scalable than the above two approaches. Here, the Badge calculation is left up to a workflow bot, which:
- Calculates the overall number of markdown checkmarks in a checklist.
- Calculates the number of checks marked positively by the reviewer, excluding the number of initial checks, since they are assumed to be completely positive before badge calculation.
- Comments appropriate badge one invoking a certain slash command in a GitHub issue.

Here is the implementation of the bot-section that calculates and displays Badges - https://github.com/bistaastha/badging-bot/blob/master/src/calculateBadge.js

Reviewer Assignment

Automating reviewer assignment allowed for getting a review set up for starting as soon as the reviewers were available to mark their observations. Right now, reviewers can work on a review if they meet the following two technical requirements:

They are a part of the badging/Reviewers team, which gives them access to edit bot created checklists.
Their GitHub username is added to the .github/reviewers.md in the respective Event or Project repositories.

Reviewer assignment works if both these conditions are met. When an Applicant opens an issue, two usernames would be picked at random from reviewers.md and will be successfully assigned an issue if they are a part of the Reviewers team.

Here is the implementation of the bot-section for assignment - https://github.com/bistaastha/badging-bot/blob/master/src/assignReviewers.js

Second pilot testing

From August 17 to August 21, the second pilot testing for D&I Badging took place. The improved front-end was a major contributor to speeding up the process. Also, it meant that the review-flow shifted to issues rather than pull requests, so applicants are no longer required to deal with creating commits, which also reduced the overhead the process had during the first pilot testing.

Future work

Going forward, I intend to work on supporting separate checklists for Virtual events and add my final changes to the project badging section.

Important links

CHAOSS Badging GitHub organization:
https://github.com/badging

Thanks for reading :)

GSoC with CHAOSS - Weeks five and six!

Aastha Bist — Wed, 22 Jul 2020 13:54:58 +0000

Hello everyone!

This is the biweekly report for weeks five and six of GSoC related to the project CHAOSS D&I Badging.

Bot

During week 5, I explored what automation could be part of the Badging review process.

Currently, there is a bot that comments the review checklist and relevant links on different Pull Request events. There are several possibilities for what kind of bots can be integrated into the process. For week five, I explored two different bots:

A bot which can assign reviewers on using a specific slash (example /assign)command in GitHub comments - This one, while sounding useful at first, is slightly unnecessary at this point since reviewers need Repository Level write permissions, and that gives them enough access to assign themselves to a PR.
The second was the bot to generate badges attached to PR link upon commenting a specific slash command and the badge level. Link - https://github.com/bistaastha/slash-bot

Pilot Testing

Week six was the time for the Pilot Testing of the D&I project. We had three mock applications initiated (2 for Events, and 1 for Projects).

This lead to gaining invaluable information about some process-related things which did not work that well in reality and need to be reworked or reordered or both in order to be useful.

Review Checklist reframing

Review Checklists are the reference list for reviewers to check for specific things in the submitted application/links.

One of the main observations during Pilot Testing was that Qualitative Checks were cumbersome to fill out. I later restructured the Checklists, removing them, leaving only checkboxes followed by what they were for.

Event Review Checklist (revised) - https://hackmd.io/@7f7uN_UHRtOhM5rnON3_HQ/Byo6owJew
Project Review Checklist (revised) -https://hackmd.io/@7f7uN_UHRtOhM5rnON3_HQ/Skqbg0WeD

Thanks for reading!

GSoC with CHAOSS - Weeks three and four

Aastha Bist — Fri, 10 Jul 2020 13:16:38 +0000

Hello reader!

This is my second bi-weekly report about my GSoC work with CHAOSS. These last two weeks included writing a new helper bot for CHAOSS Badging. This report will be a new format than the previous ones, not divided chronologically but rather a combination of the work I did during this time.

So without further delay...

Let's get started!

Templates, checklists and reframing
These last two weeks were about finalizing templates that the applicants would interact with, and the checklists the reviewers would interact with.
With Event Criteria came the new question about badge levels, since all Events are not created equal in terms of metric criteria they would fulfill. For now, we are considering only two Badge levels for Events, pending and passing

Enter the bot
Automation is going to be something that I'll be primarily focusing on in the month of July. For now, to help out with the process (reducing manual work of copying and pasting the Markdown lists from one place to another), I wrote out this small bot called list-bot-test, which does the following things:

When a PR is opened, it greets the applicant using content from .github/applicant-welcome.md of the repository the bot is installed on.
Admins / Owners of the repository (as of now) manually assign reviewers.
A checklist is generated for the assignee (tagging them by username), who has been beforehand given write level or collaborator access to the repository.

Link to the source code - https://github.com/bistaastha/list-bot-test

Coming up next...

Bots which do certain things on being called by a slash commands
Pilot testing for the CHAOSS Badging program.

GSoC with CHAOSS: First two weeks!

Aastha Bist — Wed, 17 Jun 2020 14:26:34 +0000

Hello reader!

This is going to be a biweekly work report of my work in first two weeks of Google Summer of Code with CHAOSS.

Now onto the report...

These two weeks focused on more of CHAOSS Badging content related to reviewers, and later came back to applicants.

Week two [June 8 - June 15]

Event Review Checklist and Applicant questions

Having built the basics of the criteria for Event Reviews, I started working on a checklist which would work out for events.

The initial point for these questions was earlier PR templates. Here are some pointers I used for reframing everything:

What the applicant questions have to be about?

Asking for relevant and necessary links which direct reviewers to the resources they can look into to evaluate a CHAOSS Badging application
Asking for previous work done by the applicant (Event organiser / Project maintainer) or others to make the Project/Event space more inclusive.

What reviewer questions have to be about?

Checking if the links for all necessary document are already in place
Quantitative assessments [bunch of checkboxes] about what the application fulfills, and qualitative feedback about what could be improved.

Associated links:

Event review checklist - https://gist.github.com/bistaastha/0c518c76960aad11bf95cce1f9dee6f6
Revised Project template - https://github.com/bistaastha/project-diversity-and-inclusion/blob/latest/.github/PULL_REQUEST_TEMPLATE.md
WIP Event template - https://hackmd.io/@7f7uN_UHRtOhM5rnON3_HQ/Skb29sQpU

Week one [June 1 - June 7]

Project checklist, Event criteria and reading back and forth through things

For the CHAOSS Badging process, there are going to be at least five D&I metric deliverables which will need to be met for each Project review. To keep track of this information and the milestones an application will need to clear in order to be assigned a suitable badge, the concept (and content) of a Review Checklist is used.

The inspiration for this comes from Journal of Open Source Software reviews (more about their checklists here). These checklists can be built in a way that would determine application eligibility, the path a review will take and what deliverables will be met at the same time.

Focusing on the deliverables, and working over questions asked to applicants on PR templates, I split out each deliverable under the review checklist in two parts: quantitative and qualitative.

As of now quantitative questions can be answered either as a yes or no, while qualitative questions are more about bringing what a reviewer thinks about the application (and what more it requires) in front of an Applicant.

Second part, Event Criteria is more about how events will be evaluated. Since all event related metrics won't be applicable in all cases, we thought dividing them up in different, progressive sections would make sense for Event Reviews. Later, during the weekly meeting, Matt assigned different scales to each criteria.

Associated links:
Project Checklist - https://gist.github.com/bistaastha/b7b7c8ca2139113f9cd7c58a7fa3177b
WIP Event Criteria - https://hackmd.io/@7f7uN_UHRtOhM5rnON3_HQ/H1NFDGn2L

Thanks for reading! You can also check out my community bonding report which is live here.

GSoC with CHAOSS - Community Bonding reports

Aastha Bist — Mon, 15 Jun 2020 18:14:51 +0000

Hello reader!

These are my combined Community Bonding reports related to my GSoC work with CHAOSS.

TL;DR: I had four awesome weeks figuring out more about my project.

Here is the week by week summary for everything under community bonding period :)

Week one [May 5 - May 11]

This week I focused on taking out things out of my proposed work plan and arrange them in the form of issues and to-do lists.

After taking around 48 hours processing that my proposal was selected, explaining to my family what GSoC was and maybe a little bit of excited squealing somewhere in that time, I attended my first meetings on Wednesday as a GSoC student!

From there on mentors Matt S. and Saleh spread out the meeting schedule for CHAOSS Badging work for me and Tola, and Matt G. welcomed us on the D&I WG meeting.

On the meeting, we worked on a portion of Documentation Accessibility abd discussed Badging, and how the GitHub interface might potentially end up hindering non-developer applicants.

Workflow management is an important thing here, and milestones on GitHub work well for me. Starting on I divided my tasks into lists and further created issues to keep track of progress and additional considerations. Also, I created user stories to gain a better sense about expectations and inferences related to CHAOSS Badging, and I would be making changes to them frequently in the near future.

There were a bunch of tasks from the proposal which live as issues here.

Our work and community experience during the rest of the week was discussed on the Sunday Badging Session, where I asked a bunch of questions we had acquired.

Overall, this week was super positive, and this makes me even more excited about the things to come.

Week two [May 12 to May 18]

This week I started working on roles, and on exploring a bit about how Probot based GitHub apps could become a part of the CHAOSS Badging workflow and speed it up.

Roles for CHAOSS Badging is basically categorising participants according to what they would usually do within the Badging workflow. I divided each role into two parts:

Responsibilities - what a certain kind of participant would at least be required to do in the process
GitHub permissions - what things they would be able to do on the GitHub interface

Repository permission levels allow for a more granular approach to what a member of a team on an organisation account GitHub would or would not be able to do on any certain repository, and there are certain levels defined for each role.

Week three and four [May 19 to May 31]

During this time I worked on reading more about how JOSS reviews worked, and then creating guidelines for Badging project reviews based on CHAOSS metrics. In the current form, the requirements state what a Project associated with an application would need to have at the minimum. This also lead to the beginnings of work on Review Checklist, which would be something a reviewer would compare with and mark their observations over.

To simplify roles created during the past week, Matt S. suggested that the permissions required for Badging can be represented in a simpler table. There are two parts to this represetation: the things a person in a perticular role would do under the Badging process and the things they would likely have the ability to do.

Associated links:

Roles permission table: Google Docs | GitHub
Submission Guidelines: https://github.com/bistaastha/project-diversity-and-inclusion/tree/latest/submission
Review Checklist in a Gist: https://gist.github.com/bistaastha/b7b7c8ca2139113f9cd7c58a7fa3177b