DEV Community: Kaung Zin Hein

How to contribute to a large codebase?

Kaung Zin Hein — Thu, 03 Oct 2024 03:19:56 +0000

A step-by-step guide on contributing to open-source projects with thousands of lines of code

As a budding open-source contributor, I was at a loss when I found a lack of resources on how to get started. Other than a few good guides, most of the advice is general and, often, shallow. I felt like the concrete steps were brushed off and left to the readers to figure out for themselves. Which, in a sense, makes for a great learning experience. At the same time, I felt overwhelmed by the number of projects on GitHub, by the number of issues in major projects.

Even for a developer with years of experience, approaching a new codebase takes domain-specific knowledge and patience. On top of that, errors thrown when navigating through the code maze may not even be due to a lack of skills - poorly documented build guides, platform incompatibility, etc., may explain why the repository you just cloned doesn’t work.

With that said, a pre-requisite before diving into a large codebase is to get familiar with contributing to small-to-medium-sized repositories. The best projects to contribute to are ones you’re most knowledgeable about, ones that you’re most familiar with.

But here’s the catch. A lot of what we use daily may not be a simple library that solves one thing. It may well be a full-blown framework or a full-stack application. That’s why it’s important to have honed in on the foundational concepts and the semantics of your language. I don’t mean mastery - which is a continuous pursuit, rather than an accomplishment - I’m referring to an attempt.

Your attempt at trying to master something. For example, I wrote a few npm modules to better familiarize myself with the native Node.js APIs. Why Node.js? Because my goal was/is contribute and learn from other web technologies built on the Javascript ecosystem. I’ll talk more about writing utilities in another guide.

For now, here’s my approach when dealing with a large open-source repository:

1. Use the tool so you understand, what some call, the business problem.
How is it helping customers or developers?

The ideal project to contribute to is one whose API you’ve been using regularly. Whose functions and classes you know well enough to integrate it into your applications and other projects. Familiarity with the public implementations lend itself to understanding the internal code better. If it’s a project you’re completely new to but are willing to dig into, the “Getting Started” guide from the official docs is a great starting point.

2. Start with the what.

Reading a codebase in general terms (the main functions, the entry point, the layers, etc.) is a good practice. But what’s even more effective? A targeted approach.

Trying to solve an issue and following the code trail that the problem leads to, may not cover as many lines of code as passively reading, but it puts to test your understanding of a small subset of the repository. Here's a task for you: open an issue you’ve run into (ideal) or pick up a manageable issue. This approach, compounded over time, issues, and PRs, can lead to deeper understanding.

It’s tricky to define ‘manageable’ because not every project owner assigns ‘good-first-issue’ or ‘up-for-grabs’ labels. And depending on the project, even issues with those beginner-friendly tags may still seem daunting. Which brings us to the next step.

3. Analyze how existing modules and tests are set up.

Take those as reference in laying out the structure of your code and tests. For example, when adding a new plugin, study how other integrations are written and copy the outline of the corresponding tests.

Doing so ensures that you’re following best practices that are specific to the repository at hand and staying in sync with other contributors’ code style.

4. Write down the build and test process.

Note-taking doesn’t get the credit it deserves especially in the software development circle. That’s because note-taking is a skill many haven’t mastered yet, or not willing to try at all.

Instead of looking for patterns, some mark everything down, quickly leading to burnout. Building and testing are processes at the core of most projects. They are the steps you take after making any sort of change. If you’re dealing with a repository more than once (not a YOLO contribution), it’s worth the time to jot down exactly how a change in the code is tested.

Likewise, in a mono-repo, you don’t want to waste time trying to build and test every package when the changes you’ve made only affect one or a few. But wait, can’t the compiling and testing for one package be done via a script each? Probably yes. But when you’re fiddling with more than a single package and their workspace dependencies, the processes tend to involve a couple of steps.

The practice is beneficial in 2 main ways:

A) When you return to an issue after taking a break, all the steps you’d taken last time are right before you. So, you can focus on the problem-solving part immediately, instead of trying to recall what you did the last time for building/testing/modifying configs.

B) When tackling a similar issue, you can follow the same process, instead of figuring out the steps from scratch. This saves time! Which matters a lot when your goal is to ship quality features/fixes as quick as possible.

5. Apply forensics on the commit history of a particular block of code.

When figuring out why a piece of code is implemented, reviewing not only the related PRs, but also the surrounding discussion (comments, reviews, linked issues/PRs) leaves clues - often outright solutions. This step helps you learn about the thought process and the decisions that were formed by other contributors before you arrived. For example, you can check why the author introduced a new utility function or how a bug was fixed.

6. Take feedback from maintainers/major contributors to heart.

No one’s more familiar with the codebase than those who’ve spent months and years working on it. Try to fix the pain points that they’ve brought up, especially in a code review.

On the other hand, don’t be too sorry to point out their misunderstanding of the intent of your code or to state your doubt on their arguments. Open-minded senior devs are willing to elaborate and are, well, open to feedback themselves.

Summary with Actions

1. Use the product/library/framework.
Action: If it’s not a tool you’re already familiar with, try it out to get the hang of it. Start from the official docs.

2. Follow a bottom-up approach.
Action: Open or pick up an issue. Analyze the necessary code files and functions as you try to solve the issue.

3. Look at the structure of existing modules and tests.
Action: Analyze and apply a similar (or sometimes the same) structure to your code.

4. Write down the build and test process.
Action: Record how exactly you compile and test the code involved in your PR.

5. Learn from the past.
Action: Review the commit history and the related conversations, code reviews, and comments.

6. Accept constructive feedback.
Action: Review and try to resolve the code reviews from senior contributors. Don’t hesitate to explain any misunderstanding.

Happy contributing!

Filtering in ExpressJS

Kaung Zin Hein — Fri, 16 Aug 2024 23:29:16 +0000

How I implemented the filter feature in my volunteer management application

This article is part of the series explaining the principles and algorithms implemented in my capstone web application project. You can check out the app here. Or view the source code.

Introduction

A major ingredient for modern management applications (be it content, e-commerce, customer, or financial) is the filter feature, alongside sorting and searching. Here’s how I’ve implemented it in my web app. It filters volunteer events in the backend, according to the filter category sent by the client.

The goal here is to build a custom filter feature using what Express can offer, without depending on any third-party library. So, I decided to make use of the request-response lifecycle and the “locals” property of the response.

Following the modern MCV architecture for ExpressJS applications, I’ve refactored the previous code by separating business logic from the controller. This logic is contained in service functions. Let’s take a look at the filter controller first.

Filter events controller

This controller is an array of request handlers, each wrapped in an express-async-handler function to manage asynchronous errors. Its length depends on the number of filter categories that are stored as constants. As of now, these are:

Event date (date)
Event venue (venue)
Events with open positions (isOpen)
Events that are happening in the future (isUpcoming)

Plus, the service function of the last handler is responsible for curating the filtered results (event IDs) from the above 4 filters and sorting each result based on how many filter categories it falls under. So, I’ve got an array of 5 request handlers.

Let’s examine the request handler for the event venue category (venue).

Filter Events by Venue

const filterEventsHandler = [
  //venue filter
  asyncHandler(async (req, res, next) => {
    if (!objKeysIncludes(req.body, FILTER_OPTIONS.VENUE)) {
      return next();
    }

    const { venue } = req.body;
    res.locals.filteredVenue = await filterEventsByVenue(venue);

    next();
  }),

//...more handlers

Inside the handler, there’s a helper function that checks whether or not req.body (sent from the client) includes the category “venue”. This function serves as a form of data validation for the user input.

const objKeysIncludes = (obj, checked) => {

    if(!(obj instanceof Object)){
        throw new Error('Obj param must be an object')
    }

    return Object.keys(obj).includes(checked)
}

If it’s included, the value of the venue is extracted (e.g., {venue: “External”}), after which it’s processed by the service that filters events by venue. The result is an array of event IDs, which are then stored in res.locals property. The data/variable stored() in res.locals is not shared between requests as they belong only to the current request-response cycle. If the client didn’t include the “venue” filter, this request handler is skipped by returning the next function.

if (!objKeysIncludes(req.body, FILTER_OPTIONS.VENUE)) {
      return next();
    }

    const { venue } = req.body;
    res.locals.filteredVenue = await filterEventsByVenue(venue);

    next();

Even after storing the filtered result in res.locals, next has to be called, moving on to the following handler that checks for another filter category.

Final Handler

After repeating this process for all categories, we arrive at the last handler. Its service function (filterEvents) sorts all the filtered event IDs from res.locals, attaches the filter tags that each event belongs to, and returns a sorted array of events based on the number of tags.

  //sorted filter
  asyncHandler(async (req, res) => {

    const {
      filteredResultsByKey,
      filteredAllIds,
      idsWithTags,
      sortedIdsWithTags,
    } = filterEvents(req.body, res.locals);

    return res.status(200).json({
      filteredResultsByKey,
      filteredAllIds,
      idsWithTags,
      sortedIdsWithTags,
    });
  }),

Let’s see how it works.

The values of the properties in res.locals stored by the preceding filter handlers are assigned to respective variables. Object destructing is not used because the client may not have selected all filter categories, making these variables undefined.

const filterEvents = (filterObj, resLocals) => {

  const filteredVenue = resLocals.filteredVenue;
  const filteredDate = resLocals.filteredDate;
  const filteredIsOpen = resLocals.filteredIsOpen;
  const filteredIsUpcoming = resLocals.filteredIsUpcoming;

//...
Then, the filteredResultsByKey object is declared which will house the name of the filter category (e.g., venue, isOpen) as the key, and the value of the corresponding filtered result as the property. See the example below, where each event ID qualifies as the selected venue.

// {"venue": [id1, id2, id3]}

let filteredResultsByKey = {};

Object.keys(filterObj).forEach((filterKey) => {
    switch (filterKey) {
      case FILTER_OPTIONS.DATE:
        filteredResultsByKey = {
          ...filteredResultsByKey,
          [filterKey]: filteredDate,
        };
        break;

      case FILTER_OPTIONS.IS_OPEN:
        filteredResultsByKey = {
          ...filteredResultsByKey,
          [filterKey]: filteredIsOpen,
        };

        break;

      case FILTER_OPTIONS.VENUE:
        // filteredResultsByKey[filterKey] = filteredVenue

        filteredResultsByKey = {
          ...filteredResultsByKey,
          [filterKey]: filteredVenue,
        };

        break;

      case FILTER_OPTIONS.IS_UPCOMING:
        filteredResultsByKey = {
          ...filteredResultsByKey,
          [filterKey]: filteredIsUpcoming,
        };

      default:
        break;
    }
  });

Next, the property values of filteredResultsByKey are turned into a 1-D array (from a 2-D array). This service function returns an array of event IDs, each with only one occurrence, which is stored in the filteredAllIds variable.

 const filteredAllIds = filterArrSortLoose(
    Object.values(filteredResultsByKey)
  );
//filterArrSortLoose helper
const filterArrSortLoose = (twoDArr) => {

  if (!twoDArr.every((arr) => Array.isArray(arr))) {
    throw new Error("each elem must be an arr - an arr of arr");
  }

  const oneDArr = twoDArr.flatMap((arr) => arr);

  const matchIds = [];

  oneDArr.forEach((id) => {
    if (!matchIds.includes(id)) {
      matchIds.push(id);
    }
  });

  console.log("matchIds from loose filter: ", matchIds);
  return matchIds;
};

Filter Tags

Finally, it’s time for the most challenging task yet. Assign filter tags to each event ID. To do so, we need to relate the filteredResultsByKey object and filteredAllIds array, resulting in an idsWithTags array.

//filteredResultsByKey
{
filterKey1 : [id1, id2, id3],
filterKey2 : [id3, id1, id4]
}

//filteredAllIds
[id1, id2, id3, id4]

//filteredResultsByKey + filteredAllIds -> idsWithTags
[
 {id1: [{filterKey1: filterValue}, {filterKey2: filterValue}]
 {id2: [{filterKey2: filterValue}]},
 {id3: [{filterKey1: filterValue}]}.
 ...
 }
]

Remember that an event ID can appear under more than one filter category. In this case, we assign all the associated filter tags to the event ID object.

In the following nested loop, the outer loop iterates over all filtered IDs, and the inner loop runs through each entry of the filteredResultsByKey object.

  filteredAllIds.forEach((id) => {

    Object.entries(filteredResultsByKey).forEach(([filterKey, result]) => {

      const [_, filterKeyVal] = Object.entries(filterObj).find(
        ([key, _]) => key === filterKey
      );

//...

For each event ID, we must find the corresponding filter category it belongs to by iterating through each entry in the filteredResultsByKey and cross-checking with req.body (filterObj).

If the key from the req.body equals the current key of filteredResultsByKey, we access the value of the key (e.g., in {date: “2023–12–31”, 2023–12–31 is the value). This filter category value (filterKeyVal object) and the filter key are appended as a filter tag for each qualifying event ID.

If the current event id (outer loop) is included in a property array of filteredResultsByKey (inner), there are two possible scenarios:

The event ID object is already included in the idsWithTags array, which means the event qualifies for more than one filter category.
The event ID object is not included yet.

Both scenarios are determined by finding in the idsWithTags array an object whose eventId property has the same value as the current event id. The responsible helper function returns an index number if found, and -1 if not found.

  if (result.includes(id)) {

        const isEventIdAlrExists = elemObjPropValIncludes(
          idsWithTags,
          "eventId",
          id
        );
  //...

//helper to determine if current event ID is already in idsWithTags
const elemObjPropValIncludes = (arr, propKey, checkedPropVal) => {


    //propKey must exists in all objs in the array
    if(arr.some(obj => (!Object.hasOwn(obj,propKey)))){

        throw new Error(`all elem obj in the arr must contain propKey: ${propKey} : as a property key`)
    }


    const included = arr.findIndex(obj => obj[propKey].toString() === checkedPropVal.toString())

    return included
}
Dealing with the second scenario is simple. We just push an object with “eventId” and “filterTags” entries into the parent array.

 idsWithTags.push({
            eventId: id,
            filterTags: [{ [filterKey]: filterKeyVal }],
          });

For the first case, however, there’s an extra step involved. We must map through the idsWithTags array until we arrive at the current event. Then, reassign that event ID object with the additional filter category entry it belongs to.

  const bufferArr = idsWithTags.map((event) => {

            if (event.eventId === id) {

              event = {
                ...event,
                filterTags: [
                  ...event.filterTags,
                  { [filterKey]: filterKeyVal },
                ],
              };
            }

            return event;
          });

          idsWithTags = bufferArr;

After assigning tags to each event, we must sort them in descending order based on the number of tags. At the end of the request-response cycle, this final handler returns sortedIdsWithTags to the client. It’s now the front-end’s job to manage how the events with filtered tags will be displayed.

That’s it! That’s how I’ve implemented filtering in my Express API using the nature of the request-response cycle and res.locals property.

The key takeaways I got from coming up with this feature are:

Request-response cycle
Modular middleware and the role of next()
Data Validation
Higher Order functions
Filtering logic!

Further improvements in the future:

Use Map and Set for a simpler data structure
Implementing Bloom’s filter algorithm
You can interact with the app live here. Or view the code on my GitHub repo.

Why write a library?

Kaung Zin Hein — Fri, 03 May 2024 04:01:54 +0000

This is a blog version of the talk I gave at TorontoJS Lightning Talks event on Apr 30, 2024.

My technical pet peeve

This irritates me - typing a variable’s name beside the variable in the log statement. It isn’t that big of a problem for just a few variables here and there. But many variables multiplied by many files in your app? You do the math.

That's when I realized writing my own logging library could save me some headaches.

The rant continues...

Now, who enjoys commenting out or deleting log statements before pushing their code to production? I really don't.

Fortunately, there are tools available to solve this such as the debug library which is based on the core NodeJS logging utility.

We already have a solution. So why bother? Let’s take a look at my personal use case:

More often than not, a variable is logged from within nested function calls, and across multiple files. When I wanted to quickly check from where a variable is being logged (that happens after you’re debugging and have called it a day and you return to it later), I would write the metadata (the function name) next to the log variable.

Instead of a quick check, this process became quickly tedious. Here was a use case that i hadn't found any JS library addressing (as far as I know).

Time to modify the wheel!

So, I decided to write a module that logs the value of the variable alongside its name without having to type the name manually. And I'll share with you how I did it from scratch.

Of course, this can be done using console.log() with the object destructing { } property.

But what console.log() lacks is the ability to print the function scopes of the logged variable in a developer-friendly way. Here's the output:

By developer-friendly, I mean anyone seeing this log message in their terminal can immediately tell the function scope of the variable being logged. The order of function scopes is parsed from the call stack provided by the NodeJS stack trace API. Hence the name, scope-logger.

Note: “Object.” just means the outermost function is invoked from the global object/topmost level of the file.

Here's the recipe on how to write a library by studying others (in pseudo-code!):

Steps.length = 4

Step 0: Just build it!

“Zero” not only because we are programmers, also because it’s one fundamental step before taking any further. You must be building the Minimum Viable Product (MVP) as soon as possible, or at the very least, a small chunk of it.

You’re not striving for perfection. You just need a working version.

Step 1: Start small

As a reference, pick a small library with similar use cases to your idea.
Here’s my criteria for selecting one:

Popular: not because it’s the new kid on the block, but because it’s been well-maintained for a long while with a large, active user-base.
Well-tested: because quality is important - look for integration/e2e tests and unit tests.
Challenging enough for analysis: you don’t wanna be tackling the Express framework or React on your first go. Pick a library that is considerably smaller, which you can determine by the following factor.
Number of dependencies: one way to tell if a library is not too challenging to be used as study source is based on the production dependencies count. The fewer the better. For example, I chose debug because it only has 1 dependency (ms), while the rest of the code relies on core NodeJS modules - which is exactly what I was looking for - to learn how to build a library from scratch, not off the shelf libraries with many external deps, which in turn are based on more deps. There you go, dependency hell.

Step 2: Break it down smaller

After you’ve picked a library, break it down even smaller. Study how the main functions work by replicating their behaviors: isolate those functions, provide input, and see the output.

This is one of the few times that you should allow yourself to fall into rabbit holes, and really understand the in’s and out’s of each line of code in a function.

For example:

This is the formatArgs() function of the debug library. By isolating it, I learned about ANSI colour escape codes, and where and how the properties of one debug instance is used (namespace, useColours, and this.colour)

Step 3: Publish

After you’ve studied other approaches and built your own module, all there's left to do is hit publish, right? Well, not before you complete these three things.

Semantic Versioning: for every update (major, minor, or patch) made, increment the version number according to semantic versioning.
Changelog/history: track and display the updates you’ve made so far - the version number, the date you published it, and a brief description of those changes.

Documentation: You cannot expect other developers to magically figure out how to use your library without detailed descriptions of the features. It also serves as a roadmap of what you’ve built. You can either dedicate a whole website for it or simply fill in the README.md file.

Note: I left out tests because we should be writing them as the library is being built, not just before hitting publish.

Recipe Recap

Zero: you build a working version.
One: start small by picking a small but challenging enough library to learn from
Two: analyze that library by breaking it down into even smaller pieces
Three: after attaching semantic versioning, changelog, and documentation, hit publish.

Breaks are Great!

These steps are in a loop, repeated until it’s time for a break, which is just as important as the preceding steps.

As a rule of thumb, after publishing your major or minor version, or several patch updates in a row, take your well-deserved break!

Taking a break is great for discovering more use cases as you use the library/module you just made in other projects.

As you might’ve guessed, I did use scope-logger to develop other modules. By doing so, I found features that I really needed, not just stuff that I thought would be cool to have.

When a particular use case keeps popping up, that’s my cue to update the library with a new feature to address it.

Key Takeaways:

Solve your own problems first: As a budding developer, how do you go about determining what projects to pursue? The idea for this library occurred to me as I was building projects for my portfolio. With limited industry connections and programming know-how to cater to a target audience, I felt stuck. But there lurked a hidden opportunity. If you're in the same boat, try this: ditch the latest trends, and double down on understanding the nuts and bolts of one technology, to tackle a problem that's currently bogging you down.
Reinvent the wheel, sometimes: And to solve your developer problem, contrary to conventional advice, you must reinvent the wheel to understand in-depth how the wheel is built. In the long haul, the time and effort put into studying a great library, and building one using core modules and fundamental programming principles, is what’s gonna make you stand out.
Depth && Breadth (not versus): With that said, depth is not the superior path. Tread both. The breadth part involves exploring other libraries, studying their differences, and observing patterns that show up repeatedly.

So, I invite you to try this project idea - create a library by studying great examples. As slow as it may seem, your programming craft will grow exponentially.