DEV Community: Mohamed ELIDRISSI

Exploring the Most Commonly Used Folder Names in Popular NPM Packages

Mohamed ELIDRISSI — Thu, 23 Feb 2023 13:33:50 +0000

I was listening to Episode 575 of Syntax podcast last week
when I heard Wes talk about an idea he had for the show. It was about writing a script that will fetch 8000 repos and analyze
the most common directories and then explain the purpose of each one. I thought it would be interesting to try this with NPM packages and share the result, so here it is.

Collecting the data

🏗️ Building a list of the most popular NPM packages

The first step is to gather the data from a significant number of NPM packages, this seemed easy at first because I was planning to use the NPM Registry API
to get this information, but it doesn't seem to allow searching by popularity alone. So I'm going to have to use a manually curated list of popular packages for this purpose 🤷‍♂️.

📁 Fetching the folder structure of each package

Now that we have the list of packages, we will need the repository URL of each one to get their directory structure. To accomplish this, I'm going to hit the NPM Registry API to retrieve the data about each package which should contain the repository URL, then I will request the GitHub REST API to get the contents of the root folder:

import packages from "./packages.json" assert { type: "json" };

const NPM_REGISTRY_API_URL = "https://registry.npmjs.com";
const GITHUB_API_URL = "https://api.github.com";

const pkgs = await Promise.all(
  packages.map(name =>
    fetch(`${NPM_REGISTRY_API_URL}/${name}`)
      .then(res => res.json())
  )
);

const repos = await Promise.all(
  pkgs.map(pkg => {
    const githubUrl = new URL(pkg.repository.url);
    const repoNamespace = githubUrl.pathname.replace(/\.git$/i, "");

    return fetch(`${GITHUB_API_URL}/repos${repoNamespace}/contents`)
      .then(res => res.json());
  })
);

📊 Counting the directories

We now have a two-dimensional array representing the contents of every package, the last step is to filter out individual files and aggregate the result to count the occurence of every directory:

const directories = new Map();

repos.flat()
  .filter(file => file.type === 'dir')
  .forEach(dir => {
    let count = directories.get(dir.name);
    directories.set(dir.name, count ? ++count : 1);
  });

I've made the full code available in a gist.

It's time to take a look at the result!

Analyzing the data

The above chart is enough to give you an idea about the most common directories, but we're interested in more than just numbers,
so let's dive deep and explain what's in each folder:

.github: With more than 60 occurrence, it's not surprising to see this folder on the top spot given that all of the analyzed packages are
hosted on GitHub. .github is a special folder that can contain a variety of config files and templates related to GitHub Actions workflows,
as well as other organizational files for a GitHub repository such as CONTRIBUTING.md, CODE_OF_CONDUCT.md, SECURITY.md and more. Example from axios.
tests/test/__tests__: Writing tests is an important practice in software development, and in NPM packages you'll often see them
stored in one of these folders. It can also be used to hold testing helpers and utility functions. Example from express.
docs: Documentation is an essential part of any package, as it provides users with the information they need to understand how to use it
and how it works. The documentation usually includes usage instructions, API documentation, and more. It can also be included directly
in the repository's README.md file, but it's often split into multiple files and stored in this folder for ease of navigation and maintenance.
Although the documentation files can be in any format, the most common one is Markdown. Example from node-fetch.
lib: The lib folder, short for "library", is mostly used to store the actual source code of the package, but it can also be used to store
third-party code, utilities and helpers. Example from passport.
examples: Good documentation goes well with good examples, not only does it provide a practical demonstration on how to use the package, but it also
allows developers to quickly get up to speed and start using the package in their own projects. Example from express.
src: Similar to lib, the src folder is also used to organize code, allowing for easy access to the main codebase. Example from yaml.
scripts: Maintaining a package can be a lot of work, there's lots of repetitive tasks that need to be done often such as building the package for
different targets, preparing a new release, etc. This is where automation scripts can help, and if a package has any, there's a good chance you'll find
them in this folder. Example from history.
packages: If you see a directory with this name, you're most likely looking at a monolithic repository
(monorepo). Monorepos contain the code for different projects/sub-components within a single repository, this offers several benefits such as simplified
dependency management, and improved code reusability to name a few. Example from react.
bin: Sometimes it may be desired or even crucial for a package to provide a command line interface, take a testing framework
like jest as an example. NPM allows packages to publish executable binaries
for this purpose, and as a convention they're usually placed in this directory. Example from nanoid.
benchmarks: This directory contains benchmark tests that help measure the performance of the package's code, these tests can be are very useful
when experimenting with performance optimizations, and to ensure no slowdowns are introduced between releases. Example from graphql.
.husky: Git hooks are custom scripts that run in response to some event (e.g. before a
commit is created), and they can choose to abort that event under certain conditions. One of their main drawbacks though is that they live inside
the .git folder, which means they cannot be directly versioned like the rest of the project. This folder is used by the popular Husky
package that makes it possible to include Git hooks with your project and it takes care of installing them to their appropriate location so they
can be detected by Git. Example from uuid.

Conclusion

I hope this exploration helped you gain insights about the common naming practices and conventions used in the NPM ecosystem of packages, as well as highlight their importance in improving the accessibility and readability of your project. You can use this knowledge if you plan to build your own
package, contribute to an existing one, or simply navigate your way around if you're browsing the source code.

Git, A Look Under The Hood - Prologue

Mohamed ELIDRISSI — Wed, 19 May 2021 20:16:03 +0000

Git is a free and open-source Distributed Version Control System (DVCS) initially created by Linus Torvalds in 2005 following the BitKeeper controversy after none of the existing systems met his needs. Because Linus intended to use it for the Linux Kernel development, Git was designed to take into account performance, distribution, and safeguards against data loss. Fast-forward 16 years later and it is the de facto Version Control System (VCS) for software and an indispensable tool for every Software Engineer.

You can never understand everything. But, you should push yourself to understand the system.

- Ryan Dahl (Creator of Node.js)

Git has excellent documentation and even a free book, but most people don't bother learning about it and instead only try to memorize the commands they use in their day-to-day work or rely only on GUIs that make everything seem magical, this leads to them peeling their hair in frustration as they start facing all kind of issues (looking at you, merge conflicts!). While one can certainly use Git with just basic knowledge, knowing how it all works under the hood will give you extra confidence and will make you a distinguishable engineer.

In this series, we will be going down the rabbit hole to try and demystify how Git works internally, and at the end, you'll come to know that Git is not magic, its design model is what made it the powerful tool that we all love and hate (not anymore) today, so we must understand that before going any deeper.

PS: This is not a beginner's guide so a basic knowledge of Git is required

Git is Distributed

We hear about Git being a DVCS all the time, but do you really know what that means? Before we understand the meaning of distributed, let's first talk about Centralized VCSs (CVCS).

In CVCSs, nothing is stored locally, so all actions depend on a single remote repository. To commit code, view the logs, or compare your changes with other people's, you'll need to be online. Being centralized also means having a single point of failure. If the repository is deleted by mistake or the disks on the server crash, everything will be lost.

Now let's go back to DVCSs, when you do a git clone, Git downloads the whole history of the repository and stores it in – its local database – the .git folder. This means that you can still commit your work locally when you're on an airplane or when the internet is down and push it to a remote repository when you're back online, and essentially all developers who cloned this repository now have a backup in their machine. This also has the benefit of being extremely fast, as most operations don't need to reach for a server.

Git Takes Snapshots

This is one of the key design differences between Git and other VCSs. These other systems store information as an initial file and the changes (diffs) made to it over time, Git follows a different approach and instead takes a snapshot (think a picture) of your whole files when you do a commit, now you might think that's expensive if you have a lot of files, but as we'll come to see in the next part, Git is smart enough to not store the same file twice if it hasn't changed and instead references the previous version of that file.

The important point to get from this is that Git acts like a mini filesystem, but with superpowers.

Integrity is Built-in

Before Git stores any file, it computes a 40 character long hexadecimal string known as a hash that serves as an identifier. To accomplish this, Git uses a popular hashing algorithm called SHA-1. We'll see how it does it in the next part, but it will look similar to this: a0b65939670bc2c010f4d5d6a0b3e4e4590fb92b. This makes it impossible for your files to change or get corrupted without Git knowing about it. Note that SHA1 is considered insecure nowadays, especially after Google released SHAttered –the first practical collision attack against SHA1– in 2017 and Git is planning to transition to SHA-256.

File States in Git

All files in a Git repository can either be modified, staged, or committed. Your files go into the first state when git detects that it has changed from the last version (remember the hash?) stored in its database. To save this change, you'll need to commit it, but first, you have to tell Git which changes you want to "prepare" for the next commit. This is accomplished by adding them to the staging area. Once committed, Git takes a snapshot of your files and saves it permanently in its database.

Conclusion

This was a short introduction to this series where we will uncover the nitty-gritty details behind Git internals. We started by first understanding the key aspects of the Git design model. In the next part, we'll see how this all comes down together as we discuss Git Objects.

Originally published in my personal blog: The Self-Taught Blog