`yarn global` under the hood (2) -- the source

In the previous post, we had a glance at the installation directory of yarn global and got to know how yarn manages our global packages as a project. In this post, we are going to have a look at the corresponding source code of yarn.

The file which will be talked about:

cli/commands/global.js, which defines the yarn global command

Commander flags

First, there is something familiar to us at the end of global.js. If you have not read my previous post on NodeJS CLI, and you want to know more about commander, I strongly recommend you read that.

export function setFlags(commander: Object) {
  _setFlags(commander);
  commander.description('Installs packages globally on your operating system.');
  commander.option('--prefix <prefix>', 'bin prefix to use to install binaries');
  commander.option('--latest', 'upgrade to the latest version of packages');
}

In this snippet, the description of yarn global is set, and two options are added. We can run yarn global -h in the command line to show its help message. The output will be like:

yarn global -h

Usage: yarn global [add|bin|dir|ls|list|remove|upgrade|upgrade-interactive] [flags]

Installs packages globally on your operating system.

...

--prefix <prefix>                   bin prefix to use to install binaries
--latest                            upgrade to the latest version of packages
-h, --help                          output usage information
Visit https://yarnpkg.com/en/docs/cli/global for documentation about this command.

You can see the effect of the snippet above. There are many other options, which are inherited from the global context.

Now we will look at how the subcommands are built, which are, namely, yarn global add|bin|dir|ls|list|remove|upgrade|upgrade-interactive, as is shown in the text snippet above.

const {run, setFlags: _setFlags} = buildSubCommands('global', {
// The subcommands of `yarn global`
})

The subcommands all lie in this function. Semantically, the snippet is used to build subcommands for the command global.

yarn global add

The first subcommand is add, which is the most important one, and the one that will be discussed the most.

async add(config: Config, reporter: Reporter, flags: Object, args: Array<string>): Promise<void> {
  await updateCwd(config);

  const updateBins = await initUpdateBins(config, reporter, flags);
  if (args.indexOf('yarn') !== -1) {
    reporter.warn(reporter.lang('packageContainsYarnAsGlobal'));
  }

  // install module
  const lockfile = await Lockfile.fromDirectory(config.cwd);
  const install = new GlobalAdd(args, flags, config, reporter, lockfile);
  await install.init();

  // link binaries
  await updateBins();
},

This subcommand deals with a weird situation where the user tries to do things like yarn global add yarn, via the condition args.indexOf('yarn') !== -1. You can try that, and the output will be like:

warning Installing Yarn via Yarn will result in you having two separate versions of Yarn installed at the same time, which is not recommended. To update Yarn please follow https://yarnpkg.com/en/docs/install .

Obviously, reporter.lang() has done the transformation from an error code (packageContainsYarnAsGlobal) to a human readable warning message (shown above).

updateCwd()

The updateCwd() function is defined in global.js

async function updateCwd(config: Config): Promise<void> {
  await fs.mkdirp(config.globalFolder);

  await config.init({
    cwd: config.globalFolder,
    binLinks: true,
    globalFolder: config.globalFolder,
    cacheFolder: config._cacheRootFolder,
    linkFolder: config.linkFolder,
    enableDefaultRc: config.enableDefaultRc,
    extraneousYarnrcFiles: config.extraneousYarnrcFiles,
  });
}

It will try to create the global folder and then return the path of the global folder as cwd. This ensures that no matter which directory you are currently in, there will be no difference in the effect of yarn global.

initUpdateBins()

The next part is the initUpdateBins() function.

async function initUpdateBins(config: Config, reporter: Reporter, flags: Object): Promise<() => Promise<void>> {
  const beforeBins = await getBins(config);
  const binFolder = await getBinFolder(config, flags);

  function throwPermError(err: Error & {[code: string]: string}, dest: string) {
    if (err.code === 'EACCES') {
      throw new MessageError(reporter.lang('noPermission', dest));
    } else {
      throw err;
    }
  }

  return async function(): Promise<void> {
    try {
      await fs.mkdirp(binFolder);
    } catch (err) {
      throwPermError(err, binFolder);
    }

    const afterBins = await getBins(config);

    // remove old bins
    for (const src of beforeBins) {
      if (afterBins.has(src)) {
        // not old
        continue;
      }

      // remove old bin
      const dest = path.join(binFolder, path.basename(src));
      try {
        await fs.unlink(dest);
      } catch (err) {
        throwPermError(err, dest);
      }
    }

    // add new bins
    for (const src of afterBins) {
      // insert new bin
      const dest = path.join(binFolder, path.basename(src));
      try {
        await fs.unlink(dest);
        await linkBin(src, dest);
        if (process.platform === 'win32' && dest.indexOf('.cmd') !== -1) {
          await fs.rename(dest + '.cmd', dest);
        }
      } catch (err) {
        throwPermError(err, dest);
      }
    }
  };
}

If first checks the current binaries (beforeBins) and the binary folder (binFolder), then returns an async function.

In this way, the returned function will have the value of beforeBins and binFolder stored in the closure.

When const updateBins = await initUpdateBins(config, reporter, flags); is executed, beforeBins and binFolder are recorded, but the async function will not be executed until Lockfile.fromDirectory and GlobalAdd have been executed. When the async funtion is actually called, the new packages have been installed to the global folder, so getBins(config) will return a new value, and it is assigned to afterBins.

The async funtion tries to create the binary folder, and throws a permission error when the current user has no permission to write to the destination path. Then it gets the value of afterBins.

The key point is the comparison of beforeBins and afterBins. This is done via two iterations.

In the first iteration, binaries in beforeBins are iterated. Those exist in beforeBins but not in afterBins are unlinked — they will be replaced by newer versions, while the rest remain the same — they are not involved this time.

In the second iteration, all binaries in afterBins are unlinked and then linked.

unlink is applied to all current symlinks, which seems unnecessary. What is the intention? Is it because unlink consumes fewer resources than judging whether beforeBins.has(src) in the second iteration? Or is it because there might be issue if the original symlink is not removed even when the new version is just the same?

getBins()

async function getBins(config: Config): Promise<Set<string>> {
  // build up list of registry folders to search for binaries
  const dirs = [];
  for (const registryName of Object.keys(registries)) {
    const registry = config.registries[registryName];
    dirs.push(registry.loc);
  }

  // build up list of binary files
  const paths = new Set();
  for (const dir of dirs) {
    const binDir = path.join(dir, '.bin');
    if (!await fs.exists(binDir)) {
      continue;
    }

    for (const name of await fs.readdir(binDir)) {
      paths.add(path.join(binDir, name));
    }
  }
  return paths;
}

The getBins() function iterates in all registries and collect a Set of paths of all the binaries, which are in the .bin subdirectory of each registry.

GlobalAdd

It extends the Add class, which is defined in add.js and is behind yarn add. This again reveals that yarn global add is a special case of yarn add.

linkBin() (in package-linker.js)

export async function linkBin(src: string, dest: string): Promise<void> {
  if (process.platform === 'win32') {
    const unlockMutex = await lockMutex(src);
    try {
      await cmdShim(src, dest);
    } finally {
      unlockMutex();
    }
  } else {
    await fs.mkdirp(path.dirname(dest));
    await fs.symlink(src, dest);
    await fs.chmod(dest, '755');
  }
}

yarn global add uses this function to do the symlinking. In the case of Windows, lock and unlock are required, while for OSX and Linux, the process is rather straightforward.

yarn global bin

bin is simply used to get the global binary folder.

async bin(config: Config, reporter: Reporter, flags: Object, args: Array<string>): Promise<void> {
  reporter.log(await getBinFolder(config, flags), {force: true});
},

yarn global dir

Similar to bin, dir is used to get the global folder, where the yarn global project is placed.

dir(config: Config, reporter: Reporter, flags: Object, args: Array<string>): Promise<void> {
  reporter.log(config.globalFolder, {force: true});
  return Promise.resolve();
},

yarn global list (and its deprecated version yarn global ls)

async ls(config: Config, reporter: Reporter, flags: Object, args: Array<string>): Promise<void> {
  reporter.warn(`\`yarn global ls\` is deprecated. Please use \`yarn global list\`.`);
  await list(config, reporter, flags, args);
},

async list(config: Config, reporter: Reporter, flags: Object, args: Array<string>): Promise<void> {
  await list(config, reporter, flags, args);
},

These two subcommands both call the list function which is defined as:

async function list(config: Config, reporter: Reporter, flags: Object, args: Array<string>): Promise<void> {
  await updateCwd(config);

  // install so we get hard file paths
  const lockfile = await Lockfile.fromDirectory(config.cwd);
  const install = new Install({}, config, new NoopReporter(), lockfile);
  const patterns = await install.getFlattenedDeps();

  // dump global modules
  for (const pattern of patterns) {
    const manifest = install.resolver.getStrictResolvedPattern(pattern);
    ls(manifest, reporter, false);
  }
}

Note that it will first perform install with no flags ({}). No packages will be installed unless the package.json or yarn.lock has been modified manually or by other programs. The main purpose is to get file paths, as the comment says. In install.js, there is another comment for the getFlattenedDeps() method, saying:

  /**
   * helper method that gets only recent manifests
   * used by global.ls command
   */
  async getFlattenedDeps(): Promise<Array<string>> {
    const {requests: depRequests, patterns: rawPatterns} = await this.fetchRequestFromCwd();

    await this.resolver.init(depRequests, {});

    const manifests = await fetcher.fetch(this.resolver.getManifests(), this.config);
    this.resolver.updateManifests(manifests);

    return this.flatten(rawPatterns);
  }

Then the collected patterns will be iterated via the ls() function:

function ls(manifest: Manifest, reporter: Reporter, saved: boolean) {
  const bins = manifest.bin ? Object.keys(manifest.bin) : [];
  const human = `${manifest.name}@${manifest.version}`;
  if (bins.length) {
    if (saved) {
      reporter.success(reporter.lang('packageInstalledWithBinaries', human));
    } else {
      reporter.info(reporter.lang('packageHasBinaries', human));
    }
    reporter.list(`bins-${manifest.name}`, bins);
  } else if (saved) {
    reporter.warn(reporter.lang('packageHasNoBinaries', human));
  }
}

It will list the binary files of each pattern. Note that saved is assigned false when ls() is called in list(), so only the packages having binaries will be listed, and all such packages will be treated the same, so all the output messages will be like:

info "pm2@3.2.8" has binaries:
   - pm2
   - pm2-dev
   - pm2-docker
   - pm2-runtime

yarn global remove

async remove(config: Config, reporter: Reporter, flags: Object, args: Array<string>): Promise<void> {
  await updateCwd(config);

  const updateBins = await initUpdateBins(config, reporter, flags);

  // remove module
  await runRemove(config, reporter, flags, args);

  // remove binaries
  await updateBins();
},

Similar to add, remove makes use of initUpdateBins to get beforeBins and afterBins and then perform unlinking. The removal of modules uses the runRemove() function from remove.js.

yarn global upgrade and yarn global upgrade-interactive

async upgrade(config: Config, reporter: Reporter, flags: Object, args: Array<string>): Promise<void> {
  await updateCwd(config);

  const updateBins = await initUpdateBins(config, reporter, flags);

  // upgrade module
  await runUpgrade(config, reporter, flags, args);

  // update binaries
  await updateBins();
},

async upgradeInteractive(config: Config, reporter: Reporter, flags: Object, args: Array<string>): Promise<void> {
  await updateCwd(config);

  const updateBins = await initUpdateBins(config, reporter, flags);

  // upgrade module
  await runUpgradeInteractive(config, reporter, flags, args);

  // update binaries
  await updateBins();
},

upgradeand upgrade-interactive are also similar to remove, using runUpgrade (upgrade.js) and runUpgradeInteractive(upgrade-interacitve.js), respectively.

Summary

Now we have gone through the source code of global.js. There is no magic in programming, and you just need to dive deeper.