DEV Community

Cover image for Publishing and Installing Private GitHub Packages using Yarn and Lerna
Saul Hardman
Saul Hardman

Posted on • Updated on • Originally published at viewsource.io

Publishing and Installing Private GitHub Packages using Yarn and Lerna

I have a collection of snippets and utilities that I frequently reach for when building web stuff. Up until now this code has been managed in a very adhoc fashion – copied and pasted between codebases, un-versioned, and free from the burden of tests 😉

The temptation is to publish these utilities, collectively or individually, on a package registry such as NPM. But, as rewarding and exhilarating as it can be to open source code, it does have its downsides. In particular, publicly publishing a package can signal to other developers that it's production-ready and bring with it the apparent obligation of supporting its use. Alternatively, sometimes the code is sensitive in nature or is not yet mature enough to see the light of day.

Publishing these packages privately is a good solution so long as it's economical and has an efficient enough workflow. To keep the organisational overhead low I'll keep them all in a single repository, following the monolithic repository pattern. (I can't help but feel "minilithic" would be a more appropriate name here.)

NPM doesn't allow users to publish private packages for free, but the GitHub Package Registry does (with strings attached). Given GitHub's recent acquisition of NPM this might well change in the future 🤷‍♂️

Setup the Mono-Repository

I'll use my nuxt-modules private GitHub repository, and the private packages within, as a working example.

Let's get started... In a terminal of your choice create a new project directory and initialise Git and Yarn:

> mkdir nuxt-modules
> cd nuxt-modules
> git init
> yarn init
Enter fullscreen mode Exit fullscreen mode

Enable Yarn Workspaces by configuring the "workspaces" property in package.json:

{
  "name": "nuxt-modules",
  "private": true,
  "workspaces": ["packages/*"]
}
Enter fullscreen mode Exit fullscreen mode

Initialise Lerna with independent versioning enabled:

> lerna init --independent
Enter fullscreen mode Exit fullscreen mode

Configure Lerna to play-nice with Yarn Workspaces and target the GitHub Package Registry in lerna.json:

{
  "packages": ["packages/*"],
  "version": "independent",
  "npmClient": "yarn",
  "useWorkspaces": true,
  "command": {
    "publish": {
      "conventionalCommits": true,
      "message": "chore(release): publish",
      "registry": "https://npm.pkg.github.com",
      "allowBranch": "master"
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

Feel free to customise the other properties, these are just my preferences.

Create the Packages

Populate the packages/ directory with a sub-directory for each package. The directory names shouldn't be prefixed with the scope, but the name field in the package.json should, e.g. packages/nuxt-html-validate will contain a package.json with the name field set to @saulhardman/nuxt-html-validate.

You can create packages using Lerna's lerna create command or by hand. The bare-minimum for an NPM package is a JavaScript entry-point (e.g. index.js) and a package.json.

Development dependencies that are common to all of the packages should be installed in the mono-repository root. As an example, here's the command to install ESLint, passing the -W argument to the add command:

> yarn add --dev -W eslint
Enter fullscreen mode Exit fullscreen mode

A critical step in this process is to run yarn init within each of the directories. It's then necessary to make a minor adjustment to the resulting package.json files to set the repository.directory and publishConfig.registry fields. Here is an example of the @saulhardman/nuxt-html-validate package which is located in the packages/nuxt-html-validate/ sub-directory:

{
  "repository": {
    "type": "git",
    "url": "ssh://git@github.com/saulhardman/nuxt-modules.git",
    "directory": "packages/nuxt-html-validate"
  },
  "publishConfig": {
    "registry": "https://npm.pkg.github.com/"
  }
}
Enter fullscreen mode Exit fullscreen mode

The final result should look something like this:

.
├── .gitignore
├── LICENSE.md
├── lerna.json
├── package.json
├── packages
│   ├── nuxt-html-validate
│   │   ├── README.md
│   │   ├── index.js
│   │   └── package.json
│   ├── nuxt-release
│   │   ├── README.md
│   │   ├── index.js
│   │   └── package.json
│   ├── nuxt-robotize
│   │   ├── README.md
│   │   ├── index.js
│   │   └── package.json
│   └── nuxt-rss
│       ├── README.md
│       ├── index.js
│       └── package.json
└── yarn.lock
Enter fullscreen mode Exit fullscreen mode

Authenticate with the GitHub Package Registry

The next step is to authenticate with the Github Package Registry (replace @saulhardman with your GitHub username):

> npm login --registry=https://npm.pkg.github.com --scope=@saulhardman
Enter fullscreen mode Exit fullscreen mode

To interact with the package repository API, GitHub requires you to create a Personal Access Token (PAT) which you will use in-lieu of your password. Make sure that the 'repo', 'write:packages', 'read:packages', and 'delete:packages' options are selected:

Screenshot of GitHub generate Personal Access Token page with relevant options selected

With that in-hand the .npmrc is configured to point requests for @saulhardman-scoped packages to GitHub (rather than NPM) and provide the PAT as an authToken (replace TOKEN and @saulhardman with your respective values):

//npm.pkg.github.com/:_authToken=TOKEN
@saulhardman:registry=https://npm.pkg.github.com
Enter fullscreen mode Exit fullscreen mode

Even though this Git repository will be private it's good practice not to commit keys and tokens. Accordingly, be sure to amend the .gitignore config to include the .npmrc.

Publish the Packages

Create your private GitHub repository and push your initial commit containing your packages. It's my preference to set the package.version fields to 0.0.0 to begin with. At publish-time you can pass minor or major to have 0.1.0 or 1.0.0 be the initial release version:

> yarn lerna publish minor # initial release 0.1.0
> yarn lerna publish major # initial release 1.0.0
Enter fullscreen mode Exit fullscreen mode

Once you've received a "Package published" response, you will be able to view your packages on the GitHub repository page:

Screenshot of the GitHub repository page showing '4 packages' published

Installing Private GitHub Packages

The permissions workflow surrounding private packages is... not great. There is, as far as I'm aware, no way to scope PATs to organisations, repositories, or packages. The method outlined here will allow you to install all private packages that your GitHub account has access to.

To install a private package all that's required is an .npmrc to assign an access token and configure the scopes. The PAT could be the same one used above or a different PAT with read-only permissions (replace TOKEN with your PAT and @saulhardman with your GitHub username):

//npm.pkg.github.com/:_authToken=TOKEN
@saulhardman:registry=https://npm.pkg.github.com
Enter fullscreen mode Exit fullscreen mode

Only packages in the scope @saulhardman will be installed from the GitHub Package Registry – all others will default to NPM. The yarn add command can be used as usual, e.g.:

> yarn add @saulhardman/nuxt-html-validate
Enter fullscreen mode Exit fullscreen mode

Installing Private GitHub Packages from GitHub Actions

Setting the NODE_AUTH_TOKEN environment variable on the yarn install step should be enough, but in my experience it is not. There is a thread on the GitHub Community Forum documenting a number of people's struggles.

An alternative – whether you're running yarn install directly or using a third-party action such as bahmutov/npm-install – is to construct an .npmrc dynamically using a PAT stored as an encrypted secret:

steps:
  - name: Configure NPM
    run: |
      echo "//npm.pkg.github.com/:_authToken=$NODE_AUTH_TOKEN" > .npmrc
      echo '@saulhardman:registry=https://npm.pkg.github.com' >> .npmrc
    env:
      NODE_AUTH_TOKEN: ${{ secrets.NODE_AUTH_TOKEN }}

  - name: Install Yarn Dependencies
    uses: bahmutov/npm-install@v1
Enter fullscreen mode Exit fullscreen mode

Closing Thoughts

I've created a number of private packages over the last few months – ranging from the Nuxt modules outlined above to Vue components and JavaScript utilities. I've thoroughly enjoyed it so far and I feel the initial overhead will be well worth the reward in the long term.

Discovering a bug in one usage context, fixing it, adding a test case if necessary, and then having that update trickle-down to other projects with very little friction is both satisfying and refreshing.

Additional Resources

Top comments (5)

Collapse
 
hollg profile image
Gary Holland • Edited

This has been super helpful for me, thank you! It got me past a bunch of blockers and made a lot of things clear that I didn't understand before.

I am still hitting one last issue, though: when I try to yarn add my package with yarn add @mygithubusername/mypackage I get a 401 unauthorised error even though I'm using the same PAT as I used to publish it, which has both read and write permissions for packages.

Have you experienced this by any chance?

Thanks again!

EDIT: for anyone else who has this issue in the future, the problem is with yarn. npm works fine! Something to do with yarn and private packages.

Collapse
 
saul profile image
Saul Hardman

Hey @hollg , a few questions to give me a bit more context and make sure we're not missing anything obvious:

Has the package been published to a repository belonging to you? E.g. mygithubusername/project-a containing @mygithubusername/package-a.

As this is an authorization issue, it's possibly related to the configuration of the .npmrc file. Does your local clone of the repository in which you want to yarn add @mygithubusername/package-a (e.g. mygithubusername/project-b) contain a .npmrc file that looks something like this?:

//npm.pkg.github.com/:_authToken=PAT_GOES_HERE
@mygithubusername:registry=https://npm.pkg.github.com

I'm happy to hear that you found the article useful. Hopefully we can get this last issue resolved 👍

Collapse
 
hollg profile image
Gary Holland

Has the package been published to a repository belonging to you? E.g. mygithubusername/project-a containing @mygithubusername/package-a.

Yep, that's right

Does your local clone of the repository in which you want to yarn add @mygithubusername/package-a (e.g. mygithubusername/project-b) contain a .npmrc file that looks something like this?

It sure does!

Just to give you full context:

I'm working in a private repository hosted on my github account, with this setup:

client
    some stuff
    some other stuff
server
    even more stuff
packages
    shared-types

client and server have their own package.jsons managing their own dependencies, builds etc. But I want to share some TypeScript types between them. That's what shared-types is for. So I'm trying to yarn add it to client and server.

Thanks so much for taking the time to look into this with me!

Collapse
 
anthonygushu profile image
Anthony Gushu

This is everything I was looking for and more. Thank u for ur service

Collapse
 
saul profile image
Saul Hardman

Happy it helped 👍