I have a collection of snippets and utilities that I frequently reach for when building web stuff. Up until now this code has been managed in a very adhoc fashion – copied and pasted between codebases, un-versioned, and free from the burden of tests 😉
The temptation is to publish these utilities, collectively or individually, on a package registry such as NPM. But, as rewarding and exhilarating as it can be to open source code, it does have its downsides. In particular, publicly publishing a package can signal to other developers that it's production-ready and bring with it the apparent obligation of supporting its use. Alternatively, sometimes the code is sensitive in nature or is not yet mature enough to see the light of day.
Publishing these packages privately is a good solution so long as it's economical and has an efficient enough workflow. To keep the organisational overhead low I'll keep them all in a single repository, following the monolithic repository pattern. (I can't help but feel "minilithic" would be a more appropriate name here.)
NPM doesn't allow users to publish private packages for free, but the GitHub Package Registry does (with strings attached). Given GitHub's recent acquisition of NPM this might well change in the future 🤷♂️
Setup the Mono-Repository
I'll use my nuxt-modules private GitHub repository, and the private packages within, as a working example.
Let's get started... In a terminal of your choice create a new project directory and initialise Git and Yarn:
> mkdir nuxt-modules
> cd nuxt-modules
> git init
> yarn init
Enable Yarn Workspaces by configuring the "workspaces" property in package.json:
{
"name": "nuxt-modules",
"private": true,
"workspaces": ["packages/*"]
}
Initialise Lerna with independent versioning enabled:
> lerna init --independent
Configure Lerna to play-nice with Yarn Workspaces and target the GitHub Package Registry in lerna.json:
{
"packages": ["packages/*"],
"version": "independent",
"npmClient": "yarn",
"useWorkspaces": true,
"command": {
"publish": {
"conventionalCommits": true,
"message": "chore(release): publish",
"registry": "https://npm.pkg.github.com",
"allowBranch": "master"
}
}
}
Feel free to customise the other properties, these are just my preferences.
Create the Packages
Populate the packages/ directory with a sub-directory for each package. The directory names shouldn't be prefixed with the scope, but the name field in the package.json should, e.g. packages/nuxt-html-validate will contain a package.json with the name field set to @saulhardman/nuxt-html-validate.
You can create packages using Lerna's lerna create command or by hand. The bare-minimum for an NPM package is a JavaScript entry-point (e.g. index.js) and a package.json.
Development dependencies that are common to all of the packages should be installed in the mono-repository root. As an example, here's the command to install ESLint, passing the -W argument to the add command:
> yarn add --dev -W eslint
A critical step in this process is to run yarn init within each of the directories. It's then necessary to make a minor adjustment to the resulting package.json files to set the repository.directory and publishConfig.registry fields. Here is an example of the @saulhardman/nuxt-html-validate package which is located in the packages/nuxt-html-validate/ sub-directory:
{
"repository": {
"type": "git",
"url": "ssh://git@github.com/saulhardman/nuxt-modules.git",
"directory": "packages/nuxt-html-validate"
},
"publishConfig": {
"registry": "https://npm.pkg.github.com/"
}
}
The final result should look something like this:
.
├── .gitignore
├── LICENSE.md
├── lerna.json
├── package.json
├── packages
│ ├── nuxt-html-validate
│ │ ├── README.md
│ │ ├── index.js
│ │ └── package.json
│ ├── nuxt-release
│ │ ├── README.md
│ │ ├── index.js
│ │ └── package.json
│ ├── nuxt-robotize
│ │ ├── README.md
│ │ ├── index.js
│ │ └── package.json
│ └── nuxt-rss
│ ├── README.md
│ ├── index.js
│ └── package.json
└── yarn.lock
Authenticate with the GitHub Package Registry
The next step is to authenticate with the Github Package Registry (replace @saulhardman with your GitHub username):
> npm login --registry=https://npm.pkg.github.com --scope=@saulhardman
To interact with the package repository API, GitHub requires you to create a Personal Access Token (PAT) which you will use in-lieu of your password. Make sure that the 'repo', 'write:packages', 'read:packages', and 'delete:packages' options are selected:
With that in-hand the .npmrc is configured to point requests for @saulhardman-scoped packages to GitHub (rather than NPM) and provide the PAT as an authToken (replace TOKEN and @saulhardman with your respective values):
//npm.pkg.github.com/:_authToken=TOKEN
@saulhardman:registry=https://npm.pkg.github.com
Even though this Git repository will be private it's good practice not to commit keys and tokens. Accordingly, be sure to amend the .gitignore config to include the .npmrc.
Publish the Packages
Create your private GitHub repository and push your initial commit containing your packages. It's my preference to set the package.version fields to 0.0.0 to begin with. At publish-time you can pass minor or major to have 0.1.0 or 1.0.0 be the initial release version:
> yarn lerna publish minor # initial release 0.1.0
> yarn lerna publish major # initial release 1.0.0
Once you've received a "Package published" response, you will be able to view your packages on the GitHub repository page:
Installing Private GitHub Packages
The permissions workflow surrounding private packages is... not great. There is, as far as I'm aware, no way to scope PATs to organisations, repositories, or packages. The method outlined here will allow you to install all private packages that your GitHub account has access to.
To install a private package all that's required is an .npmrc to assign an access token and configure the scopes. The PAT could be the same one used above or a different PAT with read-only permissions (replace TOKEN with your PAT and @saulhardman with your GitHub username):
//npm.pkg.github.com/:_authToken=TOKEN
@saulhardman:registry=https://npm.pkg.github.com
Only packages in the scope @saulhardman will be installed from the GitHub Package Registry – all others will default to NPM. The yarn add command can be used as usual, e.g.:
> yarn add @saulhardman/nuxt-html-validate
Installing Private GitHub Packages from GitHub Actions
Setting the NODE_AUTH_TOKEN environment variable on the yarn install step should be enough, but in my experience it is not. There is a thread on the GitHub Community Forum documenting a number of people's struggles.
An alternative – whether you're running yarn install directly or using a third-party action such as bahmutov/npm-install – is to construct an .npmrc dynamically using a PAT stored as an encrypted secret:
steps:
- name: Configure NPM
run: |
echo "//npm.pkg.github.com/:_authToken=$NODE_AUTH_TOKEN" > .npmrc
echo '@saulhardman:registry=https://npm.pkg.github.com' >> .npmrc
env:
NODE_AUTH_TOKEN: ${{ secrets.NODE_AUTH_TOKEN }}
- name: Install Yarn Dependencies
uses: bahmutov/npm-install@v1
Closing Thoughts
I've created a number of private packages over the last few months – ranging from the Nuxt modules outlined above to Vue components and JavaScript utilities. I've thoroughly enjoyed it so far and I feel the initial overhead will be well worth the reward in the long term.
Discovering a bug in one usage context, fixing it, adding a test case if necessary, and then having that update trickle-down to other projects with very little friction is both satisfying and refreshing.
Top comments (5)
This has been super helpful for me, thank you! It got me past a bunch of blockers and made a lot of things clear that I didn't understand before.
I am still hitting one last issue, though: when I try to
yarn addmy package withyarn add @mygithubusername/mypackageI get a 401 unauthorised error even though I'm using the same PAT as I used to publish it, which has both read and write permissions for packages.Have you experienced this by any chance?
Thanks again!
EDIT: for anyone else who has this issue in the future, the problem is with yarn. npm works fine! Something to do with yarn and private packages.
Hey @hollg , a few questions to give me a bit more context and make sure we're not missing anything obvious:
Has the package been published to a repository belonging to you? E.g.
mygithubusername/project-acontaining@mygithubusername/package-a.As this is an authorization issue, it's possibly related to the configuration of the
.npmrcfile. Does your local clone of the repository in which you want toyarn add @mygithubusername/package-a(e.g.mygithubusername/project-b) contain a.npmrcfile that looks something like this?:I'm happy to hear that you found the article useful. Hopefully we can get this last issue resolved 👍
Yep, that's right
It sure does!
Just to give you full context:
I'm working in a private repository hosted on my github account, with this setup:
clientandserverhave their ownpackage.jsons managing their own dependencies, builds etc. But I want to share some TypeScript types between them. That's whatshared-typesis for. So I'm trying toyarn addit toclientandserver.Thanks so much for taking the time to look into this with me!
This is everything I was looking for and more. Thank u for ur service
Happy it helped 👍