DEV Community: Owen Kelly

yarn.BUILD: a plugin to run fast parallel builds with yarn v2

Owen Kelly — Sun, 22 Nov 2020 12:00:01 +0000

TLDR -> instructions at yarn.build

Tooling in the Javascript (and Typescript) ecosystem is generally pretty good (no really). But for the longest time on piece of a puzzle I keep facing has been missing.

Most of what I build ends up being applications with more than one deployable artifact. Sometimes they're just a front-end client and a simple server. Other times it is that plus a GraphQL schema, multiple Lambdas, and so on.

Since Lerna appeared both the idea and tooling for Javascript monorepos started to take off. For me, it wasn't until Yarn that linking between local packages became a thing - so say, your front-end client and server packages could depend
on your GraphQL schema package. Yarn v2 took this to another level and added a degree of stability and correctness that make this even more enticing.

But among all of that, building the packages was still a problem. Namely, if a local package depended on another local package being built, you need to orchestrate that somehow. And as much as I tried, it always ended up feeling less than ideal - and certainly not easily repeatable.

In my dabbling into other langauges and tooling, I tried using Bazel. In some ways it's great. I used it to great success with a Golang monorepo with multiple build and testing artifacts.

But for Javascript, well, it's not pretty. Javascript's package ecosystem is reasonably mature at this point. Sure it still has it's flaws, though work is continually being done to address them. (Yarn v2 vendors your node_modules as zip files, for example.)

Javascript and Bazel mix reasonably well. But Javascript and NPM or Yarn don't. Having two systems both trying to own dependency management is just painful.

Yarn v2

At the start of 2020 I started playing around with Yarn v2.
Plug'n'Play and the zipfs approach to vendoring dependencies had me
intrigued immediately. Both are areas I've found our tooling lacking.

In practice, at the start of 2020 support was growing but still limited. Enough things worked to convince me this is a workable approach though.

And then I discovered that Yarn v2 was far more hackable than v1. Not only that, with the new focus on correctness and reproducibility the only thing missing to make a Bazel for Javascript was the build tool itself.

How it works

At a high level the plugin is pretty straight forward. Yarn's already built the dependency graph. We just need to know where to start from on that graph. That's also relatively easy. If you're in the directory of a package, well we can work out which package it is. If not, we can build everything.

Once we know what we need to build, we have a look at anything it depends on, and if they depend on anything. And so on. Once we know that, we can build a plan for how to build it all with as much parallelisation as you have threads.

Bonus Feature

Having worked all of that out, there was one last feature I really wanted to include. And to be honest, it's the main thing I've wanted from the start.

I wanted a command that will create a zip file ready for AWS Lambda, Kubernetes or Docker.

Now I hear what you're saying "what about the Serverless
framework?". While I know it's a valuable tool, and plenty of us use it with much success. It's never fit my requirements. Any abstraction over Cloudformation that obscures the actual Cloudformation templates always ended up getting in my way.

Yarn PnP makes this a bit hard. Locally linked packages make this really hard. And vendoring node_modules makes this near impossible.

Especially in a monorepo where your dependencies are shared and hoisted up. Meaning you can't just copy the adjacent node_modules folder.

We need something smarter.

Much smarter.

Once again though, we have access to the dependency graph we've already defined for Yarn. Combining this, with the zipfs tooling in Yarn v2, it was not too much extra work to get this going.

Now, in a package running yarn bundle copies the whole workspace (so likely you repository) into a temporary folder. Then, using Yarn's dependency graph, we chuck out everything we don't need. Delete the local packages that aren't used,
and the vendored packages that aren't used.

At this point we have a zip file that looks like your repo, with a bunch of stuff chucked out. Which is great, but there's two remaining issues to tackle.

The first, Yarn PnP. It's great, and it means our zip file is faster to work with and smaller than a node_modules directory. But, we need to run everything via the pnp.js file.

The second, is that as we're recreating the whole workspace in the zip file, and not just your package, you need to know exactly where it is to specify your entrypoint or index file.

The solution was pretty simple. Drop a file called entrypoint.js at the root of the zip file. Have it load pnp.js first, then load your file, referenced in
main in your package.json.

And just like that, yarn bundle can create a zip file ready to run in Lambda et al.

How to get started

This all sounds great, but how do you actually use it?

First, you have to be using Yarn v2. If you're not already here's a great getting started guide.

Next install the plugin by running the following command in your Yarn workspace:

yarn plugin import https://yarn.build/latest

This command downloads and installs (or updates) the yarn.build plugin to the latest version.

The plugin is downloaded and vendored in you repository. It's not re-downloaded on every build.

Currently there's two commands you can run.

yarn build which will run the build script defined in package.json.

And yarn bundle which will create the zip file described above, ready for Lambda et al.

There's still plenty of work to be done on this plugin, but in it's current state it's ready to start being used.

You can find the source here github.com/ojkelly/yarn.build.

And the site is at yarn.build

Let me know what you think here or on twitter @ojkelly

Photo by Danny Sleeuwenhoek on Unsplash

The Four Layers to Great Documentation

Owen Kelly — Thu, 07 Jun 2018 13:05:58 +0000

When I started to publish some open source projects, I did a deep dive into how to document them. This is the most useful method I’ve found.

Good documentation requires four layers, each building on the previous layer.

Getting Started
Guides
Concepts
API

The best way to write these is to start from the bottom (API) at work your way up to the top (Getting Started). Incidentally, readers of your documentation will start from the top, and only occasionally find their way to the bottom. Even then, starting writing from the bottom up helps to better inform the top layers.

API

This is the only section of your code that can be auto-generated.
Documentation is written exclusively for humans, and that usually means a
human needs to write it. You can auto-generate your API docs from your source
code, if you have commented it.

This should be comprehensive documentation of how to use the API of the project, typically generated from the source code.

You should be documenting your code, so that when you come back to it in 12 months you can understand it again. Be selfish, and kind to your future self, let them know what you did and why. You will not remember. (This side benefit here, is everyone else can understand too).

Concepts

What are the fundamental building blocks for this project? Try to break the project up into pieces, and write a detailed explanation of each one.

The goal here is to explain the building blocks of your project. Expect anyone who will implement your project to read through this section. You want this to be detailed and thorough. You don’t need an essay on each, a few hundred words and a diagram might be enough.

If you need to write a long page for a concept, be kind to the reader, and try to create sections for easy reference. While some will read it from top to bottom, it’s more likely it will be skimmed for the most relevant part for the reader at that time. Some headings and a table of contents is all you need.

Guides

A guide or tutorial, is the ideal way to explain how to use your project to solve a specific use case.

They should be written with the end use in mind, considering their use case and knowledge. It’s better to over explain, than not. Something may be simple and obvious to you, but unheard of to your user.

Think about why you started the project in the first place, and make a list of the use cases you wanted to solve. It may help to write a small user story for each, something like: “As a project maintainer, I would like to know how to write good documentation, so that I can get more users of my project”. The typical format to produce a user story is: “As a [role], I can [feature] so that [reason]”.

In that last paragraph, I did two things. First, I said making a list of user stories will help with working out what guides to write, but then I gave an example of a user story, and the template to write them. You could have searched to find out what a user story is, but by explaining it right there I’ve not only saved you a bit of time — I’ve removed the ambiguity of what we’re talking about. A user story may mean something slightly different to you.

But back to guides. You want to have at least 3 to 5 to explain how to use your project. It could be how to use it on each particular operating system, or how to integrate with the 3 most popular frameworks.

Or, how to integrate with a focus on a particular concept from the previous section.

Most likely you’ll have 1-3 guides when you publish your project. As issues and frequently asked questions appear, turn these into guides.

Getting Started

This is the single most important section you will write, and does not consist of a single ‘Getting Started’ page.

If you’re project is a small library, you may be able to get away with just a Getting Started section. Typically in this scenario, you would only have one main concept, and a set of examples functioning as the Guide section.

First, work out what your landing pages are. You may have a website, a GitHub readme. You might also have a page on npm, godoc, pypi, or another package manager. Normally all of these except the website will be direct copies of your readme, so lets start with that.

Readme.md

To start you need to answer the following questions clearly, and concisely:

What is this project called? What does this project do?

Then, explain how to get it. Maybe it’s a download via a package manager, or a link to a releases page containing binaries.

Now, show a simple use case. Here you’re showing the public interface of the project. For many projects the best Getting Started, is literally showing the implementation code. Don’t show everything, just enough to run it once.

We’re now in the footer of the readme. It’s nice to add information on how to develop on this project, so information on how to setup a development environment, and examples of how to run the tests should be here.

It’s strongly recommended to include information about how to contribute to the project, where to raise issues, the process for new features and pull requests.

You should include information about SemVer if you’re using it. (If you’re not using SemVer, please consider it — your users will thank you).

Many projects like to acknowledge their major authors and contributors in the footer area too.

And finally, at the very bottom include a license. This is typically a note of the license type, with a link to a LICENSE.md file. You must include a license, without one people will be wary to use your project.

Badges

You don’t need them. Although they can be useful to your users. Use your own discretion here, don’t go overboard. They can provide a bit of social proof.

Website

If you also have a website (you don’t always need one), your “Getting Started” consists of the front page, and a dedicated getting started page. If you want to include them both on the once page, just ensure theres a link that can take a reader directly to the “Getting Started” section.

The front page usually covers the same information as the Readme, and is often wrapped in a nice design. Don’t go overboard here, you’re focus is on readability above all else. Sometimes a cool graphic or animation can get a project shared around — but don’t focus on that at the expense of your actual users.

Documentation is the best marketing tool for your project, and the best indicator of the trustworthiness and maturity. It takes effort to document a project well, but if you want users it’s worth it.

As an example of this approach to documentation you can look at a project I recently published Bunjil.