DEV Community: Danielle Adams

Building a Monorepo with Yarn 2

Danielle Adams — Tue, 22 Dec 2020 21:19:13 +0000

In true JavaScript fashion, there was no shortage of releases in the JavaScript ecosystem this year. This includes the Yarn project’s release of Yarn 2 with a compressed cache of JavaScript dependencies, including a Yarn binary to reference, that can be used for a zero-install deployment.

Yarn is a package manager that also provides developers a project management toolset. Now, Yarn 2 is now officially supported by Heroku, and Heroku developers are able to take advantage of leveraging zero-installs during their Node.js builds. We’ll go over a popular use case for Yarn that is enhanced by Yarn 2: using workspaces to manage dependencies for your monorepo.

We will cover taking advantage of Yarn 2’s cache to manage monorepo dependencies. Prerequisites for this include a development environment with Node installed. To follow these guides, set up an existing Node project that makes use of a package.json too. If you don’t have one, use the Heroku Getting Started with Node.js Project.

Workspaces

First off, what are workspaces? Workspaces is Yarn’s solution to a monorepo structure for a JavaScript app or Node.js project. A monorepo refers to a project, in this case, a JavaScript project, that has more than one section of the code base. For example, you may have the following set up:

/app
 - package.json
 - /server
   - package.json
 - /ui
   - package.json

Your JavaScript server has source code, but there’s an additional front end application that will be built and made available to users separately. This is a popular pattern for setting up a separation of concerns with a custom API client, a build or testing tool, or something else that may not have a place in the application logic. Each of the subdirectory’s package.json will have their own dependencies. How can we manage them? How do we optimize caching? This is where Yarn workspaces comes in.

In the root package.json, set up the subdirectories under the workspaces key. You should add this to your package.json:

"workspaces": [
    "server",
    "ui"
]

For more on workspaces, visit here: https://yarnpkg.com/features/workspaces

Additionally, add the workspaces-tools plugin. This will be useful when running workspace scripts that you’ll use later. You can do this by running:

yarn plugin import workspace-tools

Setting up Yarn

If you’re already using Yarn, you have a yarn.lock file already checked into your code base’s git repository. There’s other files and directories that you’ll need up to set up the cache. If you aren’t already using Yarn, install it globally.

npm install -g yarn

Note: If you don’t have Yarn >=1.22.10 installed on your computer, update it with the same install command.

Next, set up your Yarn version for this code base. One of the benefits of using Yarn 2 is that you’ll have a checked in Yarn binary that will be used by anyone that works on this code base and eliminates version conflicts between environments.

yarn set version berry

A .yarn directory and .yarnrc.yml file will both be created that need to be checked into git. These are the files that will set up your project’s local Yarn instance.

Setting Up the Dependency Cache

Once Yarn is set up, you can set up your cache. Run yarn install:

yarn

Before anything else, make sure to add the following to the .gitignore:

# Yarn
.yarn/*
!.yarn/cache
!.yarn/releases
!.yarn/plugins
!.yarn/sdks
!.yarn/versions

The files that are ignored will be machine specific, and the remaining files you’ll want to check in. If you run git status, you’ll see the following:

Untracked files:
  (use "git add <file>..." to include in what will be committed)
    .gitignore
    .pnp.js
    .yarn/cache/
    yarn.lock

You’ve created new files that will speed up your install process:

.pnp.js - This is the Plug’n’Play (PnP) file. The PnP file tells your Node app or build how to find the dependencies that are stored in .yarn/cache.
.yarn/cache - This directory will have the dependencies that are needed to run and build your app.
yarn.lock - The lock file still is used to lock the versions that are resolved from the package.json.

Check all of this in to git, and you’re set. For more information about Yarn 2’s zero-install philosophy, read here: https://yarnpkg.com/features/zero-installs

Adding Dependencies to Subdirectories

Now that Yarn and the cache are set up, we can start adding dependencies. As initially shown, we have a server directory and a ui directory. We can assume that each of these will be built and hosted differently. For example, my server is written in TypeScript, using Express.js for routing, and running on a Heroku web dyno. For the front end app, it is using Next.js. The build will be run during the app’s build process.

Add express to the server dependencies.

yarn workspace server add express

Additionally, add @types/express and typescript to the devDependencies. You can use the -D flag to indicate that you’re adding devDependencies.

yarn workspace server add @types/express typescript -D

We now have our dependencies in our server workspace. We just need to create our ui workspace. Next, build a Next.js app with the yarn create command.

yarn create next-app ui

Finally, run yarn again to update the cache and check these changes into git.

Running Scripts with Workspaces

The last piece is to run scripts within the workspaces. If you look through your source code, you’ll see that there’s one global cache for all dependencies under your app’s root directory. Run the following to see all the compressed dependencies:

ls .yarn/cache

Now, lets run build scripts with workspaces. First, set up the workspace. For server, use tsc to build the TypeScript app. You’ll need to set up a TypeScript config and a .ts file first:

cd server
yarn dlx --package typescript tsc --init
touch index.ts

yarn dlx will run a command from a package so that it doesn’t need to be installed globally. It’s useful for one-off initializing commands, like initializing a TypeScript app.

Next, add the build step to the server/package.json.

"scripts": {
    "build": "tsc",
    "start": "node index.js"
},

Change directories back to the application level, and run the build.

cd ..
yarn workspace server build

You’ll see that a server/index.js file is created. Add server/*.js to the .gitignore.

Since we already have build and start scripts in our Next.js app (created by the yarn create command), add a build script at the root level package.json.

"scripts": {
    "build": "yarn workspaces foreach run build"
},

This is when the workspaces-tool plugin is used. Run yarn build from your app’s root, and both of your workspaces will build. Open a second terminal, and you’ll be able to run yarn workspace server start and yarn workspace ui start in each terminal and run the Express and Next servers in parallel.

Deploy to Heroku

Finally, we can deploy our code to Heroku. Since Heroku will run the script is in the package.json under start, add a script to the package.json.

"scripts": {
    "build": "yarn workspaces foreach run build",
    "start": "yarn workspaces server start"
},

Heroku will use the start script from the package.json to start the web process on your app.

Conclusion

There are plenty more features that Yarn, and specifically Yarn 2, offers that are useful for Heroku developers. Check out the Yarn docs to see if there are additional workspace features that may work nicely with Heroku integration. As always, if you have any feedback or issues, please open an Issue on GitHub.

Celebrating 25 Years of JavaScript

Danielle Adams — Fri, 04 Dec 2020 22:04:10 +0000

JavaScript turns 25 years old today. While it’s made an impact on my career as a developer, it has also impacted many developers like me and users around the world. To commemorate our favorite language, we’ve collected 25 landmark events that have shaped the path of what the JavaScript ecosystem looks like today.

1995

1) JavaScript is created

In 1995, Brendan Eich, a developer at Netscape, known for their Netscape browser, was tasked with building a client-side scripting language that paired well with Java. While it may not be the language that you know and love today, JavaScript was written in 10 days with features we still use today, such as first-class functions.

1997

2) ECMAScript is released

Despite JavaScript being created 2 years before, there was a need to create open standards for the language if it would be used across multiple browser types. In 1997, Netscape and Microsoft came together under Ecma International to form the first standardization of the JavaScript language, resulting in the first iteration of ECMAScript.

1999

3) Internet Explorer gets an early XMLHTTP Object

Some will recall using iframe tags in the browser to avoid reloading a user’s page with a new request. In March of 1999, Internet Explorer 5.0 is shipped with XMLHTTP, a browser API that could enable developers to take advantage of background requests.

2001

4) JavaScript gets its own data format

In 2001, JSON was first introduced via json.org. In 2006, an RFC proposing JSON, JavaScript Object Notation, was opened for review with the proposal of more than one type of HTTP call to fulfill a website: one that would fulfill a browser’s needs and the other would provide application state. Thanks to its simplicity, JSON would gain traction as the standard and continues to be used today. (Source)

2005

5) Shifts towards AJAX

After other browsers followed Internet Explorer in supporting background requests for updating clients without reloading pages, a researcher penned the term as Asynchronous JavaScript and XML, or AJAX, highlighting the shift in web development and JavaScript to asynchronous code. (Source)

2006

6) First publicly released Developer Tools

With more complexity being enabled in the browser, there was a need for tooling to keep up. Firebug was created in 2005 as the first Developer Tool to debug in Mozilla’s Firefox browser. It was the first piece of tooling that provided developers the ability to inspect and debug directly from the browser. (Source)

7) jQuery is released

jQuery can be considered the pioneer of what we know today as modern front-end web development, and it has gone to influence many libraries and frameworks today. At its height, being a JavaScript developer and being a jQuery developer were interchangeable. The library extends the JavaScript language to easily create single-page applications with DOM-traversal, event handling, and more.

2008

8) Creation of V8

As websites went from HTML pages to JavaScript applications, it was imperative that the browsers hosting these applications keep up. From 2007 to 2010, many browsers made major releases to keep up with the growing demand from JavaScript compute power. When Chrome was released, the browser’s JavaScript engine, V8, was released as a separate project. V8 was a landmark project with Its “just-in-time” compiler and would be used in future projects as a reliable and fast JavaScript runtime.

9) The first native Developer Tools

In addition to the release of V8, Chrome introduced developers to another innovation: Developer Tools that are native to the browser. At the time, features only included element inspection and looking at resources, but the tool was an upgrade from the current tooling and would influence an entire suite of developer tools for front-end development. (Source)

2009

10) CommonJS moves to standardize modules

In an effort to modularize JavaScript code and take code bases from single file scripts to multi-file source code, the CommonJS project was an effort to elevate JavaScript into language for application development. CommonJS modules would influence the Node.js module system.

11) Node.js takes JavaScript to the back-end

JavaScript had gained momentum as a language for the browser for many years before making its way to the back-end. In 2009, an engineer at Joyent, Ryan Dahl, introduced Node.js, an asynchronous event-driven JavaScript runtime at JSConf EU.

12) CoffeeScript sprinkles syntactic sugar

Long before types were popularized in JavaScript, there was CoffeeScript, a programming language that compiles to JavaScript and was inspired by Ruby, Python and Haskell. The compiler was originally written in Ruby and didn’t require compatibility from dependencies because it compiled to JavaScript, and it gained traction for exposing the good parts of JavaScript in a simple way.

2010

13) Node.js gets its first package manager

Shortly after Node.js was introduced, npm was created. npm (short for Node package manager) would eventually create the standard in managing dependencies for both front-end and back-end applications making it easier to publish, install, and manage shared source code with a project file, the package.json. npm also provided the npm registry, which would supply hundreds of thousands of applications a database to retrieve Node.js dependencies.

14) Express has it’s initial release

Inspired by Ruby’s Sinatra, Express.js was released in 2010. It was released with the intention of being a minimal, un-opinionated web framework that provided routing, middleware, and other HTTP utilities. According to GitHub, Express remains the most popular framework for back-end JavaScript developers to date.

15) Modern JavaScript MVC frameworks are born

While back-end JavaScript was gaining traction, front-end MVC frameworks were starting to pop up. Most notably, Backbone.js and AngularJS (later rewritten and released as Angular) were starting to be adopted and loved by JavaScript developers. Backbone’s approach to front-end was well-suited for mirroring an application’s business logic, while Angular took a declarative approach that enables a robust web application in the browser. Both frameworks would go on to influence later front-end libraries and frameworks, such as React, Ember.js, and Vue.js.

2011

16) Ember.js stresses convention over configuration

In 2011, a forked version of an earlier project called SproutCore, is renamed to Ember.js. Ember introduces JavaScript developers the concept of convention over configuration, in which the developer does not have to think about design decisions that can be standardized across code bases.

2012

17) Static types are introduced to JavaScript developers

2012 was a big year for static typed languages. JavaScript was, until then, a dynamically typed language by design, in that it doesn’t require the developer to declare types when initializing variables or other data structures. Enter TypeScript - an extension of JavaScript that allows developers to write typed JavaScript that is syntactically similar to JavaScript and compiles to JavaScript. Microsoft made the initial release of the project in October of 2012.

2013

18) The world reacts to React

In 2013, a developer at Facebook, Jordan Walke, presents a new JavaScript library that does not follow the then-popular MVC convention of JS frameworks. (Source) React, a component-based library that was simply the V of MVC, would go on to become one of the most popular libraries of today.

19) Electron puts Node.js into desktop applications

Additionally, with the rising popularity of Node.js, there was momentum to repurpose the runtime or other uses. GitHub made use of Node.js as a library with Chromium’s rendering engine and created Electron for desktop applications. Notable desktop applications that use Electron include GitHub Desktop, Slack, and Visual Studio Code.

2015

20) Release of ES2015/ES6

The 6th edition of ECMAScript was released in June of 2015. This specification was anticipated by many JavaScript developers for its inclusion of popular features such as support for export and import of modules (ES modules), declaring constants, and more. (Source (http://es6-features.org/)) While the previous version of ECMAScript (ES5) had been released 6 years before, much of the standards released had been worked on since ES3, which was released 16 years before. (Source)

21) GraphQL emerges as a REST alternative

In 2015, Facebook released GraphQL as an open source project, a querying language for APIs that simplifies request calls between clients and servers to resolve the differences between server-side data schemas and client-side data needs. (Source) Due to its popularity, the project would eventually be moved to its own GraphQL Foundation.

22) Node v4 is released

2015 was notable for back-end JavaScript developers because it marked the merging of io.js back into Node.js. Just a year before, Node was forked as io.js in an effort adapt quicker release cycles. When io.js was merged back in, it had already released v3, so it was natural to release Node v4 after the merge as a fresh start for the combined projects. Hereafter, Node would adapt a release cycle that would keep it up to date with the latest V8 releases.

2016

23) JavaScript developers are introduced to lock files

In the months following an infamous “left-pad” incident (Source), Yarn was released to the JavaScript ecosystem. Yarn was created out of need for more consistency across machines and offline environments running the same JavaScript applications. Yarn introduced the autogenerated lockfile to the JavaScript ecosystem, which would influence package managers to look at developer experience differently moving forward. (Source)

2019

24) Node + JS = OpenJS

After years of the JS Foundation and Node.js Foundation operating separately, the two organizations merge and become the OpenJS Foundation with goals to increase collaboration and provide a united home for projects across the JavaScript ecosystem. (Source)

2020

25) Deno makes a splash with the initial release

This year, Node.js creator, Ryan Dahl, made the initial release of Deno, a JavaScript and TypeScript engine that, again, is built on top of V8. The project has generated a lot of interest because of its first-class TypeScript support and, of course, inspiration taken from Node.js.

While these landmarks highlight some exciting moments in JavaScript history, there are countless other honorable mentions and important contributions too. The JavaScript ecosystem would not be where it was without the hard work to of developers around the world today. Every pull request, conference talk, and blog post has inspired the next innovation. For that, we thank all of you for your contributions and look forward to the bright future of JavaScript.

Node 8: Out with the old and in with the patchable

Danielle Adams — Thu, 02 Jan 2020 03:39:22 +0000

Starting today, Node 8 is officially unsupported. What does this mean for Node developers? The circuit breakers for Node 8 access don’t immediately turn off — you can still download it and use it in your source code, but be mindful that the Node team will no longer be "maintaining" the runtime. This means that new features and bug fixes will no longer be applied to the version, and this includes security patches. It's easy to assume that the biggest disadvantages of using an outdated language or runtime version are the hit to performance, but the dangerous risks are really in the security patches (or lack thereof).

Lucky for us, the Node team has a quick turnaround of version releases: every 6 months we get a new version, but that means versions are deprecated at the same rate too. The following is the most up-to-date calendar of the release schedule:

With Node 8 reaching end-of-life, the supported versions of Node will be Node 10, 12, and 13 — until April when Node 14 is released and will replace Node 13 as the "current" version.

Generally changes are expected to live in a Current release for at least 2 weeks before being backported.

Node's release plan explains that changes live in the current release (now Node 13) before being added to active releases (currently Node 10 and Node 12). These changes are made in minor and patch releases of their versions that follow semantic versioning release structure.

Once a release moves into Maintenance mode, only critical bugs, critical security fixes, documentation updates, and updates to ensure consistency and usability of the N-API across LTS releases

Up until yesterday, this meant that Node 8 mostly receives only updates for critical bugs and security patches.

How quickly should I update?

Node is built with web servers in mind, and any use of the HTTPS module is highly reliant on OpenSSL's support of TLS (Transport Layer Socket). Even libraries that are meant to secure other libraries have security vulnerabilities and reach their own end-of-life. This post discusses how Node has updated its own release schedule to stay aligned with the changes happening with OpenSSL.

As for Node upgrades, I would recommend pushing this to the top of any backlog or tech debt list if it's not already there. Unfortunately, tech debt is hard to justify because many teams find a hard time providing a quantitative return on investment, and security debt is no different. Therefore, the sooner its brought up, the better.

Because not all libraries handle dependency end-of-life as eloquently as Node, many end-of-life schedules can be found here: https://endoflife.software.

What version of OpenSSL is my application running?

Even though a supported Node version will ensure that the OpenSSL version in use is supported, some people still have questions about the releases. Node has a command line prompt that you can see all the versions being used:

node -p process.versions

Right now, my local device is using Node 13 because I like to live in the edge. If I run this command in any Node environment, I will see output that looks like this:

{
  node: '13.5.0',
  v8: '7.9.317.25-node.23',
  uv: '1.34.0',
  zlib: '1.2.11',
  brotli: '1.0.7',
  ares: '1.15.0',
  modules: '79',
  nghttp2: '1.40.0',
  napi: '5',
  llhttp: '2.0.1',
  openssl: '1.1.1d',
  cldr: '36.0',
  icu: '65.1',
  tz: '2019c',
  unicode: '12.1'
}

I'll get OpenSSL, V8, and some other useful versions I may be curious about. I use nvm to manage my Node versions locally, so I can switch between Node versions in case I need to see what the underlying differences are in dependencies between code bases.

nvm use 8
node -p process.versions

{ http_parser: '2.8.0',
  node: '8.17.0',
  v8: '6.2.414.78',
  uv: '1.23.2',
  zlib: '1.2.11',
  ares: '1.10.1-DEV',
  modules: '57',
  nghttp2: '1.39.2',
  napi: '4',
  openssl: '1.0.2s',
  icu: '60.1',
  unicode: '10.0',
  cldr: '32.0',
  tz: '2017c' }

We can see that the OpenSSL version is lower in Node 8, and using the 1.0.2s version. (Older versions of OpenSSL did not use semvar, so letters were used to represent patches with non-breaking changes.) The -p flag is short for --print, and process.versions is a subset of data from process. With node -p process, you’ll get a whole lot more metadata about the current Node binary.

Will this affect my builds on Heroku?

The short answer is "No", but as previously mentioned, it’s better to upgrade as soon as possible. You’ll still be able to use Node 8 for builds (such as with webpack) and runtime environments (like an express server) for applications on Heroku, but there will be no further updates to the version.

With that said, security patches that aren’t related to SSL/TLS are made to Node too. Just a few weeks ago npm (the package manager which gets installed with Node) had 2 CVEs patched that required quick action from both npm and Node releasers to ensure the vulnerable versions were not being installed.

One more thing…

I almost forgot to even mention this — Heroku dynos are hosted within the platform and accessed via the Heroku Router, which handles TLS/SSL on its own (ie. secure requests to https://<my-app>.herokuapp.com). The encrypted requests get sent to the Heroku router, and a Heroku client will complete the request to a user’s dyno (where the application is running) before returning a response that is sent over TLS as it leaves Heroku again.

While TLS certificates can be managed outside of Heroku and simply added to the application, this is why Node applications have to use the HTTP module for their source code. Heroku users may not even realize that this is the case because setting up an Express server does not require explicitly specifying HTTP, and configuration for HTTPS requires importing Node’s HTTPS module.

Next steps

I’d like to say that upgrading versions is as easy as changing the engine value in the package.json, but it’s a little more involving than that. Take a look at the Node CHANGELOG, test critical libraries used for production and testing, and be sure to understand all the breaking changes between versions—all of these will make an upgrade go much smoother.

Good luck out there! 🎉