15 Minute Nx on a bed of Clouds

With all the focus these days on home cooking, I thought I'd write up a recipe of my own which you can follow without even having to go into the kitchen.

For those of you craving the taste of a real monorepo, this recipe combines Nx with a continuous integration service and then adds some fresh Nx Cloud for a piquant dash of the efficiency that comes with computation memoization.

If it's not clear what computation memoization is or how it can help you, then this could be the recipe for you!

Unlike a standard recipe the product of following this set of instructions should be understanding. The quality of what you create as a result of combining all the ingredients together doesn't matter. You could love or hate the result, but what I hope you'll come away with is a little more knowledge. That's what writing this achieved for me.

I want to emphasise that I describe this as a recipe because there is no single right way to combine all these ingredients. What follows is something that worked for me. If you find any problems, errors or just have questions, feel free to tweet me @jdpearce.

⚠️ Requirements

There is only so much I can explain without making this article unwieldy, so I'll assume that anyone reading this has some understanding of JavaScript development and already has Node and npm installed. You can use another package manager if you like, but I'm only going to provide instructions for the lowest common denominator.

I am going to assume you have some idea what a monorepo actually is and why you might want to use one. Following this recipe may give you some insight into how they can be used, but I'm not going to go into detail about the pros and cons.

While I list having a GitHub account as an ingredient, I'm not going to explain how to set up new repositories and I will assume a certain amount of knowledge of git itself. There are better articles out there which can go into detail about this. If you run into problems I will always recommend "Oh Shit, Git!?!" 😁

🥬 Ingredients

• 1 x Nx (Extensible Dev Tools for Monorepos) workspace (v9.2.2 as of writing)
• 1 x Nx Cloud (Computation Memoization in the Cloud) account
• 1 x CircleCI (Continuous integration) account
• 1 x GitHub account
• Back and/or front end frameworks to taste (I'm going to use React and Express)

🥒 Preparation

Before we start, I recommend that you have accounts with CircleCI, Nx Cloud and GitHub ready. You don't want to ruin the flow of a good home cooking session by having to run out to the shops and generate new passwords.

Make sure your CircleCI account is connected to your GitHub account. You can achieve that via the Account Integrations dashboard.

I'm going to assume that you also have Nx installed globally. You don't need to do this, you can just as easily use npx nx or yarn nx to call it, but it does mean I have to write less. To install with npm:

npm install -g @nrwl/cli

Now that we're all prepared, let's get cooking!

1. Creating the Nx Workspace (Monorepo)

⏱ 3 minutes

Find a place on your file system where you want to create the new workspace and run the following command:

npx create-nx-workspace@latest nx-cloud-recipe

You'll be asked a set of questions for the kind of workspace you want to create. I'm going to choose the react-express preset which will create two applications; a React front end and an Express API. You can call the front end application whatever you like, but I'll go with todos as that seems to be fairly standard tutorial recipe fare, and I'll just select CSS as the default stylesheet format for now.

Once the creation script has worked its magic, you should find a new workspace with an initialised git repository in the nx-cloud-recipe folder.

cd nx-cloud-recipe

To check that everything is set up as expected, why not run a test:

nx test todos

On my machine this runs in a couple of seconds and the output looks like this:

Or we could run all the tests in the monorepo:

nx run-many --target=test --all

The output of that should look something like this:

You should find that the output for this command is generated almost instantaneously. This is because Nx locally caches the output of certain computational tasks, such as in this case running tests. There only happens to be one test suite in the entire monorepo and we just ran it and therefore cached it.

If we make changes to the todos application, the same command will be smart enough to recognise that something has changed and the test will be run.

You can pause here if you wish to get a feel for this particular ingredient. Try running both applications and seeing the output at http://localhost:4200 -

nx serve api & nx serve todos

⚠️ NB You'll need to run these in different instances of cmd/powershell if you're running on Windows. Alternatively, you could investigate the power of run-commands, but that's beyond the scope of this recipe for the moment.

2. Push Your Monorepo to GitHub

⏱ 2 minutes

Create a new, empty repository on GitHub, I've called mine the same as the local monorepo, but you don't have to (you could call it bob or alice if you really wanted 🤷‍♀️). After you've done that, come back to the local command line to push the generated code:

git remote add origin https://github.com/jdpearce/nx-cloud-recipe.git
git push -u origin master

⚠️ NB Don't try to push to my repo, I don't think you'll be allowed. Copy the link to your own repository from the screen that GitHub helpfully shows you with all the possible ways you can push code to it.

3. Connect to CircleCI

⏱ 2 minutes

Go to your CircleCI dashboard and select "Add Projects". This should bring you to a page something like this where you can search for the new GitHub repository that you've just created:

ℹ️ NB If the repository doesn't appear in the list, you may have to refresh the page.

Click on the "Set up Project" button and then select the "Node" configuration template:

Next we have to click the appallingly badly named "Start Building" button, which does nothing of the sort:

We're going to let CircleCI create a new branch called circleci-project-setup and commit a new file .circle/config.yml

Click on the "Add Config" button and let it do it's thing.

⚠️ NB I am not a big fan of YAML. I think it's difficult to write and more importantly difficult to read. Be warned that there is unfortunately quite a bit of it coming up. YMMV, of course...

4. Adding build and test jobs

⏱ 5 minutes

At your local command line:

git fetch
git checkout circleci-project-setup

We're going to cheat a little here. Detailed configuration of CircleCI is far beyond the scope of this recipe so I'm going to provide you with a store-bought configuration that sets up both build and test jobs. There are many ways to do this, so don't believe for a moment that this is the best or only way to achieve this goal. If you have the time artisanal, hand-crafted YAML is the way to go, but store-bought is fine for now.

ℹ️ NB Deployment is also beyond the scope of this recipe, but my colleague Rares Matei is publishing something covering deployment in the near future.

Using your editor of choice, replace the contents of the .circleci/config.yml file with the following:

version: 2.1

orbs:
  node: circleci/node@2.0.1

# Reusable Commands
commands:
  npm_install:
    description: 'Install & Cache Dependencies'
    steps:
      - run: npm install
      - save_cache:
          key: nx-cloud-recipe-{{ checksum "package-lock.json" }}
          paths:
            - ~/.cache
            - node_modules

  restore_npm_cache:
    description: 'Restore Cached Dependencies'
    steps:
      - restore_cache:
          keys:
            - nx-cloud-recipe-{{ checksum "package-lock.json" }}
            - nx-cloud-recipe- # used if checksum fails

  setup:
    description: 'Setup Executor'
    steps:
      - checkout
      - attach_workspace:
          at: ~/project

# Available Jobs
jobs:
  initialise:
    executor:
      name: node/default
    steps:
      - checkout
      - restore_npm_cache
      - npm_install
      - persist_to_workspace:
          root: ~/project
          paths:
            - node_modules
            - dist

  build:
    executor:
      name: node/default
    steps:
      - setup
      - run:
          name: Build all affected projects
          command: npx nx affected:build --base=master --head=HEAD

  test:
    executor:
      name: node/default
    steps:
      - setup
      - run:
          name: Run all affected tests
          command: npx nx affected:test --base=master --head=HEAD

workflows:
  build-and-test:
    jobs:
      - initialise
      - build:
          requires:
            - initialise
      - test:
          requires:
            - initialise

😱

This is a really scary amount of YAML and personally I find it difficult to read. CircleCI's documentation is pretty unhelpful for the novice too, so I'll try to briefly explain what's going here.

At the top level we have groupings of orbs, commands, jobs and workflows. In the orbs group we indicate that we're using the circleci/node@2.0.1 orb, which is a collection of bits and pieces for working with node projects. In particular it includes the default executor which is the environment used to run the jobs.

The commands group declares and defines three commands that can be used within jobs:

• npm_install - runs a standard dependency installation and populates a local cache
• restore_npm_cache - restores from that local cache
• setup - checks out the code and restores a workspace

The jobs group declares and defines three jobs that we can sequence within workflows:

• initialise - check out the code, restore caches, run an npm install command and then persist all this to the workspace
• build - builds all the affected projects
• test - tests all the affected projects

ℹ️ NB It's worth noting here that the build and test jobs use nx affected logic to decide which projects to build or test. Nx is smart enough to know which projects have been touched by a particular commit (or range of commits) and will run the appropriate target for those projects only. This should save you considerable amounts of time in your CI environment. In this case we're only running build and test for those projects which have changes when comparing between HEAD and the current master branch, so this should work well for pull requests.

Lastly, the workflows group defines a single workflow called build-and-test which specifies that the initialise job must be run before either build or test can be run.

If you save, commit, and push this to GitHub, you should see something like this in CircleCI:

🎉

If everything looks good as above, we can get that configuration into the master branch with a pull request.

ℹ️ NB You could make this workflow run only when a pull request is created, but that (along with many other configuration niceties) is beyond the scope of this recipe. Still, this is one of the many aspects of this recipe that you can experiment with later! 🧪

5. Connect to Nx Cloud

⏱ 3 minutes

The first step is to go to your Nx Cloud dashboard and create a new workspace. I've called mine the same as the repository, but again you don't have to do this if you're not really bothered about naming¹.

Once you've named the workspace, you'll be presented with this set of instructions for CI and local configuration. I've blocked out the tokens in the screenshot above so that nefarious web users don't store their caches in my workspace and use up my Nx Cloud Coupons (you should get 5 hours free when you first sign up).

We're now going to add that local read-only token to our nx-cloud-recipe repository:

git checkout master
git pull
git checkout -b nx-cloud-configuration
npm install @nrwl/nx-cloud && nx g @nrwl/nx-cloud:init --token=<token>

(The last line here should be copied and pasted from the "Set up for local development" section shown above)

Next we need to add the read-write token to our CircleCI setup:

(You can find this by selecting the "Workflows" section on the left and then clicking the little cog icon next to the nx-cloud-recipe workflow)

CircleCI is now ready for you to commit and push the nx-cloud-configuration branch:

git commit -am "feat: add nx cloud configuration"
git push --set-upstream origin nx-cloud-configuration

This should result in a beautiful green set of workflow steps which means you're ready to create a pull request and merge that back into master!

Time now for you to sit back, pour a glass of something enjoyable and serve the completed dish...

---

🍇 Taste the Fruits of Your Labour!

Nx and its affected logic has your back when it comes to saving time in your continuous integration environment. When it comes to saving developer time locally, that's where Nx Cloud can really shine.

Check out the repo into a new folder, e.g.

cd ..
git clone https://github.com/jdpearce/nx-cloud-recipe.git nx-cloud-recipe2
cd nx-cloud-recipe2
npm install

⚠️ NB I'd rather you didn't clone my repository, but you can if you want. I make no guarantees about it having valid coupons on the account though, so you may not see any effect.

This repo shouldn't have any local cache since we haven't run build or test here before, however, if we try building everything:

nx run-many --target=build --all

You should see almost instant output:

With the new Nx Cloud configuration in place, and because you haven't yet made any changes to the repository, the build process you run locally would be identical to the one that was run in CI, so Nx will read from the cache that was generated when the build ran in CI, saving us the amount of time it would normally have taken to run!

If we check back on the Nx Cloud site, you should be able to see that we've had an impact on the graph:

A whole MINUTE saved! 🤯

Ok, maybe that's not hugely impressive in this case, but then we did create this workspace mere minutes ago.

ℹ️ NB One minute is the smallest unit that the chart labels will actually display, but the chart is drawn with finer precision, which is why the "minute" saved in build is bigger than the "minute" saved in test 🤓

Imagine how much time you would save if this were a mature workspace with numerous apps and libraries². We've seen cases where in only weeks, teams have saved hundreds of hours of developer time. That's your time, and that means more time for you to do the things you care about.

---

1. Naming is, of course, one of the two hardest problems in computer science. Those being:  ↩

   1. Naming things
   2. Cache invalidation
   3. Off-by-one errors

2. You don't actually have to imagine this. The following chart shows that over two weeks one particular team of 20-30 saved enough time to make up a whole extra developer!  ↩

Comments

[Michael Prentice]

This is a really helpful post, thank you!

However, nx g @nrwl/nx-cloud:init --token=<token> gives me 'token' is not found in schema.