DEV Community: Alessandro Capogna

A Fullstack Javascript Monorepo example

Alessandro Capogna — Wed, 26 Aug 2020 01:38:30 +0000

You are working on a large javascript project. It is full of features and growing by the day.
You have web and mobile clients on the frontend and services of any kind on the backend side.
Every part of your application is somehow coupled with other parts to work and even starting the project in the development phase is a pain ...

If that's the case, hear what I have to say about monorepos.

(source code)

What is a Monorepo?

As Wikipedia says:

A monorepo is a software development strategy
where code for many projects is stored in the same
repository.

Simple and straightforward.

Here is a typical javascript monorepo structure:

repo-root/
  package.json
  projects/
    project-1/
      package.json
    project-2/
      package.json

Tools

When it comes with javascript we have at least two tools to work with to manage our projects in a single repository.

Yarn

Yarn is a well-known dependencies management tool (equivalent to npm). It aims to be also a project management tool by providing multi-package management primitives called workspaces:

Workspaces are a new way to set up your package architecture that’s
available by default starting from Yarn 1.0. It allows you to setup
multiple packages in such a way that you only need to run yarn install once to install all of them in a single pass.

Basically using these features we will have a single yarn.lock and a single node_modules folder at root level, which means all our project dependencies will be installed together so that yarn will be able to increase performances at installation time.

Furthermore it allows to define dependencies between internal packages with zero additional configurations.

Lerna

A tool for managing JavaScript projects with multiple packages.

Lerna offers utilities such as the ability to run custom scripts on a specific set of subprojects and integrated solutions for versioning and package publishing based on what has been changed in the repository (check my post on this topic).

For the sake of completeness, it offers all the features natively implemented by the yarn workspaces but also the possibility of integrating them: we will choose the second option.

For a more exhaustive discussion on yarn, lerna and monorepos I recommend this wonderful article.

The sample project

Our sample project is a toy application that fetches some books from the backend and displays them through a web interface.

However, to implement it I have chosen an architecture such that:

It is a microservices architecture, especially the frontend and the backend will be two separate applications.
It is also modular, therefore with the possibility of creating packages that can be shared by multiple applications.
Can be easily enhanced to cover at least one real world use case (this architecture is inspired by the Storybook Design System Architecture)

Folders structure

We are going to split our projects into two distinct folders: applications and packages.

The applications folder will contain all the components that make up our application at runtime, in our case a graphql api and a reactjs client.

The packages folder will contain modules shared by our applications, in our case a react components package (here called design-system).

The final folders structure will looks like:

repo-root/
  package.json
  packages/
    design-system/
      package.json
  applications/
    client/
      package.json
    api/
      package.json

Yarn/Lerna setup

First you need to setup the management tools for the monorepo.

Inside the root:

yarn init

Note: yarn workspaces require the root package.json to be private, so during the yarn initialization process make sure to set the private flag to true.

Then we have to install lerna:

yarn add lerna -D
yarn lerna init

I always prefer to install this kind of dependencies as devDependencies.

Next we define yarn workspaces according to our project structure:

// package.json

{  
  …
  "private": true,
  "workspaces": [
    "applications/*",
    "packages/*"
  ],
  …  
}

Then we instruct lerna how to integrate itself with yarn workspaces:

// lerna.json

{
  ...
  "packages": [
    "applications/*",
    "packages/*"
  ],
  "npmClient": "yarn",
  "useWorkspaces": true,
  ...
}

Finally we add a custom script for starting our apps during development:

// package.json

{  
  …
  "scripts": {
    "start": "yarn lerna run development:start --parallel"
  },
  …  
}

Coding the api application

For the backend I chose graphql. In particular we are going to implement the getting started tutorial of the official apollo website (with the addition of babel to take advantage of the javascript ES6 syntax).

First we have to make a new directory and cd to it:

mkdir -p applications/api
cd applications/api

Then we have to initialize our project dependencies

yarn init -y
yarn workspace applications/api add @babel/core @babel/cli @babel/node @babel/preset-env nodemon -D
yarn add apollo-server graphql
yarn install

and his files and folders

mkdir src
touch src/index.js
touch .babelrc

Next we have to add some configurations.

Here we define a script to start our graphql app:

// applications/api/package.json

{
  ...
  "scripts": {
    ...
    "development:start": "yarn nodemon --exec babel-node src/index.js ",
    ...
  },
  ...
}

Here we define presets for our Babel compiler:

// applications/api/.babelrc

{
  "presets": ["@babel/preset-env"]
}

Finally we can add the code:

// applications/api/src/index.js

import { ApolloServer, gql } from "apollo-server";

const typeDefs = gql`
  type Book {
    title: String
    author: String
  }

  type Query {
    books: [Book]
  }
`;

const books = [
  {
    title: "Harry Potter and the Chamber of Secrets",
    author: "J.K. Rowling"
  },
  {
    title: "Jurassic Park",
    author: "Michael Crichton"
  }
];

const resolvers = {
  Query: {
    books: () => books
  }
};

const server = new ApolloServer({ typeDefs, resolvers });

server.listen().then(({ url }) => {
  console.log(`🚀 Server ready at ${url}`);
});

You can now test it by running:

yarn development:start

cd ../..
yarn start

Coding the client application

For the client side we are going to build a react web app with an apollo client to work with the graphql backend.

First we bootstrap a new cra project:

npx create-react-app applications/client

Remember we want only one yarn.lock and it has to be placed on the root level, so make sure cra hasn't created a yarn.lock. Otherwise:

rm applications/client/yarn.lock

Next we install dependencies:

cd applications/client
yarn add @apollo/client graphql

Then we add some configurations:

// applications/client/package.json

{
  ...
  "scripts": {
    "development:start": "CI=true yarn react-scripts start",
    ...
  }
  ...
}

Finally, we add the code:

// applications/client/src/App.js

import React from "react";  
import { ApolloClient, ApolloProvider, InMemoryCache } from "@apollo/client";  
import Books from "./components/Books";

const client = new ApolloClient({  
  uri: "http://localhost:4000",  
  cache: new InMemoryCache()  
});

function App() {  
  return (  
    <ApolloProvider client={client}>  
      <Books />  
    </ApolloProvider>  
  );  
}

export default App;

Here we are creating the content of our app:

mkdir src/components
touch src/components/Books.js

// applications/client/src/components/Books.js

import React from "react";
import { useQuery, gql } from "@apollo/client";

const ALL_BOOKS = gql`
  query GetAllBooks {
    books {
      title
      author
    }
  }
`;

function Books() {
  const { loading, error, data } = useQuery(ALL_BOOKS);

  if (loading) return <p>Loading...</p>;
  if (error) return <p>Error :(</p>;

  return data.books.map(({ title, author }) => (
    <div key={title}>
      <p>
        {title} by {author}
      </p>
    </div>
  ));
}

export default Books;

Test it by running:

cd ../..
yarn start

Note how this will start also the api application.

Coding the design-system package

Here we are going to package some react components.

First we have to make a new directory and cd to it:

mkdir -p packages/design-system
cd packages/design-system

Then we have to init our project and his structure:

yarn init -y
yarn add react@^16.0.0 -P  
yarn add microbundle-crl -D

mkdir src
touch src/index.js
mkdir src/components
touch src/components/List.js
touch src/components/ListItem.js

Next we add some configurations:

// packages/design-system/package.json

{
  ...
  "main": "dist/index.js",
  "module": "dist/index.modern.js",
  "source": "src/index.js",
  "scripts": {
    ...
    "development:start": "yarn microbundle-crl watch --no-compress --format modern,cjs"
    ...
  },
  ...
}

Finally, we add the code:

// packages/design-system/src/index.js

import List from "./components/List";

export { List };

// packages/design-system/src/components/ListItem.js

import React from "react";
import PropTypes from "prop-types";

// I'm not using css files because they will not work when exported!
// Consider to use styled components for your project...
function ListItem(props) {
  return (
    <div
      style={{
        margin: "10px",
        padding: "10px",
        border: "1px solid #bbb",
        backgroundColor: "#eee"
      }}
    >
      <span
        style={{
          fontSize: "1.2em",
          textDecoration: "none",
          color: "#333"
        }}
      >
        {props.text}
      </span>
    </div>
  );
}

ListItem.propTypes = {
  text: PropTypes.string.isRequired
};

export default ListItem;

// packages/design-system/src/components/List.js

import React from "react";  
import PropTypes from "prop-types";  
import ListItem from "./ListItem";

function List(props) {  
  return (  
    <div>  
      {props.items.map((content, index) => (  
        <ListItem key={index} text={content || ""} />  
      ))}  
    </div>  
  );  
}

List.propTypes = {  
  items: PropTypes.arrayOf(PropTypes.string).isRequired  
};

export default List;

As last step we have to update the client app:

// applications/client/src/components/Books.js

import React from "react";
import { useQuery, gql } from "@apollo/client";
import { List } from "design-system";

const ALL_BOOKS = gql`
  query GetAllBooks {
    books {
      title
      author
    }
 }
`;

function Books() {
  const { loading, error, data } = useQuery(ALL_BOOKS);
  if (loading) return <p>Loading…</p>;
  if (error) return <p>Error :(</p>;
  return (
    <List
      items={data.books.map(({ title, author }) => `${title} by ${author}`)}
    />
  );
}

export default Books;

And its dependencies:

yarn add design-system@^1.0.0

You can now test the final app:

cd ../..
yarn start

Note: currently there seems to be a bug with react's development server. After the first start, the page must be refreshed.

Space for improvements

Our app is so simple that such a complex architecture might seem totally unjustified.

However, think this way... You want this book listing app to become the best online bookstore in the world!

On the client side, you'll need at least a store app for your customers and a dashboard for you suppliers.

On the server side the underneath data model will explode. You will have to manage your users, track orders and so on. That is, you will have to write tons of business logic line of codes and probably integrations to 3rd party systems. To preserve principles of low-coupling and high-cohesion through your code you'll need to split these logics across many applications and modules.

Your app will probably looks more like this:

According to the proposed monorepo structure it is easy to scale up the project while keeping your code manageable. You will simply create all the new packages and / or applications you need under the appropriate folders.

Conclusions

The disruptive rise of javascript in the field of web development has reached a state of the art in which it is possible to develop very complex applications in a single programming language.
This situation offers some advantages such as the possibility of centralizing project management partially described here.

I sincerely hope that my thoughts on this issue will be of help to your current or next project.

Any sort of feedback is very appreciated!

Lerna - Semantic Versioning simple project setup

Alessandro Capogna — Sat, 22 Aug 2020 15:01:10 +0000

This document contains a step by step guide to implement a minimal example to test how lerna - semantic versioning works.

Inspired by this blog post.

Source code here.

Lerna project setup

I always prefer to not install dependencies as global if it is possible. So here i'm using lerna as a dev dependency:

yarn init
yarn add lerna -D
yarn lerna init --independent

Note: on initialization it is essential to set the independent flag. It allows us to manage sub-projects versioning independently instead of having a single version for all of them.

Lerna configs setup

With Semantic Versioning there is not one and only rule that establishes how the version of a project should be increased. However, lerna in combination with the Conventional Commits format proposes a solution to automatically determine semantic version bumps and generate change logs. In summary, if a project (or sub-project in our case) has been updated and its commit message implies:

a BREAKING CHANGE, then bumps the MAJOR number
a FEATURE, then bumps the MINOR number
a REFACTOR / BUG-FIX, then bumps the PATCH number

Conventional Commits flag setup

In the process of releasing changes, that is achieved by running

yarn lerna publish

lerna will check the updated sub-projects and will create and push a new commit with version bumps and change logs (it will also update packages inter-dependencies).

To instruct lerna that we have chosen to follow the Conventional Commits format we must specify:

// lerna.json
...
"command": {
  "publish": {
    "conventionalCommits": true
  }
}
...

Publish message setup

It can be also useful to customize the publish commit message like this

// lerna.json
...
"command": {
  "publish": {
    ...
    "message": "chore(release): publish"
    ...
  }
}
...

Otherwise, it will be "Publish" by default

Github private package registry setup (optional)

As final step, updated packages (if public, that is "private" field is setted to false or just missing in the relative package.json) will be published to a registry. To have them published on our private github registry you simply need to add this:

// lerna.json
...
"command": {
  "publish": {
    ...
    "registry": "https://npm.pkg.github.com/"
    ...
  }
}

Then, before a release, make sure you are logged into your github-npm account. Otherwise run

npm login --registry=https://npm.pkg.github.com/

Where your password will be a repo and packages scoped github personal access token.

Note: other configurations related to this step will also be made at sub-projects level. We will see it in the next section ...

An important note on the lerna publish command

As mentioned above when "private" field in a package.json is missing or setted to false lerna will try to publish your package on a registry.
If it is so and there is no configuration related to the registry url, lerna will try to publish it in the npm public registry by default.

This could be an unwanted behavior! 🙀

If you don't want to publish any package then consider using one of these two commands for the project release:

yarn lerna publish --skip-npm

yarn lerna version

Otherwise make sure you wrote all the necessary configurations: define registry urls for what you want to publish or set packages as private for what you don't want to publish!

Final result

Your final config file will look like this:

// lerna.json
{
  "packages": ["packages/*"],
  "version": "independent",
  "command": {
    "publish": {
      "conventionalCommits": true,
      "message": "chore(release): publish",
      "registry": "https://npm.pkg.github.com/"
    }
  }
}

Coding packages

Here is the code for our three subprojects (thanks to which I will explain some concepts in the next section):

alpha
beta
usage

Where alpha and beta are independent packages, while usage depends on alpha and beta.

Note: remember to change @ your-username and github repo occurrencies

alpha

mkdir packages/alpha
touch packages/alpha/index.js
touch packages/alpha/package.json

// packages/alpha/index.js

module.exports = "alpa";

// packages/alpha/package.json

{
  "name": "@your-username/alpha",
  "repository": "git@github.com:your-username/your-repo-name.git",
  "version": "0.0.0",
  "publishConfig": {
    "registry": "https://npm.pkg.github.com/"
  }
}

beta

mkdir packages/beta
touch packages/beta/index.js
touch packages/beta/package.json

// packages/beta/index.js

module.exports = "beta";

// packages/beta/package.json

{
  "name": "@your-username/beta",
  "repository": "git@github.com:your-username/your-repo-name.git",
  "version": "0.0.0",
  "publishConfig": {
    "registry": "https://npm.pkg.github.com/"
  }
}

usage

mkdir packages/usage
touch packages/usage/index.js
touch packages/usage/package.json

// packages/usage/index.js

const alpha = require("@your-username/alpha");
const beta = require("@your-username/beta");
console.log(alpha + " " + beta);

// packages/usage/package.json

{
  "name": "@your-username/usage",
  "version": "0.0.0",
  "repository": "git@github.com:your-username/your-repo-name.git",
  "dependencies": {
    "@your-username/alpha": "^0.0.0",
    "@your-username/beta": "^0.0.0"
  },
  "publishConfig": {
    "registry": "https://npm.pkg.github.com/"
  }
}

From the project root directory, run

yarn lerna bootstrap

to install dependencies. Note that internal dependencies will be automatically linked by lerna.

Now cd to the usage package and run

node index

You should see "alpa beta" logged into the terminal.

Working with repo

Because commit messages affects how version is bumped and change logs are generated, each commit will be scoped to a particular feat or fix for a particular package.

Monorepo structure

We are going to commit monorepo boilerplate first

git add .
git reset -- packages

so that we are staging all files except our packages. Then

git commit -m "feat: created a lerna monorepo"

Note that for this particular commit the Conventional Commits format is irrelevant, but it can be a good exercise to apply it right now.

alpha package

git add packages/alpha
git commit -m "feat: added export of package name"

beta package

git add packages/beta
git commit -m "feat: added export of package name"

usage package

git add packages/usage
git commit -m "feat: added export of concatenated package dependencies name"

First publish

git push origin master
yarn lerna publish

Now we have our packages published at version 0.1.0 (remember how bumping works) and change logs generated per sub-project level.

Check it out!

Time to fix

You may have already noticed that the alpha package has a typo in the name it exports. is "alpa" instead of "alpha". We need to fix it

// packages/alpha/index.js

module.exports = "alpha";

Now we can release a patched version for alpha

git commit -am "fix: typo in message"
git push origin master
yarn lerna publish

You can see also usage package has bumped because of a dependency to alpha that has been automatically updated by lerna.

Conclusions

If you work in a team on a fairly complex project, I think having a code versioned according to these standards can simplify your life and save it in some cases! First, you have a history to go through and if the commits are self-explanatory you also have good documentation for free. Second (valid in general for any type of versioning), in case of emergency in production you can always switch back to a stable version of your product!