Exploring the Monorepo #3: Build the source

Table Of Contents

• Make it build
• Does it work?
• The Good
• The Bad
  • What can we solve with scripting?

Today we're going to try something that will definitely work, it's just a question of how bad a developer-experience we'll end up with: We'll compile our Typescript libraries into Javascript.

This will work because it cuts Typescript entirely out of the dependency equation, but it'll also drive a huge wedge into our workflow because now changes to source-code has to be compiled before it can be seen by consumers. And I suspect we'll find other downsides too.

Hopefully we can fix or alleviate those problems with some tooling or scripting, but this article isn't meant to uncover the golden solution that'll end this article-series… It's just that I'm so very tired of hitting errors that I want to end today with something that works. Sometimes we have to take a leap to understand the problem better, so let's dive into it!

Make it build

To figure out what it means to build Typescript to Javascript let's first try it out on the libs/types package. It's quite straightforward to set up compiling:
1) Ensure tsconfig.json has the outDir field specified, that's the folder Javascript gets output to:

$ cd libs/types
$ cat tsconfig.json
  "compilerOptions": {
    "outDir": "./dist"

2) Add a build script:

$ cat package.json
  "scripts": {
    "build": "tsc -b"
$ pnpm build
$ tree dist 
dist
├── index.d.ts
├── index.js
└── index.js.map

3) Ensure package.json entry-point fields are set to point to the files in the dist folder:

$ cat package.json 
  "main": "dist/index.js",
  "types": "dist/index.d.ts",

With that in place this library can now be used as a normal Javascript dependency, and consumers don't have to know it's written in Typescript. Next we just have to apply this to all the code!

Does it work?

So, the result is the usual good overview:

webby
├── apps
│  ├── api
│  │  ├── package.json
│  │  ├── prisma/
│  │  ├── src/
│  │  └── tsconfig.json
│  └── web
│     ├── package.json
│     ├── src/
│     ├── tsconfig.json
│     └── typings/
├── libs
│  ├── analytics
│  │  ├── jest.config.js
│  │  ├── package.json
│  │  ├── src/
│  │  └── tsconfig.json
│  ├── logging
│  │  ├── package.json
│  │  ├── src/
│  │  └── tsconfig.json
│  └── types
│     ├── package.json
│     ├── src/
│     └── tsconfig.json
├── package.json
└── tsconfig.json

ℹ️ This project is prepared on GitHub1s if you'd like to familiarize yourself with the code.

I've kept the pnpm workspace configuration because it was such an easy tool to work with, making installing dependencies and running the build script across all packages quite easy:

$ cd ../..
$ pnpm install
Scope: all 6 workspace projects
$ pnpm -r run build
Scope: all 6 workspace projects

So does web work now?

$ cd apps/web
$ pnpm start
[razzle] > Started on port 3000

Good, good. And api?

$ cd ../api
$ pnpm start
[api] > prisma generate && nodemon -w src/* -x 'ts-node src/api.ts'
[api] Error: Command failed with exit code 1: npm install -D prisma@2.26.0
[api]  ERROR  Command failed with exit code 1.

Oh no! But wait, why does it say Command failed: npm install when we're using pnpm??

Turns out this is a known issue the nice people at Prisma are working on, the workaround for now is to install a specific version (as I write this they recommend using version 2.27.0-integration-fix-sdk-pnpm.2).

If we make that change to package.json's dependencies, does that make api work?

$ pnpm install
- @prisma/client 2.26.0
+ @prisma/client 2.27.0-integration-fix-sdk-pnpm.2
- prisma 2.26.0
+ prisma 2.27.0-integration-fix-sdk-pnpm.2
$ pnpm start
[api] api started at http://localhost:3002

Oh my goodness, hooray! 🎉

Putting it all together we can now raise our product entirely from the root:

$ cd ../..
$ git clean -dxi .; # this prompts you for what to delete so it's safe to run
$ pnpm install && pnpm build
$ pnpm start
apps/web start: [razzle] > Started on port 3000
apps/api start: [api] api started at http://localhost:3002

We did it!

The Good

Taking a step back there are some things I very much like about this pattern:

• By building the code we are no longer bound to writing it in Typescript. Any language that compiles to Javascript will do. So encapsulation of each project has increased, which I'll count as wonderful.

• This allows us a lot of flexibility in what a library produces: For this article-series the libraries are just groupings of code, their built code is identical in function to the source. But what if we imagine we wanted to generate something different than the source-code? What if we had a library whose source-code downloads Swagger documentation from some remote API and generate a Javascript client? To do that we must have a build step, and with this article's approach building is now a "first-class concept" so we don't have to make weird one-off exceptions to support something like that.

• I really appreciate the simplicity of boiling everything down to Javascript, there's just that much less chance of anything going wrong.

Do you see other good things about this pattern? I'd love to hear your take on this.

But there are some big drawbacks too! 😓

The Bad

• We now have a workflow where changes to a library aren't reflected in consumers until the library gets rebuilt. So we have to remember to run pnpm build after every change 😬. That's not good because it's so easy to forget, and then whatever work we've just done will seem like it's missing in the consumer in possibly subtle and confusing ways. I don't know you so maybe you wouldn't have a problem with this, but I think for newcomers and juniors it'll be that little extra annoyance that we are so desperately trying to avoid.

• We end up with boilerplate code and configurations that are identical across all projects, e.g. tsconfig.json must specify outDir and package.json must have a build script + specify main & types fields… it's just an annoying amount of small details that have to be exactly right and it gets worse the more projects we add.

Are there other downsides you can think of? I'd love to hear them!

What can we solve with scripting?

We first and foremost need to not manually rebuild all the time. I see two paths ahead:

1. Dependency rebuilds are invoked whenever consumers run their scripts. So every time apps/web runs start it would first go out and rebuild its dependencies.
2. Rebuild dependencies via a watcher, so every time a package's code changes it rebuilds itself.

Can you think of other proposals?

We also would benefit from some solution to the boilerplate code and configurations, e.g. if a script could check all packages and fix or warn about misconfigurations then we'd probably have alleviated the issue well enough.

This isn't the article where we write the scripts or even decide exactly how to do it, but maybe that's a topic for the next article? At this point I'd very much like to hear from you, so please leave a comment with your thoughts or suggestions.

Comments

[Raif Harik]

Hi, I know this is a couple years old, but maybe that means you have some new solutions :). I was wondering if you stuck with this workflow and if so how you managed the developer experience. You mention some options but don't speak to what you used. Then in the next article, you focus on docker.

[Jon Lauridsen]

Hi Raif, yeah I did stick with the "compile to js" approach, I applied it across several projects and honestly never looked back. The drawbacks are there but they never really bit us, and pnpm's dependency graph made it all simple to navigate. I did end up adding a couple pnpm aliases to my shell to more easily (re-)build up and downstream:

alias pbd="pnpm -r --depth=-1 --filter={.}^... build"
alias pbu="pnpm -r --depth=-1 --filter=...{.} build"

I've since switched towards Deno and so haven't developed any of this any further, but I think the basics still hold up.

And thanks so much for reading!!