DEV Community: Nathan Cook

Optimize Yarn 1.2 Monorepo Docker Development with Turborepo: Tips for Prisma and Dockerfile build filtering

Nathan Cook — Fri, 02 Jun 2023 09:44:52 +0000

This yarn 1.2 monorepo demonstrates

Why putting Prisma in its own workspace is a good idea
Docker container build time difference between turbo run build --filter=service and turbo run build --filter=service^...

Setup

bash setup.sh

#!/bin/bash

docker pull node:20.2.0-alpine3.17
docker pull postgres:15.3-alpine3.17

docker image build -f dockerfiles/Dockerfile.node -t custom-node:latest dockerfiles

yarn
yarn workspace prisma-client build
yarn

This monorepo has the following workspaces:

apps/frontend
apps/backend
packages/prisma-client

Why put Prisma in a workspace?

App workspaces can access Prisma types

apps/backend/src/app.service.ts

import {Injectable} from "@nestjs/common";
import {PrismaService} from "nestjs-prisma";
import type {User} from "prisma-client";

@Injectable()
export class AppService {
  constructor(private readonly prisma: PrismaService) {}
  getHello(): string {
    return "Hello World!";
  }

  getUsers(): Promise<User[]> {
    return this.prisma.user.findMany();
  }
}

apps/frontend/src/use-data.ts

import React from "react";
import {ofetch} from "ofetch";
import type {User} from "prisma-client";

export default function useData() {
  const [data, setData] = React.useState<User[]>([]);

  React.useEffect(() => {
    (async () => {
      const data = await ofetch("http://localhost:3333/users");
      setData(data);
    })();
  }, []);

  return data;
}

In frontend/backend package.json

{
  "dependencies": {
    "prisma-client": "*"
  }
}

Simplifies generating the prisma client in Docker containers

packages/prisma-client/package.json

{
  "name": "prisma-client",
  "version": "1.0.0",
  "main": "dist/index.js",
  "files": ["src/index.ts"],
  "repository": "https://github.com/moofoo/turborepo-tricks-prisma-workspace.git",
  "author": "moofoo <nathancookdev@gmail.com>",
  "license": "MIT",
  "private": true,
  "scripts": {
    "build": "npx -y rimraf dist/* && prisma generate && npx tsc"
  },
  "dependencies": {
    "@prisma/client": "^4.15.0"
  },
  "devDependencies": {
    "@types/node": "^20.2.5",
    "prisma": "^4.15.0",
    "rimraf": "^5.0.1",
    "typescript": "^5.1.3"
  }
}

With prisma generate in the build script, turbo run build ... will generate the prisma client for workspaces that have prisma-client as a dependency. This is particularly useful when it comes to writing Dockerfiles, since it makes it easier to write generic Dockerfiles for development.

Build time difference between `--filter=service` and `--filter=service^...`

If you compare docker-compose.good.yml and docker-compose.bad.yml, you'll see that the only difference is whether the Node.js services use Dockerfile.good or Dockerfile.bad.

The only difference between those Dockerfiles is on line 30

Dockerfile.good

RUN turbo run build --no-cache --filter=${APP}^...

Dockerfile.bad

RUN turbo run build --no-cache --filter=${APP}

The latter RUN command is equivalent to running yarn workspace ... build for the given app workspace and all dependent workspaces

PLEASE NOTE THAT THE `compare.sh` BASH SCRIPT RUNS `docker system prune -f` TO ACCURATELY COMPARE BUILD TIMES

Run compare.sh to see the difference:

$ bash compare.sh
Cleaning up
Building bad
yarn run v1.22.19
$ docker compose -f docker-compose.bad.yml build --no-cache -q
Done in 155.18s.

real    2m35.339s
user    0m0.531s
sys     0m0.230s
Cleaning up
Building good
yarn run v1.22.19
$ docker compose -f docker-compose.good.yml build --no-cache -q
Done in 94.58s.

real    1m34.768s
user    0m0.498s
sys     0m0.165s

`--filter=workspace` : 2m35.339s

`--filter=workspace^...` : 1m34.768s

(I intentionally didn't follow dependency best practices with the Frontend app to increase its build time, to make the difference between the two build commands more clear. The basic point is don't build workspaces if you don't need to)

Fully featured report generation in Node with the JSReports rendering core

Nathan Cook — Fri, 26 May 2023 05:39:03 +0000

(Check out this repo to get an idea of what's possible)

If you're interested in a free, self-hosted NodeJS report generation solution that doesn't compromise on templating functionality, you should check out JSReport, specifically the functionality one can build using the jsreport 'rendering core'.

JSReport is open-source, and unlike many SaaS providers in this space, they do not gatekeep ANY of their product's core functionality behind tiered subscription pricing. While JSReport does charge for their web-based 'Template Studio' and SaaS offerings, the report generation framework that powers these products is open source and free to use however you want.

Let's get started with a basic example.

Open up your terminal (I'm assuming linux, no promises if you're in Windows-land) and run the following:

git clone https://github.com/moofoo/basic-jsreport-core-example.git && \
cd basic-jsreport-core-example && \
yarn

First, lets look at the package.json

{
  "name": "basic-jsreport-core-example",
  "version": "1.0.0",
  "main": "index.js",
  "license": "MIT",
  "dependencies": {
    "@jsreport/jsreport-core": "^3.11.4",
    "@jsreport/jsreport-docx": "^3.7.1",
    "@jsreport/jsreport-handlebars": "^3.2.1",
    "@jsreport/jsreport-unoconv": "^3.0.1",
    "handlebars": "^4.7.7"
  }
}

@jsreport/jsreport-core is the "minimalist jsreport rendering core", on to which we'll add 'extensions' (plugins, what have you), to build up the functionality we want.

JSReport categorizes extensions into three categories: Engines, Recipes, Everything Else

Engines - These are templating languages, i.e. handlebars, ejs, etc. The current example uses handlebars, by way of the @jsreport/jsreport-handlebars extension. For any given Engine you will also need the actual templating language in your dependencies - handlebars@4.7.7 in this case.
Recipes

"A recipe represents the technique used to print the document. It can be an HTML to pdf conversion, DOCX rendering, and others."

The current example employs the docx recipe, so we have the @jsreport/jsreport-docx extension as a dependency.

3, Everything Else - for instance, this repo uses the @jsreport/jsreport-unoconv extension to do PDF conversion via unoconv.

Now let's look at index.js. Here's that file:

const { readFileSync, writeFileSync, unlinkSync } = require("fs");
const templateData = require('./data.json');

// remove existing report files, if they exist
try {
  unlinkSync("./report.docx");
  unlinkSync("./report.pdf");
} catch (err) {
  console.log("no report files");
}

// Create jsreport instance
const jsreport = require("@jsreport/jsreport-core")();

// Add plugins: docx recipe, handlebars engine and unoconv extension for pdf conversion
jsreport.use(require("@jsreport/jsreport-docx")());
jsreport.use(require("@jsreport/jsreport-handlebars")());
jsreport.use(require("@jsreport/jsreport-unoconv")());

(async () => {
  // Initialize jsreport instance
  await jsreport.init();

  // Read template.docx, base64 encoded
  const content = readFileSync("./template.docx", {encoding: "base64"});

  // Render a .docx report from template
  const resultDocx = await jsreport.render({
    template: {
      content: "",
      recipe: "docx",
      engine: "handlebars",
      docx: {
        templateAsset: {
          content,
          encoding: "base64",
        },
      },
    },
    data:templateData
  });

  // Write report.docx to disk
  await writeFileSync("./report.docx", resultDocx.content, {encoding: "utf-8"});
  console.log("report.docx created");

  // Render a .pdf report from template using unoconv extension
  const resultPdf = await jsreport.render({
    template: {
      content: "",
      recipe: "docx",
      engine: "handlebars",
      docx: {
        templateAsset: {
          content,
          encoding: "base64",
        },
      },
      unoconv: {
        format: "pdf",
        enabled: true,
      },
    },
    data: templateData
  });

  // Write report.pdf to disk
  await writeFileSync("./report.pdf", resultPdf.content, "binary");
  console.log("report.pdf created");
})();

Let's go over the JsReport-relevant lines. We start with some initialization boilerplate:

// Create jsreport instance
const jsreport = require("@jsreport/jsreport-core")();

// Add plugins: docx recipe, handlebars engine and unoconv extension for pdf conversion
jsreport.use(require("@jsreport/jsreport-docx")());
jsreport.use(require("@jsreport/jsreport-handlebars")());
jsreport.use(require("@jsreport/jsreport-unoconv")());

The next lines initialize the instance given the added extensions and gets the template file:

  // Initialize jsreport instance
  await jsreport.init();

  // Read template.docx, base64 encoded
  const content = readFileSync("./template.docx", {encoding: "base64"});

Note the base64 encoding.

At this point, the jsReport instance is ready to render docx (or pdf) reports from .docx templates:

// Render a .docx report from template
  const resultDocx = await jsreport.render({
    template: {
      content: "",
      recipe: "docx",
      engine: "handlebars",
      docx: {
        templateAsset: {
          content,
          encoding: "base64",
        },
      },
    },
    data:templateData
  });

  // Write report.docx to disk
  await writeFileSync("./report.docx", resultDocx.content, {encoding: "utf-8"});
  console.log("report.docx created");

Let's go over the options passed to jsreport.render, specifically the template config:

content: "",

While the content param isn't used here, JsReport will throw an error if it is undefined. An empty string suffices.

recipe: "docx"

This sets the render 'recipe'. Regarding naming conventions:

"The convention is that jsreport repository extension starts with jsreport-xxx, but the extension real name and also the recipes or engines it registers excludes the jsreport- prefix. This means if you install extension @jsreport/jsreport-handlebars the engine's name you specify in the render should be handlebars."

Accordingly, @jsreport/jsreport-docx transmutes to 'docx', here.

engine: "handlebars"

same story

docx: {
  templateAsset: {
    content,
    encoding: "base64",
  },
},

content is just the template.docx file, base64 encoded, and encoding informs the render method about that encoding.

The next jsreport.render call is identical to the previous one, with one addition to the template options:

 unoconv: {
    format: "pdf",
    enabled: true,
 },

So, when this render runs, a PDF will be created, rather than a .docx file. Note that enabled is assumed to be false if undefined, so it must be set to true here.

Using whatever document viewer you have handy, check out whats in template.docx. This is just an invoice example I pulled from the JSReport Playground

Ok, let's run this thing:

node index.js

Assuming nothing goes horribly wrong, you should now have two report files in the repo root, report.pdf and report.docx.

--------------------

While the code presented here is nothing fancy, the real magic is how this extremely simple setup gives you full access to JSReport's robust templating features, for example:

And that's just for .docx based templates.

That's all for now!

NestJS + JSReport integration will be covered in a future article, however the repo for that article is already complete: https://github.com/moofoo/nestjs-jsreport-examples

Creating a development dockerfile and docker-compose.yml for yarn 1.22 monorepos using Turborepo.

Nathan Cook — Wed, 17 May 2023 03:01:31 +0000

(Be sure to check out the example repo)

If you've ever worked in a yarn monorepo project with inter-dependent workspaces, you know that creating docker containers for development in a way that avoids unnecessary rebuilds can be a challenge.

Turborepo, "...an intelligent build system optimized for JavaScript and TypeScript codebases", provides tools that make this task much easier.

Let's assume a project that looks like something like this:

- /project-directory
    - /apps
        - /frontend
            - package.json
            - other files...
        - /backend
            - package.json
            - other files...
    - /packages
        - /shared-stuff
            - package.json
            - other files...
    - .dockerignore
    - package.json
    - turbo.json
    - Dockerfile.dev
    - docker-compose.yml

.dockerignore

**/node_modules
**/.next
**/dist

This ensures that node_modules and build artifacts are excluded when copying directories to the container from the host in our Dockerfile.

package.json

{
  "name": "turbo-docker-monorepo",
  "version": "1.0.0",
  "main": "index.js",
  "license": "MIT",
  "private": true,
  "workspaces": ["apps/*", "packages/*"],
  "packageManager": "yarn@1.22.19",
  "devDependencies": {
    "turbo": "^1.9.6"
  }
}

This is the minimal package.json you'll need for a yarn monorepo that makes use of turborepo with workspaces in the 'apps' and 'packages' directories. Not much else to say!

turbo.json

{
  "$schema": "https://turbo.build/schema.json",
  "pipeline": {
    "build": {
      "dependsOn": ["^build"],
      "outputs": ["dist/**", ".next/**", "!.next/cache/**"]
    },
    "lint": {},
    "dev": {
      "cache": false,
      "persistent": true
    }
  }
}

This turbo.json file is taken from the basic turborepo example.

Let's assume that the backend app has the packages/shared-stuff workspace as a dependency (meaning the backend package.json has "shared-stuff": "*" in its list of dependencies).

We can use turborepo to build backend in a way that respects its dependence on the shared-stuff package with the command turbo run build --filter=backend, which will build (in order) the shared-stuff package and then backend app. Cool!

Dockerfile.dev

# syntax=docker/dockerfile:1.5.2
# based on: https://github.com/vercel/turbo/blob/main/examples/with-docker/apps/api/Dockerfile

FROM node:20.2-alpine3.17 as base

# adding apk deps to avoid node-gyp related errors and some other stuff. adds turborepo globally
RUN apk add -f --update --no-cache --virtual .gyp nano bash libc6-compat python3 make g++ \
      && yarn global add turbo \
      && apk del .gyp

#############################################
FROM base AS pruned
WORKDIR /app
ARG APP

COPY . .

# see https://turbo.build/repo/docs/reference/command-line-reference#turbo-prune---scopetarget
RUN turbo prune --scope=$APP --docker

#############################################
FROM base AS installer
WORKDIR /app
ARG APP

COPY --from=pruned /app/out/json/ .
COPY --from=pruned /app/out/yarn.lock /app/yarn.lock

# Forces the layer to recreate if the app's package.json changes
COPY apps/${APP}/package.json /app/apps/${APP}/package.json

# see https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/reference.md#run---mounttypecache
RUN \
      --mount=type=cache,target=/usr/local/share/.cache/yarn/v6,sharing=locked \
      yarn --prefer-offline --frozen-lockfile

COPY --from=pruned /app/out/full/ .
COPY turbo.json turbo.json

# For example: `--filter=frontend^...` means all of frontend's dependencies will be built, but not the frontend app itself (which we don't need to do for dev environment)
RUN turbo run build --no-cache --filter=${APP}^...

# re-running yarn ensures that dependencies between workspaces are linked correctly
RUN \
      --mount=type=cache,target=/usr/local/share/.cache/yarn/v6,sharing=locked \
      yarn --prefer-offline --frozen-lockfile

#############################################
FROM base AS runner
WORKDIR /app
ARG APP
ARG START_COMMAND=dev

COPY --from=installer /app .

CMD yarn workspace ${APP} ${START_COMMAND}

Let's go over what happens in each layer. For the following discussion, assume that the backend app service is being created, which has shared-stuff as a dependency.

base

FROM node:20.2-alpine3.17 as base

RUN apk add -f --update --no-cache --virtual .gyp nano bash libc6-compat python3 make g++ \
      && yarn global add turbo \
      && apk del .gyp

This layer adds container dependencies to ensure NPM modules that use node-gyp build correctly. It also adds turborepo globally. The remaining layers are built from 'base'

pruned

FROM base AS pruned
WORKDIR /app
ARG APP

COPY . .

RUN turbo prune --scope=$APP --docker

This layer copies over the project files and runs turbo prune for the service in question. The $APP argument will be either 'frontend' or 'backend' (assuming the later, for this example), which is set in docker-compose.yml (this will be covered last). Regarding the --docker flag:

With the docker flag, the prune command will generate folder called out with the following inside of it:

A folder json with the pruned workspace's package.jsons

A folder full with the pruned workspace's full source code, but only including the internal packages that are needed to build the target.

A new pruned lockfile that only contains the pruned subset of the original root lockfile with the dependencies that are actually used by the packages in the pruned workspace.

Assuming we're building the backend service container, here is what the 'out' directory in the ‘pruned’ layer would look like at this point:

- /out
    - /full
       - /apps
           - /backend (all files)
       - /packages
           - /shared-stuff (all files)
       - .gitignore
       - package.json
       - turbo.json
    - /json
        - /apps
            - /backend
                - package.json
        - /packages
            - /shared-stuff
                - package.json
        - package.json
    - yarn.lock

Note that the 'frontend' app is not present and (according to the turborepo docs) its dependencies were excluded from yarn.lock, which is what we want!

installer

FROM base AS installer
WORKDIR /app
ARG APP

# COPY 1
COPY --from=pruned /app/out/json/ .
COPY --from=pruned /app/out/yarn.lock /app/yarn.lock
COPY apps/${APP}/package.json /app/apps/${APP}/package.json

# RUN 1
RUN \
      --mount=type=cache,target=/usr/local/share/.cache/yarn/v6,sharing=locked \
      yarn --prefer-offline --frozen-lockfile

# COPY 2
COPY --from=pruned /app/out/full/ .
COPY turbo.json turbo.json

# RUN 2
RUN turbo run build --no-cache --filter=${APP}^...

# RUN 3
RUN \
      --mount=type=cache,target=/usr/local/share/.cache/yarn/v6,sharing=locked \
      yarn --prefer-offline --frozen-lockfile

The COPY 1 command series begins by copying the json directory in out from the pruned layer, as well as the 'scoped' yarn.lock file. It also redundantly copies the app's package.json from the host, which I've found necessary for the container to recreate correctly when shared workspace dependencies change.

The first run command (RUN 1) runs yarn and uses a cache mount targeting the yarn cache directory, which will help speed up subsequent re-builds.

The next two statements (COPY 2) copy the contents of out/full from the pruned layer as well as turbo.json from the host.

Since this container will run in a dev environment, it means we can safely skip building the backend app. However, we do need to build packages/shared-stuff, which in the current example is a dependency of backend. This is done with the command in RUN 2: RUN turbo run build --no-cache --filter=${APP}^.... In this case, since we're building the backend service container, the filter flag will resolve to --filter=backend^.... What this means is that all workspace dependencies of backend, but not backend itself, will be built, which is what we want.

This is a huge time saver for certain frameworks (looking at you, Next.js).

The final RUN command (RUN 3) is just a copy of the first (RUN 1). This final yarn call ensures that dependencies between workspaces are linked correctly.

runner

FROM base AS runner
WORKDIR /app
ARG APP
ARG START_COMMAND=dev

COPY --from=installer /app .

CMD yarn workspace ${APP} ${START_COMMAND}

The final layer copies over the installer layer and runs the command to start the app (based on the START_COMMAND argument).

Now let's look at docker-compose.yml, specifically the config for the backend service (almost done!)

version: '3.8'

x-defaults:
  &defaults
  init: true
  tty: true
  networks:
    - my_monorepo_network

networks:
  my_monorepo_network:

services:
    backend:
        <<: *defaults
        ports:
          - "3333:3333"
        command: yarn workspace backend dev
        environment:
          - PORT=3333
        build:
          args:
            APP: backend
            START_COMMAND: dev
          context: .
          dockerfile: ./Dockerfile.dev
        volumes:
          - ./apps/backend:/app/apps/backend
          - /app/apps/backend/node_modules
    ...other services...

This is a pretty typical definition for a Node service. Here you can see the build args, which are used in Dockerfile.dev.

Changing dependencies

tl;dr: docker compose up -d -V --build <service>

When you change dependencies for your node services on the host, you need to rebuild the container for it to reflect those changes.

You don't need to use docker compose build --no-cache <service> for this. There are better methods which don't require completely busting cache and starting from scratch.

The complicating factor for node services, as commonly configured in docker compose projects, is anonymous volumes:

...

volumes:
   - ./apps/backend:/app/apps/backend
   - /app/apps/backend/node_modules # <-- this guy

...

In the context of yarn 1.22 monorepos, apps/backend/node_modules contains development dependencies that are unique to that workspace.

The point of the anonymous volume /app/apps/backend/node_modules is to make it so node_modules in the backend workspace within the container is excluded from the bind mount ./apps/backend:/app/apps/backend.

Because of that volume, simply running docker compose build <service> or docker compose up -d --build <service> won't work like you expect it to when development dependencies for backend change.

There are two ways I've found to reliably recreate containers when deps change, while dealing with anonymous volumes:

1 The -V, --renew-anon-volumes flag

docker compose up -d -V --build <service>

You don't need to stop or restart the service.

2 rm container and build

docker compose rm -f -s <service> && docker compose up -d --build <service>

Either of these methods will do the trick.

NestJS + Prisma + PostgreSQL ~ RLS multi-tenancy using nestjs-prisma, nestjs-cls and Prisma Client Extensions

Nathan Cook — Thu, 11 May 2023 23:33:00 +0000

(Github repository for this article)

Here's a simple way to do multi-tenancy in NestJS with Prisma using Client Extensions and AsyncLocalStorage (via nestjs-cls).

This post describes the basics of a generic NestJS implementation. Look at this repo for a complete and functional example app. Checkout branch async-hooks to see the AsyncLocalStorage-based implementation (which relates to this post). The repo also demonstrates how to use request-scoped providers (main branch) and durable request-scoped providers (durable branch).

Step #1: Setup AsyncLocalStorage using nestjs-cls in your bootstrap function:

// main.ts
import {ClsMiddleware} from 'nestjs-cls';

async function bootstrap() {
  // init app...

  app.use(
    new ClsMiddleware({
      async setup(cls, req) {
        cls.set('TENANT_ID', req.params('tenant_id'));
      },
    }).use
  );

  // etc
}

Step #2: Add ClsModule.forRoot({ global:true }) to your App Module (app.module.ts) imports.

Step #3: Create a file that exports a custom Factory Provider that returns an extended Prisma client.

// prisma-tenancy.provider.ts
import { PrismaModule, PrismaService } from 'nestjs-prisma';
import { ClsService } from 'nestjs-cls';

const useFactory = (prisma: PrismaService, store: ClsService) => {
    return prisma.$extends({
        query: {
            $allModels: {
                async $allOperations({ args, query }) {
                    const tenantId = store.get('TENANT_ID');


                    const [, result] = await prisma.$transaction([
                        prisma.$executeRaw`SELECT set_config('tenancy.tenant_id', ${`${tenantId || 0}`}, TRUE)`,
                        query(args),
                    ]);

                    return result;
                },
            },
        },
    });
};

export type ExtendedTenantClient = ReturnType<typeof useFactory>;

export const TENANCY_CLIENT_TOKEN = Symbol('TENANCY_CLIENT_TOKEN');

export const PrismaTenancyClientProvider = {
    provide: TENANCY_CLIENT_TOKEN,
    imports: [PrismaModule],
    inject: [PrismaService, ClsService],
    useFactory
};

Step #4: Create a module that exports the above Factory Provider

// prisma-tenancy.module.ts
import { Module, Global } from '@nestjs/common';
import { PrismaTenancyClientProvider, TENANCY_CLIENT_TOKEN } from './prisma-tenancy.provider';
import { PrismaModule } from 'nestjs-prisma';

@Global()
@Module({
    imports: [PrismaModule],
    providers: [PrismaTenancyClientProvider],
    exports: [TENANCY_CLIENT_TOKEN]
})
export class PrismaTenancyModule { }

Step #5: Add the module above to your root app.module.ts imports.

Now, you can inject the extended client using the token for your custom provider (TENANCY_CLIENT_TOKEN, in this case).

import {Injectable, Inject} from "@nestjs/common";
import {
  TENANCY_CLIENT_TOKEN,
  ExtendedTenantClient,
} from "./prisma-tenancy.provider";

@Injectable()
export class SomeService {
  constructor(
    @Inject(TENANCY_CLIENT_TOKEN) private readonly prisma: ExtendedTenantClient
  ) {}

  // etc
}

Useful Links

Introducing DNV - Frictionless Node development with Docker Compose

Nathan Cook — Tue, 03 Aug 2021 00:23:21 +0000

Hello! If you're a Node developer who uses Docker Compose, you should check out DNV.

DNV works behind the scenes to keep dependencies in your Docker container in-sync with your local project.

It also comes with a custom made, featureful ncurses-like UI designed for use when developing apps using Docker Compose.

This was my 'covid isolation' personal project to learn more about Node fundamentals, and I think it turned out pretty good! If you do check it out and run into any problems, please create an issue.

Here's a showcase recorded using Asciinema which shows how you can shell in to containers, install and run programs with full interactivity, open the README.md for your dependencies, show a live 'metrics' display with CPU, memory and event loop stats of the Node process, view a list of and run .sh scripts and script entries from your package.json in the container, and more!

DNV Showcase