DEV Community: Omar khairy

Setup monorepo with pnpm, typescript and turborepo

Omar khairy — Tue, 27 Feb 2024 15:43:28 +0000

Introduction

The past few weeks i was swtich some backend project to use typescript and the frontend during that time was
alredy typescript based and it contains types and interfaces the most of them represent the backend models.
So we switched to typescript on the backend and we started to share the types and interfaces between the backend and the frontend.

The two problems

when setup or share files between folders sepecially with typescript you will face two problems:

Add the right package.json setup to each folder(package) to be able to build and run each folder separately.
Setup the right tsconfig.json to automatically make ts to watch the changes and build the files.

Fix the first problem (Adding the right package.json setup to each folder) with workspaces

The first problem can be solved by using the workspaces feature in the package.json file.
The workspaces feature allows you to share dependencies between the packages and it will be installed in the root node_modules folder.

yarn workspaces vs pnpm workspaces

On the project we already use yarn as a package manager so i went with yarn workspaces.
The main difference between yarn and pnpm is that pnpm uses a single node_modules folder and hard links to the packages that are shared between the packages.
So in that way pnpm is faster than yarn and npm because it doesn't need to install the same package multiple times.
Form storage wise sepecially on large projects pnpm is the best choice.

From my experience i found that pnpm is faster than yarn workspaces.

here is the project structure:


project
├── app
│   ├── src
│   ├── package.json
│   ├── tsconfig.json
├── server
│   ├── src
│   ├── package.json
│   ├── tsconfig.json
├── packages
│   ├── shared-types
    │   ├── src
    │   ├── package.json
    │   ├── tsconfig.json
├── package.json

app/package.json

{
  "name": "app",
  "private": true,
  "version": "0.0.0",
  "type": "module",
  "scripts": {
    "dev": "vite",
    "build": "tsc && vite build"
  },
  "dependencies": {
    "react": "^18.2.0",
    "react-dom": "^18.2.0"
  },
  "devDependencies": {
    // some dev dependencies
  }
}

server/package.json

{
  "name": "server",
  "version": "1.0.0",
  "main": "index.js",
  "scripts": {
    "start": "nodemon --exec ts-node src/index.ts",
    "build": "tsc"
  },
  "dependencies": {
    "express": "^4.18.2"
  },
  "devDependencies": {
    "@types/express": "^4.17.21",
    "@types/node": "^20.11.17",
    "nodemon": "^3.0.3",
    "ts-node": "^10.9.2",
    "typescript": "^5.2.2"
  }
}

packages/shared-types/package.json

{
  "name": "shared-types",
  "version": "1.0.0",
  "main": "dist/index.js",
  "types": "dist/index.d.ts",
  "files": ["dist"],
  "scripts": {
    "build": "tsc"
  },
  "devDependencies": {
    "typescript": "^5.2.2"
  }
}

on shared-types/package.json i added the types field to point to the dist/index.d.ts file
so when i import the shared-types package in the app or server it will automatically import the types and interfaces.

Adding pnpm workspaces(add pnpm-workspace.yaml file to the root folder)

packages:
  - app
  - server
  - packages/*

run pnpm install to install the packages and the dependencies.

Add shared-types to the app and server as a dependency

cd server && pnpm add shared-types

cd app && pnpm add shared-types

Fix the second problem (Setup the right tsconfig.json)

To automatically make ts to watch the changes and build the files.

the composite option

on package/shared-types/tsconfig.json i added the composite option to true.

{
  "compilerOptions": {
    "module": "commonjs",
    "target": "es6",
    "rootDir": "./src",
    "outDir": "./dist",
    "esModuleInterop": true,
    "declaration": true,
    "composite": true
  }
}

The composite option is used to enable the package to be used with package references(we will use it later to reference the shared-types package in the app and server tsconfig.json file).

Add the references field to the app and server tsconfig.json file
on the app and server tsconfig.json file i added the references field to point to the shared-types package.

app/tsconfig.json

{
  "compilerOptions": {
    "target": "ES2020",
    "useDefineForClassFields": true,
    "lib": ["ES2020", "DOM", "DOM.Iterable"],
    "module": "ESNext",
    "skipLibCheck": true
    // some other options
  },
  "include": ["src"],
  "references": [
    {
      "path": "../packages/shared-types"
    }
  ]
}

server/tsconfig.json

{
  "compilerOptions": {
    "module": "commonjs",
    "target": "es6",
    "rootDir": "./src",
    "outDir": "./dist",
    "esModuleInterop": true,
    "composite": true
  },
  "references": [
    {
      "path": "../packages/shared-types"
    }
  ]
}

Trying imort the shared-types package in the app and server

import express from 'express'

import { Product } from 'shared-types'

const app = express()
const port = 3000

const products: Product[] = [
  { id: '1', name: 'Product 1', price: 100 },
  { id: '2', name: 'Product 2', price: 200 },
  { id: '3', name: 'Product 3', price: 300 },
]

The final folder structure

project
├── app
│   ├── src
│   ├── package.json
│   ├── tsconfig.json
|   ├── pnpm-lock.yaml
├── server
│   ├── src
│   ├── package.json
│   ├── tsconfig.json
|   ├── pnpm-lock.yaml
├── packages
│   ├── shared-types
    │   ├── src
    │   ├── package.json
    │   ├── tsconfig.json
    |   ├── pnpm-lock.yaml
├── package.json
├── pnpm-lock.yaml
├── pnpm-workspace.yaml

Why do they use turborepo on large open source projects?

Turborepo is a tool that makes it easy to manage monorepos with pnpm and typescript.
On large open source porject like cal.com they use it for fast building or running developing tasks like testing or linting.
Turborepo depend havily on caching so it would reduce signficantly the time to build or run the tasks as well as CI/CD pipelines time and cost.

I am still exploring turborepo i may write a post about it in the future, but for simple projects i think pnpm workspaces is enough.

The Full code is available on my repo

Coding Backwards

Omar khairy — Wed, 12 Apr 2023 09:40:04 +0000

Coding Backwards

I do really love the working backward concept, that framework it is the amazon way to build new products I got it from the book Working backward by Colin Bryar and Bill Carr.
For me I found it really useful to reduce the vagueness of a new project or feature that I am working on, it helps me to focus on the most important things first and to avoid going in the wrong direction.

As Bezos said:

The Working Backwards process is a huge amount of work. But, it saves you even more work later.
The Working Backwards process is not designed to be easy, it’s designed to save huge amounts of work on the backend,
and to make sure that we’re actually building the right thing.”

Working Backward in actions

A few weeks ago I was working on a new feature. the feature was we need to let the user know what is going on in the background.
Basically, the project was uploading and mainpulating large pdf files. The user uploads a file and the file is processed in the background.
we did not want to show a loading spinner or a progress bar because the processing time is not predictable.

So we decided to introduce real-time updates to the users with WebSocket. Most of our products depend on Cloudflare services and we use Cloudflare workers to handle the requests and the WebSocket connection.

We decided to use the Cloudflare durable objects to store the state processing with other metadata about the current processing step.

But the problem was that we did not use durable objects before it is a relatively new database in the Cloudflare ecosystem and when it comes to databases a lot of question comes to mind: in context concurrency, scaling, isolation levels, etc.

Coding Backwards in actions: What is the goal of the feature?

The goal of the feature is to let the user know what is going on in the background. We want to show the user real-time updates about the progress of the processing.

What is the technical solution?

With Cloudflare durable and WebSockets.

General Technical Q&A

Question: Does Durable-Objects have a limit on the number of WebSockets that can be opened?
Answer: No, there is no limit on the number of WebSockets that can be opened.

Question: Does durable objects support concurrent writes or isolation?
Answer: Yes, durable objects support concurrent writes it has a single-threaded execution model also it provides transactional API

const fileStatus = await DURABLE_OBJECTS.get("fileStatus");
await fileStatus.transaction(async (txn) => {
    await txn.put("file1", "processing");
    await txn.put("file3", "finished");
    await txn.put("file2", "failed");
});

Question: Does Durable-Objects have a limitation for key-value?
Answer: Yes, Keys are limited to a max size of 2048 bytes, and values are limited to 128 KiB (131072 bytes).

Question: How can we store files processing status?
Answer: We can store files processing status in a key-value store. the key it would be the file name and the value it would be the status of the file processing.

Question: How can we add a new status for an file?
Answer: We can add a new status for an file by using the transactional API.

Question: How can we get all statuses of an file?
Answer: We can get the status of an file by using the transactional API. it would add latency to the request. But it provides isolation and consistency.

Question: Can we control caching for Durable Objects?
Answer: Yes, we can control caching it provides memory cache for the same key, also we can disable it by setting noCache to true

Question: How can we get the status of all filess?
Answer: we can get all files status's by using list() method.

In Depth Technical Q&A

Question: How can Node.js server update the status of processing an file?
Answer: Internally, in the Node.js Server, We can use an event emitter to emit an event then in the event listener we can update the status of an file by HTTP request.
Also, the event emitter, makes the code cleaner and achieves the separation of concerns between the event emitter and the event listener.

const eventEmitter = new EventEmitter();

// in the event listener
eventEmitter.on("fileStatus", (fileName, status) => {
  // update the status of an file
});

// do some work like downloading an file
// emit an event
eventEmitter.emit("files downloaded", fileName, status);

// do some work like process an file
// emit an event
eventEmitter.emit("files processed", fileName, status);

Question: Event emitter is synchronous and it would block the execution of the code, how can we solve this problem?
Answer: We can use emittery to solve this problem because it is asynchronous.

Question: How can durable objects listen to events?
Answer: Durable objects provide a way to handle HTTP requests directly from the worker script.

For example, we can use the following code to handle HTTP requests directly from the worker script.

export class Counter {
  constructor(state, env) {
    this.state = state;
  }

 // Handle HTTP requests from clients.
  async fetch(request) {
    let url = new URL(request.url);
    let value = (await this.state.storage.get("value")) || 0;

    switch (url.pathname) {
      case "/increment":
        ++value;
        break;
      case "/decrement":
        --value;
        break;
      case "/":
        // Just serve the current value.
        break;
      default:
        return new Response("Not found", { status: 404 });
    }

    await this.state.storage.put("value", value);
    return new Response(value);
  }
}

Question: How can we handle WebSocket requests from clients?

Answer: We can use the following code to handle WebSocket requests from clients.


export class Counter {
  constructor(state, env) {
    this.state = state;
  }

  // Handle WebSocket connections from clients.
  async webSocketUpgrade(request) {
    let socket = await request.accept();
    let value = (await this.state.storage.get("value")) || 0;

    socket.send(value);

    for await (let {data} of socket) {
      if (data === "increment") {
        ++value;
      } else if (data === "decrement") {
        --value;
      } else {
        socket.close(1008, "Invalid message");
        return;
      }

      await this.state.storage.put("value", value);
      socket.send(value);
    }
  }
}

Wrapping up

A few months ago, I was working another feature. we decided to go with cloudlfre key-value store but after weeks of development we facing a problem with kev-value store which is it was caching the data in edge nodes and we can not control it so we were reading the stale data.
So using working backwards we can avoid these kind of problems and we can make the right decision from the beginning.