DEV Community: Pavel Litkin

Ultimate Node.js starter that scales with DI, native TypeScript, super fast unit tests and all batteries included

Pavel Litkin — Sun, 10 Jul 2022 16:03:13 +0000

Ultimate Node.js Starter that Scales with Native TypeScript, Super Fast Unit Tests, DI and more Batteries Included

The purpose of this post is to provide you with a tool to start your new node.js projects with an emphasis on scalability and developer experience.

The main idea is to use minimum dependencies, easier maintenance, better re-compiling times, faster testing, less boilerplate.

Quick Start

Clone the repository with

git clone --depth=1 https://github.com/bfunc/nodejs-ulitmate-template.git

Install the dependencies with you favorite package manager

npm install

Run the application in development mode with

npm run dev

ts-node-dev will effectively restart node process on files change

Access

http://localhost:4000

Map of example routes:
/docs - swagger docs
/orders - sample api route
/products - example api route
/products/:id - example api route

└── /
    ├── docs (GET)
    │   docs (HEAD)
    │   └── / (GET)
    │       / (HEAD)
    │       ├── * (GET)
    │       │   * (HEAD)
    │       ├── uiConfig (GET)
    │       │   uiConfig (HEAD)
    │       ├── initOAuth (GET)
    │       │   initOAuth (HEAD)
    │       ├── json (GET)
    │       │   json (HEAD)
    │       ├── yaml (GET)
    │       │   yaml (HEAD)
    │       └── static/
    │           └── * (GET)
    │               * (HEAD)
    ├── orders (GET)
    │   orders (HEAD)
    └── products (GET)
        products (HEAD)
        └── /
            └── :id (GET)
                :id (HEAD)

Run the application in production mode

npm start

You're ready to go!

Additional commands

Run unit tests

npm run test

Run test coverage

npm run coverage

Auto format all project files with prittier

npm run format

Run ESlint on all project files

npm run lint

Tooling

Native TypeScript

We can avoid cumbersome compiling step with intermediate artifacts and get native TypeScript execution for node.js with ts-node

With ts-node you can run any _.ts directly as you are running regular _.js script with node.

ts-node index.ts

It comes with a price of small performance overhead on first file read at runtime, so if this is a concern for your application in production you can use ts-node together with SWC (in order of magnitude faster TypeScript transpiler implemented in Rust) without typechecking.

Path mapping
Very handy tsconfig-paths library
allows to import modules from the filesystem without prefixing them with "./".

Watch Mode
We are going to use ts-node-dev to watch files and restart application on change, ts-node-dev is tweaked version of node-dev that uses ts-node under the hood. It restarts target node process but shares Typescript compilation process between restarts. This significantly increases speed of restarting comparing to node-dev or nodemon.

ESLint

Nothing special here, ESLint config extends @typescript-eslint/recommended rules.

Run lint command run linter on whole project

Environment

Use .env file to simplify setting environment variables for development, it will be
picked up by dotenv.
Env files may contain values such as database passwords or API keys. It is bad practice committing .env files to version control.

Logging

pino json logger, because it is standard in most enterprise applications.

Webserver

Fastify web framework, becasue it is highly focused on providing the best developer experience with the least overhead.

Unit Test

Testing is very important part of development process, that is why here we are going to bet on new player on unit test frameworks field Vitest. In this case benefits are more important than potential risk choosing less established solution in enterprise (in any case it is worth a try because Vitest and Jest APIs and snapshots are compatible).

Benefits of using `Vitest` over `Jest`

Main benefit is speed, in testing speed is important, especially if you tend to work in TDD/BDD style, every millisecond matters and Vitest is way faster than Jest in watch mode.
It understands TypeScript natively, no need to run transpiler
Everything is in the box, assertions, mocking, coverage - no need to maintain bloated list of dependencies.
Vitest UI, test dashboard interface. demo

Warning though, Vitest is in active development and still considered as not fully stable. Checkout doc page for more info.

Project structure

Two of the most commonly used approaches to structure projects are: Folder-by-type and Folder-by-feature.

Examples:

`Folder-by-type`

src
├── controllers
│    ├── UserController.ts
│    └── PetController.ts
├── repositories
│    ├── UserRepository.ts
│    └── PetRepository.ts
├── services
│    ├── UserService.ts
│    └── PetService.ts
│
└── index.ts

`Folder-by-feature`

src
├── pet
│    ├── Pet.ts
│    ├── PetController.ts
│    ├── PetRepository.ts
│    └── PetService.ts
├── user
│    ├── User.ts
│    ├── UserController.ts
│    ├── UserRepository.ts
│    └── UserService.ts
│
└── index.ts

Natively, when we are starting a new project, we tend to follow Folder-by-type approach, because when there is small amount of functionality it looks cleaner and requires less thinking. But what actually happens is that when project grows it basically turns into one big feature without clean separation of concerns inside.

It turns out that
Folder-by-type works well on small-scale projects and Folder-by-feature better suits large applications, because it provides higher modularity and easier code navigation.

We are aiming for scale with this starter, so it is based on Folder-by-feature structure and when project will became really big and amount of files in feature will become too high, structure can be improved a bit by taking an advantage of Folder-by-type structure inside features.

It may look like this:

`Folder-by-feature-by-type`

src
├── pet
│    ├── controllers
│    │    ├── PetGenericController.ts
│    │    └── PetSpecificController.ts
│    └── services
│         ├── PetGenericService.ts
│         └── PetSpecificService.ts
├── user
│    ├── controllers
│    │    ├── UserGenericController.ts
│    │    ├── UserPrivateController.ts
│    │    └── UserPublicController.ts
│    └── services
│         ├── UserGenericService.ts
│         ├── UserPrivateService.ts
│         └── UserPublicService.ts
│
└── index.ts

Dependency Injection

The idea behind dependency injection is really simple, it is basically providing list of dependencies as parameters instead of having hardcoded imports.

The base of our dependency injection is a design pattern called composition root, it is located in the src/container.ts file. Container is getting created with provided collection of dependencies, dependancy can be anything constant, function or class.
Example:


function getUserService({ UserModel }) {
  return {
    getUserWithBooks: userId => {
      ...
      UserModel.getBooksByUserId(userId)
    },
  }
}

container.register({
  // the `userService` is resolved by invoking the function.
  userService: asFunction(getUserService)
})

Take a look at awilix docs for more information.

Automatic module loading

Automatic module loading from filesystem (like pages in next.js) is used. The convention is that before container creation script will look into modules folder, traverse its content and auto load dependencies of defined types, like models, controllers, services etc. Check src/index.ts for list of filenames that will be automatically loaded.

For now dependenciesLoader.ts script is very basic, for more advanced scenarios with nested folders or glob patterns you can use built-in awilix loadModules function.

Swagger documentation generator

Automatically generated Swagger docs from your model schemas. Zod instance is automatically converted to JSON schema that is provided to Fastify route in order to generate docs, no code duplication.

Final words

Ultimate Starter was designed to be as much flexible as less opinionated as possible, that is why Database drivers, ORMs or authentication libraries were not included as part of the starter, despite there is strong temptation to add at least integration with supabase.

It is not easy to find the Golden Mean, here is list of things that are currently missing, sorted by importance.

Error handling
GraphQL
Authentication
Commit hooks
Deployment guidelines

If there is something that is missing to achieve the best developer experience possible, please do not hesitate and leave a comment. Your comments may be extremely valuable, other people may encounter the same things you do. Sharing is caring :)

Simple TypeScript tricks for scalable apps

Pavel Litkin — Sun, 01 May 2022 17:27:05 +0000

TypeScript gains popularity each day. We use TypeScript because it makes development safer and faster. Here is a set of tricks and guidelines for a more rigorous use of TypeScript. You need to get used to them once - and they will save a lot of time in the future.

Do not use `any`

The trick is, instead of ignoring the problem - "legalize" it:
type TODO_ANY = any

Pros:

• You do not have to be desperate and write props.
• You are deliberately showing that the code should be improved in the future.
• This type is easy to find by searching for the project.

Restrictions:

• Using this technique too often is unlikely to be beneficial.
• Conscientiousness and responsibility are needed to remove such gaps in the future.

You may also find it useful:
export type TODO_UNKNOWN = unknown;

In the case of any, the compiler will allow almost any action. In the case of unknown, on the contrary, almost any use will lead to an error.

Last word, do not use any even if type does not matter, for example, instead of this:
const getLength = (arr: any []) => arr.length

you can write this:
const getLength = (arr: { length: number }) => arr.length
It is not ideal, but is much better than any.

Of course best solution in this specific case is to use generics:
const getLength = <T>(arr: T[]) => arr.length;

Use `readonly`

It is bad practice to mutate the structures you work with. For example, in the Angular world, this immediately leads to a number of consequences in the application: problems appear with checking changes in components, after which the display is not updated after mutating the data.
But can we easily prevent all attempts to mutate data? For myself, I just formed a habit of writing readonly everywhere.

What is to be done?

There are likely many places in your application where you can replace unsafe types with readonly alternatives.

Use readonly on interfaces:

// Before
export interface Person {
  name: string;
}

// After
export interface Person {
  readonly name: string;
}

Give preference to readonly in the types:

// Before
export type UnsafeType = { prop: number }

// After
export type SafeType = Readonly<{ prop: number }>

Use readonly fields class wherever it is possible:

// Before
class UnsafeComponent {
  loader$ = new Loader<boolean>(true);
}

// After
class SafeComponent {
  readonly loader$ = new Loader<boolean>(true);
}

You can use readonly on all data structures, like:
const numsFromServer = readonly number[] = [1, 2, 42]

as const

TypeScript v3.4 introduces const-assertions. It is a stricter tool than readonly types because it will pack your constant with the most precise type possible. Now you can be sure: no one and nothing can change this.

Also, when using as const, your IDE will always show you the exact type of the entity being used.

Use strict mode

TypeScript has a cool strict mode that is unfortunately disabled by default. It includes a set of rules for safe and comfortable working with TypeScript.
With strict mode, you forget about errors like undefined is not a function and cannot read property X of null. Your types will be accurate and correct.

What is to be done?

If you start a new project, just turn on strict right away and be happy.

If you already have a project without strict mode and now you want to enable it, then you will face a number of difficulties. It is very difficult to write strict code when the compiler does not signal problems in any way. So you will probably have a lot of buggy spots and the migration process makes you bored very quickly.

Generally, the best approach is to break this huge task into chunks. The strict mode itself is a set of six rules. You can enable one of them and fix any errors. Your project is now a little more rigorous. Next time, turn on another rule, correct any mistakes and keep working. In one day, you will collect the entire strict mode!

But in large projects everything is not so smooth, then you can act more iteratively. Enable the flag, and above all conflicts, add @ts-ignore and a TODO comment. The next time you work with the file, correct the type as well.

Setting --strict to true, sets all of the following options to true:

--noImplicitAny: If TypeScript can’t infer a type, we must specify it. This mainly applies to parameters of functions and methods: With this settings, we must annotate them.
--noImplicitThis: Complain if the type of this isn’t clear.
--alwaysStrict: Use JavaScript’s strict mode whenever possible.
--strictNullChecks: null is not part of any type (other than its own type, null) and must be explicitly mentioned if it is a acceptable value.
--strictFunctionTypes: enables stronger checks for function types.
--strictPropertyInitialization: Properties in class definitions must be initialized, unless they can have the value undefined.

Use Utility Types

TypeScript has a set of types that are shortcuts for conversions of other types.

I advise you to study in detail all the official documentation on Utility Types and start actively introducing them into your applications. They also save a lot of time.

From my point of view, most useful utility types are Pick and Omit, they allow you to avoid creating not essential intermediate types.

`Pick<Type, Keys>`

interface Todo {
  title: string;
  description: string;
  completed: boolean;
}

type TodoPreview = Pick<Todo, "title" | "completed">;

const todo: TodoPreview = {
  title: "Clean room",
  completed: false,
};

`Omit<Type, Keys>`

interface Todo {
  title: string;
  description: string;
  completed: boolean;
  createdAt: number;
}

type TodoInfo = Omit<Todo, "completed" | "createdAt">;

const todoInfo: TodoInfo = {
  title: "Pick up kids",
  description: "Kindergarten closes at 5pm",
};

Conclusion

These are just a few of the principles we use, but in our team, they are perhaps the most basic. Of course, this is not the only correct way to write code in TypeScript, but once we get used to it, we just write the code this way without thinking. From time to time, this saves us from unexpected tricky mistakes in large projects.

Best practices for file uploads in Nodejs and HTML

Pavel Litkin — Wed, 21 Apr 2021 08:37:30 +0000

Although it seems like uploading a file to the server is a simple task, there are many ways to do it and there are some pitfalls and edge cases, so the purpose of this article is to overview the whole process end to end and to take a closer look at each aspect.

Let's begin from the

Creating upload form

To upload files to the server from the browser generally we need to create a html form. The most stupid simple example of the upload form that can be is:

<form method="POST" enctype="multipart/form-data">
  <input type="text" name="myText" />
  <input type="file" name="myImage" />
  <input type="submit" />
</form>

It works as is. No javascript needed.

However most likely we want something more sophisticated, for example if we are uploading images we may want to show preview thumbnails with a file size label or we may want to see a progress bar with pause/resume button or whatever else it may be.

Possibly the most common requirement is to replace default html <input type="file"> with nice dropzone area, there are javascript libs that can do that, but in reality you may not even need library, it can be done with simple input type="hidden" (to prevent that less-than-attractive user interface from being displayed). Because it is not affecting functionality we will skip form ui and styling here, there are plenty of good tutorials in the web (MDN) on this topic.

The html of basic upload form looks like this:

<form>
  <h4>Multipart from data upload example</h4>
  <input id="text" type="text" name="myText" />
  <input id="image" type="file" name="myImage" 
                                multiple accept="image/*" />
  <input id="submit" type="submit" value="Upload" />
  <hr />
  <h4>Preview</h4>
  <div id="preview"></div>
</form>

Couple of interesting points here:

has accept attribute that is handy to limit input from accepting undesired file types. Another multiple attribute allows input to accept multiple files, omit it if you want to limit it to one (In this tutorial we will upload a single image).
Example of text input here was added just for example, it may be handy to pass a caption with image in the same payload or some initial data that is needed to create an entry in db.

Let's add some javascript to bring form alive:

window.onload = function () {
  const submitButton = document.getElementById('submit')
  const imageInput = document.getElementById('image')
  const textInput = document.getElementById('text')
  const preview = document.getElementById('preview')

  submitButton.addEventListener('click', async e => {
    e.preventDefault()

    const [file] = imageInput.files

    if (!file) {
      throw new Error('File was not selected')
    }

    // TODO - send file
  })
}

Nothing special, just some boring boilerplate to get values from inputs and register click handler for the submit button.

Select file

To add some spice we can show preview for the image, when the user drops it into input.

imageInput.addEventListener('change', e => {
  // we are taking only the first element
  // because we are doing single file upload
  const [file] = imageInput.files
  function updatePreviewImage(file){
    // TODO - update preview <img> src 
  }
})

Now when we have reference to the selected file we need to create a DOMString for <img> tag. There is browser interface method URL.createObjectURL() that can take underlying ArrayBuffer and create DOMString that represents the specified File object.

function updatePreviewImage(file) {
  const url = URL.createObjectURL(file)
  preview.innerHTML = `<img src="${url}" />`
}

Let’s take a look on src attribute of appeared <img> tag:
<img src="blob:http://localhost:3000/1b2a4ac9-4bd4-4726-b302-d74e6ed2ba48">

As you can see, url of page where our html is hosted is part of DOMString, if page will be opened with file:/// protocol and not being hosted, then DOMString will work but look like this:
<img src="blob:null/f8111cf8-d598-4305-9bdd-4ba5b7db22f7">.

This illustrates that URL lifetime is tied to the document in the window on which it was created. That means that we have to release an object URL, by calling revokeObjectURL() after the form submission.

Building FormData

Building FormData is straightforward:

const formData = new FormData()
formData.append('myImage', file)
formData.append('myText', textInput.value || 'default text')

First parameter is the name of the property in the request.body object when we will get a request later on server.
Second is the value and there is an optional third parameter that may hold the original filename of the Blob object if we are sending Blob.

Adding file as a Blob

Sometimes we need to upload files that are not images, like 3d models, audio records or any other binary file format. In some cases it may be useful to treat them as Blobs, example:

const [file] = imageInput.files
const arrayBuffer = await file.arrayBuffer()
const myBlob = new Blob([new Uint8Array(arrayBuffer)], {
  type: file.type,
})
formData.append('myBlob', myBlob, file.name)

Sending FormData to server

We can send data to server with simple fetch

await fetch(uploadUrl, {
  method: 'POST',
  body: formData,
})

There is small pitfall however:

Warning: When using FormData to submit POST requests using XMLHttpRequest or the Fetch_API with the multipart/form-data Content-Type (e.g. when uploading Files and Blobs to the server), do not explicitly set the Content-Type header on the request. Doing so will prevent the browser from being able to set the Content-Type header with the boundary expression it will use to delimit form fields in the request body.

Open DevTools and take a look on request headers, you will see that browser automatically adds Content-Type: multipart/form-data and then it appends random boundary value that is used to separate parts of form-data

Content-Type:
  multipart/form-data; boundary=---WebKitFormBoundaryrHPgSrtbIrJmn

Display progress bar

Unfortunately currently it is not possible to get file upload progress for fetch() method chromestatus.com.

Solution for now is to use good ol' fella XMLHttpRequest

let request = new XMLHttpRequest()
request.open('POST', '/upload')

// upload progress event
request.upload.addEventListener('progress', function (e) {
  // upload progress as percentage
  console.log((e.loaded / e.total) * 100) 
})

// request finished event
request.addEventListener('load', function (e) {
  // HTTP status message (200, 404 etc)
  console.log(request.status)
})

// send POST request to server
request.send(formData)

Server side

Before we dive into the code let's stop and think for a while.

Do we need to store files on the server?
What exactly has to be done with files on the server, is there any part of that responsibility that can be passed to third-party tools?
Can move files to external storage like AWS S3 or Azure Blob Storage without temporarily saving them on the server?

Libraries for processing file uploads

To process uploaded files we need a library that knows how to do it in an efficient and secure way. There is good comparison article on this (link at the bottom of the page).

We gonna stick with busboy becasue it is considered the most production stable solution (other libraries using it under the hood) and becasue it do not creates temporary files.

If we do need to save files occasionally, we can stream file contents to disk like this:

const imageBuffer = req.raw.files['myImage'].data;
  const fileName = getRandomFileName();
  const stream = fs.createWriteStream(fileName);
  stream.once('open', function (fd) {
    stream.write(imageBuffer);
    stream.end();
  });

Or we can can take multer library that is based on busboy too and it has option to automatically save files on disk upon receiving.

Web Framework

In this tutorial we will use a web framework, despite the fact that we do not need a web framework to receive uploaded files.

Why? It’s because in real projects, almost always we do need a web framework, unless we are doing something very specific, so we want to know how to properly tie our library with the framework.

Official Fastify plugin for uploading files is fastify-file-upload, if will take a closer look at it’s repo, we will see that is it nothing more than a wrapper around another library express-fileupload, that is by itself a wrapper around busboy.

So for Fastify we gonna use fastify-file-upload and for Express express-fileupload. Using wrapper is convinient for example, you may define validation schema for formdata, but we use busboy directly without wrapper too. Let's write our own wrapper around busboy library.

Writing Fastify wrapper for busboy

Writing a wrapper is really simple task, there is only one tiny thing that Fastify out of the box supports only the application/json context-type, so we need to define our parser for multipart/form-data

fastify.addContentTypeParser('multipart/form-data', function (request, payload, done) {
  done(err, payload)
})

Fasity exposes original nodejs request and response under request.raw and reply.raw

fastify.post('/upload', function (request, reply) {
  const req = request.raw
  const res = reply.raw
  // TODO - copy paste code from busboy example as is, 
  // example: new Busboy({ headers: req.headers }) ...
  // full example in the demo repo

We just put some code in route handler, and it works, but this is not the right approach, Fastify gives us much cleaner was to do it, to register our hander as a plugin.

Blobs

There is nothing special in receiving Blobs, the same server code works as is, the only difference is that it may be missing original filename if it was not provided as third parameter to formData.append

Security

It is important to know that there are many types of vulnerabilities that may be exploited if there is a bug in the processing buffers, more information here.

It is considered a good practice to take out upload endpoints to separate microservice that will have an additional layer of security.

Secure file uploads rules

Always create a new unique filename, never use one provided by a client, because it may intentionally include paths to critical system files on the server.
Never host uploaded files from the same document root, better to host them on totally different machines in different networks.
Any file may be malicious, extension doesn’t mean anything, it is better to perform some third-party malware scan if it is possible.
Keep control of permissions, files should not be executable
Authenticate file uploads, limit number of uploaded files per session, limit file size range

Link to example repository

https://github.com/bfunc/file-upload-example

DEV Community: Pavel Litkin

Ultimate Node.js starter that scales with DI, native TypeScript, super fast unit tests and all batteries included

TOC

Ultimate Node.js Starter that Scales with Native TypeScript, Super Fast Unit Tests, DI and more Batteries Included

Quick Start

Additional commands

Tooling

Native TypeScript

ESLint

Environment

Logging

Webserver

Unit Test

Benefits of using Vitest over Jest

Project structure

Examples:

Folder-by-type

Folder-by-feature

Folder-by-feature-by-type

Dependency Injection

Automatic module loading

Swagger documentation generator

Final words

Simple TypeScript tricks for scalable apps

Do not use any

Pros:

Restrictions:

Use readonly

What is to be done?

as const

Use strict mode

What is to be done?

Use Utility Types

From my point of view, most useful utility types are Pick and Omit, they allow you to avoid creating not essential intermediate types.

Pick<Type, Keys>

Omit<Type, Keys>

Conclusion

Best practices for file uploads in Nodejs and HTML

Creating upload form

Select file

Building FormData

Adding file as a Blob

Sending FormData to server

Display progress bar

Server side

Libraries for processing file uploads

Web Framework

Writing Fastify wrapper for busboy

Blobs

Security

Secure file uploads rules

Link to example repository

Further reading

Benefits of using `Vitest` over `Jest`

`Folder-by-type`

`Folder-by-feature`

`Folder-by-feature-by-type`

Do not use `any`

Use `readonly`

`Pick<Type, Keys>`

`Omit<Type, Keys>`