Build a NestJS Module for Knex.js (or other resource-based libraries) in 5 Minutes

John is a member of the NestJS core team

Ever wanted to integrate your favorite library into NestJS? For example, while Nest has broad built-in support for database integration, what if you want to use your own favorite library and no Nest package exists? Well, why not build your own?

This might seem like a daunting task at first. But if you've been following my blog posts, you saw a design pattern for NestJS dynamic modules in my last post that points you in the right direction. OK, you say, but it still seems like a lot of work to integrate a library.

I have good news! Using the awesome power of the Nest CLI, you can generate a complete, custom dynamic module template, following NestJS best practices, with a single command! You can then literally have your library integrated in about 5 minutes! This technique works for many resource-based libraries that export what I'll call an API object, which naturally work well with Nest's built-in singleton provider model for sharing a global resource. Follow along with such an adventure as we build a quick integration to Knex.js below.

And stay tuned for more on the wonders of the vastly under-appreciated NestJS CLI. Built on top of the Angular CLI, the potential for this tool to revolutionize the way you use Nest is limitless. I have lots more planned for this topic in the near future!

Intro

In my How to build completely dynamic NestJS modules article last week, I covered a design pattern that is used throughout standard Nest packages like @nestjs/passport, @nestjs/jwt, and @nestjs/typeorm. The point of the article was two-fold:

• Provide a roadmap to a basic pattern used within common NestJS packages to help you be a better consumer of those packages.
• Encourage Devs to think about how to make use of this pattern in their own code to enable more easily composable modules.

Some might consider the pattern a bit complex, and I wouldn't necessarily argue. However, gaining a good understanding of it is useful, and helps in the quest for mastery of some core concepts of Nest, especially around leveraging the module system and dependency injection. Nevertheless, there's a fair amount of boilerplate code in the pattern. Wouldn't it be great to simplify the process?

Buckle up! NestJS CLI custom schematics to the rescue! 🚀

What We'll Build

In this tutorial, we'll build a module that exports a direct API to the full Knex.js library. This is a powerful DB integration library used widely across the Node.js ecosystem. Here's what we'll do. The following represent the exact same steps you can use to integrate any other basic callable API (for example, ioredis, Cassandra, Neo4J, Elasticsearch, LevelDb to name just a few).

1. Use the dynpkg custom schematic to generate a customized package (the schematic automates the dynamic module pattern I've been discussing above).
2. Fill in a few quick details to customize the package for Knex.js.
3. Use the auto-generated test client to demonstrate that it works.

Thanks to the power of the Nest CLI, coupled with a custom schematic, we can complete these steps in about 5 minutes! As a bonus, we'll cover a few more advanced features at the end of this article, including how to publish the generated package on npm with a single click.

If you want to see a completed version of the steps presented in the remainder of this article (a fully useable NestJS/Knex.js module), you can find a github repo with the completed code here.

Nest CLI and Schematics

You probably already use the Nest CLI on a regular basis. nest new myProject and nest generate controller user are common examples. They use the @nestjs/schematics package that ships with the CLI out-of-the-box. What's very cool is that you can easily use other schematics to do highly customized work. A feature common to all schematics is that they understand your project, and have a smart way of scaffolding new architectural components and wiring them into your project. Think of an individual schematic as a blueprint, and of the CLI machinery as a set of standards for how each blueprint works. Because of this, new schematics "just work", and inherit all of the features provided by the CLI and its standard capabilities.

You can write your own schematics, and I plan to show you how in some upcoming blog posts. For now, I've built one that you can use to do the library integration project we've undertaken today. Let's get started!

To use any external schematics, you of course need to install them. One important note: you must install a schematics collection as a global package, due to the way the CLI works.

Installing a Custom Schematics Collection

Schematics are packaged as collections and can be bundled up as npm packages, so installing them is simple. You can do so now with:

npm install @nestjsplus/dyn-schematics -g

Using a Custom Schematics Collection

Using schematics from a custom collection is straightforward. The standard Nest CLI command structure looks like this:

nest commandOrAlias [-c schematicCollection] requiredArg [options]

• commandOrAlias is: new or generate (alias: g) or add
• schematicCollection is optional, and defaults to the built-in NestJS schematics; you can optionally specify a globally installed npm package
• requiredArg is the architectural element being generated/added, e.g., controller, module, or soon, in our case dynpkg
• options are global to all schematics; they can be --dry-run, --no-spec, or --flat

Once it's installed, you use the @nestjsplus/dyn-schematics package like this:

1. Make sure you're in the folder you want to have as the parent of the project. With this schematic, we're creating a new complete nest package, meaning it's a new standalone project. So it will create a folder using the name you provide, and put all of the component parts inside that folder.
2. Run the schematic with the CLI

nest g -c @nestjsplus/dyn-schematics dynpkg nest-knex

This runs using the custom schematics collection from @nestjsplus/dyn-schematics and specifying the dynpkg schematic to execute (the schematic identifies the thing to generate - in this case, a dynamic module package identified as dynpkg), and giving it a name of nest-knex. If you want to see what this would do without adding files to your filesystem, just add --dry-run at the end of the command.

This should prompt you with the question Generate a testing client?. Answer yes to this to make testing easier. You'll see the following output, and you'll then be able to open this project in your IDE from the nest-knex folder.

► nest g -c @nestjsplus/dyn-schematics dynpkg nest-knex
? Generate a testing client? Yes
CREATE /nest-knex/.gitignore (375 bytes)
CREATE /nest-knex/.prettierrc (51 bytes)
CREATE /nest-knex/README.md (2073 bytes)
CREATE /nest-knex/nest-cli.json (84 bytes)
CREATE /nest-knex/package.json (1596 bytes)
CREATE /nest-knex/tsconfig.build.json (97 bytes)
CREATE /nest-knex/tsconfig.json (430 bytes)
CREATE /nest-knex/tslint.json (426 bytes)
CREATE /nest-knex/src/constants.ts (54 bytes)
CREATE /nest-knex/src/index.ts (103 bytes)
CREATE /nest-knex/src/main.ts (519 bytes)
CREATE /nest-knex/src/nest-knex.module.ts (1966 bytes)
CREATE /nest-knex/src/nest-knex.providers.ts (262 bytes)
CREATE /nest-knex/src/nest-knex.service.ts (1111 bytes)
CREATE /nest-knex/src/interfaces/index.ts (162 bytes)
CREATE /nest-knex/src/interfaces/nest-knex-module-async-options.interface.ts (532 bytes)
CREATE /nest-knex/src/interfaces/nest-knex-options-factory.interface.ts (194 bytes)
CREATE /nest-knex/src/interfaces/nest-knex-options.interface.ts (409 bytes)
CREATE /nest-knex/src/nest-knex-client/nest-knex-client.controller.ts (732 bytes)
CREATE /nest-knex/src/nest-knex-client/nest-knex-client.module.ts (728 bytes)

At this point, you can install the generated code:

cd nest-knex
npm install

Now start up the app with:

npm run start:dev

If you now browse to http://localhost:3000 you should get:

Hello from NestKnexModule!

Great! At this point, we've scaffolded the framework of our dynamic module, and we're ready to begin the Knex.js integration.

Integrating with Knex.js

You now have a project scaffolded to do the library integration. We have to make just a few edits to perform the actual integration. Let's talk them through:

1. We need to install knex as a dependency. Note, in order to run this project, you'll need access to a live SQL database. Knex.js doesn't bundle any database libraries, so you'll need to install the appropriate one (you can read more here). In this tutorial, I'll use PostgreSql, which I have available on localhost. If you do not have a local DB available, you can consider using a docker setup.
2. We need a way to provide the configuration options to the Knex.js API.
3. We need to call the Knex.js API in the way it wants to be initialized, returning a handle to a knex object which we can use to access Knex.js features.
4. To test things out, we can use the client controller that was auto-generated by the schematic.
5. Once it works, we can use this package directly. Or we can publish the package to a registry (e.g., an internal package registry or publicly to npmjs.com)

Install Dependencies

The knex package is required. You will also need a database API library to run the module. Below, I use pg for PostgreSql, but you can choose whatever you want (from the list here).

cd nest-knex
npm install knex pg

Knex.js Options

As discussed in the dynamic modules article, we provide options to the service using an async options provider. The code to do all this has been scaffolded for you, but to pass the appropriate options in a native TypeScript way, we need to modify the NestKnexOptions interface. This is contained in the file src/interfaces/nest-knex-options.interface.ts.

We want to describe the available Knex.js options that will be passed to the library API. The easiest way to do this is to use the types already provided by Knex.js. Since this interface is exported by the knex package, we can simply import it, and our job is nearly done! We'll simply alias it to make it visible to our generated package. Open up src/interfaces/nest-knex-options.interface.ts and edit it to look like this:

// src/interfaces/nest-knex-options.interface.ts
import { Config } from 'knex';

export interface NestKnexOptions extends Config {}

Knex Connections

The job of our module is pretty simple: connect to the API and return a re-usable knex object for interacting with the database. In the case of PostgreSql, knex (on top of pg) returns a connection pool, which means we can connect once and return a singleton knex object that will automatically balance multiple queries across the connection pool. This works seamlessly with NestJS and its notion of singleton providers!

Note: I'm less familiar with Knex than other DB libraries, and also less familiar with other DBs like MySQL, so if you are using this module for such databases, be sure you understand the right pattern for sharing a knex connection object within your application. This topic is not a NestJS issue -- it's all about the architecture of the Knex.js library and the database client library.

Now we'll implement a simple pattern to return our knex object. Open up src/nest-knex.service.ts. Have a quick look to see what the generated code looks like, then replace that code with the following:

// src/nest-knex.service.ts
import { Injectable, Inject, Logger } from '@nestjs/common';
import { NEST_KNEX_OPTIONS } from './constants';
import { NestKnexOptions } from './interfaces';

const Knex = require('knex');

interface INestKnexService {
  getKnex();
}

@Injectable()
export class NestKnexService implements INestKnexService {
  private readonly logger: Logger;
  private _knexConnection: any;
  constructor(@Inject(NEST_KNEX_OPTIONS) private _NestKnexOptions: NestKnexOptions) {
    this.logger = new Logger('NestKnexService');
    this.logger.log(`Options: ${JSON.stringify(this._NestKnexOptions)}`);
  }

  getKnex() {
    if (!this._knexConnection) {
      this._knexConnection = new Knex(this._NestKnexOptions);
    }
    return this._knexConnection;
  }
}

Almost all of this is unchanged from the generated boilerplate. We simply added a property to store our connection object (_knexConnection), and a getKnex() method to instantiate the object the first time it's requested, and then cache it for future use.

Test the Module

As mentioned, you'll need a local database instance available to test the module. If you don't have one handy, consider using docker to do so. It's easy!

The @nestjsplus/dyn-schematics dynpkg schematic built a small test client module for us (assuming you answered yes to the prompt). We can use this to quickly test our nest-knex module.

First, open src/nest-knex-client/nest-knex-client.module.ts and add the needed Knex.js options in the register() method. Adjust yours appropriately:

// src/nest-knex-client/nest-knex-client.module.ts
...
@Module({
  controllers: [NestKnexClientController],
  imports: [
    NestKnexModule.register({
      client: 'pg',
      connection: {
        host: 'localhost',
        user: 'john',
        password: 'mypassword',
        database: 'nest',
        port: 5432,
      },
    }),
  ],
})
...

Now, open src/nest-knex-client/nest-knex-client.controller.ts and plug in some queries. Of course this is not really a great design pattern (invoking database services directly from your controller), but is really just a quick test that the Knex.js integration works. In reality, you'll want to delegate any such access to true NestJS services as per NestJS best practices.

Here's an idea of what you can try. This test relies on the following database table being available (syntax below works with PostgreSql, but may require slight tweaks for other databases):

CREATE TABLE cats
(
   id     serial    NOT NULL,
   name   text,
   age    integer,
   breed  text
);

ALTER TABLE cats
   ADD CONSTRAINT cats_pkey
   PRIMARY KEY (id);

And here's a sample of what you could try in your test controller. Note that you have the full power of Knex.js available via the knex object. See here for lots more interesting samples of things you can do with Knex. Since you have a handle to the knex object, all of those Knex Query Builder methods should just work! In this admittedly silly example, we are both creating and querying some cats in our index route, using the knex object to access the database.

// src/nest-knex-client/nest-knex-client.controller.ts
import { Controller, Get } from '@nestjs/common';
import { NestKnexService } from '../nest-knex.service';

@Controller()
export class NestKnexClientController {
  constructor(private readonly nestKnexService: NestKnexService) {}

  @Get()
  async index() {
    const knex = this.nestKnexService.getKnex();
    const newcat = await knex('cats').insert({
      name: 'Fred',
      age: 5,
      breed: 'tom cat',
    });

    const cats = await knex.select('*').from('cats');

    return cats;
  }
}

Now, fire up the app with npm run start:dev, and browse to http://localhost:3000, and you should see the results of your query!

If you want to test the module with a separate app -- really demonstrating the power of the reusable library module you just built -- read on to see how to upload it as an npm package in one step. And for convenience, you can download a full client app that imports the module here and use it to test your newly minted Knex.js module. In fact, knex-cats is a fully working example, so if you've been reading along without coding, that's the easiest way to see the finished product.

Bonus Section

Hopefully I kept my promise of showing you how quickly you can generate a dynamic NestJS module that wraps an external resource library in just a few minutes! We're done -- you have a full-fledged Knex.js API at your disposal now! The queries we did in the previous section are just the tip of the iceberg, as Knex.js has a boat load of cool features.

Let's cover a couple more topics to round out the discussion.

A Better API

As mentioned, it's not good practice to access the knex object directly from our controller. We can make things a little bit better by creating a service to house our database logic. We can also make our Knex.js module a little easier to use by adding a higher level API. Let's do that. We're going to add a provider that let's us directly inject the knex object into any service. You'll see how this cleans the API up in a moment.

First, create a file called nest-knex-connection.provider.ts inside the src folder. Add the following code:

// src/nest-knex-connection.provider.ts
import { KNEX_CONNECTION } from './constants';
import { NestKnexService } from './nest-knex.service';

export const connectionFactory = {
  provide: KNEX_CONNECTION,
  useFactory: async nestKnexService => {
    return nestKnexService.getKnex();
  },
  inject: [NestKnexService],
};

To follow best practices for providers, we are using the KNEX_CONNECTION constant as our provider injection token. Be sure to add this line to src/constants.ts:

export const KNEX_CONNECTION = 'KNEX_CONNECTION';

See what we're doing here? We're binding an injection token (KNEX_CONNECTION) to a factory so that we can use our knex API object directly, avoiding the need to instantiate the service. For example, once we complete one more step (below), we'll be able to access the knex object in our controller as follows (notice the slightly simplified technique between old and new in the sample code):

// src/nest-knex-client/nest-knex-client.controller.ts
import { Controller, Inject, Get } from '@nestjs/common';
import { KNEX_CONNECTION } from '../constants';

@Controller()
export class NestKnexClientController {
  // new
  constructor(@Inject(KNEX_CONNECTION) private readonly knex) {}
  // old
  // constructor(private readonly nestKnexService: NestKnexService) {}

  @Get()
  async index() {
    // following line no longer needed
    // const knex = this.nestKnexService.getKnex();
    const newcat = await this.knex('cats').insert({
      name: 'Fred',
      age: 5,
      breed: 'tom cat',
    });

    const cats = await this.knex.select('*').from('cats');

    return cats;
  }
}

To wire this new provider into our module, open up src/nest-knex.module.ts and make the following changes:

1. Import the connectionFactory
2. Add the connectionFactory to the @Module metadata providers and exports properties. This section of the file will now look like this:

import { connectionFactory } from './nest-knex-connection.provider';

@Global()
@Module({
  providers: [NestKnexService, connectionFactory],
  exports: [NestKnexService, connectionFactory],
})

The complete implementation of our best practices for database access would also have us create a service that accesses the KNEX_CONNECTION provider, rather than doing that in a controller, but I'll leave that as an exercise for the reader. 😉

Dynamic Registration (with a Config Factory)

In the dynamic modules article, we talked about using asynchronous options providers. By that, we meant that rather than use a statically declared object containing our connection params like this:

    NestKnexModule.register({
      client: 'pg',
      connection: {
        host: 'localhost',
        user: 'john',
        password: 'mypassword',
        database: 'nest',
        port: 5432,
      },
    }),

We'd instead like to supply our connection options in a dynamic fashion. Good news! This is already built-in to the generated package. Let's test it out. For simplicity, we'll just use an in-place factory, but you can use the full power of class-based, factory based, and existing providers. To test it, we'll update the registration of the NestKnexModule in src/nest-knex-client/nest-knex-client.module.ts to look like this:

// src/nest-knex-client/nest-knex-client/module.ts
    NestKnexModule.registerAsync({
      useFactory: () => {
        return {
          debug: false,
          client: 'pg',
          connection: {
            host: 'localhost',
            user: 'john',
            password: 'mypassword',
            database: 'nest',
            port: 5432,
          },
        };
      },
    }),

It's not a terribly exciting example, but you get the idea. You can take advantage of any of the asynchronous provider methods to supply the configuration data to the module. If you check out the knex-cats sample, you'll see some more robust use of a config module to configure our Knex module dynamically.

Publish the Package

If you read my earlier Publishing NestJS Packages with npm article, you may have noticed that the structure of this package (e.g., the package.json file, the tsconfig.json file, the presence of the index.ts file in the root folder) follows that pattern exactly. As a result, publishing this package is as simple as this (assuming you have an npmjs.com account, that is):

npm publish

Seriously, that's it!

Of course, you can always use this ready-made @nestjsplus/knex package, built exactly as describe in this article.

Not Just External APIs!

The @nestjsplus/dyn-schematics package not only supports generating a standalone package, as described so far, but also generating a regular dynamic module inside an existing project. I won't cover that in detail here, but the use-case is simple. Let's say you're building a ConfigModule for use within a project. Simply scaffold it with the dynmod schematic (instead of dynpkg), and it will generate a new dynamic module into your existing project, fully ready for you to implement the details of your register() and registerAsync() static methods.

This works exactly like the normal CLI commands for adding elements like controllers, services, and modules. In fact, its behavior is very parallel to nest generate module. It builds the dynamic module and all its parts (which look quite similar to what we've seen in this article, minus the top level Nest application components) in a folder inside your project, and wires them up to the rest of the project just as the normal module schematic does. Read more about it at the github page for @nestjsplus/dyn-schematics. This use-case (adding a dynamic module to an existing project) is covered here. You can easily give it a quick dry run inside an existing Nest project with:

nest g -c @nestjsplus/dyn-schematics dynmod myNewModule --dry-run

Conclusion

If you've been following this series, you've previously learned a powerful architectural pattern for building modular NestJS services and modules, and composing them into applications. Now, with the power of schematics, you can automatically generate the boilerplate code to implement this powerful pattern. You can use this schematic to customize your own internal modules, or to easily integrate external APIs. In future articles, I'll show you how to build your own schematics to extend the power of NestJS and the NestJS CLI even further!

Feel free to ask questions, make comments or suggestions, or just say hello in the comments below. And join us at Discord for more happy discussions about NestJS. I post there as Y Prospect.

Comments

[Mike Taylor]

Great article. I'm using Typescript and it was giving me this error in the getKnex() function:

src/next-knex/nest-knex.service.ts:40:30 - error TS2350: Only a void function can be called with the 'new' keyword.

40       this._knexConnection = new Knex(this._NestKnexOptions);
                                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

did a bit of quick research and the fix is this:

      this._knexConnection = new (Knex as any)(this._NestKnexOptions);

Keep up the great work!

[umasankar-swain]

dev.to/umasankarswain/how-to-updat...

[Frédéric Woelffel]

Another great article @johnbiundo 👏 This series is really interesting. Thanks for you work 👍

I have a few questions that might sound dumb but I'm going to ask them anyway:

• I've noticed that the generated module is decorated as Global by default. Is that the way to go? Shouldn't we avoid global modules? Let's say I need a specific connection to a Postgres instance in module A and another connection to another instance in module B. If the NestKnexModule is global, won't that be an issue?
• Second question is simply about the naming of the register and registerAsync methods: I thought that global modules were supposed to expose forRoot (and its async variant) and that module-scoped modules were supposed to expose register (and its async variant). Have I been wrong this whole time? Or is it just a matter of personal preference?

Again, thanks for your work! I wish those schematics were released sooner as they would have saved me some time :)
Can't wait for your next post.

[John Biundo]

@fwoellfel, thanks! Glad it has been helpful!

Not dumb at all! I should probably discuss the use of @Global() in the article, so thanks for bringing it up. The main reason I use it is, to be honest, in a "utility" type module I find it convenient to register it once (usually in the root module), and then have it globally available. However you bring up an excellent point, and @Global() definitely has some side effects (such as breaking module encapsulation) and is usually to be avoided. I want to give this some more thought, and I have an idea or two on how to handle your Module A/Module B question, but need to test a few things out. I'll report back!

And your second question is good as well! To be honest, I hadn't quite sussed out that pattern of register vs. forRoot, but now that you mention it, I'll have to go back and look more closely.

Thanks so much for the thoughtful feedback. It really helps to have sharp eyes from people who have worked on dynamic modules. I hope to continue to improve these articles as a useful resource!

[umasankar-swain]

dev.to/umasankarswain/how-to-updat...

[Mike Taylor]

I am getting this error from the /.constants. Why is this throwing an error for a constant?

Nest can't resolve dependencies of the NestKnexService (?). Please make sure that the argument NEST_KNEX_OPTIONS at index [0] is available in the NestKnexService context.

Potential solutions:
- If NEST_KNEX_OPTIONS is a provider, is it part of the current NestKnexService?
- If NEST_KNEX_OPTIONS is exported from a separate @Module, is that module imported within NestKnexService?
  @Module({
    imports: [ /* the Module containing NEST_KNEX_OPTIONS */ ]
  })

[Ozan Ulutaş]

Hi. For the future readers;
Error complains about NestKnexService can not resolve injected dependency. In this case the dependancy is NEST_KNEX_OPTIONS which is a custom provider. To make the dependency injection work, the provider needs to be registered in the knex module. NEST_KNEX_OPTIONS provider is not included in this article. It is included in github link author provided.

import { connectionFactory } from './nest-knex-connection.provider';
import { createKnexProviders } from './knex.providers';

@Global()
@Module({
  providers: [
    KnexService,
    connectionFactory,
    ...createKnexProviders({/*...knex options*/}),
  ],
  exports: [KnexService, connectionFactory],
})
export class KnexModule {}

[John Biundo]

Hi Mike,

Can you provide some more context for this error? Do you have a repository that you can share?

Best,
John

[gates17]

hi, im new to nextjs and knex and im trying to build a custom api with both of these apps.
However, while following your guide, after running npm install im getting the following error
npm ERR! nest-knex@1.0.0 build: tsc
npm ERR! Exit status 2
which are related to jest-diff module.
Can u maybe point me out any directions on how to fix this and finish the current integration of knex with nestjs?
ty

[gates17]

apparently this error seems to be a conflict between @types/jest and typescript versions, and can be solved by updating both with jest v26.0.4 and typescript above v3.8 (v3.9.6 currently).
sry for the bother

[Shamilw]

Good afternoon, I am using your package github.com/nestjsplus/knex
and I wanted to wrap the user.service in a repository and I got errors

Potential solutions:

• If UserRepository is a provider, is it part of the current UserService?
• If UserRepository is exported from a separate @Module, is that module imported within UserService? @Module({ imports: [ /* the Module containing UserRepository */ ] }

[Pini Cheyni]

Have you tried newer version of knex ^1.0.3 ?
There are some breaking changes for me line there is no exported interface "Config".

[Monirul Islam]

is it possible to get knex debug data into knexmodule?

[Zyles]

Can you update this article? It seems out of date.

What is in constants.ts ?

And I get error 'knex_1.Knex is not a constructor'

Maybe a working git repo would be nice. Something is missing.

[Mário Monteiro]

another awesome post from John :)
follow the tutorial and works in current date only fails in bellow fix

import { Config } from 'knex';

with

import { Knex } from 'knex'

export interface NestKnexOptions extends Knex.Config {}

and one must update this dependencies

"devDependencies": {
  "@types/jest": "^26.0.24",
  "typescript": "^4.3.5"
}    

thanks John

[Promise Ihunna]

Can't seem to get past "nest g -c @nestjsplus/dyn-schematics dynpkg nest-knex".

[Arran Boyd]

Having the same issue, did you manage to get the package to work?

[Promise Ihunna]

I think I did, I will have to read through my code and find out. I'll get back to you

[umasankar-swain]

dev.to/umasankarswain/how-to-updat...