DEV Community: Josu Martinez

Lightweight Abstract Repository for NestJS

Josu Martinez — Wed, 19 Jul 2023 16:00:00 +0000

In this article, I will explain how to develop a sample application using NestJS and MongoDB, leveraging the functionalities provided by the monguito, a lightweight and type-safe library to seamlessly create custom database repositories for Node.js applications.

The sample application is a book manager that includes features such as creating, updating, deleting, and finding books. I will also provide step-by-step instructions for installing and running the application.

For those eager to get to the code of the sample application, you can find it at this Github repository. You may also find a further explanation on the monguito library at this other article.

The Domain Model

The application domain model is pretty simple: Book is a supertype that specifies two subclasses i.e., PaperBook and AudioBook. Here is its definition:

export class Book implements Entity {
  readonly id?: string;
  readonly title: string;
  readonly description: string;

  constructor(book: {
    id?: string;
    title: string;
    description: string;
  }) {
    this.id = book.id;
    this.title = book.title;
    this.description = book.description;
  }
}

export class PaperBook extends Book {
  readonly edition: number;

  constructor(paperBook: {
    id?: string;
    title: string;
    description: string;
    edition: number;
  }) {
    super(paperBook);
    this.edition = paperBook.edition;
  }
}

export class AudioBook extends Book {
  readonly hostingPlatforms: string[];

  constructor(audioBook: {
    id?: string;
    title: string;
    description: string;
    hostingPlatforms: string[];
  }) {
    super(audioBook);
    this.hostingPlatforms = audioBook.hostingPlatforms;
  }
}

Entity is an interface created to assist developers in the implementation of type-safe domain models. It specifies an id field that all Book or subclass instances must include. This is because id is assumed to be the primary key of any stored book. However, you do not need to implement Entity if you do not want to; simply make sure that your superclass includes an id field in its definition.

The Repository

Here a simple definition of a custom repository for books:

@Injectable()
export class MongooseBookRepository
  extends MongooseRepository<Book>
  implements Repository<Book>
{
  constructor(@InjectConnection() connection: Connection) {
    super(
      {
        Default: { type: Book, schema: BookSchema },
        PaperBook: { type: PaperBook, schema: PaperBookSchema },
        AudioBook: { type: AudioBook, schema: AudioBookSchema },
      },
      connection,
    );
  }
}

Repository is a Typescript generic interface that specifies some common CRUD database operations that can be executed on any persistent domain object. MongooseRepository, on another hand, is a Mongoose-based implementation of the interface that provides you with all the boilerplate code required to execute those operations. You can find further details on both Repository and MongooseRepository in this article. I would personally recommend you read the section Some Important Implementation Details to gain some required knowledge on the logic expected for your custom repository constructor, specially if your domain model is polymorphic i.e., if you want to store instances of a type and/or its subtypes in the same MongoDB collection.

You may also have some further questions regarding the Injectable and InjectConnection decorators. Please be patient, I will address them soon enough in this article 😉.

The Controller

The controller is the entry point of external requests to the book manager application. Here is how it looks like:

type PartialBook = { id: string } & Partial<Book>;

function deserialise<T extends Book>(plainBook: any): T {
  let book = null;
  if (plainBook.edition) {
    book = new PaperBook(plainBook);
  } else if (plainBook.hostingPlatforms) {
    book = new AudioBook(plainBook);
  } else {
    book = new Book(plainBook);
  }
  return book;
}

@Controller('books')
export class BookController {
  constructor(
    @Inject('BOOK_REPOSITORY')
    private readonly bookRepository: Repository<Book>,
  ) {}

  @Get()
  async findAll(): Promise<Book[]> {
    return this.bookRepository.findAll();
  }

  @Post()
  async insert(
    @Body({
      transform: (plainBook) => deserialise(plainBook),
    })
    book: Book,
  ): Promise<Book> {
    return this.save(book);
  }

  @Patch()
  async update(
    @Body()
    book: PartialBook,
  ): Promise<Book> {
    return this.save(book);
  }

  @Delete(':id')
  async deleteById(@Param('id') id: string): Promise<boolean> {
    return this.bookRepository.deleteById(id);
  }

  private async save(book: Book | PartialBook): Promise<Book> {
    try {
      return await this.bookRepository.save(book);
    } catch (error) {
      throw new BadRequestException(error);
    }
  }
}

You may argue several things here. For starters, you may think that an enterprise application should delegate business/domain logic to a layer of service objects as described in e.g., Domain-Driven Design (tactical design). I have decided not to do so for simplicity purposes; the book manager presented in this article is such an extremely simple CRUD application that introducing services would be over-engineering. I rather implement the minimum amount of code necessary for the sake of maximising the actual purpose of the article: illustrate how to integrate the monguito library on a NodeJS-based enterprise application.

Moreover, you would probably not write a deserialise function to enable the transformation of JSON request bodies into domain objects when dealing with POST requests. Instead, you would rather use a NestJS pipe to do so, thus properly complying with the Single Responsibility principle. Once again, I wanted to share the simplest possible working example at the expense of not conveying to the recommended practices in NestJS application construction. That being said, I would highly recommend you to read this section on how to use class-validator and class-transformer for the validation and deserialisation of JSON request bodies in the development of complex enterprise applications.

Okay! So all is missing now is how to tell the controller to use the custom book repository at runtime. This is where we are going next.

The Application Module

NestJS implements the Dependency Inversion principle; developers specify their component dependencies and NestJS uses its built-in dependency injector to inject those dependencies during component instantiation.

So, how do we specify the dependencies of the components that compose the book manager sample application? There are two easy steps that we need to take: The first step consists of writing some decorators in the MongooseBookRepository and BookController classes, as I already did in the code definition for both. The former class specifies that its instances are Injectable to other components. It also specifies that to instantiate a book repository, NestJS needs to inject a Mongoose connection. This is done with the InjectConnection decorator related to the connection constructor input parameter.

On another hand, the definition of BookController specifies that, during instantiation, the controller consumes an instance of a Repository<Book> defined by the BOOK_REPOSITORY custom token.

The definition of the custom token is part of the second step: writing the last required class: AppModule. The definition of this class is as follows:

@Module({
  imports: [
    MongooseModule.forRoot('mongodb://localhost:27016/book-repository'),
  ],
  providers: [
    {
      provide: 'BOOK_REPOSITORY',
      useClass: MongooseBookRepository,
    },
  ],
  controllers: [BookController],
})
export class AppModule {}

The Module decorator of this class specifies the Mongoose connection required to instantiate MongooseBookRepository at the imports property. It also determines that any component that has a dependency with the provider identified by the BOOK_REPOSITORY custom token is to get an instance of MongooseBookRepository. Finally, it determines that BookController is the sole controller for the book manager application.

Running the Application

Once you have implemented the book manager in your own or cloned the monguito GitHub project, and assuming that you have NestJS installed in your local machine, there are two further prerequisites to satisfy before you can run the application.

You must install all the project dependencies specified at the package.json file. To do so, simply execute the yarn install command from the root folder of the project. If you have cloned the library project, that folder is examples/nestjs-mongoose-book-manager.

The second prerequisite is installing Docker Desktop (in case you have not done it yet) and launching it in your local machine. I wrote a docker-compose.yml file to create a Docker container with an instance of the latest MongoDB version for your convenience, but you may create your own otherwise.

Then, all you need to do is run yarn start:dev. You will then be able to access all the endpoints specified at BookController via e.g., the curl command or Postman.

Conclusion

After following the several steps required to build an easy web application that integrates monguito, I hope that you are now able to leverage your knowledge and build your own. If you are willing to create a complex application, I also hope I has been able to provide you with a bit of guidance. Otherwise, please leave me a comment and I will get back to you ASAP 😊.

Cover image by Michael Sum on Unsplash.

MongoDB Abstract Repository for Node

Josu Martinez — Sun, 04 Jun 2023 16:54:17 +0000

monguito is a lightweight and type-safe library to seamlessly create custom database repositories for Node.js applications. It provides developers with some built-in CRUD operations, enabling them to focus in adding their own operations, and ensures a clean separation between persistence and domain logic. Despite being database technology agnostic by nature, it currently includes an implementation for MongoDB.

Note

This article introduces monguito, but it does not include all of its details. You may find the full documentation and some usage examples at this GitHub repository, including an example of how to use this library in a NestJS-based application.

Why Build Yet Another Repository Library

Joining ClimatePartner made me switch my gears and start to develop JS/TS-based web applications. My very first assignment was connecting a small backend NestJS app with a MongoDB database. We had big plans for our app and we soon realised that the domain model owned by my team was going to grow in complexity, and quickly. Furthermore, we saw that this was also the reality in other teams that wanted to use the same kind of solution.

All considered, we decided to incorporate a database access utility that implements the Repository pattern to cleanly decouple persistence from domain logic while keeping query logic duplication to the bare minimum. We also wanted it to be of abstract nature so that we could reuse the database access logic that is common to all custom repositories (i.e., CRUD operations over any persistent domain object). Finally, we needed it to implement the Polymorphic pattern to support subtyping-based domain models.

So we started researching the state-of-the-art and discovered several libraries that could fit our requirements. The best candidates we were able to find were Mongoose, Typegoose, and TypeORM. Mongoose is a well-known Node.js library for MongoDB that implements the Data Mapper pattern and lets developers define schemas to constraint the data models associated with their domain objects. However, Mongoose works with concrete data models, which in a complex domain model scenario results in query logic duplication. Typegoose is a type-safe Mongoose wrapper that allows schema constraint declaration at domain object field level via JS decorators. Unfortunately, those very decorators leak persistence logic into the domain model. Besides, Typegoose also implements the Data Mapper pattern, thus sharing the same drawbacks of Mongoose. TypeORM, on another hand, implements the Repository pattern and provides some basic support for MongoDB. However, TypeORM presents several limitations compared to Mongoose.

Although none of the existing alternatives were exactly fitting our needs, we saw value on Mongoose's solid, well-documented, and complete set of features. On the top of that, we wanted the most lightweight solution possible. That's why we decided to create monguito.

Cut to the Chase

Say we have a domain model that specifies Book as the supertype domain object and PaperBook and AudioBook as subtypes. Here is one possible definition for that domain model:

class Book implements Entity {
  readonly id?: string;
  readonly title: string;
  readonly description: string;
  readonly isbn: string;

  constructor(book: {
    id?: string;
    title: string;
    description: string;
    isbn: string;
  }) {
    this.id = book.id;
    this.title = book.title;
    this.description = book.description;
    this.isbn = book.isbn;
  }
}

class PaperBook extends Book {
  readonly edition: number;

  constructor(paperBook: {
    id?: string;
    title: string;
    description: string;
    isbn: string;
    edition: number;
  }) {
    super(paperBook);
    this.edition = paperBook.edition;
  }
}

class AudioBook extends Book {
  readonly hostingPlatforms: string[];
  readonly format?: string;

  constructor(audioBook: {
    id?: string;
    title: string;
    description: string;
    isbn: string;
    hostingPlatforms: string[];
  }) {
    super(audioBook);
    this.hostingPlatforms = audioBook.hostingPlatforms;
  }
}

Now, you want to be able to persist and retrieve instances of Book and any of its subtypes from a MongoDB database. MongooseBookRepository is a custom repository that specifies Book-related database operations. Here is its definition:

class MongooseBookRepository
  extends MongooseRepository<Book>
  implements BookRepository
{
  constructor() {
    super({
      Default: { type: Book, schema: BookSchema },
      PaperBook: { type: PaperBook, schema: PaperBookSchema },
      AudioBook: { type: AudioBook, schema: AudioBookSchema },
    });
  }

  async findByIsbn<T extends Book>(isbn: string): Promise<Optional<T>> {
    if (!isbn)
      throw new IllegalArgumentException('The given ISBN must be valid');
    return this.entityModel
      .findOne({ isbn: isbn })
      .exec()
      .then((book) => Optional.ofNullable(this.instantiateFrom(book) as T));
  }
}

Voilà! MongooseBookRepository inherits a series of CRUD database operations from MongooseRepository and adds its own i.e., findByIsbn. Now you can simply instantiate MongooseBookRepository and execute any of the database operations as follows:

const bookRepository = new MongooseBookRepository();
const books: Book[] = bookRepository.findAll();

No more leaking of persistence logic into your domain/application logic!

Some Important Implementation Details

You may be wondering about certain elements exposed in the previous example. Starting from the top part of the MongooseBookRepository definition, the first of these is MongooseRepository<Book>. This is one of the two main elements exposed at monguito: a generic template that implements several common CRUD database operations. Furthermore, this class also handles the creation of a Mongoose data model capable of handling subtyping-based book data objects. All this complexity is hidden from you; all you need to know is that you have entityModel at your disposal to implement your own database operations, as shown at findByIsbn.

The only extra effort required is that you define a map that specifies the types that compose your domain model and their related Mongoose schemas, as specified at the constructor of MongooseBookRepository. It is also important to note that the key of the entry that specifies your domain object supertype must be named Default, and the key of each other entry must be named after the type name of the subtype domain object it refers to. This is a naming convention required by the inner implementation details of MongooseRepository. If you do not have any domain object subtype in your domain model then simply instantiate a map with a unique entry that specifies a Default key and the type-schema values of your domain object.

Another element that may catch your eye is BookRepository. This is an optional interface that extends Repository, the second main element exposed at monguito: a database technology agnostic generic interface that specifies all the supported CRUD database operations to be implemented by any repository. The following snippet shows this relation between these two interfaces:

interface BookRepository extends Repository<Book> {
  findByIsbn: <T extends Book>(isbn: string) => Promise<Optional<T>>;
}

Handling instances of interface types instead of classes is a good practice in object-oriented applications; depending on abstractions instead of implementations makes your code scalable and better suited to change. Therefore, I would highly recommend that you define your own repository interfaces, or in the case you are only interested in exposing CRUD operations, instantiate your custom repositories as objects of type Repository<T>, where T is your domain object supertype.

On another hand, Book implements an interface called Entity. This interface models any persistent domain object type. The interface defines an optional id field. The optional nature of the field is due to the fact that its value is internally set by Mongoose. If you do not want your domain model to include this dependency, though, you can just ensure that your domain objects specify an id?: string field. And there is nothing wrong doing that, as id is the canonical primary key in MongoDB documents.

Finally, many database operations return objects of type Optional. This type is inspired in the Java Optional type. Its purpose is to model the concept of "no result". This type is an excellent alternative to returning null objects or throwing an exception. The usage of Optional is safer and clearer than having to specify T | null as result type or having to handle an exception that does not represent an application exceptional condition. So yes, this the feature that makes monguito a bit opinionated, but I believe that is worth it.

Conclusion

The monguito library provides the means to create custom repositories for Node.js-based applications in a fast and easy manner. It is conceived to achieve effective decoupling between persistence and domain logic. Its current implementation is a Mongoose wrapper that includes all the boilerplate code required to perform common CRUD operations over polymorphic domain objects in MongoDB. Thus, developers can focus in writing domain-specific database operations at custom repositories.

Moreover, the library has been designed to allow developers use many interesting Mongoose features (e.g., creation and validation of polymorphic models as well as hook implementations, just two mention two) without having to deal with most of Mongoose's inner complexity.

Cover image by Alev Takil on Unsplash.

Effective TDD

Josu Martinez — Mon, 05 Dec 2022 12:43:51 +0000

It has been six months since I started working at ClimatePartner building a new enterprise application. My new team follows some agile practices such as pair programming and Test-Driven Development (TDD), and I have the honest feeling that the sun is shining for us!

Well, almost.

There are some issues that I have been facing both now and in past industrial experiences on the topic of building solid enterprise applications that I would like to explain to you in this article.

Furthermore, I will also propose a simple test-first-based methodology to build enterprise applications that enhance team communication and promote faster high-quality code deliveries.

Without further ado, let's get to it!

Lights and Shadows of TDD

Agile practices are highly beneficial for the fast prototyping of software. TDD sits at the core of such practices, providing software with a paramount property: robustness. Writing tests upfront forces developers to think about the expected and exceptional behaviour of the software components they build, enhancing code quality and ensuring the fulfilment of functional requirements.

While TDD is no silver bullet as one must comply with good practices such as the SOLID principles, it is a powerful practice that enables developers not to be afraid to fix, clean, and adapt their code to functional requirement updates. And that is all great. However, it is not that easy to effectively apply TDD.

At the beginning of the construction of any new enterprise application, we (developers) gather several functional requirements. Then, we derive a set of use cases that will satisfy such functional requirements. After that, we develop software components sitting at different layers, from the highest to the lowest, drilling down to the heart of the application i.e., its domain model. That is the definition of the top-down software development process.

However, the bottom-up software development process fits better with TDD. Compared to the top-down alternative, bottom-up is a more pragmatic approach, since it enables us to incrementally cover all levels of indirection starting with the most basic (i.e., the domain model) and gradually moving towards higher layers of abstraction. It allows the production of sounder application code foundations, which in turn makes us gain great confidence in our work.

To apply TDD in a top-down approach, however, one should first write tests for software components located at the higher layers. Thus, to make tests pass, developers must mock any dependencies to any lower-layer components, since they simply do not exist just yet. The need to create such dependencies is already a problem because mocking lower layer components is not always possible or, in the best case scenario, feels counter-intuitive e.g., imagine yourself having to mock the logic of a domain object for the sake of producing a service component test.

Furthermore, I personally doubt that it would bring any value at all, as I think that intermediate component validation should always exercise all dependencies except those to external services (e.g., a database), which can be mocked. More importantly, due to the inherent complexity to realise non-trivial functional requirements as code, one does not fully understand the technical implications that some given functional requirements have on the domain model until one starts reasoning about them during the implementation of the domain model.

Start to write intermediate software components tests may thus not be a good idea, since many of those tests (if not all) are likely to be thrown to the dustbin once the lower layer software artefacts are actually implemented. Furthermore, I have seen software developers (especially junior teammates) giving up and implementing some proof of concept for the use case at hand without writing any validation logic whatsoever. This is using the code-first practice, which defeats the purpose of TDD. Also, without following proper Continuous Delivery practices, there is a high risk of ending up pushing non-validated code to our version control repository.

So, how could we effectively apply TDD in the production of enterprise applications given a set of functional requirements?

Hexagonal Architecture to the Rescue

There is plenty of literature on Hexagonal Architecture on the Internet. I would especially recommend reading the white paper on the topic written by Alistair Cockburn.

For the purpose of the present article, please allow me to tell you a real-life short story aimed to briefly explain the motivation and main benefits of Hexagonal Architecture: In the many years that I have been developing enterprise applications I have seen many people (myself included) starting new projects focusing on other topics rather than in our real mission. Such a mission consists of providing actual value to the companies we work for. That value is on the domain logic of our applications.

Paraphrasing Uncle Bob in his book Clean Architecture, everything else is a distraction, an implementation detail that can (and should) be postponed, ideally to the end of development. Examples of implementation details are database technologies, controller logic, or frontend technology. Even the backend framework is an implementation detail that we could pick later in the development process if we really wanted. Hexagonal Architecture, also called Ports and Adapters, is an architectural pattern aimed to decouple software application core logic from outer implementation details.

We developers should focus on the core logic of enterprise applications and postpone the implementation of the logic required to communicate with external services. To achieve that goal, all we really need to do is to write some interfaces (so-called ports) and mock the components (so-called adapters) that actually communicate with the external services. Thus, adapter implementation can come later in the development process. And the later they come, the better, since all the insights that we gain while we produce the core logic proves really useful during the decision-making of which technologies to pick.

Consider the elements that compose the inner hexagon. Apart from the domain model, there is also a layer of application services. These software components do not specify any domain logic. Instead, they are simple coordinators of adapter and domain model logic. An application service realises a use case that takes care of a subset of functional requirements for the enterprise application. This is important data to keep in mind for what comes next.

How to Effectively Use TDD in Hexagonal Architecture-based Enterprise Applications

As I stated earlier, applying TDD is easier when following the bottom-up software development process. However, many of us find it easier to reason about a system design following the top-down approach. And although it seems that we are incurring a conflict of interest, that is fine because we can start designing (i.e., sketching in pseudocode or some UML diagram) our application services top-down without writing a single line of code for them; not until we complete the implementation of the domain model.

Before we start coding, we could interpret application services as some software design guidelines to perform vertical slices of the enterprise applications we are to build. Each vertical slice represents the whole execution path of a use case, from the action performed by an upstream external service or a user in a UI to any operation executed on a downstream external service. By the time we finish with the design of an application service, we have identified which adapters and domain components we need to implement. Now that we have visibility on the domain model, we can next implement its fundamental components by applying TDD.

Then, we can implement the application service following the test-first approach, creating a port to any external service adapter and mocking its actual implementation. By the time we are done with the implementation of the application service and the related domain model, we can ascertain that such implementation is valid i.e., likely to be bug-free and matching its functional requirements. Finally, we can implement the adapter logic, also applying TDD.

This methodology enables fast and gradual implementation of enterprise applications, allowing developers to gain confidence in the validity of all the components they create without throwing any test away. Moreover, it does not impose any restrictions on functional requirement updates.

This diagram depicts the methodology to write the logic of one use case. Notice that I do not talk about specific test types since this is a pretty controversial subject, although I would recommend following the conventions and terminology used in The Practical Test Pyramid article.

The proposed methodology simplifies team task distribution, no matter whether they work alone or in pairs. Some of its steps can be done in parallel e.g., a teammate/pair can build the core logic mocking any reference to an external dependency, whereas another can be working on the development of the integration code with such external dependency, since these two parts are decoupled, thus shortening delivery time. All they need to do is convey some API for the external dependency realised as a port. Both the mock and the actual external dependency need to implement the aforementioned port interface. Similarly, another team/teammate/pair can implement the frontend part for the use case, if that is what the enterprise application demands.

On another hand, due to its structured nature, communicating the state of development of a concrete use case among peers is easy. When handing over a use case development, the departing entity can simply point the new one to the current application service they were designing or implementing. The new entity then can simply trace anything that has been already created on adapter or domain logic in the code base.

A final note on implementation details: We could update our domain model to specify some of those details by writing annotations/decorators of the database technology and input data validation resources from the underlying framework that adjusts best to the nature of our application. I would however advise against it, as that would be leaking implementation details into our domain model, which is not the best practice since implementation details and the domain model tend to change for different reasons and frequency. On another hand, as explained by Vaughn Vernon in his book Implementing Domain Driven Design (DDD), Hexagonal Architecture is closely related to DDD. However, you do not need to follow the set of DDD practices and patterns to build complex enterprise applications that are based on Hexagonal Architecture, although I would highly recommend you do. But those decisions, after all, are entirely up to you.

Conclusion

Test-Driven Development is a powerful practice in the construction of robust enterprise applications. TDD is best applied following the bottom-up software development process. However, because developers are used to reasoning about such applications following a top-down approach, applying it in practice is challenging. This article exposes a simple methodology to help developers effectively apply TDD in the development of enterprise applications.

Cover photo by Aedrian on Unsplash.