Redis Streams + NestJS: Part 1 | Setup

Intro

This is part 1 of a 3-part series, where we will explore how to use Redis streams with NestJS.

It is structured in 3 parts:

• Part 1 - Setting up the NestJS application and connecting to Redis
• Part 2 -Populating Redis streams and reading from in fan-out mode
• Part 3 - Using consumer groups to handle one stream from multiple actors in a way that one message is sent to and processed only by a single actor (consumer)

By the end of this series, you will have the knowledge and tools necessary to create your own NestJS app that utilizes Redis streams to handle real-time data.

Full code is available on the github

Assumed prior knowledge

Before diving into this post, it is assumed that you have a basic understanding of NestJS and Redis.
Familiarity with JavaScript generators is also recommended, as they will be used in the examples to listen to the Redis stream continuously.
We will also use Docker and Docker Compose to create and run our Redis server and application.
NestJS is a typescript framework, so you should be familiar with it too.
To familiarize yourself with these things, I recommend:

• NestJS docs
• TypeScript
• Redis:
  • Redis streams intro
  • The tutorial
• Generators:
  • JavaScript Generators reference
  • Dr. Axel Rauschmayer's book's "JavaScript for impatient programmers" sections on synchronous and asynchronous generators (available free online)
• Docker:
  • Docker
  • Docker Compose

Side note

It's also important to note that while this post explicitly covers the integration of Redis streams with NestJS, the concepts and techniques discussed here can also be applied to other frameworks and languages. Redis streams is a versatile tool that can be used in a wide range of applications and architectures. Therefore, even if you are not using NestJS, the general concepts and approaches outlined in this post can still apply to your projects and stack.

Setup

Creating a new NestJS application

To get started, we will need to create a new NestJS application. This can be done using the NestJS CLI, which can be installed via npm by running npm i -g @nestjs/cli. Once installed, you can create a new application by running nest new redis-streams, where "redis-streams" is the name of your application.
This will create a new directory with the same name, containing the basic structure of a NestJS application.

Docker setup

To simplify things, we are going to use docker-compose.yml file to include both our app and Redis server:

# docker-compose.yml

version: '3'

services:
  app:
    container_name: app
    image: node:18
    environment:
      HTTPS_METHOD: noredirect
    ports:
      - 8081:3000
    volumes:
      - ./:/usr/src/app/
    working_dir: /usr/src/app
    command: npm run start:dev

  redis:
    container_name: redis
    image: redis:7
    restart: always

Now you can run docker-compose up -d to build and run your services. To see console output, use docker-compose logs
You should see the following message:

 LOG [NestApplication] Nest application successfully started +7ms

Connecting to Redis

Setting up node-redis client library

To call Redis from we are going to use the official NodeJS Redis client library node-redis

$ npm i redis

There are also other libraries, e.g. ioredis is noteworthy alternative. You can see the list of clients on the Redis website

Redis module

Finally, we can start working with Redis from our application.
First, will create a new module for Redis-related services.

$ nest g module redis

You should see this:

// redis.module.ts

import { Module } from '@nestjs/common';

@Module({
  providers: [],
  exports: [],
})
export class RedisModule {}

RedisClient factory

To use RedisClient from node-redis library we are going to create a factory provider for it:

// redis-client.factory.ts

import { FactoryProvider } from '@nestjs/common';
import { createClient } from 'redis';
import { RedisClient, REDIS_CLIENT } from './redis-client.type';

export const redisClientFactory: FactoryProvider<Promise<RedisClient>> = {
  provide: REDIS_CLIENT,
  useFactory: async () => {
    const client = createClient({ url: 'redis://redis:6379/0' });
    await client.connect();
    return client;
  },
};

Let's break it down:
We create a FactoryProvider that will call async function provided in useFactory:

// redis-client.factory.ts
// --snip--
  useFactory: async () => {
    const client = createClient({ url: 'redis://redis:6379/0' });
    await client.connect();
    return client;
  },
// --snip--

1. We call createClient function from redis library, and pass it the URL consisting of {protocol}://{host}:{port}/{database}, where:
   • protocol= redis
   • host = redis - this is specified in docker-compose.yml with container_host: redis. Usually, you would create an environment variable with your Redis instance IP and use it here.
   • port = 6379 - default Redis port
   • database = 0 default database
2. We connect to the Redis server await client.connect();
3. Return the created & connected client.

You may have noticed that we did not provide an instance of RedisClient type but REDIS_CLIENT, which is our injection token. Also, RedisClient is our custom type, not redis.
This is due to node-redis on v4 not exporting the RedisClient type, so we need to create our own in /redis-client.type.ts file:

export type RedisClient = ReturnType<typeof createClient>;
export const REDIS_CLIENT = Symbol('REDIS_CLIENT');

All that is left is to add this to our module:

// redis.module.ts

//  --snip--
@Module({
  providers: [redisClientFactory],
})
export class RedisModule {}

Creating RedisService

Next, let's add a RedisService that will create a layer of abstraction from RedisClient.

$ nest g service redis

In there we will inject our RedisClient

// redis.service.ts

// --snip--
@Injectable()
export class RedisService implements OnModuleDestroy {
  public constructor(
    @Inject(REDIS_CLIENT) private readonly redis: RedisClient,
  ) {}

  onModuleDestroy() {
    this.redis.quit();
  }
}

Not to leave hanging connections, we are going to close the connection to the Redis by calling this.redis.quit() on OnModuleDestroy lifecycle event.

Ping for Redis server

To check that we have successfully connected to Redis, let's add an API endpoint that calls call Redis ping

// redis.service.ts

  ping() {
    return this.redis.ping();
  }

Let's export RedisService so that we can use it in other modules:

// redis.module.ts

// --snip--
  exports: [RedisService],
// --snip--

Now we will import it into AppService and pass through our call to ping redis:

// app.service.ts

@Injectable()
export class AppService {
// --snip--
  constructor(
    private readonly redisService: RedisService,
  ) {}

  redisPing() {
    return this.redisService.ping();
  }
// --snip--

Finally, we can add a new endpoint to AppController that will execute a ping to the Redis server and send the response to the user:

// app.controller.ts

// --snip--

@Controller()
export class AppController {
  constructor(private readonly appService: AppService) {}

  @Get('redis-ping')
  redisPing() {
    return this.appService.redisPing();
  }
}

Now, with our app and Redis server running, we can open our /redis-ping endpoint http://localhost:8081/redis-ping, and you should see the response:

Congratulations! We have finished part 1 of the 3-part series and created a NestJS application with a working connection to our Redis server! In part 2 we are going to create, populate and read from Redis streams.