DEV Community: Juan Castillo

Building a Webhook Systems with NestJS: Handling Retry, Security, Dead-letter Queues, and Rate Limiting ✨

Juan Castillo — Fri, 04 Apr 2025 19:49:02 +0000

Webhooks are a powerful way to enable real-time communication between services. But what happens if a webhook fails? Or if a malicious user floods your endpoint with spam? In this guide, we’ll build a resilient webhook system in NestJS that includes:

Webhook sender: Securely signing and sending webhooks.
Webhook receiver: Verifying, processing, and retrying failed webhooks.
Dead-letter queues (DLQ): Handling persistent failures gracefully.
Rate limiting: Preventing spam and abuse.

1. Setting Up the Webhook Sender 🔄

The sender application is responsible for dispatching events to the webhook receiver. To ensure security, we sign each payload before sending it.

Signing and Sending Webhooks

import * as crypto from 'crypto';
import axios from 'axios';

// Replace with your secret key, use environment variables in production
const SECRET = 'your-secret-key'; 

function signPayload(payload: any): string {
  return crypto.createHmac('sha256', SECRET)
    .update(JSON.stringify(payload))
    .digest('hex');
}

async function sendWebhook(url: string, event: any) {
  const signature = signPayload(event);

  await axios.post(url, event, {
    headers: {
      'X-Signature': signature,
    },
  });
}

// Example usage:
sendWebhook('http://webhook-receiver.com/events', { event: 'USER_CREATED', data: { id: 123 } });

2. Implementing the Webhook Receiver 🏠

The receiver application verifies the signature and processes incoming webhooks.

Verifying Webhook Signatures

import { Controller, Post, Headers, Body, HttpException, HttpStatus } from '@nestjs/common';
import * as crypto from 'crypto';

const SECRET = 'your-secret-key';

function verifySignature(payload: any, signature: string): boolean {
  const expectedSignature = crypto.createHmac('sha256', SECRET)
    .update(JSON.stringify(payload))
    .digest('hex');
  return expectedSignature === signature;
}

@Controller('webhooks')
export class WebhookController {
  @Post()
  async handleWebhook(@Headers('X-Signature') signature: string, @Body() payload: any) {
    if (!verifySignature(payload, signature)) {
      throw new HttpException('Invalid signature', HttpStatus.UNAUTHORIZED);
    }

    console.log('Received webhook:', payload);
    // Process webhook...
  }
}

3. Implementing Retry Logic with Bull 📥

If webhook processing fails, we should retry it using BullMQ, a Redis-based queue system.

Installing Dependencies:

npm install @nestjs/bull bull redis

Configuring Bull in `app.module.ts`

import { BullModule } from '@nestjs/bull';
import { Module } from '@nestjs/common';

@Module({
  imports: [
    BullModule.forRoot({
      redis: {
        host: 'localhost',
        port: 6379,
      },
    }),
    BullModule.registerQueue({ name: 'webhook-queue' }),
  ],
})
export class AppModule {}

Webhook Retry Queue Processor

import { Process, Processor } from '@nestjs/bull';
import { Job } from 'bull';
import axios from 'axios';

@Processor('webhook-queue')
export class WebhookProcessor {
  @Process('retry')
  async handleFailedWebhook(job: Job) {
    const { payload, url } = job.data;
    try {
      await axios.post(url, payload);
    } catch (error) {
      throw new Error('Webhook retry failed');
    }
  }
}

Adding Failed Webhooks to the Queue

import { InjectQueue } from '@nestjs/bull';
import { Queue } from 'bull';
import { Controller, Post, Body } from '@nestjs/common';

@Controller('webhooks')
export class WebhookController {
  constructor(@InjectQueue('webhook-queue') private webhookQueue: Queue) {}

  @Post()
  async handleWebhook(@Body() payload: any) {
    try {
      console.log('Processing webhook:', payload);
      // Process webhook normally...
    } catch (error) {
      console.error('Failed webhook, adding to queue');
      await this.webhookQueue.add('retry', { payload, url: 'http://webhook-receiver.com/events' }, { attempts: 3 });
    }
  }
}

4. Implementing Dead-Letter Queues (DLQ) ☠️

If a webhook fails after multiple retries, it should be moved to a Dead-letter queue for manual intervention.
This is useful for logging and debugging, and allows you to handle persistent failures gracefully.

@Processor('webhook-queue')
export class WebhookProcessor {
  @Process('retry')
  async retryWebhook(job: Job) {
    try {
      await axios.post(job.data.url, job.data.payload);
    } catch (error) {
      if (job.attemptsMade >= job.opts.attempts) {
        console.error('Moving webhook to DLQ:', job.data);
        await this.deadLetterQueue.add(job.data);
      }
      throw error;
    }
  }
}

5. Implementing Rate Limiting ⏳

To prevent spam attacks, we’ll use NestJS Throttler in our webhook receiver.
This will limit the number of requests a user can make in a given time frame.

Installing Dependencies:

npm install @nestjs/throttler

Configuring Rate Limiting

import { Module } from '@nestjs/common';
import { ThrottlerModule } from '@nestjs/throttler';

@Module({
  imports: [
    ThrottlerModule.forRoot({
      ttl: 60, // 1 minute
      limit: 5, // Max 5 requests per minute per user
    }),
  ],
})
export class AppModule {}

Applying Rate Limiting to the Webhook Controller

import { Throttle } from '@nestjs/throttler';
import { Controller, Post, Body } from '@nestjs/common';

@Controller('webhooks')
export class WebhookController {
  @Throttle(5, 60)
  @Post()
  async handleWebhook(@Body() payload: any) {
    console.log('Processing webhook:', payload);
  }
}

Conclusion 🎯

By implementing secure webhooks with retries, dead-letter queues, and rate limiting, we ensure our system is reliable, fault-tolerant, and secure. 🚀

Security: Signed payloads prevent unauthorized requests.
Retries: Failed webhooks are reprocessed automatically.
Dead-letter queues: Webhooks that repeatedly fail are logged for manual intervention.
Rate limiting: Protects against spam and DoS attacks.

Got questions or improvements? Drop them in the comments! 💬 Happy coding! 🔥

🚀 Handling Cron Jobs in NestJS with Multiple Instances using Bull

Juan Castillo — Sun, 23 Mar 2025 05:13:32 +0000

A Complete Guide with Workers, Rate Limiting, Retries & Dead-Letter Queues (DLQs) 🚀🔥

When your NestJS application grows, running cron jobs becomes… tricky.
Running multiple instances means your cron jobs will run multiple times, which is usually very bad 😅.

To avoid duplicate jobs and chaos, we’ll build a robust distributed cron system using:

NestJS
BullMQ (modern queue system)
Redis
Worker processes
Retry logic
Dead-letter queues (DLQs)
Rate-limiting

And all wrapped in a multi-instance setup using Docker Compose.
Let’s go. 🛠️✨

🧱 Architecture Overview

We will build two separate NestJS applications:

                        ┌────────────────────┐
                        │  Scheduler Service │  ← runs cron jobs
                        │  (NestJS App #1)   │
                        └──────────┬─────────┘
                                   │  pushes jobs
                                   ▼
                            ┌─────────────┐
                            │   BullMQ    │
                            │  (Redis)    │
                            └──────┬──────┘
                                   │  workers pull jobs
                                   ▼
                        ┌────────────────────┐
                        │   Worker Service   │  ← processes jobs
                        │  (NestJS App #2)   │
                        └────────────────────┘

Only the Scheduler runs cron jobs
Only the Worker processes jobs
Both communicate through Redis
All instances stay synchronized
No duplicated tasks 🎉

🧰 Step 1 — Install Redis

The easiest method is Docker:

docker run -d --name redis -p 6379:6379 redis:7

📦 Step 2 — Install Dependencies

Install BullMQ & NestJS integration:

npm i bullmq @nestjs/bullmq
npm i -D @types/bull

Install scheduling support:

npm i @nestjs/schedule

(Optional) install Bull Dashboard:

npm i @bull-board/api @bull-board/express

🔌 Step 3 — Create a Shared Redis Config

Create src/bull-config.ts:

export const redisConfig = {
  connection: {
    host: process.env.REDIS_HOST ?? 'localhost',
    port: Number(process.env.REDIS_PORT ?? 6379),
  },
};

You’ll reuse this in both apps.

🕒 Step 4 — Build the Scheduler App

(The app that runs cron jobs and pushes them into queues)

Enable ScheduleModule:

// scheduler/app.module.ts
@Module({
  imports: [
    ScheduleModule.forRoot(),
    BullModule.forRoot(redisConfig),
    BullModule.registerQueue({
      name: 'email_queue',
    }),
  ],
})
export class AppModule {}

Create a Cron Job

// scheduler/email-cron.service.ts
import { Cron, CronExpression } from '@nestjs/schedule';
import { InjectQueue } from '@nestjs/bullmq';
import { Queue } from 'bullmq';

@Injectable()
export class EmailCronService {
  constructor(@InjectQueue('email_queue') private emailQueue: Queue) {}

  @Cron(CronExpression.EVERY_10_SECONDS)
  async handleCron() {
    console.log('⏰ Adding job to email queue...');

    await this.emailQueue.add(
      'send-welcome-email',
      { userId: Math.floor(Math.random() * 1000) },
      {
        attempts: 3,
        backoff: { type: 'exponential', delay: 3000 },
        removeOnComplete: true,
        removeOnFail: false,
        rateLimiter: {
          max: 5,
          duration: 1000,
        },
      }
    );
  }
}

What we added:

🔄 Retries (3 attempts)
⏳ Exponential backoff
🚦 Rate-limiting (5 jobs per second)
🧹 Cleanup completed jobs

🛠️ Step 5 — Build the Worker App

(This app consumes & executes queue jobs)

// worker/app.module.ts
@Module({
  imports: [
    BullModule.forRoot(redisConfig),
    BullModule.registerQueue({
      name: 'email_queue',
    }),
  ],
  providers: [EmailProcessor],
})
export class AppModule {}

Create the Job Processor

// worker/email.processor.ts
import { Processor, WorkerHost } from '@nestjs/bullmq';
import { Job } from 'bullmq';

@Processor('email_queue')
export class EmailProcessor extends WorkerHost {
  async process(job: Job<any>): Promise<any> {
    try {
      console.log(`📨 Processing email job for User ${job.data.userId}`);

      if (Math.random() < 0.3) {
        throw new Error('Simulated email failure!');
      }

      return { success: true };
    } catch (err) {
      console.error('❌ Job failed:', err);
      throw err;
    }
  }
}

💀 Step 6 — Dead-Letter Queue (DLQ)

Create a secondary queue called email_queue:dlq.

Modify worker:

@Processor('email_queue')
export class EmailProcessor extends WorkerHost {
  async onFailed(job: Job, error: any) {
    const dlq = new Queue('email_dlq', redisConfig);

    await dlq.add('dlq-job', {
      failedJob: job.id,
      data: job.data,
      error: error.message,
    });

    console.log(`💀 Moved job ${job.id} to DLQ`);
  }
}

Now any job that fails after max retries goes into the dead-letter queue,
ready for manual review or reprocessing.

🧱 Step 7 — Dockerize Everything

Here is the full working docker-compose.yml:

version: "3.9"

services:
  redis:
    image: redis:7
    ports:
      - "6379:6379"

  scheduler:
    build: ./scheduler
    depends_on:
      - redis
    environment:
      REDIS_HOST: redis
    deploy:
      replicas: 1

  worker:
    build: ./worker
    depends_on:
      - redis
    environment:
      REDIS_HOST: redis
    deploy:
      replicas: 3

What this achieves:

Scheduler has 1 replica → cron runs once
Worker can have as many replicas as needed → job processing scales horizontally

📊 Optional: Add BullMQ Dashboard

Inside the worker app:

import { createBullBoard } from '@bull-board/api';
import { BullMQAdapter } from '@bull-board/api/bullMQAdapter';
import { ExpressAdapter } from '@bull-board/express';

@Module({})
export class DashboardModule implements OnModuleInit {
  onModuleInit() {
    const serverAdapter = new ExpressAdapter();
    const emailQueue = new Queue('email_queue', redisConfig);

    createBullBoard({
      queues: [new BullMQAdapter(emailQueue)],
      serverAdapter,
    });

    serverAdapter.setBasePath('/dashboard');
    app.use('/dashboard', serverAdapter.getRouter());
  }
}

Open dashboard:

http://localhost:3000/dashboard

🎉 Final Result: A Rock-Solid Cron Job System

You now have:

✅ A Scheduler app that runs cron jobs
✅ A Worker app that processes tasks
❌ No duplicated cron executions
🚦 Rate-limiting
🔁 Retry logic with exponential backoff
💀 Dead-letter queues
🐳 Horizontal scaling with Docker
🎛️ Optional admin dashboard

All production-ready.
All beautifully decoupled.
All powered by Redis & BullMQ.

Cursor-Based Pagination in NestJS with TypeORM 🚀

Juan Castillo — Thu, 13 Mar 2025 19:25:03 +0000

Hey there, devs! 👋 If you've ever struggled with paginating large datasets efficiently, you're in the right place. Today, we'll implement cursor-based pagination in a NestJS API using TypeORM. This approach is far superior to offset-based pagination when dealing with large databases. Let's dive in! 🏊‍♂️

What We'll Cover 🔥

Using a createdAt cursor to fetch records efficiently.
Implementing a paginated endpoint in NestJS.
Returning data with a cursor for the next page.

1️⃣ Creating a DTO for Pagination Parameters

First, let's define a DTO to handle pagination parameters:

import { IsOptional, IsString, IsNumber } from 'class-validator';
import { Transform } from 'class-transformer';

export class CursorPaginationDto {
  @IsOptional()
  @IsString()
  cursor?: string; // Receives the `createdAt` of the last item on the previous page

  @IsOptional()
  @Transform(({ value }) => parseInt(value, 10))
  @IsNumber()
  limit?: number = 10; // Number of items per page (default: 10)
}

2️⃣ Implementing the Query in the Service

Now, let's create the logic in our service:

import { Injectable } from '@nestjs/common';
import { InjectRepository } from '@nestjs/typeorm';
import { Repository } from 'typeorm';
import { User } from './user.entity';
import { CursorPaginationDto } from './dto/cursor-pagination.dto';

@Injectable()
export class UserService {
  constructor(
    @InjectRepository(User)
    private readonly userRepository: Repository<User>,
  ) {}

  async getUsers(cursorPaginationDto: CursorPaginationDto) {
    const { cursor, limit } = cursorPaginationDto;

    const queryBuilder = this.userRepository
      .createQueryBuilder('user')
      .orderBy('user.createdAt', 'DESC')
      .limit(limit + 1); // Fetching one extra record to check if there's a next page

    if (cursor) {
      queryBuilder.where('user.createdAt < :cursor', { cursor });
    }

    const users = await queryBuilder.getMany();

    const hasNextPage = users.length > limit;
    if (hasNextPage) {
      users.pop(); // Remove the extra item
    }

    const nextCursor = hasNextPage ? users[users.length - 1].createdAt : null;

    return {
      data: users,
      nextCursor,
    };
  }
}

3️⃣ Creating the Controller

Finally, let's expose our paginated endpoint:

import { Controller, Get, Query } from '@nestjs/common';
import { UserService } from './user.service';
import { CursorPaginationDto } from './dto/cursor-pagination.dto';

@Controller('users')
export class UserController {
  constructor(private readonly userService: UserService) {}

  @Get()
  async getUsers(@Query() cursorPaginationDto: CursorPaginationDto) {
    return this.userService.getUsers(cursorPaginationDto);
  }
}

4️⃣ Defining the Database Model

Here's our User entity:

import { Entity, Column, PrimaryGeneratedColumn, CreateDateColumn } from 'typeorm';

@Entity()
export class User {
  @PrimaryGeneratedColumn('uuid')
  id: string;

  @Column()
  name: string;

  @CreateDateColumn()
  createdAt: Date;
}

How Cursor-Based Pagination Works ⚡

1️⃣ The first request to GET /users does not include a cursor. It fetches the first limit records.

2️⃣ The backend returns a nextCursor, which is the createdAt timestamp of the last user in the response.

3️⃣ To fetch the next page, the frontend makes a request to GET /users?cursor=2024-03-09T12:34:56.000Z, and the backend will return users created before that timestamp.

4️⃣ This process continues until nextCursor is null, meaning there are no more records left.

Example JSON Response 📝

{
  "data": [
    { "id": "1", "name": "John", "createdAt": "2024-03-09T12:00:00.000Z" },
    { "id": "2", "name": "Anna", "createdAt": "2024-03-09T11:45:00.000Z" }
  ],
  "nextCursor": "2024-03-09T11:45:00.000Z"
}

Why Use Cursor-Based Pagination? 🤔

✅ Better Performance: Avoids OFFSET, which slows down large datasets.

✅ Scalability: Works seamlessly with millions of records.

✅ Optimized Queries: Using indexed fields like createdAt makes queries lightning-fast. ⚡

Conclusion 🎯

Cursor-based pagination is a game-changer for handling large datasets in APIs. 🚀 It's faster, more efficient, and ensures a smoother experience for your users. Now you’re ready to implement it in your own NestJS project! 💪

Got questions or improvements? Drop them in the comments! 💬 Happy coding! 😃

Implementing Rate Limiting in NestJS with Redis for Scalable Applications 🚀

Juan Castillo — Tue, 04 Mar 2025 01:53:47 +0000

Introduction

Rate limiting is essential to prevent abuse and ensure fair usage of your API. When running multiple instances of your NestJS application, using Redis as a centralized store allows rate limits to be shared across all instances. In this guide, we'll implement rate limiting in NestJS using Redis.

Why Use Redis for Rate Limiting?

When deploying your NestJS application in a distributed environment (multiple instances behind a load balancer), storing rate limit counters in memory won't work consistently. Instead, using Redis provides:

Shared state across all instances 🏢
High performance for quick lookups ⚡
Persistence even after app restarts 🔄

Setting Up Redis and Dependencies

First, install Redis and the required dependencies:

npm install redis ioredis rate-limiter-flexible

If you haven't installed Redis yet, you can run it using Docker:

docker run -d --name redis -p 6379:6379 redis

Implementing Rate Limiting in NestJS

1. Create a `RateLimitMiddleware`

Create a new middleware file:

mkdir src/middleware && touch src/middleware/rate-limit.middleware.ts

Now, implement rate limiting logic using rate-limiter-flexible:

import { Injectable, NestMiddleware, HttpException, HttpStatus } from '@nestjs/common';
import { Request, Response, NextFunction } from 'express';
import { RateLimiterRedis } from 'rate-limiter-flexible';
import Redis from 'ioredis';

const redisClient = new Redis({
  host: 'localhost', // Change if running Redis in a different host
  port: 6379,
  enableOfflineQueue: false,
});

const rateLimiter = new RateLimiterRedis({
  storeClient: redisClient,
  keyPrefix: 'middleware',
  points: 10, // 10 requests
  duration: 60, // per 60 seconds
});

@Injectable()
export class RateLimitMiddleware implements NestMiddleware {
  async use(req: Request, res: Response, next: NextFunction) {
    try {
      await rateLimiter.consume(req.ip); // Identify clients by IP
      next();
    } catch (rejRes) {
      throw new HttpException('Too Many Requests', HttpStatus.TOO_MANY_REQUESTS);
    }
  }
}

2. Apply Middleware Globally

Modify AppModule to register the middleware globally:

import { Module, MiddlewareConsumer, NestModule } from '@nestjs/common';
import { RateLimitMiddleware } from './middleware/rate-limit.middleware';

@Module({})
export class AppModule implements NestModule {
  configure(consumer: MiddlewareConsumer) {
    consumer.apply(RateLimitMiddleware).forRoutes('*');
  }
}

3. Running and Testing 🛠️

Start Redis and your NestJS app:

docker start redis  # If using Docker
npm run start

Then, send multiple requests to test rate limiting:

curl -X GET http://localhost:3000

After 10 requests within 60 seconds, you should get:

{"statusCode":429,"message":"Too Many Requests"}

Conclusion

With Redis and rate-limiter-flexible, we implemented a scalable rate-limiting solution for a distributed NestJS application. This setup ensures consistent request limits across multiple instances, preventing abuse while maintaining high performance. 🚀🔥

Happy coding! 🎉

🚀 Performance Testing: Types and Implementation with k6

Juan Castillo — Thu, 06 Feb 2025 15:13:04 +0000

🎯 Introduction

Performance testing is crucial to ensure your app can handle traffic like a champ! Whether you're preparing for Black Friday or just want to avoid angry users, different types of tests help identify bottlenecks and improve system stability. In this article, we’ll explore four key performance tests and show you how to implement them using k6 (an open source tool and cloud service that makes load testing easy for developers and QA engineers).

🐳 Setting Up k6 with Docker

Want to automate and visualize your test results? Set up k6 with Docker, InfluxDB, and Grafana! 📊

src
 |_scripts/
 |_docker-compose.yml
 |_grafana-dashboard.yaml
 |_grafana-datasource.yaml

🛠️ Docker Compose Configuration

networks:
  k6:
  grafana:

services:
  influxdb:
    image: influxdb:1.8
#    entrypoint: /bin/sh
#    user: root
    networks:
      - k6
      - grafana
    ports:
      - "8086:8086"
    environment:
      - INFLUXDB_DB=k6

  grafana:
    image: grafana/grafana:8.5.21
#    entrypoint: /bin/sh
#    user: root
    networks:
      - grafana
    ports:
      - "3000:3000"
    environment:
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_BASIC_ENABLED=false
      - GF_SERVER_SERVE_FROM_SUB_PATH=true
    volumes:
      - ./dashboards:/var/lib/grafana/dashboards
      - ./grafana-dashboard.yaml:/etc/grafana/provisioning/dashboards/dashboard.yaml
      - ./grafana-datasource.yaml:/etc/grafana/provisioning/datasources/datasource.yaml

  k6:
    image: grafana/k6:latest
#    entrypoint: /bin/sh
#    user: root
    networks:
      - k6
    ports:
      - "6565:6565"
    environment:
      - K6_OUT=influxdb=http://influxdb:8086/k6
    volumes:
      - ./scripts:/scripts

🛠️ Grafana configuration

## grafana-dashboard.yaml
apiVersion: 1
providers:
  - name: 'default'       
    org_id: 1             
    folder: ''            
    type: 'file'          
    options:
      path: /var/lib/grafana/dashboards

## grafana-datasource.yaml
apiVersion: 1

datasources:
  - name: k6influxdb
    type: influxdb
    access: proxy
    database: k6
    url: http://influxdb:8086
    isDefault: true

With this setup, you can store k6 test results in InfluxDB and analyze them in Grafana like a pro! 📈

Run InfluxDb and Grafana

Run the command

docker-compose up influxdb grafana -d

After that you can access to the Grafana dashboard in localhost:3000 also you can change the port in the docker-compose file.

🤔 What Does Performance Mean for an API?

Before diving into testing, let's define what "performance" means for your API. Ask yourself:

What will the API be used for?
How much traffic per instance is acceptable?
What are the maximum acceptable response times?

Answering these questions helps set benchmarks for successful testing. ✅

🏋️ Types of Performance Tests

1️⃣ Load Testing

Goal: Measure system behavior under normal and maximum expected loads.

Use Case: Find out how many users your system can handle before things start breaking. 😨

Example Implementation:

import http from 'k6/http';
import { check, sleep } from "k6";

const isNumeric = (value) => /^\d+$/.test(value);
const default_vus = 5;
const target_vus_env = `${__ENV.TARGET_VUS}`;
const target_vus = isNumeric(target_vus_env) ? Number(target_vus_env) : default_vus;

export let options = {
  stages: [
      { duration: "5s", target: target_vus },
      { duration: "10s", target: target_vus },
      { duration: "5s", target: 0 }
  ],
  thresholds: {
        http_req_failed: ['rate<0.01'], // 99% of request should be success
        http_req_duration: ['p(95)<500'], // 95% of response time < 500ms 
    },
};

export default function () {
  const response = http.get("https://swapi.dev/api/people/30/");
  check(response, { "status is 200": (r) => r.status === 200 });
  sleep(.300);
}

2️⃣ Stress Testing

Goal: Find out when your system will break under extreme conditions. 🏗️🔥

Use Case: Push your API to the limit and see when it starts crying for help. 😭

Example Implementation:

import http from 'k6/http';
import { sleep, check } from 'k6';

export const options = {
    stages: [
        { duration: '1m', target: 100 },
        { duration: '2m', target: 100 },
        { duration: '1m', target: 200 },
        { duration: '2m', target: 200 },
        { duration: '1m', target: 500 },
        { duration: '2m', target: 500 },
        { duration: '1m', target: 0 },
    ],
};

export default function () {
    const res = http.get("https://swapi.dev/api/people/30/");
    check(res, { 'status was 200': (r) => r.status === 200 });
    sleep(1);
}

3️⃣ Soak Testing

Goal: Check long-term system stability under constant load. 🕰️

Use Case: Find memory leaks or slow performance over time. 🐢

Example Implementation:

import http from 'k6/http';
import { sleep, check } from 'k6';

export const options = {
    stages: [
        { duration: '1m', target: 200 },
        { duration: '4h', target: 200 },
        { duration: '1m', target: 0 },
    ],
};

export default function () {
    const res = http.get("https://swapi.dev/api/people/30/");
    check(res, { 'status was 200': (r) => r.status === 200 });
    sleep(1);
}

4️⃣ Spike Testing

Goal: Test how your system handles sudden bursts of traffic. 🚀

Use Case: Simulate a flash sale and see if your API survives. 🏷️💥

Example Implementation:

import http from 'k6/http';
import { sleep, check } from 'k6';

export const options = {
    stages: [
        { duration: '30s', target: 100 },
        { duration: '1m', target: 2000 },
        { duration: '10s', target: 2000 },
        { duration: '1m', target: 100 },
        { duration: '30s', target: 0 },
    ],
};

export default function () {
    const res = http.get("https://swapi.dev/api/people/30/");
    check(res, { 'status was 200': (r) => r.status === 200 });
    sleep(1);
}

🏃 Run test

Now we can run any of the tests, for this example each Javascript file must be inside the "scripts" folder in the root of the project

Windows

docker-compose run k6 run //scripts//load.js

Linux / Mac

docker-compose run k6 run /scripts/load.js

🎬 Conclusion

Performance testing is your secret weapon for keeping users happy and preventing downtime. By running Load, Stress, Soak, and Spike tests with k6 and leveraging Docker for monitoring, you’ll ensure your system stays fast and reliable. 🚀 Happy Coding!

How to Log with Loki and Grafana Using NestJS 🚀

Juan Castillo — Thu, 23 Jan 2025 20:14:15 +0000

Introduction

Have you ever wondered how to keep a detailed log of what's happening in your NestJS application? Logs are essential for monitoring and debugging applications, and with Loki and Grafana, we can take our logging to the next level. In this tutorial, you'll learn how to integrate Loki with NestJS to log and visualize messages efficiently. Let's get started! 📝✨

Prerequisites

Before we begin, make sure you have the following:

Basic knowledge of NestJS.
Node.js and npm installed on your machine.
A Grafana account and Loki set up.

Step 1: Setting Up the NestJS Project

First, let's create a new NestJS project. Open your terminal and run the following command:

npx @nestjs/cli new logging-project
cd logging-project

Step 2: Installing Dependencies

We need to install axiosto send HTTP requests to Loki. Run the following command:

npm install axios

Step 3: Creating the Logging Service

We'll create a logging service in NestJS that will send logs to Loki. Create a new file named logging.service.ts in the src/logging/ directory and add the following code:

import { Injectable, ConsoleLogger } from '@nestjs/common';
import axios from 'axios';

@Injectable()
export class LokiLogger extends ConsoleLogger {
  private static lokiUrl = process.env.LOKI_URL;
  private static lokiToken = process.env.LOKI_TOKEN;
  private static defaultLabels: any = {
    app: process.env.APP_NAME || 'not-set',
    env: process.env.NODE_ENV || 'development',
  };
  private static gzip = false;
  private static onLokiError: (error: any) => void = () => {};

  private static sendLokiRequest = (
    labels: Record<string, string>,
    message: string,
  ): any => {
    const data = JSON.stringify({
      streams: [
        {
          stream: labels,
          values: [[(Date.now() * 1000000).toString(), message]],
        },
      ],
    });

    axios({
      method: 'POST',
      url: `${LokiLogger.lokiUrl}/loki/api/v1/push`,
      headers: LokiLogger.gzip
        ? {
            'Content-Type': 'application/json',
            'Content-Encoding': 'application/gzip',
          }
        : {
            'Content-Type': 'application/json',
            Authorization: `Bearer ${LokiLogger.lokiToken}`,
          },
      data: data,
    })
      .then()
      .catch((error) => {
        if (LokiLogger.onLokiError) {
          LokiLogger.onLokiError(error);
        } else {
          console.error('error', error.message, error?.response?.data);
        }
      });
  };

  error(
    message: string,
    trace?: string,
    context?: string,
    labels?: Record<string, string>,
  ): void {
    LokiLogger.sendLokiRequest(
      {
        ...LokiLogger.defaultLabels,
        ...labels,
        context: context ?? this.context,
        level: 'error',
      },
      message,
    );
    super.error(message, trace, context);
  }

  log(
    message: string,
    context?: string,
    labels?: Record<string, string>,
  ): void {
    LokiLogger.sendLokiRequest(
      {
        ...LokiLogger.defaultLabels,
        ...labels,
        context: context ?? this.context,
        level: 'info',
      },
      message,
    );
    super.log(message, context);
  }

  warn(
    message: string,
    context?: string,
    labels?: Record<string, string>,
  ): void {
    LokiLogger.sendLokiRequest(
      {
        ...LokiLogger.defaultLabels,
        ...labels,
        context: this.context,
        level: 'warn',
      },
      message,
    );
    super.warn(message, context);
  }

  debug(
    message: string,
    context?: string,
    labels?: Record<string, string>,
  ): void {
    LokiLogger.sendLokiRequest(
      {
        ...LokiLogger.defaultLabels,
        ...labels,
        context: this.context,
        level: 'debug',
      },
      message,
    );
    super.debug(message, context);
  }

  verbose(
    message: string,
    context?: string,
    labels?: Record<string, string>,
  ): void {
    LokiLogger.sendLokiRequest(
      {
        ...LokiLogger.defaultLabels,
        ...labels,
        context: this.context,
        level: 'verbose',
      },
      message,
    );
    super.verbose(message, context);
  }
}

Step 4: Using the Custom Logger in main.ts

Now, let's use this service as a custom logger in the main.ts file:

import { NestFactory } from '@nestjs/core';
import { AppModule } from './app.module';
import { LokiLogger } from './logging/logging.service';

async function bootstrap() {
  const app = await NestFactory.create(AppModule, {
    logger: new LokiLogger(),
  });
  await app.listen(3000);
}
bootstrap();

Step 5: Using the Logger in Your Controllers

Since we have set up LokiLoggeras a custom logger, we can use it directly in our controllers without needing to instantiate it manually. For example, in app.controller.ts:

import { Controller, Get, Logger } from '@nestjs/common';

@Controller()
export class AppController {
  private readonly logger = new Logger(AppController.name);

  @Get()
  getHello(): string {
    this.logger.log('This is an info message');
    this.logger.warn('This is a warning');
    this.logger.error('This is an error');
    return 'Hello World!';
  }
}

Step 6: Viewing Logs in Grafana

With Loki receiving our logs, it's time to view them in Grafana:

Log in to your Grafana instance.
Create a new dashboard and add a panel.
Configure a query to view the logs from your NestJS application, filtering by the labels defined.

Conclusion

And that's it! We have set up a simple but effective logging system using NestJS and Loki, with visualization in Grafana. Now, our applications can tell their story, and we can ensure everything is running smoothly. Happy logging! 🚀📊

References

NestJS Documentation
Loki Documentation

Understanding JavaScript's ?? (Nullish Coalescing) and || (Logical OR) Operators 🤔💡

Juan Castillo — Mon, 20 Jan 2025 00:53:58 +0000

Introduction

In JavaScript, the ?? (nullish coalescing) and || (logical OR) operators are commonly used to provide default values when a variable is "falsy" or null/undefined. Although they might seem similar, there's a key difference in how they determine when to provide the default value. Let's dive into each operator and explore their differences with some examples! 🛠️📚

The ?? (Nullish Coalescing) Operator

The ?? operator returns the right-hand side value only if the left-hand side value is null or undefined. This is particularly useful when you want to provide a default value specifically when a variable is null or undefined.

let a = null;
let b = a ?? "default"; // "default"

The || (Logical OR) Operator

The || operator returns the right-hand side value if the left-hand side value is "falsy". In JavaScript, "falsy" values include false, 0, "" (empty string), null, undefined, and NaN.

let a = 0;
let b = a || "default"; // "default"

Key Differences

1. Falsy Values Considered:

|| considers false, 0, "", null, undefined, and NaNas "falsy" values.
?? considers only null and undefined as "nullish" values.

2. Common Uses:

Use ?? when you want to specifically handle null and undefined and allow other "falsy" values like 0 or "".
Use || when you want a default value for any "falsy" value.

Examples

1. Using ??:

let user = {
    name: "Alice",
    age: 0,
    country: null
};

let age = user.age ?? 18;         // 0, because 0 is not null or undefined
let country = user.country ?? "Unknown"; // "Unknown", because it is null

2. Using ||:

let user = {
    name: "Alice",
    age: 0,
    country: null
};

let age = user.age || 18;         // 18, because 0 is "falsy"
let country = user.country || "Unknown"; // "Unknown", because it is null

Summary

??: Use this operator when you only care about null or undefined.
||: Use this operator when you want to cover all "falsy" values.

Understanding when to use each of these operators will help you handle default values more precisely and avoid logical errors in your code. Happy coding! 🎉✨

Centralizing Query and Mutation Configurations with TanStack Query + React 😎

Juan Castillo — Fri, 17 Jan 2025 03:09:45 +0000

TanStack Query is a powerful tool for managing server state in React applications. However, configuring queries and mutations repeatedly in a project can lead to code duplication and inconsistency. In this tutorial, we’ll centralize the configuration of queries and mutations using two custom hooks: useApiGet and useApiSend. 🚀

These hooks will:

Provide consistent retry logic.
Handle success and error scenarios.
Allow query invalidation post-mutation.
Minimize boilerplate across your project.

Why Centralize Query and Mutation Configurations?

Centralizing these configurations ensures:

Reusability: Use the same logic for all queries and mutations.
Consistency: Avoid discrepancies in retry mechanisms or error handling.
Maintainability: Update logic in one place and propagate it across your application.

The Code

Here’s the implementation of the useApiGet and useApiSend hooks:

1. Custom Query Hook: useApiGet

This hook simplifies useQuery with retry logic and default options.

import { isAxiosError } from 'axios';
import { useQuery, QueryKey, QueryFunction } from '@tanstack/react-query';

const HTTP_STATUS_TO_NOT_RETRY = [400, 401, 403, 404, 500, 501, 502, 503, 504, 505, 506, 507, 508, 510, 511];
const MAX_RETRY = 3;

type ApiGetProps = {
  key: QueryKey;
  fn: Promise<any> | QueryFunction<unknown, QueryKey, never> | undefined;
  options?: any;
};

const useApiGet = <T>({ key, fn, options }: ApiGetProps) =>
  useQuery<T>({
    queryKey: key,
    queryFn: fn,
    refetchOnWindowFocus: false,
    retry: (failureCount, error: any) => {
      if (failureCount >= MAX_RETRY) return false;
      if (isAxiosError(error) && HTTP_STATUS_TO_NOT_RETRY.includes(error.response?.status ?? 0)) return false;
      return true;
    },
    ...options,
  });

Key Features:

Retry Mechanism: Retries up to MAX_RETRY times for network issues.
Avoids Retry on Certain Status Codes: Stops retrying for non-network errors like 404 or 500.
Customizable Options: Allows overriding default behaviors via the options prop.

2. Custom Mutation Hook: useApiSend

This hook builds on useMutation and includes features like query invalidation and callbacks for success/error handling.

import { useMutation, useQueryClient, InvalidateQueryFilters } from '@tanstack/react-query';
import { isAxiosError } from 'axios';

type ApiSendProps<T> = {
  fn: ((data: T) => Promise<any>) | (() => Promise<any>);
  success?: (data: any) => void;
  error?: (error: any) => void;
  invalidateKey?: InvalidateQueryFilters[] | undefined;
  options?: any;
};

const useApiSend = <T>({ fn, success, error, invalidateKey, options }: ApiSendProps<T>) => {
  const queryClient = useQueryClient();

  return useMutation<unknown, Error, T>({
    mutationFn: fn,
    onSuccess: (data) => {
      if (invalidateKey) {
        invalidateKey.forEach((key) => queryClient.invalidateQueries(key));
      }
      if (success) success(data);
    },
    onError: error,
    retry: (failureCount, error: any) => {
      if (failureCount > 2) return false;
      if (isAxiosError(error) && HTTP_STATUS_TO_NOT_RETRY.includes(error.response?.status ?? 0)) return false;
      return true;
    },
    ...options,
  });
};

Key Features:

Query Invalidation: Automatically refreshes specific queries when a mutation succeeds.
Custom Callbacks: Executes success or error callbacks for further handling.
Retry Logic: Similar to useApiGet, retries only on network-related errors.

How to Use These Hooks

Fetching Data with useApiGet
Use this hook to fetch data consistently across your application.

import { useApiGet } from './apiHooks';

const UserProfile = () => {
  const { data, isLoading, error } = useApiGet({
    key: ['user', 'profile'],
    fn: axios.get('/api/profile'),
  });

  if (isLoading) return <p>Loading...</p>;
  if (error) return <p>Error loading profile</p>;

  return <div>{data?.name}</div>;
};

Sending Data with useApiSend

Use this hook to send data while managing invalidation and success/error callbacks.

import { useApiSend } from './apiHooks';

const UpdateProfile = () => {
  const { mutate, isLoading } = useApiSend({
    fn: (data) => axios.post('/api/profile', data),
    success: () => alert('Profile updated successfully!'),
    error: (err) => console.error('Failed to update profile:', err),
    invalidateKey: [['user', 'profile']],
  });

  const handleSubmit = () => {
    mutate({ name: 'New Name' });
  };

  return (
    <button onClick={handleSubmit} disabled={isLoading}>
      Update Profile
    </button>
  );
};

Conclusion

Centralizing the configuration of queries and mutations using TanStack Query is a game-changer for managing server state in React applications. With useApiGet and useApiSend, you can streamline logic, improve consistency, and reduce code duplication.

Start implementing these hooks today, and let TanStack Query simplify your state management journey!. Happy coding! 🎉✨

DEV Community: Juan Castillo

Building a Webhook Systems with NestJS: Handling Retry, Security, Dead-letter Queues, and Rate Limiting ✨

1. Setting Up the Webhook Sender 🔄

Signing and Sending Webhooks

2. Implementing the Webhook Receiver 🏠

Verifying Webhook Signatures

3. Implementing Retry Logic with Bull 📥

Installing Dependencies:

Configuring Bull in app.module.ts

Webhook Retry Queue Processor

Adding Failed Webhooks to the Queue

4. Implementing Dead-Letter Queues (DLQ) ☠️

5. Implementing Rate Limiting ⏳

Installing Dependencies:

Configuring Rate Limiting

Applying Rate Limiting to the Webhook Controller

Conclusion 🎯

🚀 Handling Cron Jobs in NestJS with Multiple Instances using Bull

A Complete Guide with Workers, Rate Limiting, Retries & Dead-Letter Queues (DLQs) 🚀🔥

🧱 Architecture Overview

🧰 Step 1 — Install Redis

📦 Step 2 — Install Dependencies

🔌 Step 3 — Create a Shared Redis Config

🕒 Step 4 — Build the Scheduler App

Enable ScheduleModule:

Create a Cron Job

🛠️ Step 5 — Build the Worker App

Create the Job Processor

💀 Step 6 — Dead-Letter Queue (DLQ)

🧱 Step 7 — Dockerize Everything

📊 Optional: Add BullMQ Dashboard

🎉 Final Result: A Rock-Solid Cron Job System

Cursor-Based Pagination in NestJS with TypeORM 🚀

What We'll Cover 🔥

1️⃣ Creating a DTO for Pagination Parameters

2️⃣ Implementing the Query in the Service

3️⃣ Creating the Controller

4️⃣ Defining the Database Model

How Cursor-Based Pagination Works ⚡

Example JSON Response 📝

Why Use Cursor-Based Pagination? 🤔

Conclusion 🎯

Implementing Rate Limiting in NestJS with Redis for Scalable Applications 🚀

Introduction

Why Use Redis for Rate Limiting?

Setting Up Redis and Dependencies

Implementing Rate Limiting in NestJS

1. Create a RateLimitMiddleware

2. Apply Middleware Globally

3. Running and Testing 🛠️

Conclusion

🚀 Performance Testing: Types and Implementation with k6

🎯 Introduction

🐳 Setting Up k6 with Docker

🛠️ Docker Compose Configuration

🛠️ Grafana configuration

Run InfluxDb and Grafana

🤔 What Does Performance Mean for an API?

🏋️ Types of Performance Tests

1️⃣ Load Testing

2️⃣ Stress Testing

3️⃣ Soak Testing

4️⃣ Spike Testing

🏃 Run test

Windows

Linux / Mac

How to Log with Loki and Grafana Using NestJS 🚀

Understanding JavaScript's ?? (Nullish Coalescing) and || (Logical OR) Operators 🤔💡

Key Differences

1. Falsy Values Considered:

2. Common Uses:

1. Using ??:

2. Using ||:

Centralizing Query and Mutation Configurations with TanStack Query + React 😎

Configuring Bull in `app.module.ts`

1. Create a `RateLimitMiddleware`