DEV Community: Alfian Rivaldi

Easy way to process waterfall functions

Alfian Rivaldi — Wed, 03 Jul 2024 15:48:40 +0000

Have you ever encountered a process that flows like a waterfall? This is an example.

Even though we only hit 1 endpoint, we have to process all of them.

Yes, that's easy, but what if step 2 fails and we want everything to be repeated. But the data has already entered the database? Just delete it anyway, but it's lazy to handle it all.

Maybe the method I provide can make all of that easier, even if there are dozens of steps.

Disclaimer first, I implement this on nestjs. And I hope those of you who read already understand and are proficient in using nestjs.

We first prepare the base service, which contains the execute command to run all the steps in order.

import { Injectable } from '@nestjs/common';
import { FALLBACK_STEP_METADATA_KEY } from 'src/decorators/fallback.decorator';
import { STEP_METADATA_KEY } from 'src/decorators/step.decorator';
import { uid } from 'uid';

@Injectable()
export class WaterfallService {
  private steps: { methodName: string; order: number }[] = [];
  private fallbacks: { methodName: string; order: number }[] = [];

  constructor() {
    this.collectSteps();
  }

  private collectSteps() {
    const methods = Object.getOwnPropertyNames(Object.getPrototypeOf(this));
    methods.forEach((methodName) => {
      const order = Reflect.getMetadata(STEP_METADATA_KEY, this, methodName);
      if (order !== undefined) {
        this.steps.push({ methodName, order });
      }

      const fallbackOrder = Reflect.getMetadata(
        FALLBACK_STEP_METADATA_KEY,
        this,
        methodName,
      );
      if (fallbackOrder !== undefined) {
        this.fallbacks.push({ methodName, order: fallbackOrder });
      }
    });

    this.steps.sort((a, b) => a.order - b.order);
    this.fallbacks.sort((a, b) => a.order - b.order);
  }

  async executeSteps(params?) {
    const eventId = uid(6);
    let executedSteps = [];
    let returnedData: any;
    try {
      for (const step of this.steps) {
        let paramPassed = params;
        if (step.order > 1) {
          paramPassed = returnedData;
        }
        const result = await (this as any)[step.methodName](
          eventId,
          paramPassed,
        );
        returnedData = result;
        executedSteps.push(step);
      }
    } catch (error) {
      await this.executeFallbacks(executedSteps, eventId);
      throw error; // Re-throw the error after handling fallbacks
    }
  }

  private async executeFallbacks(executedSteps, eventId) {
    // Execute fallbacks in reverse order
    for (let i = executedSteps.length - 1; i >= 0; i--) {
      const step = executedSteps[i];
      const fallback = this.fallbacks.find((f) => f.order === step.order);
      if (fallback) {
        await (this as any)[fallback.methodName](eventId);
      }
    }
  }
}

Next, we will create a decorator to make it easier for the execute function to run the steps in the order we want.

import 'reflect-metadata';

export const STEP_METADATA_KEY = 'step_order';

export function Step(order: number): MethodDecorator {
  return (target, propertyKey, descriptor) => {
    Reflect.defineMetadata(STEP_METADATA_KEY, order, target, propertyKey);
  };
}

export const FALLBACK_STEP_METADATA_KEY = 'fallback_step_order';

export function Rollback(order: number): MethodDecorator {
  return (target, propertyKey, descriptor) => {
    Reflect.defineMetadata(FALLBACK_STEP_METADATA_KEY, order, target, propertyKey);
  };
}

The last one we implement into our service.

import { BadRequestException, Injectable } from '@nestjs/common';
import { WaterfallService } from './commons/waterfall/waterfall.service';
import { Step, Rollback } from './decorators/step.decorator';

@Injectable()
export class AppService extends WaterfallService {
  @Step(1)
  async logFirst(eventId) {
    console.log('Step 1 [eventId]:', eventId);
  }

  @Rollback(1)
  async fallbackFirst(eventId) {
    console.log('Rollback 1 [eventId]:', eventId);
  }

  @Step(2)
  async logSecond(eventId, data) {
    console.log('Step 2 [eventId]:', eventId);
  }

  @Rollback(2)
  async fallbackSecond(eventId) {
    console.log('Rollback 2 [eventId]:', eventId);
  }

  @Step(3)
  async logThird(eventId) {
    console.log('Step 3 [eventId]:', eventId);
  }

  @Rollback(3)
  async fallbackThird(eventId) {
    console.log('Rollback 3 [eventId]:', eventId);
  }

  async execute() {
    await this.executeSteps();
    return 'Step Executed';
  }
}

Now in the controller don't forget to call the execute function, like this example

import { Controller, Get } from '@nestjs/common';
import { AppService } from './app.service';

@Controller()
export class AppController {
  constructor(private readonly appService: AppService) {}

  @Get()
  getHello() {
    return this.appService.execute();
  }
}

Then run the nestjs. For the test hit [GET]http://localhost:3000

Later in the log terminal it will be like this

Now it's easy, right? You just add as many function steps as you need.

Now what if in step 3 there is an error and want to rollback the previous processes that have been run?
Take it easy, as long as there is a function with the decorator @Rollback(number)

The function will run if an error occurs. For example, here's an example.

import { BadRequestException, Injectable } from '@nestjs/common';
import { WaterfallService } from './commons/waterfall/waterfall.service';
import { Step, Rollback } from './decorators/step.decorator';

@Injectable()
export class AppService extends WaterfallService {
  @Step(1)
  async logFirst(eventId) {
    console.log('Step 1 [eventId]:', eventId);
  }

  @Rollback(1)
  async fallbackFirst(eventId) {
    console.log('Rollback 1 [eventId]:', eventId);
  }

  @Step(2)
  async logSecond(eventId, data) {
    console.log('Step 2 [eventId]:', eventId);
  }

  @Rollback(2)
  async fallbackSecond(eventId) {
    console.log('Rollback 2 [eventId]:', eventId);
  }

  @Step(3)
  async logThird(eventId) {
    throw new BadRequestException('Something error in step 3');
  }

  @Rollback(3)
  async fallbackThird(eventId) {
    console.log('Rollback 3 [eventId]:', eventId);
  }

  async execute() {
    await this.executeSteps();
    return 'Step Executed';
  }
}

Later, if it is run again, the results will be like this

I intentionally add eventId in each step, if there is a case you insert data into the database, also save the eventId to identify that the data has the eventId owner. So when you rollback and want to delete the data, you don't get confused about which data is being rolled back.

For the example case of data in the first function return that is passed to the next function, just return the function, then the return will be passed to the next function. This is the example

import { BadRequestException, Injectable } from '@nestjs/common';
import { WaterfallService } from './commons/waterfall/waterfall.service';
import { Step, Rollback } from './decorators/step.decorator';

@Injectable()
export class AppService extends WaterfallService {
  @Step(1)
  async logFirst(eventId, data) {
    console.log('Step 1 [eventId]:', eventId);
    console.log('Step 1 [data]:', data);

    return {
      step: 1,
      message: 'this data from step 1',
    };
  }

  @Rollback(1)
  async fallbackFirst(eventId) {
    console.log('Rollback 1 [eventId]:', eventId);
  }

  @Step(2)
  async logSecond(eventId, data) {
    console.log('Step 2 [eventId]:', eventId);
    console.log('Step 2 [data]:', data);

    return {
      step: 2,
      message: 'this data from step 2',
    };
  }

  @Rollback(2)
  async fallbackSecond(eventId) {
    console.log('Rollback 2 [eventId]:', eventId);
  }

  @Step(3)
  async logThird(eventId) {
    throw new BadRequestException('Something error in step 3');
  }

  @Rollback(3)
  async fallbackThird(eventId) {
    console.log('Rollback 3 [eventId]:', eventId);
  }

  async execute() {
    const data = { step: 0, message: 'this data from initial function' };
    await this.executeSteps(data);
    return 'Step Executed';
  }
}

Then the terminal will look like this

I added a repo example here.

That's all my method of running the waterfall function, Maybe if there are questions and collaboration, you can contact me.

Stop using one table for your client's

Alfian Rivaldi — Thu, 27 Jun 2024 05:31:49 +0000

Do you have a saas? And you have a lot of clients? I know you hate managing them, don't you?
Follow my way to manage it easily, no need to use difficult logic and make you frustrated.

All you need is nestjs, postgres. Yes, that's enough.
We will use multi tenancy method with postgres multi schema. Maybe if it is described it will be like this

Let's say you have 100 clients, and each of them has 100 data items. If you use the old method of 1 table for all clients then your table contains 10,000 jumbled data. You need to think about database performance, api performance, think about the logic to filter, add indexes to certain fields.

But if you use this method, you don't need to think about all that.

"Hey, if only talking is easy", oh yes take it easy, I have prepared an example of a repo, you just need to clone it and modify it. repo here.
"Hey then how do I use it?", well I will explain it from the beginning here.

Before that, I assume you are already proficient in using typescript.

You must clone the repo to your local.

This repo example uses the nestjs monorepo method, for those who don't know, maybe you can read the nestjs documentation.
In this example repo has 2 services, namely identity and inventory

Identity Service is very important, to store your client's identity data (Identity Service only has crud Users).
In addition, it is a service that will become multi tenancy in this example I created an Inventory Service (Inventory Service only has crud Items)

Install depedency using npm yes npm install
Create .env file using .env.example and fill it according to your database
Run the npm run migrate command, this function is for migrating the User table in the public schema
Run the 2 services using the commands npm run start:dev identity and npm run start:dev inventory
Create one user using the endpoint [POST]http://localhost:3000/users, save the created id for later.
To create a new schema automatically you can simply hit the [GET]http://localhost:3000/users/migrate endpoint.
Now you check, there will definitely be a new schema, right?
To use the endpoint on the Inventory service, just use the [GET]http://localhost:3001/items endpoint, but don't forget to use x-client=id user in the headers with the value of the id created earlier.
You can repeat step 6 onwards to create a new client schema.

There may be a few things to note to extend or modify this example repo.

If you want to add new migrations you can simply run the command npx mikro-orm migration:create --config ./apps/identity/mikro-orm.config.ts for service Identity and npx mikro-orm migration:create --config ./apps/inventory/mikro-orm.config.ts for service Inventory
Especially Inventory migrations you need to modify by following the first migration in the folder. Because the migration process on Inventory uses dyanmic schema and Mikroorm does not support it yet. So I modified it the way it is now.

I think that's enough. I think the rest you understand better. Maybe if there are questions and collaboration, you can contact me.