DEV Community: satyajit nayak

Ensure Idempotency in concurrent POST request with different request id's

satyajit nayak — Sat, 28 Dec 2024 06:05:53 +0000

In a payment processing system, a client may accidentally send the same payment request multiple times due to network retries or user error. Each request includes a unique request_id, but the actual payment details (e.g., amount, recipient, and transaction metadata) might be identical. How can we ensure the system avoids processing such duplicate requests, even if they come with different request_id values, while still maintaining consistency and idempotency?

Anyone with practical experience please write your thoughts...

I attempted to ensure idempotency for concurrent POST requests by generating unique request IDs for each client request. However, I noticed that identical requests with different request IDs were still being processed multiple times, leading to duplicate entries in the database.

I expected the system to detect duplicate requests based on the content of the payload (e.g., identical payment details) and process only one of them, regardless of the request ID, to ensure no duplicate resources or transactions were created.

Search & Replace Texts in DOCX

satyajit nayak — Sun, 09 Jun 2024 10:53:00 +0000

Hey guys, I have created a package name edit-office-file which can search & replace multiple text strings inside a DOCX file as well other office files.

First install the pkg: npm i edit-office-files.
Usage style:


import {SearchAndReplace} from 'edit-office-files';

async function main() {
  const searchTexts = ['Hello World', 'are You', 'coloured'];
  const replacementTexts = ['REPLACEMENT1', 'REPLACEMENT2', 'REPLACEMENT3'];
  const reader = new SearchAndReplace('assets/document.docx', searchTexts, replacementTexts, 'updated.docx');
  await reader.process();
}

main();

Here is the github link for the project.

understand priorities of queues in NodeJS

satyajit nayak — Fri, 07 Jun 2024 05:10:30 +0000

Guess the output!!

console.log('1');

setImmediate(() => {
  console.log('3');
});

setTimeout(() => {
  console.log('4');
}, 0);

process.nextTick(() => {
  console.log('2');
});

OUTPUT:

Guess How?? (Try to run the code here.)

Output depends on the priorities of different queues involves in the execution.

Priority of [ process.nextTick() > Timer Queue (setTimeout) > Check Queue (setImmediate) ]

Run Kafka Locally using Docker

satyajit nayak — Sat, 23 Dec 2023 13:23:53 +0000

Create a file with name docker-compose.yml

version: "3.7"
services:
  zookeeper:
    restart: always
    image: docker.io/bitnami/zookeeper:3.8
    ports:
      - "2181:2181"
    volumes:
      - "zookeeper-volume:/bitnami"
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
  kafka:
    restart: always
    image: docker.io/bitnami/kafka:3.3
    ports:
      - "9093:9093"
    volumes:
      - "kafka-volume:/bitnami"
    environment:
      - KAFKA_BROKER_ID=1
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CLIENT:PLAINTEXT,EXTERNAL:PLAINTEXT
      - KAFKA_CFG_LISTENERS=CLIENT://:9092,EXTERNAL://:9093
      - KAFKA_CFG_ADVERTISED_LISTENERS=CLIENT://kafka:9092,EXTERNAL://localhost:9093
      - KAFKA_CFG_INTER_BROKER_LISTENER_NAME=CLIENT
    depends_on:
      - zookeeper
volumes:
  kafka-volume:
  zookeeper-volume:

Command to run kafka inside docker

docker-compose build // run this first time only
docker-compose up

Create a file to test if kafka working locally - name the file check_kafka.js

const { Kafka } = require('kafkajs');

// Define your Kafka broker(s)
const kafkaBrokers = ['localhost:9093'];

// Create a Kafka instance
const kafka = new Kafka({
  clientId: 'kafka-test-client',
  brokers: kafkaBrokers,
});

// Create a Kafka producer
const producer = kafka.producer();

// Create a Kafka consumer
const consumer = kafka.consumer({ groupId: 'kafka-test-group' });

// Function to produce a test message
async function produceMessage() {
  await producer.connect();
  await producer.send({
    topic: 'test-topic',
    messages: [{ value: 'Hello, Kafka!' }],
  });
  await producer.disconnect();
}

// Function to consume messages
async function consumeMessages() {
  await consumer.connect();
  await consumer.subscribe({ topic: 'test-topic', fromBeginning: true });

  await consumer.run({
    eachMessage: async ({ message }) => {
      console.log(`Received message: ${message.value}`);
    },
  });

  // Uncomment the following line if you want to stop the consumer after a certain period
  // setTimeout(async () => await consumer.disconnect(), 5000);
}

// Run the test
async function runTest() {
  try {
    await produceMessage();
    await consumeMessages();
  } catch (error) {
    console.error('Error:', error);
  } finally {
    await producer.disconnect();
  }
}

// Run the test
runTest();

connnect with kafka container (to check if it's runnig)

# to find containers with name
docker ps --format '{{.Names}}'

docker exec -it setup-kafka-kafka-1 bin/bash #setup-kafka-kafka-1 is my kafka container name

# find kafka ip
docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' <container_id_or_name>

# to check if kafka working in machine
node check_kafka.js

On running kafka using docker you will get output something like this

If kafka working fine then you will get output something like this

For detail you can find more in my github repo

Random Data Generator Website

satyajit nayak — Wed, 08 Nov 2023 02:10:16 +0000

Random Data Generator Website

Are you a developer looking for a quick and easy way to generate random data for testing and development purposes? Look no further! Our Random Data Generator Website is designed to help you effortlessly create random data based on predefined schemas. Whether you need sample data for your application or just want to experiment with different data structures, this tool has got you covered.

Features

Our Random Data Generator Website comes packed with a variety of features to make data generation a breeze:

Define Custom Schemas: Tailor your data generation needs by defining custom schemas. You can specify the data fields you want, along with their types.
Data Types Supported: As of now, our generator supports three common data types:
- String: For text data.
- Number: For numerical data.
- Generic: For other miscellaneous data.
Instant JSON Output: The generated data is presented in a clean and easily readable JSON format, making it convenient for use in your projects.
User-Friendly Interface: Our website features a user-friendly interface that allows you to create schemas and generate data with ease.
Real-Time Data Generation: See the data being generated in real-time as you define your schema. This instant feedback can be incredibly helpful in fine-tuning your data requirements.
Copy Generated Content: Need to quickly grab the generated data? Just click on the right-hand side (RHS) of the generated content, and it's automatically copied to your clipboard.
JSON Repair: If you encounter broken JSON data, our tool can help you fix it, ensuring that your data is valid and ready for use.

Schema Definition

The heart of our Random Data Generator is the schema definition. You can specify the data fields you need and their respective types. The current schema structure looks like this:

[
  {
    "name": "full_name",
    "type": "string"
  },
  {
    "name": "age",
    "type": "number"
  },
  {
    "name": "rating",
    "type": "generic"
  }
]

In this example, we've defined three fields: "full_name" (string), "age" (number), and "rating" (generic). You can customize this schema to suit your specific needs.

Example Output

Once you've defined your schema and clicked the generate button, you'll receive data in the following format:

{
  "full_name": "Innovatech Ltd",
  "age": 25,
  "rating": "4.7"
}

The generated data adheres to the schema you've defined, making it suitable for testing, prototyping, or any other purpose you may have in mind.

It can also fix your broken JSON on the go & generate mock data:

INPUT

[
  {
    name: "full_name",
    "type": string"
  },
  {
    "name": age",
    "type": number"
  },
  {
    "name": rating",
    type: "generic"
  }
]

OUTPUT

{
  "full_name": "Innovatech Ltd",
  "age": 25,
  "rating": "4.7"
}

How to Use

Getting started with our Random Data Generator Website is a straightforward process:

Load the Website: Visit our website (provide the URL) to access the data generator tool.
Enter the Desired Schema: Use the provided form to enter the schema for your data. Specify the field names and their data types.
Generate Data: Click the "Generate" button to see the data being generated based on your schema. It will appear on your screen in JSON format.
Copy Generated Content: To quickly copy the generated data for use in your projects, just click on the right-hand side (RHS) of the data, and it will be automatically copied to your clipboard.

With our Random Data Generator Website, you can save time and effort by generating data that matches your requirements in an instant. Whether you're a developer, a tester, or anyone in need of sample data, this tool is here to simplify your data generation process.
Give it a try and experience the convenience of creating random data with ease. Happy generating! Github Link

Natural Cron ⏰⏰ - Create cron expressions with human readable APIs

satyajit nayak — Tue, 07 Nov 2023 03:03:57 +0000

Natural Cron ⏰⏰

Natural Cron is an easy-to-use Node.js library that simplifies the creation and validation of cron expressions with human-readable APIs. It's designed to make working with cron jobs in your Node.js applications more intuitive and efficient. In this article, we'll explore the features of Natural Cron and provide examples of how you can use it to create custom cron expressions for various scheduling scenarios.

Features

Natural Cron offers a range of features to help you work with cron expressions effectively:

Chainable Methods: Constructing complex cron expressions is made easy with chainable methods that allow you to specify minute, hour, day of the month, month, and day of the week.
Validation: Natural Cron provides validation for each component of a cron expression, ensuring that your expressions are valid. If you provide invalid input, it will generate helpful error messages.
TypeScript Support: For those who value strong typing and intelligent code completion, Natural Cron offers TypeScript support.

Now, let's dive into how you can install and use Natural Cron in your Node.js projects.

Installation

You can install Natural Cron using npm or yarn. Here's how to do it:

Using npm:

npm install natural-cron

Using yarn:

yarn add natural-cron

Usage

Natural Cron provides an intuitive API for creating custom cron expressions. Let's explore some common scheduling scenarios and how you can use the library to generate the corresponding cron expressions.

const { CronExpressionBuilder, CronValidators } = require('natural-cron');
const schedule = new CronExpressionBuilder();

Examples

Here are several examples of scheduling scenarios and the corresponding Natural Cron expressions:

Run a job at 5:30 PM every day:

 schedule
  .atTime('17:30')
  .compile(); // Generates: 30 17 * * *

Run at 9 AM on weekdays (Monday to Friday):

 schedule
  .atTime('09:00')
  .onWeekDays([1, 2, 3, 4, 5])
  .compile(); // Generates: 0 9 * * 1-5

Run at noon on the 1st and 15th of the month:

 schedule
  .atTime('12:00')
  .onDaysOfMonth([1, 15])
  .compile(); // Generates: 0 12 1,15 * *

Run at midnight during January and July:

 schedule
  .atTime('00:00')
  .duringMonths([1, 7])
  .compile(); // Generates: 0 0 * 1,7 *

Run every 15 minutes:

 schedule
  .everyX(15, CronTimeUnit.Minute)
  .compile(); // Generates: */15 * * * *

Run every day at noon:

 schedule
  .every('day')
  .atHours([12])
  .compile(); // Generates: 0 12 * * *

Run every Sunday at 5 PM:

 schedule
  .onWeekDays([0])
  .atHours([17])
  .compile(); // Generates: 0 17 * * 0

Run the 1st day of every month at 1 AM:

 schedule
  .onDaysOfMonth([1])
  .every('month')
  .atHours([1])
  .compile(); // Generates: 0 1 1 * *

Run every weekday at 8:30 AM:

 schedule
  .atTime('08:30')
  .onWeekDays([1, 2, 3, 4, 5])
  .compile(); // Generates: 30 8 * * 1-5

Run every 6 hours:

 schedule
  .everyX(6, CronTimeUnit.Hour)
  .compile(); // Generates: 0 */6 * * *

Run every quarter at midnight:

 schedule
  .duringMonths([1, 4, 7, 10])
  .onDaysOfMonth([1])
  .atTime('00:00')
  .compile(); // Generates: 0 0 1 1,4,7,10 *

Run every Saturday and Sunday at 10:15 AM:

 schedule
  .atTime('10:15')
  .onWeekDays([6, 0])
  .compile(); // Generates: 15 10 * * 6,0

Run at 9 AM, 12 PM, and 3 PM every day:

 schedule
  .every('day')
  .atHours([9, 12, 15])
  .compile(); // Generates: 0 9,12,15 * * *

Run at 7 AM, 2 PM, and 10 PM on Tuesdays:

 schedule
  .atHours([7, 14, 22])
  .onWeekDays([2])
  .compile(); // Generates: 0 7,14,22 * * 2

Run at 20 past every hour on the 5th of July:

 schedule
  .atMinutes([20])
  .onDaysOfMonth([5])
  .duringMonths([7])
  .compile(); // Generates: 20 * 5 7 *

Run every 5 minutes during office hours - using every():

 schedule
  .every('minute')
  .atMinutes([0, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55])
  .atHours([9, 10, 11, 12, 13, 14, 15, 16])
  .compile(); 
  // Generates: 0,5,10,15,20,25,30,35,40,45,50,55 9-16 * * *

Run every 5 minutes during office hours - using everyX():

 schedule
 .everyX(5, CronTimeUnit.Minute)
 .atHours([9, 10, 11, 12, 13, 14, 15, 16])
 .compile(); // Generates: */5 9-16 * * *

Run at quarter past and quarter to every hour:

 schedule
  .every('hour')
  .atMinutes([15, 45])
  .compile(); // Generates: 15,45 * * * *

Please note that the schedule object is an instance of CronExpressionBuilder used for generating the cron expressions.

API Reference

`CronExpressionBuilder`

atMinutes(minutes: Array<number>): this
atHours(hours: Array<number>): this
atTime(time: string): this
every(unit: string): this
everyX(interval: number, unit: ScheduleUnit): this
onWeekdays(days: Array<number>): this
onDaysOfMonth(days: Array<number>): this
duringMonths(months: Array<number>): this
compile(): string - Get the cron expression

`CronValidators`

Static methods for validating cron expression components:

validateMinute(minute: number)
validateHour(hour: number)
validateDayOfMonth(day: number)
validateMonth(month: number)
validateDayOfWeek(day: number)
validateTime(time: string)

`CronTimeUnit`

Use the CronTimeUnit enum for ScheduleUnit:

export enum CronTimeUnit {
  Minute = 'minute',
  Hour = 'hour',
  DayOfMonth = 'dayOfMonth',
  Month = 'month',
  DayOfWeek = 'dayOfWeek',
}

Natural Cron simplifies the process of working with cron expressions, making it easier to schedule tasks in your Node.js applications. Whether you need to create a simple daily job or a complex, customized schedule, Natural Cron has you covered. With its human-readable API and validation features, it's a powerful tool for managing cron jobs with confidence.

Give Natural Cron a try in your Node.js projects and enjoy a more natural way to work with cron expressions!

Converting Handlebars AST to Template Strings with hbs-ast-to-str

satyajit nayak — Mon, 23 Oct 2023 04:47:00 +0000

Handlebars.js is a popular templating engine that allows you to create dynamic templates for your web applications. When you use Handlebars, you often work with Abstract Syntax Trees (ASTs) to manipulate and process templates. But have you ever needed to convert an AST back into a template string? That's where the "hbs-ast-to-str" package comes in.

Introduction to `hbs-ast-to-str`

The hbs-ast-to-str package is a NodeJS library that lets you convert Handlebars ASTs back into template strings with optional modifications. It's a handy tool for those times when you need to reverse the process and generate a template string from an AST.

Installation

You can install hbs-ast-to-str via npm as well as yarn:

npm install hbs-ast-to-str

yarn add hbs-ast-to-str

Basic Usage

Let's take a look at how you can use hbs-ast-to-str in your projects.

Without Modifications

If you just want to convert an AST to a template string without any modifications, here's how you can do it:

const { convertAstToString } = require('hbs-ast-to-str');

const ast = /* Your Handlebars AST using Handlebars.parse(template_str) */;

const template = convertAstToString(ast);

With Modifications

In some cases, you may want to make specific modifications during the conversion process. You can achieve this with the "ModificationOptions" interface. Here's an example:

const { convertAstToString } = require('hbs-ast-to-str');

const ast = /* Your Handlebars AST using Handlebars.parse(template_str) */;

const options = {
  helper: 'filter',
  paramType: HbsNodeTypes.SubExpression,
  paramPosition: 2,
  modifiers: [(d) => `'${d}'`],
};

const template = convertAstToString(ast, options);

The ModificationOptions Interface

The heart of this package is the "ModificationOptions" interface, which defines a set of options for making modifications to specific nodes or elements in a Handlebars template. Let's delve into the properties of this interface:

helper: An optional string that specifies the name of the helper function. This can be used to associate a modification with a particular helper.
paramType: An enumeration that represents the type of parameter you want to modify. You can use HbsNodeType to specify the node type.
paramPosition: An integer that specifies the position of the parameter you want to modify. This is particularly useful when dealing with helper parameters.
modifiers: An array of functions that define the modifications to be applied. Each function in the array should implement the logic for the modification.

Here's an example of how you can use the "ModificationOptions" interface in TypeScript:

const options = {
  helper: 'customHelper',
  paramType: HbsNodeType.Block,
  paramPosition: 2,
  modifiers: [(parameterValue) => {
    // Implement your modification logic here.
    // Modify the 'parameterValue' as needed.
  }],
};

Conclusion

The hbs-ast-to-str package is a powerful tool for developers working with Handlebars templates. Whether you need to convert an AST back into a template string or make specific modifications during the process, this package has you covered. It simplifies the task of working with Handlebars ASTs and empowers you to create dynamic and flexible templates for your web applications.

To learn more and explore the details, refer to the spec file included with the package.

So, if you find yourself dealing with Handlebars ASTs in your projects, give hbs-ast-to-str a try and streamline your templating workflow.

DEV Community: satyajit nayak

Ensure Idempotency in concurrent POST request with different request id's

Search & Replace Texts in DOCX

understand priorities of queues in NodeJS

Run Kafka Locally using Docker

Random Data Generator Website

Random Data Generator Website

Features

Schema Definition

Example Output

It can also fix your broken JSON on the go & generate mock data:

How to Use

Natural Cron ⏰⏰ - Create cron expressions with human readable APIs

Natural Cron ⏰⏰

Features

Installation

Usage

Examples

API Reference

CronExpressionBuilder

CronValidators

CronTimeUnit

Converting Handlebars AST to Template Strings with hbs-ast-to-str

Introduction to hbs-ast-to-str

Installation

Basic Usage

Without Modifications

With Modifications

The ModificationOptions Interface

Conclusion

`CronExpressionBuilder`

`CronValidators`

`CronTimeUnit`

Introduction to `hbs-ast-to-str`