DEV Community: Amit Kumar Rout

How to Open a 14 GB File in Under a Second — The Lazy Loading Technique Most Developers Never Use

Amit Kumar Rout — Mon, 27 Apr 2026 15:54:05 +0000

You double-click a file. Your app freezes. The spinning cursor appears. Ten seconds pass. Twenty. Eventually one of two things happens — either the data finally appears, or the app crashes with an out-of-memory error.

If you have ever built or used a tool that works with large files, you have been on both sides of this experience. And if you have ever tried to fix it, you have probably reached for the same solutions everyone reaches for: load less data, use a faster language, buy more RAM.

None of those are the real fix. The real fix is a change in how you think about the problem.

The Naive Approach and Why It Fails

When most developers build a tool that reads a file, they follow the same mental model:

Open the file
Read all of it into memory
Process it
Show the result This works fine for small files. For a 1 MB config file or a 10 MB dataset, reading everything upfront is fast and convenient. But the approach has a fundamental flaw that only reveals itself at scale: it couples the size of the file to the amount of memory your program consumes and the time the user has to wait before seeing anything.

A 14 GB CSV file means 14 GB of data has to be read, parsed, and held in memory before the user sees a single row. On a machine with 16 GB of RAM, this either works barely (leaving almost nothing for the rest of the system) or triggers heavy disk swapping that makes everything grind to a halt. On a machine with 8 GB of RAM, it simply crashes.

But here is the thing that makes this approach not just slow but fundamentally wrong: the user never needs all 28 million rows at once.

A monitor is typically 1080 to 1440 pixels tall. A row in a data viewer is typically 30 to 40 pixels tall. That means a user can see at most 40 to 50 rows at any given moment. Out of 28 million.

We are loading 28 million rows to show 50. That is the problem.

The Insight: Knowing Where Data Is Versus Having the Data

The key mental shift is this: there is a difference between knowing where something is and having it in hand.

Imagine a library with 10,000 books. You walk in and ask the librarian for a specific passage. The librarian does not read every book into memory before answering you. They use a catalogue a small, fast index that tells them exactly which shelf, which book, which page. Then they go directly to that location and retrieve just what you need.

Your program can work the same way.

Instead of reading the entire file into memory, you make a single fast pass through it to build an index. The index does not contain the data, it contains the location of the data. Specifically, for every row in the file, you record the byte position where that row starts.

For a 28 million row file, this index is a list of 28 million numbers. Each number is 8 bytes (a 64-bit integer is large enough to address any position in any file you will encounter in practice). That is 225 megabytes — large, but manageable, and completely independent of how much actual data is in the file.

Once you have this index, you can jump to any row in the file instantly, without reading anything in between.

Building the Index

The index-building pass has one job: scan the file from start to finish, and every time you find the start of a new row, record its byte position.

The tricky part is doing this correctly. A naive implementation might just scan for newline characters but real-world files have edge cases. Fields can contain quoted newlines. A CSV field like "line 1\nline 2" contains a newline inside the field itself, and a naive scanner would treat that as a row boundary when it is not.

The index builder needs to understand the file format well enough to distinguish real row boundaries from newlines inside quoted fields. This is not complex it just requires tracking whether you are currently inside a quoted field as you scan.

The pass is fast. It does not parse field values, does not allocate strings, does not do anything except count bytes and record positions. On modern hardware with an SSD, you can scan several gigabytes per second. A 14 GB file takes 20 to 30 seconds to index. Not instant, but the window stays responsive throughout, you can show a progress indicator, and crucially, you only pay this cost once per file open.

On-Demand Reading

Once you have the index, reading any row is a two-step operation:

Look up the byte position of that row in the index, this is an array lookup, essentially free
Seek to that position in the file and parse just that one row When the user opens the file, you show them the first 50 rows. You read only those 50 rows. As they scroll down, you read the next batch. As they scroll up, you read the previous batch. At any given moment, you have only a small window of parsed rows in memory — typically the visible rows plus a small buffer above and below to make scrolling feel smooth.

Memory usage is now roughly:

index size  +  (visible rows × row size)

For our 28 million row file, that is about 225 MB for the index plus a few hundred kilobytes for the visible rows. The 14 GB file itself is never fully in memory. Total working set: well under 1 GB, regardless of file size.

The user experience is transformed. The file "opens" the moment the index is built. Scrolling is instant. Jumping to row 14,000,000 takes the same time as jumping to row 2,look up the position, seek, parse.

A Tool That Makes This Efficient: Memory Mapping

One technique that makes on-demand reading significantly faster in practice is memory mapping, a feature available in virtually every modern operating system.

When you memory-map a file, the OS maps the file's bytes directly into your program's address space. The file appears as an array of bytes you can access by index, without any explicit read calls. The OS handles paging behind the scenes, loading pages from disk only when you actually access them.

This is a natural fit for the lazy loading pattern. Your index stores byte positions. Accessing a row is as simple as indexing into the mapped array at that position. The OS efficiently handles the underlying disk reads, using its own cache so that recently accessed regions are served from RAM on subsequent accesses.

Memory mapping is not specific to any programming language or platform. The concept exists everywhere, it is how text editors handle large files, how databases handle their data pages, and how operating systems load executable code.

The Tradeoffs — Being Honest

This pattern is not free. There are real tradeoffs to understand before applying it.

Index build time is not zero. For a 14 GB file, the initial scan takes 20 to 30 seconds. If your users are opening the same large file repeatedly, you can cache the index to disk so subsequent opens are instant. If files change frequently, caching becomes harder to manage.

Operations that touch every row still take time. Sorting a column requires extracting that column's value from every row to build the sort key. Searching for a string requires checking every row. These operations still do a full pass, they just do it more efficiently because they only extract what they need rather than materializing the entire dataset. On a 14 GB file, a sort or search takes seconds rather than the instant response users might expect from in-memory data.

Random access has higher per-row overhead than sequential access. When data is in memory, accessing row 1,000,000 is a simple array index. When it is on disk, it requires a seek, which is fast on SSDs but not free. For workloads that access rows in random order at high frequency, in-memory storage is still faster. Lazy loading is optimized for the common case of a user looking at a small window of data and occasionally jumping around.

This pattern is best suited for read-heavy workloads. If users need to edit cells, you need to track modifications on top of the on-disk representation, which adds complexity. For pure viewing and analysis, the pattern is a natural fit.

The Mental Model Shift

The change required is not a new algorithm or a clever library. It is a shift in how you think about the relationship between files, memory, and what you show the user.

The old model:

File → Memory → Screen

Load everything, then show it.

The new model:

File → Index → Screen (fetching on demand)

Know where everything is, then fetch only what you need to show.

This shift has compounding benefits. A tool built on the new model does not just handle large files, it handles files of any size with the same performance characteristics. A 100 MB file and a 100 GB file feel identical to the user. The index for a 100 GB file is larger, but it is still orders of magnitude smaller than the file itself. The on-demand reads are the same either way.

The constraint you are designing around is no longer "how much memory can I allocate" but "how fast can I seek and read a small amount of data" and that constraint is much more forgiving on modern hardware.

Applying This to Your Own Tools

If you are building any tool that reads large files — a log viewer, a data explorer, a CSV editor, a file differ, a code search tool — the lazy loading pattern is worth considering from the start. It is significantly harder to retrofit into an existing tool than to design around from the beginning.

The core implementation has three parts:

The index builder — a single sequential pass through the file that records row or record boundaries. This is the only place where you read the entire file, and you only do it once.

The random access reader — a function that takes a row number, looks up its byte position in the index, seeks to that position, and parses just that row. This function should be fast and stateless — given the same row number, it always returns the same result.

The viewport manager — the layer between your UI and the reader that decides which rows to fetch based on what the user can currently see. It maintains a small cache of recently read rows and fetches new ones as the user scrolls.

These three pieces together give you a tool that opens any size file in the time it takes to build the index, uses memory proportional to what is visible rather than what exists, and stays responsive throughout.

The technique is not new. It is not exotic. It is how high-performance tools have always handled data at scale. The reason most applications do not use it is not that it is hard — it is that the naive approach works well enough at small sizes, and by the time files get large enough to matter, the architecture is already baked in.

Start with the right model, and large files stop being a problem worth solving. They become just another input your tool handles without complaint.

If you want to see this pattern in action, I built Columnar — a desktop CSV viewer that uses exactly this approach to open files with tens of millions of rows. The source is available at https://github.com/amitkroutthedev/Columnar. If you like the project please star the project such that more and more people find the project and if you like my work follow me on Github

Drop a comment if you've hit this problem before or solved it a different way — I'd love to hear your approach

Happy Coding!

Event-Driven Architecture in Node.js: Building Scalable and Responsive Systems

Amit Kumar Rout — Tue, 02 Dec 2025 08:40:12 +0000

Event-Driven Architecture has become one of the key patterns in designing modern, scalable applications. Node.js, due to its intrinsic asynchrony and non-blocking I/O model, especially fits the purposes when building event-driven systems. Mastering event-driven architecture will completely reshape your view of application structuring, whether you are working on microservices, real-time applications, or high-concurrency platforms.

In this comprehensive guide, we are going to look into some of the key concepts of event-driven architecture, go through practical examples, cover best practices, and point out some common pitfalls to avoid.

What is Event-Driven Architecture?

Event-driven architecture is a design paradigm in which the flow of the application is driven by events—distinct messages or signals that indicate something has happened in the system. Instead of components directly calling other components (tightly coupled), they emit events and listen for events from other components (loosely coupled).

Core Components

Event Emitters – Components that produce events when specific actions or state changes occur.

Event Listeners (Consumers) – Components that subscribe to events and execute logic in response.

Events – Payloads that describe what happened (e.g., userRegistered, orderPlaced, paymentProcessed).

Event Bus/Message Broker – Central infrastructure that routes events from producers to consumers (in-process or distributed).

Why Node.js is Ideal for Event-Driven Architecture

Node.js is designed on an event-driven, non-blocking I/O model. At its core, the EventEmitter class provides the base functionality for building event-driven architectures within applications. It also integrates seamlessly with external message brokers such as RabbitMQ, Kafka, and Redis, hence ideal for distributed event-driven systems.

Getting Started: EventEmitter Basics

The EventEmitter class is the foundation of event-driven programming in Node.js. Let's start with simple examples.

Basic EventEmitter Usage

const EventEmitter = require('events');

// Create an event emitter instance
const emitter = new EventEmitter();

// Register a listener for 'userRegistered' event
emitter.on('userRegistered', (user) => {
  console.log(`Welcome email sent to ${user.email}`);
});

// Emit the event
emitter.emit('userRegistered', { 
  id: 1, 
  email: 'john@example.com', 
  name: 'John Doe' 
});
// Output: Welcome email sent to john@example.com

Creating Custom Emitter Classes

For larger applications, you'll want to create custom classes that inherit from EventEmitter:

const EventEmitter = require('events');

class UserService extends EventEmitter {
  constructor() {
    super();
  }

  registerUser(userData) {
    // User registration logic
    const user = {
      id: Date.now(),
      ...userData
    };

    // Emit event after successful registration
    this.emit('userRegistered', user);

    return user;
  }

  deleteUser(userId) {
    // User deletion logic
    this.emit('userDeleted', { userId });
  }
}

// Usage
const userService = new UserService();

userService.on('userRegistered', (user) => {
  console.log(`User registered: ${user.email}`);
  // Send welcome email
});

userService.on('userRegistered', (user) => {
  console.log(`New user created: ${user.id}`);
  // Update analytics
});

userService.registerUser({ 
  email: 'jane@example.com', 
  name: 'Jane Smith' 
});

Notice that we can register multiple listeners for the same event. Each listener executes independently.

Real-World Example: Order Processing System

Let's build a practical example showing how event-driven architecture improves system design. We'll create an order processing system where different services react to order events.

Basic Implementation

const EventEmitter = require('events');

class OrderService extends EventEmitter {
  constructor() {
    super();
    this.orders = new Map();
  }

  createOrder(orderData) {
    const order = {
      id: Date.now(),
      status: 'pending',
      ...orderData,
      createdAt: new Date()
    };

    this.orders.set(order.id, order);

    // Emit event when order is created
    this.emit('orderCreated', order);

    return order;
  }

  completeOrder(orderId) {
    const order = this.orders.get(orderId);

    if (!order) {
      this.emit('error', new Error('Order not found'));
      return;
    }

    order.status = 'completed';
    this.emit('orderCompleted', order);
  }
}

// Email Service (listens to order events)
class EmailService {
  constructor(orderService) {
    // When an order is created, send confirmation email
    orderService.on('orderCreated', (order) => {
      console.log(`Confirmation email sent for Order #${order.id}`);
    });

    // When order is completed, send shipment notification
    orderService.on('orderCompleted', (order) => {
      console.log(`Shipment notification sent for Order #${order.id}`);
    });
  }
}

// Inventory Service (listens to order events)
class InventoryService {
  constructor(orderService) {
    orderService.on('orderCreated', (order) => {
      console.log(`Inventory updated for Order #${order.id}`);
      console.log(`Reduced stock for ${order.items.length} items`);
    });
  }
}

// Payment Service (listens to order events)
class PaymentService {
  constructor(orderService) {
    orderService.on('orderCreated', (order) => {
      console.log(`Processing payment of $${order.total}`);
      // Charge customer
    });
  }
}

// Usage
const orderService = new OrderService();
const emailService = new EmailService(orderService);
const inventoryService = new InventoryService(orderService);
const paymentService = new PaymentService(orderService);

// Error handling
orderService.on('error', (err) => {
  console.error(`Error: ${err.message}`);
});

// Create an order
const order = orderService.createOrder({
  customerId: 'CUST123',
  items: [
    { productId: 'PROD1', quantity: 2, price: 50 },
    { productId: 'PROD2', quantity: 1, price: 75 }
  ],
  total: 175
});

console.log('Order created:', order);

// Complete the order
orderService.completeOrder(order.id);

Output:

Order created: { id: 1733120640123, status: 'pending', ... }
Confirmation email sent for Order #1733120640123
Inventory updated for Order #1733120640123
Processing payment of $175
Shipment notification sent for Order #1733120640123

Key Advantages Demonstrated

Loose Coupling: Email, Inventory, and Payment services don't know about each other. They only listen to order events.

Scalability: Adding new services is simple—just listen to the events you care about.

Single Responsibility: Each service has one job and does it well.

Maintainability: Changes to one service don't require changes to others.

Handling Multiple Event Listeners and Event Namespacing

As your application grows, you need strategies to organize events effectively.

Event Namespacing

const EventEmitter = require('events');

class ApplicationBus extends EventEmitter {}

const bus = new ApplicationBus();

// Use namespacing with colon convention
bus.on('user:created', (user) => {
  console.log('User created event handler');
});

bus.on('user:updated', (user) => {
  console.log('User updated event handler');
});

bus.on('user:deleted', (user) => {
  console.log('User deleted event handler');
});

// Emit namespaced events
bus.emit('user:created', { id: 1, email: 'john@example.com' });
bus.emit('user:updated', { id: 1, email: 'john.doe@example.com' });
bus.emit('user:deleted', { id: 1 });

Removing Event Listeners

const handler = (data) => {
  console.log('Handler called');
};

bus.on('myEvent', handler);

// Remove specific listener
bus.off('myEvent', handler);

// Remove all listeners for an event
bus.removeAllListeners('myEvent');

// Remove all listeners
bus.removeAllListeners();

// Get listener count
console.log(bus.listenerCount('myEvent')); // 0

Listening Once

// Listen only once, then automatically remove
bus.once('connectionEstablished', () => {
  console.log('Connected! This prints only once.');
});

bus.emit('connectionEstablished');
bus.emit('connectionEstablished'); // Won't trigger handler

Distributed Event-Driven Architecture with Message Brokers

For applications spanning multiple services or servers, in-process EventEmitter isn't sufficient. This is where message brokers like RabbitMQ, Kafka, and Redis come in.

Redis Pub/Sub Implementation

Redis offers a lightweight pub/sub mechanism perfect for event distribution across services.

const redis = require('redis');

// Publisher
const publisher = redis.createClient();

async function publishOrder(order) {
  await publisher.connect();

  await publisher.publish('orders', JSON.stringify({
    type: 'orderCreated',
    data: order
  }));

  await publisher.quit();
}

// Subscriber
const subscriber = redis.createClient();

async function subscribeToOrders() {
  await subscriber.connect();

  await subscriber.subscribe('orders', (message) => {
    const event = JSON.parse(message);

    if (event.type === 'orderCreated') {
      console.log('Order created, sending confirmation email');
      // Handle order creation
    }
  });
}

// Usage
publishOrder({ 
  id: 1, 
  customerId: 'CUST123', 
  total: 100 
});

subscribeToOrders();

RabbitMQ Implementation

RabbitMQ provides more advanced routing and reliability features through exchanges and queues.

const amqp = require('amqplib');

const RABBITMQ_URL = 'amqp://localhost';

// Producer
async function publishEventToRabbitMQ(event) {
  const connection = await amqp.connect(RABBITMQ_URL);
  const channel = await connection.createChannel();

  const exchange = 'orders_exchange';
  const routingKey = event.type;

  await channel.assertExchange(exchange, 'direct', { durable: true });

  channel.publish(
    exchange,
    routingKey,
    Buffer.from(JSON.stringify(event))
  );

  console.log(`Event published: ${event.type}`);

  await channel.close();
  await connection.close();
}

// Consumer
async function consumeOrderEvents() {
  const connection = await amqp.connect(RABBITMQ_URL);
  const channel = await connection.createChannel();

  const exchange = 'orders_exchange';
  const queue = 'order_notifications_queue';
  const routingKeys = ['orderCreated', 'orderCompleted'];

  await channel.assertExchange(exchange, 'direct', { durable: true });
  await channel.assertQueue(queue, { durable: true });

  // Bind queue to exchange with specific routing keys
  for (const key of routingKeys) {
    await channel.bindQueue(queue, exchange, key);
  }

  // Consume messages
  await channel.consume(queue, async (msg) => {
    if (msg) {
      const event = JSON.parse(msg.content.toString());

      console.log(`Processing event: ${event.type}`);

      if (event.type === 'orderCreated') {
        console.log(`Sending confirmation email for order ${event.data.id}`);
      }

      // Acknowledge message (remove from queue)
      channel.ack(msg);
    }
  });
}

// Usage
publishEventToRabbitMQ({
  type: 'orderCreated',
  data: { id: 1, customerId: 'CUST123', total: 100 }
});

consumeOrderEvents();

Apache Kafka Implementation

Kafka is ideal for high-throughput, distributed systems with event streaming capabilities.

const { Kafka } = require('kafkajs');

const kafka = new Kafka({
  clientId: 'my-app',
  brokers: ['localhost:9092']
});

// Producer
async function publishToKafka(event) {
  const producer = kafka.producer();
  await producer.connect();

  await producer.send({
    topic: 'orders',
    messages: [
      {
        key: event.data.orderId,
        value: JSON.stringify(event)
      }
    ]
  });

  console.log(`Event sent to Kafka: ${event.type}`);
  await producer.disconnect();
}

// Consumer
async function consumeFromKafka() {
  const consumer = kafka.consumer({ groupId: 'order-service-group' });
  await consumer.connect();

  await consumer.subscribe({ topic: 'orders', fromBeginning: false });

  await consumer.run({
    eachMessage: async ({ topic, partition, message }) => {
      const event = JSON.parse(message.value.toString());

      console.log(`Received event: ${event.type}`);

      if (event.type === 'orderCreated') {
        console.log(`Processing order ${event.data.orderId}`);
        // Handle event
      }
    }
  });
}

// Usage
publishToKafka({
  type: 'orderCreated',
  data: { orderId: 'ORD123', customerId: 'CUST123', total: 100 }
});

consumeFromKafka();

Error Handling in Event-Driven Systems

Proper error handling is critical because unhandled exceptions in event listeners can crash your application.

Basic Error Handling

const EventEmitter = require('events');

class SafeEventEmitter extends EventEmitter {
  constructor() {
    super();

    // Listen for errors
    this.on('error', (err) => {
      console.error('EventEmitter error:', err.message);
      // Log error, send alert, etc.
    });
  }
}

const emitter = new SafeEventEmitter();

// Emit error event
emitter.emit('error', new Error('Database connection failed'));

// Listeners with try-catch
emitter.on('dataProcessed', async (data) => {
  try {
    // Process data
    await performDatabaseOperation(data);
  } catch (err) {
    emitter.emit('error', err);
  }
});

Dead Letter Queues

Dead Letter Queues (DLQ) store messages that fail processing, allowing for later inspection and reprocessing.

const amqp = require('amqplib');

async function setupDLQ() {
  const connection = await amqp.connect('amqp://localhost');
  const channel = await connection.createChannel();

  // Main exchange and queue
  const exchange = 'main_exchange';
  const mainQueue = 'main_queue';
  const dlxExchange = 'dlx_exchange';
  const dlQueue = 'dead_letter_queue';

  // Setup DLX (Dead Letter Exchange)
  await channel.assertExchange(dlxExchange, 'direct', { durable: true });
  await channel.assertQueue(dlQueue, { durable: true });
  await channel.bindQueue(dlQueue, dlxExchange, 'failed');

  // Setup main queue with DLX
  await channel.assertExchange(exchange, 'direct', { durable: true });
  await channel.assertQueue(mainQueue, {
    durable: true,
    arguments: {
      'x-dead-letter-exchange': dlxExchange,
      'x-dead-letter-routing-key': 'failed',
      'x-message-ttl': 60000 // 1 minute
    }
  });

  await channel.bindQueue(mainQueue, exchange, 'events');

  // Consume from main queue
  await channel.consume(mainQueue, async (msg) => {
    if (msg) {
      try {
        const event = JSON.parse(msg.content.toString());
        console.log(`Processing: ${event.type}`);

        // Simulate processing that might fail
        if (event.data.shouldFail) {
          throw new Error('Processing failed');
        }

        channel.ack(msg);
      } catch (err) {
        console.error(`Error processing message: ${err.message}`);
        // Reject and send to DLQ
        channel.nack(msg, false, false);
      }
    }
  });

  // Process dead letter messages
  await channel.consume(dlQueue, async (msg) => {
    if (msg) {
      const event = JSON.parse(msg.content.toString());
      console.log(`Dead letter message: ${event.type}`);
      // Notify ops team, log for manual review
      channel.ack(msg);
    }
  });
}

setupDLQ();

Best Practices for Event-Driven Architecture

1. Define Clear Event Schemas

// Use a consistent event structure
const eventSchema = {
  type: 'string', // 'userCreated', 'orderPlaced', etc.
  eventId: 'uuid', // Unique identifier for this event
  timestamp: 'ISO8601', // When event occurred
  source: 'string', // Which service emitted it
  data: 'object', // Event-specific payload
  version: 'integer' // Schema version for versioning
};

// Example event
const event = {
  type: 'userCreated',
  eventId: '550e8400-e29b-41d4-a716-446655440000',
  timestamp: '2025-12-02T10:31:00Z',
  source: 'user-service',
  data: {
    userId: '123',
    email: 'user@example.com',
    createdAt: '2025-12-02T10:31:00Z'
  },
  version: 1
};

2. Keep Event Payloads Small

// Bad - too much data
bus.emit('userCreated', {
  userId: '123',
  email: 'user@example.com',
  profile: { /* large object */ },
  preferences: { /* large object */ },
  history: [ /* large array */ ]
});

// Good - minimal necessary data
bus.emit('userCreated', {
  userId: '123',
  email: 'user@example.com'
});

3. Handle Event Versioning

class EventHandler {
  handleUserCreated(event) {
    switch(event.version) {
      case 1:
        return this.handleUserCreatedV1(event);
      case 2:
        return this.handleUserCreatedV2(event);
      default:
        throw new Error(`Unknown event version: ${event.version}`);
    }
  }

  handleUserCreatedV1(event) {
    console.log(`Processing v1 event for user: ${event.data.email}`);
  }

  handleUserCreatedV2(event) {
    console.log(`Processing v2 event for user: ${event.data.email}`);
    // Handle new fields
  }
}

4. Ensure Idempotency

Events may be delivered multiple times. Make operations safe to repeat:

const processedEventIds = new Set();

async function handleEvent(event) {
  // Check if event was already processed
  if (processedEventIds.has(event.eventId)) {
    console.log(`Event ${event.eventId} already processed, skipping`);
    return;
  }

  try {
    // Process event
    await processOrderEvent(event);

    // Mark as processed
    processedEventIds.add(event.eventId);
  } catch (err) {
    console.error(`Failed to process event: ${err.message}`);
    // Let message broker handle retry
  }
}

5. Implement Proper Monitoring and Logging

class MonitoredEventEmitter extends EventEmitter {
  emit(eventName, ...args) {
    const startTime = Date.now();

    console.log(`[Event] Emitting: ${eventName}`);

    try {
      const result = super.emit(eventName, ...args);
      const duration = Date.now() - startTime;

      console.log(`[Event] ${eventName} completed in ${duration}ms`);

      return result;
    } catch (err) {
      console.error(`[Event] Error in ${eventName}:`, err);
      throw err;
    }
  }
}

const emitter = new MonitoredEventEmitter();

emitter.on('orderProcessed', async (order) => {
  // Process order
});

emitter.emit('orderProcessed', { id: 1, total: 100 });

Common Pitfalls and How to Avoid Them

Pitfall 1: Creating a "Distributed Ball of Mud"

Problem: Too many events with unclear relationships lead to chaotic systems.

Solution: Document event flows, establish clear ownership, and use domain-driven design principles.

// Good: Clear, intentional events
const events = {
  USER: {
    REGISTERED: 'user.registered',
    VERIFIED: 'user.verified',
    DELETED: 'user.deleted'
  },
  ORDER: {
    CREATED: 'order.created',
    SHIPPED: 'order.shipped',
    DELIVERED: 'order.delivered'
  }
};

Pitfall 2: Not Handling Event Failures

Problem: Failed events are silently dropped or cause system crashes.

Solution: Implement retry logic, dead letter queues, and proper error handling.

async function processWithRetry(eventHandler, event, maxRetries = 3) {
  for (let attempt = 1; attempt <= maxRetries; attempt++) {
    try {
      await eventHandler(event);
      return; // Success
    } catch (err) {
      if (attempt === maxRetries) {
        console.error(`Failed after ${maxRetries} attempts:`, err);
        throw err; // Send to DLQ
      }

      // Exponential backoff
      const delay = Math.pow(2, attempt) * 1000;
      console.log(`Retry in ${delay}ms`);
      await new Promise(resolve => setTimeout(resolve, delay));
    }
  }
}

Pitfall 3: Tight Coupling Between Event Producers and Consumers

Problem: Adding new consumers requires changes to producers.

Solution: Use a centralized event bus or message broker.

// Bad: Tight coupling
class OrderService {
  createOrder(data) {
    const order = new Order(data);

    // Service knows about email, inventory, payment
    new EmailService().sendConfirmation(order);
    new InventoryService().updateStock(order);
    new PaymentService().processPayment(order);
  }
}

// Good: Event-driven and loosely coupled
class OrderService {
  constructor(eventBus) {
    this.eventBus = eventBus;
  }

  createOrder(data) {
    const order = new Order(data);

    // Only emit event, let consumers handle it
    this.eventBus.emit('orderCreated', order);
  }
}

Pitfall 4: Infinite Event Loops

Problem: Event A triggers Event B, which triggers Event A again.

Solution: Implement event tracking and careful flow analysis.

class EventTracker {
  constructor(maxDepth = 5) {
    this.maxDepth = maxDepth;
    this.currentDepth = 0;
  }

  async handleEvent(event, handler) {
    if (this.currentDepth >= this.maxDepth) {
      throw new Error('Maximum event depth exceeded - possible loop detected');
    }

    this.currentDepth++;

    try {
      await handler(event);
    } finally {
      this.currentDepth--;
    }
  }
}

Performance Considerations

Event Throttling

function throttleEvent(emitter, eventName, delay) {
  let lastEmit = 0;
  const originalEmit = emitter.emit.bind(emitter);

  emitter.emit = function(name, ...args) {
    if (name === eventName) {
      const now = Date.now();
      if (now - lastEmit < delay) {
        return;
      }
      lastEmit = now;
    }
    return originalEmit(name, ...args);
  };
}

const emitter = new EventEmitter();
throttleEvent(emitter, 'dataUpdated', 1000); // Max once per second

// These rapid updates will be throttled
for (let i = 0; i < 100; i++) {
  emitter.emit('dataUpdated', { value: i });
}

Event Batching

class BatchedEventEmitter extends EventEmitter {
  constructor(batchSize = 10, batchDelay = 1000) {
    super();
    this.batchSize = batchSize;
    this.batchDelay = batchDelay;
    this.batch = [];
    this.timeout = null;
  }

  batchEmit(eventName, data) {
    this.batch.push(data);

    if (this.batch.length >= this.batchSize) {
      this.flush(eventName);
    } else if (!this.timeout) {
      this.timeout = setTimeout(() => this.flush(eventName), this.batchDelay);
    }
  }

  flush(eventName) {
    if (this.batch.length > 0) {
      this.emit(eventName, this.batch);
      this.batch = [];
    }

    if (this.timeout) {
      clearTimeout(this.timeout);
      this.timeout = null;
    }
  }
}

const emitter = new BatchedEventEmitter(50, 2000);

emitter.on('dataProcessed', (batch) => {
  console.log(`Processing batch of ${batch.length} items`);
});

for (let i = 0; i < 1000; i++) {
  emitter.batchEmit('dataProcessed', { id: i });
}

Comparing Event-Driven Approaches

Aspect	In-Process EventEmitter	Redis Pub/Sub	RabbitMQ	Apache Kafka
Scale	Single process	Multiple services	Multiple services	Distributed systems
Persistence	No	No	Optional	Yes
Ordering	Guaranteed	No	No	Guaranteed per partition
Throughput	Very high	High	High	Very high
Setup	Trivial	Simple	Moderate	Complex
Learning Curve	Beginner	Beginner	Intermediate	Advanced
Use Case	Monolith, simple apps	Cache, notifications	Microservices, reliability	Streaming, analytics

Conclusion

Event-Driven Architecture brings significant benefits to Node.js applications: improved scalability, loose coupling, and responsiveness. Whether you start with in-process EventEmitter for a monolithic application or scale to distributed message brokers like Kafka for microservices, the event-driven paradigm provides a powerful foundation.

Key takeaways:

Use EventEmitter for simple, in-process event handling
Graduate to Redis Pub/Sub or RabbitMQ for microservices communication
Implement proper error handling with retry logic and dead letter queues
Define clear event schemas and maintain event versioning
Ensure idempotency in event handlers
Monitor and log events extensively
Avoid common pitfalls like tight coupling and infinite loops

By mastering event-driven architecture, you'll build systems that are more maintainable, scalable, and resilient to change. Start small with EventEmitter, gradually adopt message brokers as your system grows, and always keep your event flows clear and well-documented.

Happy event-driven coding!

LangCompiler: A Cloud-Native, High-Performance Code Execution Service

Amit Kumar Rout — Fri, 14 Feb 2025 16:07:19 +0000

Introduction

Smooth, secure, and scalable execution of code is of paramount concern in modern software development for a variety of applications, from education systems to enterprise applications. LangCompiler has been created to cater to this demand by providing a cloud-native, high-performance environment for code execution across a variety of programming languages.

What is LangCompiler?

LangCompiler is an elegantly crafted service that provides secure and scalable code execution across a wide spectrum of programming languages. It currently supports Python, Java, JavaScript, and C++, with further expansion planned in response to community requirements. Its versatility makes it a developer, educator, and enterprise favorite.

Key Features

Multi-Language Support: The service provides code execution across four popular languages (Python, Java, JavaScript, and C++), with further expansion planned.
Dynamic Resource Constraints: It is scalable, managing workloads efficiently through resource-aware execution.
Advanced Error Handling: The service includes extensive logging and debugging to guarantee seamless execution.
Comprehensive Execution Metrics: Users can monitor performance, detect inefficiencies, and optimize workloads.
Secure and Isolated Execution: It includes security features to prevent unauthorized access and code protection.

Built with Golang and Docker

To ensure high performance and reliability, LangCompiler is built using Golang, leveraging its efficiency and concurrency features. Additionally, it is Dockerized to provide an isolated execution environment, ensuring consistent and secure execution across different workloads. The service is hosted on Render, allowing seamless deployment, scaling, and management.

Try LangCompiler Today!

We invite developers, educators, and enterprises to explore LangCompiler and take advantage of its robust capabilities. To further enhance API development, we encourage you to check out our premium plans.

🔗 Visit LangCompiler's Product Page and start coding with confidence!

Happy Coding!!

Dockerizing Your Frontend Application with Nginx for Seamless Deployment

Amit Kumar Rout — Thu, 26 Dec 2024 09:54:02 +0000

Introduction

Most often, deploying a frontend application requires separating the serving of static assets from backend APIs. A good approach to this is to couple Nginx as the reverse proxy and web server with Docker for containerization. This guide takes you through the process of how to deploy a frontend application using Nginx and Docker.

Prerequisites

How to Deploy a Frontend Application Using Nginx and Docker
Introduction
Efficiently deploying a frontend application often involves separating the serving of static assets from backend APIs. A powerful approach is combining Nginx as a reverse proxy and web server with Docker for containerization. This guide walks you through the process of deploying a frontend application using Nginx and Docker.

Prerequisites
To follow this guide, you should have:

Basic knowledge of React (or any frontend framework).
Familiarity with Docker and Nginx.

Setting Up the Application

A clear project structure simplifies deployment. Place all necessary files in a folder to build your Docker image. These files should include:

build/ folder (contains the production-ready static files).
Dockerfile(defines how the image is built).
nginx.conf (custom Nginx configuration).
sites-enabled/ (optional additional Nginx configurations).

For an in-depth explanation of the Nginx configuration, refer to this

Elevating React: Unleashing the Power of Nginx for Effortless Deployment

Amit Kumar Rout ・ Dec 9 '23

#javascript #tutorial #react #nginx

Steps:

Build the production version of your application:

npm run build

This command creates a build/ directory containing optimized static files.

Create a Dockerfile: Write the following Dockerfile to define the container:

FROM nginx:latest

# Clean the default HTML directory and remove default nginx.conf
RUN rm -rf /usr/share/nginx/html/*
RUN rm /etc/nginx/nginx.conf

# Copy custom Nginx configuration
COPY nginx.conf /etc/nginx/

# Copy the production build files into Nginx's HTML directory
COPY build/ /usr/share/nginx/html/

# Copy additional Nginx configurations (if any)
COPY ./sites-enabled/* /etc/nginx/sites-enabled/

# Expose port 80
EXPOSE 80

# Start Nginx
CMD ["nginx", "-g", "daemon off;"]

Building the Docker Image

Once your files are ready, build the Docker image:

docker build -t my-app-img:prod .

If deploying on another VM, you can package the image as a .tar file:

docker save my-app-img > my-app-img.tar

Running the Docker Image

Option 1: Running the image directly

Load the Docker image from a .tar file (if transferring between systems):

cat my-app-img.tar | docker load

Run the container:

docker run -itd -p 80:80 --name my-app my-app-img

This starts the container, exposing it on port 80.

Option 2: Running the Docker Compose

Creating a docker-compose.yml file:

version: "3"
services:
  web:
    container_name: my_app
    image: my-app-img
    ports:
      - "80:80"
    volumes:
      - ./template:/etc/nginx/templates/
    environment:
      NGINX_ENVSUBST_OUTPUT_DIR: /etc/nginx/sites-enabled

Then, start the container:

docker-compose up -d

Conclusion

The use of Nginx and Docker to deploy a frontend application gives a strong, scalable solution. This is because it separates static assets from backend APIs, giving the best performance. Customize your Nginx configuration to meet your needs, such as:

Adding SSL/TLS for HTTPS support.
Turn on caching for high performance.

This setup enables your frontend application to be reliable in production-ready environments.

If you have any suggestions for improving the process, please leave them in the comments section. If you found this post helpful, please like and share it.

Happy Coding!

Implementing Redux in Next.js (App Router): A Comprehensive Guide

Amit Kumar Rout — Tue, 16 Apr 2024 06:17:30 +0000

Introduction

Next.js, a popular React framework, offers flexibility and control over state management. However, it lacks a built-in solution. This is where Redux/Zustand comes in.

Redux and Zustand are both state management solutions for Next.js apps, but they differ in complexity and approach. Redux is more complex but offers predictability, scalability, and a larger community. Zustand is simpler but less predictable and scalable, with a smaller community. Choose Redux for large and complex applications requiring strict data flow and a well-defined architecture. Choose Zustand for smaller applications where simplicity and performance are essential. The best choice depends on your project's specific needs and preferences.

This article is more focused on Redux than Zustand.

Prerequisites

react
redux
redux-toolkit
redux-persist

Folder Structure

I am keeping redux codes in /store folder and creating a wrapper(Provider) component in StoreProvider.tsx file.

/app
  layout.tsx
  page.tsx
  StoreProvider.tsx
/store
  store.ts
  hooks.ts
  storage.ts
  storage
  /actions
   Actions.ts
 /reducers
   Reducers.ts
   RootReducers.ts

I am assuming that you are already have basic knowledge on Redux, Actions, Reducers, combineReducers. I am also using redux-persist for persistence of the data.

in /store/storage.ts:

import createWebStorage from "redux-persist/lib/storage/createWebStorage";


const createNoopStorage = () => {
  return {
    getItem(_key: any) {
      return Promise.resolve(null);
    },
    setItem(_key: any, value: any) {
      return Promise.resolve(value);
    },
    removeItem(_key: any) {
      return Promise.resolve();
    },
  };
};

const storage = typeof window !== "undefined" ? createWebStorage("local") : createNoopStorage();

export default storage;

The storage function specifies the storage engine to be used for persisting the state(localStorage in web).

in store/store.ts:

import { configureStore } from "@reduxjs/toolkit";
import { persistReducer } from "redux-persist";
import storage from "./storage";
import RootReducer from "./reducers/RootReducers";

const persistConfig = {
  key: "root",
  storage,
  blacklist: ["tracking"],
};

const persistedReducer = persistReducer(persistConfig, RootReducer)

export const makeStore = () => {
  return configureStore({
    reducer: persistedReducer,
    devTools: process.env.NODE_ENV !== "production",
    middleware: (getDefaultMiddleware) =>
      getDefaultMiddleware({
        serializableCheck: {
          ignoredActions: [FLUSH, REHYDRATE, PAUSE, PERSIST, PURGE, REGISTER],
        },
      }),
  });
};
export type AppStore = ReturnType<typeof makeStore>;
export type RootState = ReturnType<AppStore["getState"]>;
export type AppDispatch = AppStore["dispatch"];

This function makeStore configures a Redux store with a persisted reducer, conditionally enables Redux DevTools based on the environment, and attempts to customize the middleware using redux-thunk.

To utilise the function(makeStore), we need to create a client component that will initiate redux store and share with application using the React-Router Provider component.

In app/StoreProvider.tsx:

'use client'
import { useRef } from 'react'
import { Provider } from 'react-redux'
import { makeStore,AppStore } from '../store/store'
import { PersistGate } from "redux-persist/integration/react";
import { persistStore } from "redux-persist";

export default function StoreProvider({ children }: {
  children: React.ReactNode
}) {
  const storeRef = useRef<AppStore>()
  if (!storeRef.current) {
    storeRef.current = makeStore()
  }
  const persistor = persistStore(storeRef.current);

  return (
    <Provider store={storeRef.current}>
      <PersistGate loading={null} persistor={persistor}>
        {children}
      </PersistGate>
    </Provider>
  );
}

Then we will use this wrapper component in mainlayout in our application.

In app/layout.tsx:

import type { Metadata } from "next";
import { Inter } from "next/font/google";
import "./globals.css";
import StoreProvider from "./StoreProvider";

const inter = Inter({ subsets: ["latin"] });

export const metadata: Metadata = {
  title: "Next-Redux",
  description: "Implementing Redux in Nextt(App Router)",
};

export default function RootLayout({
  children,
}: Readonly<{
  children: React.ReactNode;
}>) {
  return (
   <html lang="en">
      <body className={inter.className}>
        <StoreProvider>
          {children}
        </StoreProvider>
      </body>
    </html>
 );
}

Conclusion

The Redux state has been implemented with persistence in our Next.js application. Redux-Persist effectively manages the application state and restores data even when the page refreshes or navigates. Additionally, Thunk middleware allows you to write action creators that return a function instead of an action.
I hope this will helpful for implementing Redux in Next.js application. If you have any suggestions for improving the code, please leave them in the comments section. If you found this post helpful, please like and share it.

Happy Coding!

Upgrading to 'create-rp-app': A Better way to create fast react app

Amit Kumar Rout — Thu, 18 Jan 2024 04:07:44 +0000

Introduction

Some times ago, I published a npx package create-sra(blog) - a powerful tool for building React applications integrated with webpack.But missed the mark on certain features like TypeScript integration and the flexibility to choose a preferred package manager. Moreover, let's face it, 'create-sra' (sra=>super react app) is kind of a mundane name.

Say hello to create-rp-app(rp=>reactopack).This update isn't just a change in name; it's a leap forward, delivering a refined, feature-rich experience. Now, creating a fast and secure React app is not only seamless but also customizable to your preferences.

In this blog post, I'll walk you through the evolution of create-rp-app, highlighting the reasons behind this transformation, the added functionalities that make it a game-changer, and guide you on effortlessly transitioning from create-sra to the latest and greatest in our npm package lineup.

Why the change

Enhancing Developer Experience:
The decision to evolve from create-sra to create-rp-app stems from our commitment to enhancing the developer experience. While create-sra was a robust tool for React application development, we wanted to make the process even more enjoyable and user-friendly.
Introducing Typescript Integration:
One of the standout features of create-rp-app is the seamless integration of Typescript. Recognising the growing popularity and benefits of Typescript in modern web development, we wanted to provide developers with a straightforward way to incorporate Typescript into their React projects.
Flexibility in Package Mangement:
create-sra was tied to a specific package manager(npm), limiting the choices available to developers. With create-rp-app, we've introduced a new level of flexibility, allowing you to choose your preferred package manager. Whether you're a fan of npm or yarn, create-rp-app adapts to your workflow.
Future-Proofing Your Project:
As technology evolves, so do the best practices and tools in the developer ecosystem. By migrating to create-rp-app, you're not only embracing the latest and greatest features but also future-proofing your projects. Stay ahead of the curve with a tool designed to adapt to the ever-changing landscape of web development.

Benefits of `create-rp-app`

Seamless Typescript Integration
Flexible Package Management
Tailored Project Configurations
Enhanced Developer Experience
Future-Proof Your Project

How to use

To use create-rp-app just open terminal and type

npx create-rp-app

Conclusion

In conclusion, the evolution from create-sra to create-rp-app marks a significant stride forward in the realm of React app development. The transition is designed to be smooth, and with it comes a host of new possibilities for your React projects.
And as a single developer, I believe in the power of collaboration, and I'm excited to invite you to join me in this ongoing journey. If you're passionate about React development and want to contribute, share ideas, or simply code together, your collaboration is more than welcome. Together, we can make React app development an even more exciting and rewarding experience.

Feel free to comment below with any questions, suggestions, or feedback.

Additional Resources:

Github - https://github.com/amitkroutthedev/create-reactopack
NPM pacakge - https://www.npmjs.com/package/create-rp-app

Happy Coding!!!

Elevating React: Unleashing the Power of Nginx for Effortless Deployment

Amit Kumar Rout — Sat, 09 Dec 2023 07:50:19 +0000

In the dynamic world of web development, efficient implementation of React applications is critical to providing a smooth user experience. One powerful tool that can greatly improve the deployment process is Nginx.
In this blog post, we'll explore the benefits of using Nginx to deploy React applications, introduce its features that make deployment easier, and provide a step-by-step guide to help you integrate Nginx into your React deployment pipeline.

Why Nginx?

Choosing Nginx for deploying React applications is a strategic decision rooted in its robust capabilities. Nginx's efficient load balancing ensures seamless distribution of user requests, optimizing performance, and preventing server overload. Its built-in caching mechanisms significantly enhance load times, providing users with a faster and more responsive experience. Moreover, Nginx's versatility as a reverse proxy and its ability to handle SSL termination make it a reliable choice for securing and scaling React applications. In summary, Nginx's powerful features make it a compelling and practical solution for elevating the deployment of React applications.

Integrating Nginx into Your React Deployment

Step-1: Installing Nginx in the VM server

sudo apt update
sudo apt install ngnix

After installation, to start the nginx service first adjust the firewall by -

sudo ufw app list
sudo ufw allow 'Nginx HTTP'
sudo ufw status

And to check if ngnix service status

systemctl status nginx

Step-2: Preparing UI

When the previous development is complete, run "npm build" to compile and optimize your React application for production and create an improved version ready for deployment.

Step-3: Adding build folder and setting up reverse proxy in nginx

Two input files required to change which can be found in /etc/nginx/ that are

sites-enabled/default
nginx.conf

$ cat /etc/nginx/sites-enabled/default

server {
    listen 80 default_server;
    listen [::]:80 default_server;

    root /usr/share/nginx/html;

    index index.html index.htm index.nginx-debian.html;

    server_name _;
    client_max_body_size 5M;
    location /static {
        alias /usr/share/nginx/html/static;
    }
     location /favicon.ico {
        alias /usr/share/nginx/html/favicon.ico;
    }
    location / {   //Root Location
            try_files $uri /index.html;
        }
    location /v1/ {  //Proxy to Backend Location         
    proxy_pass http://101.244.148.101:5003; 
        proxy_buffering off;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Port $server_port;

    }
}

In above code,

root /usr/share/nginx/html - Specifies the root directory for the server, where it will look for files to serve.
index index.html index.htm index.nginx-debian.html; - Defines the order in which index files are tried when a directory is accessed. For example, if a user requests a directory, Nginx will try to serve index.html first.
location / {} - This block is used to handle general requests. If the requested file doesn't exist, it falls back to serving /index.html.
location /v1/{} - This block is used to proxy requests with the path prefix /v1/ to a backend server at http://101.244.148.101:5003. It also sets some headers (X-Real-IP, X-Forwarded-Host, X-Forwarded-Port) to preserve client information when passing the request to the backend serve

After making changes, then we will make changes in nginx configuration file.

$ cat /etc/nginx/nginx.conf

Add this code block inside "http{}"

http{
...                  //default code 
add_header X-Frame-Options "SAMEORIGIN";
add_header Strict-Transport-Security "max-age=31536000; includeSubdomains; preload";
add_header Content-Security-Policy "default-src 'self' http: https: data: blob: 'unsafe-inline'" always;
add_header X-XSS-Protection "1; mode=block";
proxy_hide_header X-Powered-By;
etag off;
...                //default code
}

add_header X-Frame-Options "SAMEORIGIN";
- Specifies the X-Frame-Options header to mitigate clickjacking attacks.
add_header Strict-Transport-Security "max-age=31536000; includeSubdomains; preload"
- Configures HTTP Strict Transport Security (HSTS) to enforce secure connections over HTTPS.
add_header Content-Security-Policy "default-src 'self' http: https: data: blob: 'unsafe-inline'" always;
- Defines a Content Security Policy (CSP) to mitigate cross-site scripting (XSS) attacks.
add_header X-XSS-Protection "1; mode=block";
- Enables the browser's built-in Cross-Site Scripting (XSS) protection.
proxy_hide_header X-Powered-By;
- Hides the X-Powered-By header in responses from proxied servers.
- Enhances security by reducing information about the server software.
etag off;
- Disables the use of Entity Tags (ETags) in HTTP responses.
- ETags are used for web cache validation; disabling them can improve performance and reduce unnecessary server load.

After saving the file, copy for build folder content inside /usr/share/nginx/html

Restart the nginx service by - sudo systemctl reload nginx

You can now access your website or application through your browser.

If any issues arise, check the Nginx error logs for potential error messages.

Conclusion:

By seamlessly integrating Nginx into your React deployment strategy, you unlock a suite of powerful tools that not only boost performance and security but also scale your application seamlessly. This post offers a comprehensive overview of Nginx's capabilities, coupled with a step-by-step guide for effortless integration into your React deployment pipeline.

Happy Coding!!

Created My First npx Package: "create-sra"

Amit Kumar Rout — Mon, 02 Oct 2023 04:48:09 +0000

Introduction

I am excited to introduce my first npx package, create-sra!This package was born out of a desire to streamline the process of creating Single-Page React Applications (SRAs) quickly and efficiently. You can find the package on npm here.SRA means "SUPER-REACT-APP"

Motivation

As a frontend developer, my go-to tools are React, often used with Create-React-App or NextJS. However, building projects with Create-React-App can be a slow process, and it sometimes introduces vulnerabilities. Recognizing these pain points, I identified an opportunity to simplify this process by crafting a tool that automates the initial setup, including webpack configuration. This not only saves time and effort but also grants users greater flexibility.

Package Overview

create-sra is a command-line tool that swiftly generates a basic Single-Page React Application, powered by webpack, with just a few straightforward commands. It establishes the essential files and folder structure based on the user's preferences, allowing you to dive straight into crafting your application logic.

Installation

To use create-sra just open terminal and type

npx create-sra

This will create a new directory and all the necessary files and dependencies to kickstart your project.

Key Features

-Quick Setup: create-sra establishes a basic SRA with a default configuration, allowing you to start coding right away.
-Built-in Development Server: The package includes a development server, enabling you to see your changes in real-time as you code.
-Clean React Project: No unnecessary packages are installed by default.
-No Vulnerabilities: There are no package vulnerabilities.
-CSS Framework Support: You can easily download various CSS frameworks.
-Superfast: Quickly create, install, and set up folders.

Conclusion

Creating create-sra has been an incredibly rewarding journey. It's gratifying to know that this package has the potential to save developers a significant amount of time and effort in setting up new React projects.
I encourage you to give create-sra a try and let me know your thoughts. Your feedback is invaluable and will play a crucial role in the package's ongoing development.

Feel free to comment below with any questions, suggestions, or feedback.

Happy Coding

Bun 1.0 ⚡:All-in-One Toolkit

Amit Kumar Rout — Wed, 13 Sep 2023 09:13:52 +0000

JavaScript has always been a dynamic language with a vibrant ecosystem. Over the years, we've seen a plethora of tools designed to make JavaScript development easier, faster, and more efficient.

Currently, there is a new development: Bun 1.0, which serves as a runtime, package manager, test runner, and bundler for JavaScript and TypeScript.

How fast is Bun compared to node?

To test it I am using

Node - v20.6.1
Bun - v1.0.0

1. Running a performance test

I am using this code to test performance

console.time('test');
for (let i = 0; i < 10000; i++) console.log(i)
console.timeEnd("test");

I have created a node file by npm init

Running the index.js script

I am getting around 154.889ms.

To create in bun, we will use bun init

Running the same index.js script

I am getting around 61.35ms⚡.

I have changed console logs to create difference between node and bun

2. Creating Ecosystem(create-vite-app, create-react-app)

We will run same commands in npx and bunx to measure creating time.

I created react project using create-react-app
$npx create-react-app node-text-react

$bun x create-react bun-text-react

Both are adding same number of packages(1455) but npx does it in 35s whereas bunx(bun x) does it in 19s⚡.

After CRA, I tried in vite project
$npm create vite@latest

$bun x create-vite vite-project-bun

Both added 262 pakages but node does it in 11s whereas bunx(bun x) does it in 1.2s⚡.

Then I tried creating next app with bun and node
$bun x create-next-app

$npx create-next-app@latest

Bun installed 321 packages in 7.85s⚡ where as node installed 325 packages in 22s.

Conclusion

Bun is very fast as compared with node and yarn, node, deno (according to documentation). It is based on Javascript Core Engine and Zig. Still there are some disadvantages like it can't run on Windows machine and also bugs as it is first stable release. With the introduction of Bun as a new contender in the Node ecosystem, developers now have viable alternatives to Node.

If you have any suggestions, please leave them in the comments section. If you found this post helpful, please like and share it.

Happy Coding!

Resetting Reducer State in React and Redux for Better State Management

Amit Kumar Rout — Fri, 26 May 2023 07:12:21 +0000

Introduction:
In React and Redux applications, managing state efficiently is crucial for optimal performance. One common issue developers face is the persistence of reducer state in useSelector hooks, causing unnecessary re-renders even when actions are not called. In this article, we'll explore the importance of resetting the reducer state and how to implement it in your application.

Section 1: Understanding the Problem
The useSelector hook is a powerful tool in Redux that allows components to access the Redux store's state. However, when the reducer state is not reset, useSelector can cause unnecessary re-renders, impacting performance.
Consider the following example:

import { useDispatch, useSelector } from "react-redux";
import {useState,useEffect} from "react";
import {addUser} from "../actions";

function Page(){
const dispatch = useDispatch()
const user = useSelector((state) => state.user);
const addUser = useSelector((state) => state.addUser);
//setting a snackbar for user to see the response on adding the new user
const [snackBarIsOpen, setSnackBarIsOpen]=useState(false);
const [snackMsg, setSnackBarMsg] = useState("");
const [usersData, setUsersData] = useState([]);
useEffect(() => {
  if (user.response) {
    setUserData(user.response.data)
  }
}, [user]);
useEffect(() => {
  if (addUser.response) {
    setSnackBarIsOpen(true)
    setSnackBarMsg(addUser.response.message)
  }
 if (addUser.error) {
    setSnackBarIsOpen(true)
    setSnackBarMsg(addUser.error.data.message)
  }
}, [addUser]);

function addUserButton(name){
   dispatch(addUser(name))       //dispatching the action
}

/*
//remaining code
*/

In the above example, we are retrieving a user list from reducers and rendering it on a Page along with an 'Add' button to add a user. If it is a new user, it will be added to the list with the message User added successfully.However, if it is a duplicate user, an error message Duplicate user not allowed will be displayed. In this scenario, a snackbar is used to show the messages. Additionally, we are considering the use of redux-persist to maintain the viewer's logged-in state.

The problem arises when a user is being added (let's consider a successful entry). The snackbar displays the success message retrieved from the useSelector hook. However, if the page is re-rendered, the snackbar still shows the message, even if the user has not been added.

Section 2: Resetting Reducer State
To prevent unnecessary re-renders, we can reset the reducer state after performing actions like delete, edit, or create. This ensures that the useSelector hook only triggers re-renders when relevant actions are called.

Section 3: Implementing Reset Actions
First let's create a reset state in reducer:


export const addUserReducer = (state = {}, action) => {
  switch (action.type) {
    case "addUser-api-request":
      return { loading: true }
    case "addUser-api-success":
      return { loading: false, response: action.payload }
    case "addUser-api-fail":
      return { loading: false, error: action.payload }
    case "addUser-api-reset": //added new case returning empty object
      return {}                 
    default:
      return state;
  }
};

Now, dispatch the reset action in our ui page

///rest of code 
<Snackbar
  anchorOrigin={{ "bottom", "left" }}
  open={snackBarIsOpen}
  autoHideDuration={3000}
  onClose={()=>{setSnackBarIsOpen(false);dispatch({type:'addUser-api-reset'})}}
  message={snackMsg}
/> 
</>
)}
export default Page;

Now, whenever the Snackbar is active and closes after 3 seconds, it will also empty the object in the addUser reducer, effectively resetting the state. You can also reset all states at once, but if it is only necessary to reset one state, this is the recommended approach.

Conclusion:
Resetting the reducer state in React and Redux applications is essential for efficient state management and preventing unnecessary re-renders. By implementing reset actions and handling them in your reducers, you can improve your application's performance and ensure a smoother user experience. Give this technique a try in your projects and see the difference it makes!

If you have any suggestions for improving the code, please leave them in the comments section. If you found this post helpful, please like and share it.

Happy Coding!

How to protect routes for different user roles with restricted access?

Amit Kumar Rout — Tue, 18 Apr 2023 07:05:47 +0000

Introduction
Web applications often require different levels of access for various users. For instance, an administrator may need access to specific features that a regular user may not. One way to manage user access in your web application is by setting up protected routes.

In React, we use the 'react-router' package to set up public and protected routes. To install 'react-router' in a React application, you can use the following command: npm i react-router.

Prerequisites

Knowledge on React,React-Router
node > = 16.19.0
react >= 18.2.0
react-router-dom >= 6.9.0

Protected Route
In most cases, for protected routes, users are logged out automatically if they are not authenticated. However, in a role-based application, if an authenticated user of a different role tries to access a route that is not permitted, the protection mechanism may be violated.

The goal is to redirect the user back to the previous URL if they try to access a route that is not permitted for their role.
Firstly, the project file structure

In PageRouter.js



import {
    BrowserRouter as Router,
    Route,
    Routes,
} from "react-router-dom";

import LoginPage from "../pages/LoginPage";
import UserRegistration from "../pages/UserRegistration";
import AdminPage from "../pages/admin/AdminPage"
mport UserPage from "../pages/user/UserPage"

export const PageRouter = () => {

    return (
        <Router>
            <Routes>
                <Route exact path="/" element={<LoginPage />} />
                <Route exact path="/userReg" element={<UserRegistration />} />
                <Route exact path="/adminPage" element={<AdminPage />} />
                <Route exact path="/userPage" element={<UserPage />} />
            </Routes>
        </Router>
    )
}

Import PageRouter.js in app.js

Currently, there are two roles - "Admin" and "User" - for which we have created two files, ProtectedRouteAdmin.js and ProtectedRouteUser.js.

In ProtectedRouteAdmin.js



import {useEffect} from "react";
import { Route,Redirect, Navigate, Outlet, useNavigate, useLocation} from "react-router-dom";
import jwtDecode from "jwt-decode";
import axios from "axios";

const ProtectedRouteAdmin = (props) => {
  const token = localStorage.getItem("token");
  const navigate = useNavigate();
  function presentPage() {
    navigate(-1);
  }

  if (!token) return <Navigate to="/" />;

  useEffect(()=>{
    if(token && jwtDecode(token).role!== "admin"){ 
      presentPage()
      }
  },[token && jwtDecode(token).role!== "admin"])

  const decodedData = jwtDecode(token);


  if (decodedData.role === "admin") {
    return <Outlet {...props} />;
  }
 else if(decodedData.role!=="admin"){
   presentPage()
  }
};

export default ProtectedRouteAdmin;

And in ProtectedRouteUser.js



import {useEffect} from "react";
import { Route,Redirect, Navigate, Outlet, useNavigate, useLocation} from "react-router-dom";
import jwtDecode from "jwt-decode";
import axios from "axios";

const ProtectedRouteUser = (props) => {
  const token = localStorage.getItem("token");
  const navigate = useNavigate();
  function presentPage() {
    navigate(-1);
  }

  if (!token) return <Navigate to="/" />;

  useEffect(()=>{
    if(token && jwtDecode(token).role!== "user"){ 
      presentPage()
      }
  },[token && jwtDecode(token).role!== "user"])

  const decodedData = jwtDecode(token);


  if (decodedData.role === "user") {
    return <Outlet {...props} />;
  }
 else if(decodedData.role!=="admin"){
   presentPage()
  }
};

export default ProtectedRouteUser;

Updating code in PageRoute.js



//other imports
import ProtectedRouteAdmin from "./ProtectedRouteAdmin";
import ProtectedRouteUser from "./ProtectedRouteUser";

export const PageRouter = () => {
    return (
        <Router>
            <Routes>
                <Route exact path="/" element={<LoginPage />} />
                <Route exact path="/userReg" element={<UserRegistration />} />
               <Route element={<ProtectedRouteAdmin/>}>
                  <Route exact path="/adminPage" element={<AdminPage />} />
</Route>
               <Route element={<ProtectedRouteUser/>}>
                  <Route exact path="/usPage" element={<AdminPage />} />
</Route>
            </Routes>
        </Router>
    )
}

Conclusion
The protected routes have been set up. Test them to ensure they are working correctly by logging in with different user roles and checking if they have access to the appropriate routes. If necessary, you can add Redux to check authorized routes.
I hope this helps you to write Protected Routes in a better way! If you have any suggestions for improving the code, please leave them in the comments section. If you found this post helpful, please like and share it.

Happy Coding!

Building my first VS Code Extension

Amit Kumar Rout — Mon, 10 Apr 2023 11:57:57 +0000

I recently built my first VS Code extension and published it. It consists of basic code snippets that include commonly used React packages.

What are Code Snippets?
Code snippets are reusable blocks of code that can be inserted into your code editor with a single command or keyboard shortcut. Snippets can be used to save time by automating common tasks, such as creating boilerplate code or filling in repetitive code patterns. Snippets can also improve code quality by ensuring that code adheres to best practices or standards.

Building a Code Snippets Extension

Install the yo and generator-code packages using the following command:

npm install -g yo generator-code

Create a new code snippets extension using the following command:

yo code

Follow the prompts to configure your extension. When prompted, select "New Code Snippets" as the project type.

The VS Code snippet editor will open. Here, you can define your code snippets using a JSON format.
Define your code snippets in the 'prefix', 'body', and 'description' properties within the 'snippets.code-snippets' file in the 'snippets' folder
Test your code snippets extension using the VS Code debugger.

Publishing VS Code extension

Install vsce with npm install -g vsce
Create an account in VS Code MarketPlace and create an organisation
Add the publisher name in package.json file

  "publisher": "publisherName",

Run the command

vsce package

It will create a file in .vsix.
If it warns for empty repository the add

"repository": {
    "private": true
  },

and if it ask for license then add a LICENSE.md file

Click on "+ New Extension" and add .vsix file.
It will take some time for verifications and finally available in marketplace

Extension in VS Code Marketplace

Conclusion

Building a code snippets extension on VS Code is a great way to share your code snippets with others and improve your own productivity. With a few simple steps, you can create a custom extension that automates common tasks and adheres to best practices.

Hope you use this extension, like it, and review it. If you want to add more code snippets, please comment on the post.

Happy Coding!

DEV Community: Amit Kumar Rout

How to Open a 14 GB File in Under a Second — The Lazy Loading Technique Most Developers Never Use

The Naive Approach and Why It Fails

The Insight: Knowing Where Data Is Versus Having the Data

Building the Index

On-Demand Reading

A Tool That Makes This Efficient: Memory Mapping

The Tradeoffs — Being Honest

The Mental Model Shift

Applying This to Your Own Tools

Event-Driven Architecture in Node.js: Building Scalable and Responsive Systems

What is Event-Driven Architecture?

Core Components

Why Node.js is Ideal for Event-Driven Architecture

Getting Started: EventEmitter Basics

Basic EventEmitter Usage

Creating Custom Emitter Classes

Real-World Example: Order Processing System

Basic Implementation

Key Advantages Demonstrated

Handling Multiple Event Listeners and Event Namespacing

Event Namespacing

Removing Event Listeners

Listening Once

Distributed Event-Driven Architecture with Message Brokers

Redis Pub/Sub Implementation

RabbitMQ Implementation

Apache Kafka Implementation

Error Handling in Event-Driven Systems

Basic Error Handling

Dead Letter Queues

Best Practices for Event-Driven Architecture

1. Define Clear Event Schemas

2. Keep Event Payloads Small

3. Handle Event Versioning

4. Ensure Idempotency

5. Implement Proper Monitoring and Logging

Common Pitfalls and How to Avoid Them

Pitfall 1: Creating a "Distributed Ball of Mud"

Pitfall 2: Not Handling Event Failures

Pitfall 3: Tight Coupling Between Event Producers and Consumers

Pitfall 4: Infinite Event Loops

Performance Considerations

Event Throttling

Event Batching

Comparing Event-Driven Approaches

Conclusion

LangCompiler: A Cloud-Native, High-Performance Code Execution Service

Introduction

What is LangCompiler?

Key Features

Built with Golang and Docker

Try LangCompiler Today!

Dockerizing Your Frontend Application with Nginx for Seamless Deployment

Introduction

Prerequisites

Setting Up the Application

Elevating React: Unleashing the Power of Nginx for Effortless Deployment

Amit Kumar Rout ・ Dec 9 '23

Steps:

Building the Docker Image

Running the Docker Image

Option 1: Running the image directly

Option 2: Running the Docker Compose

Conclusion

Implementing Redux in Next.js (App Router): A Comprehensive Guide

Introduction

Prerequisites

Folder Structure

Conclusion

Upgrading to 'create-rp-app': A Better way to create fast react app

Introduction

Why the change

Benefits of create-rp-app

How to use

Conclusion

Additional Resources:

Elevating React: Unleashing the Power of Nginx for Effortless Deployment

Why Nginx?

Integrating Nginx into Your React Deployment

Benefits of `create-rp-app`