DEV Community: Tannu Goyal

CONSISTENT HASHING -Through the Architecture of Amazon Dynamo

Tannu Goyal — Wed, 11 Mar 2026 15:02:49 +0000

This was just a quick overview. Read the full blog here

Consistent Hashing in Distributed Systems

When systems scale to millions of users, storing data on a single database becomes impossible.

Problems appear quickly:

Storage limits
Performance bottlenecks
Single point of failure
Expensive vertical scaling

The solution is horizontal scaling — distributing data across multiple servers.

But this introduces a new question:

How do we decide which server should store a particular piece of data?

Traditional Hashing

A simple approach is:

node = hash(key) % N

Where:

key → userId, productId, etc.
N → number of servers

This distributes keys fairly well across servers.

However, it has a major flaw.

If the number of servers changes, most keys get reassigned.

At large scale, this can trigger massive cache misses and overload databases.

Consistent Hashing

Consistent hashing solves this by mapping both servers and keys onto a circular hash ring.

A key is assigned to:

The first server encountered while moving clockwise on the ring.

The big advantage:

Adding or removing a server only affects a small portion of keys
Most of the system remains untouched

This makes scaling far more stable.

Virtual Nodes

Basic consistent hashing can create uneven load.

Systems like Amazon Dynamo solve this using virtual nodes.

Instead of placing each server once on the ring, the system places it multiple times, creating better load distribution and smoother rebalancing.

Where It’s Used

Consistent hashing powers many large-scale systems:

Distributed databases (Cassandra, DynamoDB)
Distributed caches (Memcached, Redis)
CDNs and load distribution systems

It’s one of the key ideas that makes modern distributed infrastructure scalable.

CQRS - Design Pattern

Tannu Goyal — Mon, 09 Mar 2026 07:51:14 +0000

When you open TikTok, your feed loads instantly. A new video appears with every swipe, perfectly tailored to your interests.

But behind that smooth experience lies a massive distributed system handling millions of requests per second.

The challenge is simple but brutal: TikTok is an extremely read-heavy system.

For every user uploading a video, liking content, or posting a comment, thousands of others are simply scrolling.

Reads : Writes ≈ 1000 : 1

Traditional CRUD architectures struggle with this kind of workload. When reads and writes share the same database model, the system faces issues like:

database locking under high concurrency
heavy indexing slowing down writes
complex joins for read queries
difficulty scaling reads independently

To solve this, large-scale platforms often adopt CQRS (Command Query Responsibility Segregation).

CQRS separates write operations (commands) from read operations (queries) so each side can be optimized independently.

In a TikTok-like architecture:

Commands handle actions like publishing a video, liking a video, or posting a comment.
These actions generate events that flow through an event stream such as Kafka.
Multiple services consume those events and update specialized read models.
The read side uses optimized stores like Redis for feeds, Elasticsearch for search, and document databases for metadata.

This architecture allows the system to handle massive traffic while keeping user interactions fast.

I wrote a full deep dive explaining how this works, including:

Why traditional architectures break at scale
How CQRS separates reads and writes
Using Kafka to synchronize read models
Event Sourcing and event-driven architectures
How TikTok builds and serves its feed pipeline

👉 Read the full article here

Node.js Event Loop: A Deep Dive

Tannu Goyal — Fri, 20 Oct 2023 18:57:46 +0000

Node.js is celebrated for its non-blocking, asynchronous architecture, and at the heart of this magic is the Event Loop. In this blog, we'll explore how the Event Loop works and why it's integral to Node.js's performance.

What is the Event Loop?
Phases of the Event Loop
Why It Matters
Common Misconceptions
Best Practices
Conclusion

What is the Event Loop?

The Event Loop is the mechanism that allows Node.js to perform non-blocking I/O operations, despite the fact that JavaScript is single-threaded. By offloading operations to the system kernel whenever possible, it ensures the efficiency of our applications.

fs.readFile('/file.txt', (err, data) => {
    if (err) throw err;
    console.log(data);
});

console.log('Reading file...');

In Node.js, the system doesn't wait for file reading to complete. Instead, it continues with the next operation (console.log('Reading file...')), and processes the callback once the reading is done.

Phases of the Event Loop

The Event Loop in Node.js goes beyond just a simple loop. It encompasses various phases:

Timers Phase: Handles callbacks from setTimeout and setInterval.
Pending Callbacks Phase: Executes I/O callbacks deferred from the last cycle.
Idle, Prepare Phase: This phase is only for internal use.
Poll Phase: Fetches new I/O events and runs their callbacks.
Check Phase: Processes callbacks from setImmediate().
Close Callbacks Phase: Manages close events like socket.on('close', ...).

Why It Matters

Understanding the Event Loop is crucial:

Performance Optimization: It aids in maximizing the efficiency of your application.
Debugging: Assists in debugging asynchronous operations.
Insights: Gives a deeper understanding of the inner workings of Node.js.

Common Misconceptions

Node.js is Multithreaded: Though Node.js can utilize threads via the worker_threads module, the primary code execution remains single-threaded.
Priority of setImmediate() vs setTimeout(): Their priority relies on the event loop phase they are invoked from.

Best Practices

Avoid Blocking the Event Loop: Intense CPU operations can hinder the event loop. It's advisable to shift them to worker threads or utilize a job queue.
Use Async/Await: Employing async/await can simplify the structure of modern asynchronous operations.
Monitor the Event Loop: Tools such as node-clinic are invaluable for overseeing your event loop's health.

Conclusion

The Event Loop is Node.js's secret ingredient enabling its non-blocking behavior. Grasping its mechanics not only elevates your skills as a developer but also empowers you to develop effective and scalable solutions.

Happy coding!

Implementing Authentication in Node.js

Tannu Goyal — Fri, 20 Oct 2023 18:52:30 +0000

Authentication is a crucial aspect of web applications, especially when building a blog where users need to create accounts, log in, and manage their content securely. In this tutorial, we'll walk you through implementing authentication in a Node.js blog using the popular authentication library, Passport.js, along with a MongoDB database for storing user information.

Prerequisites

Before we begin, make sure you have the following prerequisites installed:

Node.js: You can download and install Node.js from nodejs.org.
MongoDB: Install MongoDB from mongodb.com.
A code editor of your choice (e.g., Visual Studio Code).

Setting Up the Project

Initialize Your Project: Create a new directory for your project and run the following command to initialize a Node.js project with npm:
```
npm init -y
```

Install Dependencies: Install the necessary packages:

npm install express passport passport-local express-session mongoose bcrypt

Create Project Structure:
Create the following project structure:

├── app.js
├── models
│   └── User.js
├── routes
│   ├── auth.js
│   └── index.js
├── views
│   ├── login.ejs
│   ├── register.ejs
│   ├── dashboard.ejs
│   └── home.ejs
├── public
│   └── styles.css
└── .env

Initialize MongoDB Connection:
In app.js, set up your MongoDB connection using Mongoose:
```
const mongoose = require('mongoose');

mongoose.connect('mongodb://localhost/nodejs-blog-auth', {
  useNewUrlParser: true,
  useUnifiedTopology: true,
});
```
Replace 'mongodb://localhost/nodejs-blog-auth' with your MongoDB connection string.

Implementing Passport.js for Authentication

Configure Passport.js: In app.js, configure Passport.js for authentication:

const passport = require('passport');
const LocalStrategy = require('passport-local').Strategy;

// Initialize Passport
app.use(passport.initialize());
app.use(passport.session());

Create User Model: Define a User model in models/User.js:

const mongoose = require('mongoose');
const Schema = mongoose.Schema;

const userSchema = new Schema({
  username: String,
  password: String,
});

module.exports = mongoose.model('User', userSchema);

Implement Passport Local Strategy: Create a Passport Local Strategy for user authentication in config/passport.js:

const LocalStrategy = require('passport-local').Strategy;
const User = require('../models/User');

module.exports = function (passport) {
  passport.use(
    new LocalStrategy((username, password, done) => {
      User.findOne({ username: username }, (err, user) => {
        if (err) return done(err);
        if (!user) return done(null, false, { message: 'Incorrect username' });
        if (user.password !== password)
          return done(null, false, { message: 'Incorrect password' });

        return done(null, user);
      });
    })
  );

  passport.serializeUser((user, done) => {
    done(null, user.id);
  });

  passport.deserializeUser((id, done) => {
    User.findById(id, (err, user) => {
      done(err, user);
    });
  });
};

Create Authentication Routes: Create authentication routes for login and registration in routes/auth.js:

const express = require('express');
const router = express.Router();
const passport = require('passport');
const User = require('../models/User');

// Register
router.post('/register', (req, res) => {
  // Implement user registration logic here
});

// Login
router.post(
  '/login',
  passport.authenticate('local', {
    successRedirect: '/dashboard',
    failureRedirect: '/login',
    failureFlash: true,
  })
);

module.exports = router;

Create Views for Authentication: Create login and registration forms in the views directory (login.ejs and register.ejs).

Set Up Routes:
In routes/index.js, set up your main application routes, including the authentication routes:

const express = require('express');
const router = express.Router();

// Define your routes here

module.exports = router;

Include Passport Configuration: Include the Passport configuration in app.js:

require('./config/passport')(passport);

Start the Server: Start your Express server in app.js:

const PORT = process.env.PORT || 3000;

app.listen(PORT, () => {
  console.log(`Server is running on port ${PORT}`);
});

Conclusion
In this tutorial, we've laid out the foundation for implementing authentication in a Node.js blog using Passport.js and MongoDB. You can now expand upon this foundation to add user registration, password reset, and user profile functionality. Remember to handle security best practices, such as hashing passwords securely and implementing user input validation.

Feel free to explore more features like user roles, JWT authentication, and authorization to enhance the security and functionality of your Node.js blog.