DEV Community: Asad Zaman

Scaling a Keyword Marking System: Partitioned vs Non-Partitioned Database Approaches

Asad Zaman — Wed, 23 Jul 2025 16:55:02 +0000

Scalable Keyword Marking System

Building a realtime keyword highlighting system for millions of users and billions of keywords presents significant architectural challenges. In this document I will try to address database partitioning and efficient keyword matching.

Problem Overview

1 Million Users: Each user can store up to 10,000 keywords.
10 Billion Keywords: Assume total system capacity.
Real-time Requirement: Sub-second retrieval and highlighting of all user keywords on any given page, including overlapping matches (e.g., "Dhaka," "Dhaka City").

Source code

https://github.com/asadpstu/keyword_match

Database: PostGres

You can run as a docker service. Just copy and paste in the terminal. Within a few moment, your pgres database will be up and running

docker run -d --name my-postgres-db --network my_pg_network -p 5434:5432 -e POSTGRES_DB=mydatabase -e POSTGRES_USER=myuser -e POSTGRES_PASSWORD=mypassword -v pg_data:/var/lib/postgresql/data postgres:16-alpine

Download dbever or you can use your favorite db tools.

`CREATE TABLE keywords (user_id BIGINT NOT NULL,keyword TEXT NOT NULL,PRIMARY KEY (user_id, keyword)) PARTITION BY HASH (user_id);`

Creating partitions of keywords table e.g. 256 partitions

DO $$
BEGIN
  FOR i IN 0..255 LOOP
    EXECUTE format('CREATE TABLE keywords_p%s PARTITION OF keywords FOR VALUES WITH (MODULUS 256, REMAINDER %s);', i, i);
  END LOOP;
END;
$$;

I am attaching some queries too. It might come handy if you feel it is worthy too practice.

SELECT * FROM keywords WHERE user_id = 1;
SELECT * FROM keywords_p125 WHERE user_id=1;

SELECT tableoid::regclass AS partition_name, *
FROM keywords
WHERE user_id = 1;

EXPLAIN SELECT user_id, keyword FROM keywords WHERE user_id=1;

Why Database Partitioning is important

Without partitioning, a single, massive table may lead to:

Slow Queries: Indexing and scanning across billions of rows become prohibitive.
Large, Inefficient Indexes: Unable to fit in memory, causing I/O bottlenecks.
High Maintenance Overhead: VACUUM, ANALYZE, REINDEX operations are lengthy and disruptive.
Slow Data Management: Deleting old user data is resource-intensive.

With hash partitioning on user_id (e.g., into 256 partitions), we achieve:

Fast Queries: Queries only hit a relevant, small partition (1/256th of the data).
Smaller, In-Memory Indexes: Faster lookups.
Efficient Maintenance: Operations run quickly per partition, with less impact.
Instant Data Pruning: Dropping old user data by dropping entire partitions is instantaneous.
Enhanced Parallelism: Concurrent access and maintenance across partitions improve throughput.

This directly translates to sub-second query times even at massive scale, unlike the multi-second or more queries seen without partitioning.

Keyword Matching: The Role of Aho–Corasick Algorithm (Trie Data structure)

Retrieving keywords is just one part. Efficiently matching thousands of keywords against page content is solved by the Aho–Corasick algorithm. This algorithm builds a Trie of keywords, enabling linear-time scanning of text to detect all occurrences, including overlapping matches, which is vital for real-time performance.

Summary: Partitioning vs. Non-Partitioning

Feature	Without Partitioning	With Partitioning
Query Speed	Degrades with scale (seconds+)	Stays fast (sub-second) due to smaller scope
Index Size	Huge, inefficient, disk-bound	Smaller, efficient, fits in memory per partition
Maintenance Ops	Slow, blocking	Fast, isolated, less impactful
Data Pruning	Painfully slow row-by-row	Instant (drop partition)
Scalability	Quick bottlenecks	Efficient, leverages parallelism

Conclusion

Database partitioning, especially by user_id, combined with an efficient keyword matching algorithm like Aho–Corasick can help building a performant and scalable keyword marking system. I would like to suggest partitioning your table early for any system that may handle large scale per user data.

Handling Race Conditions Using Node.js — Yes, You Read It Right.

Asad Zaman — Mon, 21 Apr 2025 15:13:07 +0000

Have you ever thought about what could happen if two people tried to withdraw from the same bank account at the exact same time?

That’s a classic case of a race condition — and if not handled properly, it can lead to real money vanishing (or duplicating!) out of thin air.

In this post, I’ll walk you through:

What a race condition is (in the context of banking)
How to simulate one in Node.js
How to fix it using a mutex (lock).

What Is a Race Condition?

A race condition occurs when two or more operations access shared data at the same time, and the final result depends on the order in which they execute.

In banking terms:
Two people transfer money from the same account at the same time. If not handled properly, both could think the money is still there — and withdraw it — causing the account to go negative or become inconsistent.

Simulating the Problem in Node.js

Let’s say we have a shared in-memory balance of $100, and two transfer requests come in at the same time:

const express = require('express');
const app = express();
const port = 3000;
let accountBalance = 100; 

function transfer(amount) {
    const balance = accountBalance;
    if (balance < amount) {
        throw new Error('Insufficient funds');
    }
    // Simulate some delay in processing. It can be many reason
    setTimeout(() => {
        accountBalance = balance - amount;
        console.log(`Balance after transfer: $${accountBalance}`);
    }, 1000);
}

app.post('/transfer', (req, res) => {
    const amount = 50;
    try {
        transfer(amount);
        res.send('Transfer completed');
    } catch (error) {
        res.status(400).send(error.message);
    }
});

app.listen(port, () => {
    console.log(`Server running on http://localhost:${port}`);
});

From your terminal -

curl -X POST http://localhost:3000/transfer & curl -X POST http://localhost:3000/transfer &

Did you notice? You still have 50$!
This is broken! If two requests run simultaneously, both read $100, both think there's enough, and both deduct $50 — ending up with a final balance of $50, when it should be $0.

Let's fix the problem -

const { Mutex } = require('async-mutex');
const mutex = new Mutex();

async function transfer(amount) {
    const release = await mutex.acquire();
    try {
        const currentBalance = accountBalance;
        if (currentBalance < amount) {
            throw new Error('Insufficient funds');
        }
        await new Promise(resolve => setTimeout(resolve, 100));
        accountBalance = currentBalance - amount;
        console.log(`Transfer successful. New balance: $${accountBalance}`);
    } finally {
        release();
    }
}

Now, only one transfer runs at a time. The second request waits until the first finishes. And you may have noticed - your balance is now zero.

This is a hypothetical example, but race conditions like this do happen in real-world applications — such as counting page views, managing product stock in e-commerce platforms, and more. The scary part? They often go unnoticed until they cause serious issues.

Optimized Email lookup for millions of users

Asad Zaman — Sat, 12 Apr 2025 18:26:59 +0000

Fast, Scalable, and Reliable — How to solve the email registration problem at scale for millions of users without compromising on performance or data integrity.so, here’s the deal.

When you’re dealing with millions of users, one tiny problem starts eating your system alive — checking if an email already exists. It sounds small, but if you're doing it 10 million times? Boom, memory gone. Performance drops. It is not good.

But don’t worry. We got this. Let’s mix a bit of RedisBloom and PostgreSQL, and build a system that’s fast, scalable, and doesn't chew up all your memory.

RedisBloom: The probabilistic memory-saver

Memory efficiency: It doesn’t store the full emails. It just uses a bit array and some math magic (Bloom Filter logic). So, 10 million emails = only ~30MB memory. Yup. Not kidding.
Crazy fast: Check if an email exists in constant time, O(1). Like blink-of-an-eye fast.
Lightweight: No need to store actual emails in Redis. Less load, more speed.\
Downside? Well... false positives are possible (~1%).

RedisBloom’s cool, but to be honest — 1% false positive is still a problem. So, when RedisBloom thinks the email is already there, we just double-check in PostgreSQL.

PostgreSQL: The Trusty Double Checker

Accurate: It’s our final truth source.
Fast checks: We use SELECT 1 instead of pulling the whole email to keep things lean.

The Workflow: How it all comes together

Check RedisBloom: Check if the email might exist. If RedisBloom says no — we’re good. Move forward and register.
Check PostgreSQL: If RedisBloom says “maybe”, we run a simple SELECT 1 in Postgres. Just to be safe.
Insert into Both: If the email doesn’t exist in Postgres, insert it in both Postgres and RedisBloom to keep them in sync.

Why it will work

Efficient: RedisBloom handles most checks. Saves RAM. Super snappy.
Reliablity: PostgreSQL catches the rare false positive. You get 100% accuracy.
Scalable: Works smooth with 100 or 100 million users. Future-proof.
Secure: We hash emails (SHA-256) so we’re not storing plain data anywhere.

Real World Impact

If you’re dealing with around 10 million emails, using RedisBloom can literally save you hundreds of megabytes of memory. Instead of storing every email directly, RedisBloom squeezes all that down into around 30MB. That’s crazy efficient.

On the other hand, PostgreSQL still does its job reliably, but only when we really need it. We’re not hammering the database for every check—only when RedisBloom gives us a “maybe.” And even then, we’re just doing a lightweight SELECT 1, which is super fast and doesn’t pull any unnecessary data.

For performance — each RedisBloom check is almost instant, like around 0.001 to 0.005 milliseconds. PostgreSQL might take a tad longer, maybe 0.005 to 0.02 milliseconds, but again, that’s only for a small chunk of the checks. Combine them and even if you do 10 million checks, it’s done in under a few minutes. So,

You're saving a lot of memory
You're not burning your database with unnecessary query
our system stays fast and clean even at a huge scale.

Boom. That’s what good architecture looks like.

SHA-256: Keeping emails private

Don’t save raw emails. Before storing anything, we hash each email using SHA-256, so even if someone breaks into your DB, they won’t be able to read the email addresses. Please check the following code, I hope this is understandable.

async function **emailExists**(email: string): Promise\<boolean> {
    const hash = hashEmail(email);
    const mightExist = await redis.bf.exists('email\_bloom', hash);
    if (!mightExist) return false;
    const res = await pg.query(
        'SELECT 1 FROM registered\_emails WHERE email\_hash = $1', \[hash]
    );
    return res.rowCount > 0;
}

async function **registerEmail**(email: string) {
    if (await emailExists(email)) {
      console.log('Email already exists.');
      return;
    }
    const hash = hashEmail(email);
    await pg.query(
       'INSERT INTO registered\_emails (email\_hash) VALUES ($1)', \[hash]
    );
    await redis.bf.add('email\_bloom', hash);
    console.log('Email registered!');

}

Summary

Use RedisBloom for fast, memory-efficient lookups.
Use PostgreSQL for accurate, final checks.
Hash your emails with SHA-256 for privacy.
Get the best of both worlds: speed and integrity.

This setup is battle-tested. It’s like the perfect combo of fast-n-loose and strict-n-safe. It works at scale, it respects your resources, and your users are protected.

Give it a try. Scale with confidence.