Generating Millions of Unique IDs in TypeScript — Fast, Scalable & Collision-Free

✨ Why I Wrote This

While scaling a backend service recently, I hit a challenge — generating billions of unique IDs efficiently, without collisions, and without blowing up memory.
UUIDs were too long, and auto-increment IDs broke under distributed systems.

That’s when I explored Snowflake algorithms and combined them with Base62 encoding — producing short, globally unique IDs that can be generated in crores per minute.

Here’s what I learned 👇

---

1. Introduction

Ever wondered how Twitter, YouTube, or TikTok generate billions of unique IDs every day — instantly and safely?

Traditional methods like UUID or AUTO_INCREMENT are not scalable across distributed databases.
As systems grow, we need an ID generator that’s:

• ⚡ Fast
• 🧠 Memory-efficient
• 🧩 Collision-free
• 🔢 Compact & Sortable

Let’s explore how to build one — and even generate crores of unique IDs in seconds using TypeScript.

---

🧠 2. Key Requirements for a Scalable ID Generator

An ideal ID generator should be:

• Unique — zero collision risk
• Sortable — time-based for sequencing
• Fast — able to generate millions/sec
• Compact — fixed length (~13 chars)
• Flexible — predictable or opaque

---

⚙️ 3. Popular Approaches

a) UUID / NanoID

• ✅ Simple, plug-and-play
• ❌ Not sortable
• ❌ UUIDv4 = 36 chars long
• ⚡ NanoID shorter but slower at massive scale

b) Snowflake Algorithm (used by Twitter, TikTok)

Snowflake generates a 64-bit integer using:

• Timestamp
• Machine ID
• Process ID
• Sequence Counter

It’s ultra-fast, distributed-safe, and scales horizontally across servers.

Example in TypeScript:

class Snowflake {
  private lastTimestamp = 0n;
  private sequence = 0n;
  private machineId: bigint;

  constructor(machineId: number) {
    this.machineId = BigInt(machineId) & 0b11111n; // 5 bits
  }

  private timestamp() {
    return BigInt(Date.now());
  }

  nextId() {
    let now = this.timestamp();
    if (now === this.lastTimestamp) {
      this.sequence = (this.sequence + 1n) & 0b111111111111n; // 12 bits
      if (this.sequence === 0n) now++;
    } else {
      this.sequence = 0n;
    }
    this.lastTimestamp = now;

    const id =
      ((now - 1700000000000n) << 17n) | (this.machineId << 12n) | this.sequence;
    return id.toString();
  }
}

---

🔢 4. Enhancing Snowflake with Base36/Base62 Encoding

To make IDs shorter, URL-safe, and more random-looking, encode them in Base62.

function toBase62(num: bigint): string {
  const chars = "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ";
  let str = "";
  while (num > 0n) {
    const rem = num % 62n;
    str = chars[Number(rem)] + str;
    num = num / 62n;
  }
  return str.padStart(13, "0"); // fixed length 13
}

Now combine both:

const snow = new Snowflake(1);
const id = toBase62(BigInt(snow.nextId()));
console.log(id); // e.g. "00bX5ztuG8p3L"

✨ Result:
Short, unique, URL-safe, and crazy fast 🔥

---

⚡ 5. Performance Comparison

Method	Length	Sortable	Collision Risk	Speed	Memory Use
UUIDv4	36	❌	Low	Medium	Medium
NanoID	~21	❌	Low	Medium	Medium
Snowflake	19 digits	✅	Very Low	⚡ Very Fast	Low
Snowflake + Base62	11–13 chars	✅	Very Low	⚡⚡ Blazing	Low

---

🔒 6. Predictability vs Randomness

• Snowflake IDs are time-sequential → predictable but easy to sort.
• Adding Base62 encoding makes them opaque.
• For extra randomness, add a salt or hash layer (SHA1 / MD5 hash slice).

---

🌍 7. Real-world Usage

• 🐦 Twitter, 💬 Discord, 📸 Instagram → use Snowflake-like systems
• 🔗 TinyURL, Bitly → use Base62 for compact links
• ☁️ Firebase / Firestore → time-based + random push IDs

---

🧮 8. Generating in Chunks (for Crores of IDs)

You can generate crores of IDs efficiently by batching them into chunks.
This avoids memory overload and stays lightning fast ⚙️

import fs from "fs";

const snow = new Snowflake(1);
const CHUNK_SIZE = 100_000; // 1 lakh IDs
const TOTAL = 10_000_000;   // 1 crore IDs

const output = fs.createWriteStream("ids.txt", { flags: "a" });

async function generateIDs() {
  console.time("Generation");
  for (let i = 0; i < TOTAL; i += CHUNK_SIZE) {
    let chunk = "";
    for (let j = 0; j < CHUNK_SIZE; j++) {
      const id = toBase62(BigInt(snow.nextId()));
      chunk += id + "\n";
    }
    output.write(chunk);
    console.log(`✅ Generated ${i + CHUNK_SIZE} IDs`);
  }
  output.end();
  console.timeEnd("Generation");
}

generateIDs();

💡 This can easily scale to tens of crores of unique IDs with constant memory usage.

---

🧭 9. Best Choice

Use Case	Recommended Method
Distributed systems (Twitter, Discord)	Snowflake + Base62
Small apps / prototypes	NanoID
Human-readable short links	Random Base62 + Collision Check

---

🏁 10. Conclusion

A strong ID generation system is the backbone of scalability.

By combining Snowflake with Base62 encoding, we get IDs that are:

• ⚡ Fast
• 🧩 Compact
• 🌍 Globally unique
• ⏱️ Time-sortable

Perfect for e-commerce, URL shorteners, and microservice architectures.

💬 Would you like me to write a follow-up on multi-threaded ID generation (using Node.js Workers) to scale to 100+ crore IDs? Comment below!

---

💡 If you found this helpful, follow me for more backend & TypeScript scalability insights!

---