Snowflake IDs Will Blow Your Mind 🤯

Subscribe for FREE in my blog: https://marcosmendess.substack.com/

Twitter (X) created Snowflake in 2010, but the idea went far beyond it. Today, Discord directly uses Twitter’s format, while other companies created their own approaches inspired by the same problem, and Sony created Sonyflake, an implementation explicitly inspired by Snowflake.

In 2010, Twitter was growing fast. Really fast. The data no longer fit comfortably on a single machine. They were distributing data, partitioning databases, using Cassandra, multiple MySQL instances and, suddenly, they needed a way to generate IDs that would work across all of this.

They could no longer simply ask a database: “Hey, what’s the next number?”

Every server needed to be able to generate its own ID, so Twitter (X) created Snowflake, and the interesting part is that a Snowflake isn’t exactly an ID. It’s a small data structure pretending to be a number.

A traditional Snowflake looks something like this:

The timestamp tells you approximately when that ID was created. The worker identifies which machine generated it. And the sequence allows multiple IDs to be generated within the same millisecond by the same worker. This combination of timestamp, worker, and sequence is precisely the solution described by Twitter (X) in the original Snowflake announcement.

fun-fact: 12 bits = 4,096, so each worker can generate up to 4,096 IDs in the same millisecond before it needs to wait for the next millisecond.

Put all of this together and you have a 64-bit identifier that can be generated independently by multiple machines.

No central ID server. No need to make a request to the database just to find out what the next number is. No machine telling the rest of the system: “Hold on, let me see who gets 123456789.”

You simply generate it. And that’s basically the trick, the server doesn’t need to know what everyone else is doing. It just needs to know who it is and what time it is.

That’s why Snowflake ended up becoming interesting far beyond Twitter. Discord, for example, uses Twitter’s Snowflake format for its own IDs. Users, messages, channels, servers… many of the things you find there are identified this way. Discord’s own documentation explicitly says that it uses Twitter’s Snowflake format.

And there’s a really cool detail here. Because the timestamp is inside the ID, the IDs also end up being approximately ordered by the time they were created, so instead of having something like:

550e8400-e29b-41d4-a716-446655440000

we have:

1234567890123456789

1234567890123456790

1234567890123456791

These numbers aren’t random. They’re carrying information. In fact, you can take a Snowflake and figure out approximately when that object was created. Discord even takes advantage of this property for things like pagination, and YES, it’s kind of weird to think about 😭

You look at 1234567890123456789 and think: “It’s just a number.” No, it isn’t. It’s a timestamp, a machine identifier, and a counter hidden inside a number.

But, of course, nothing is free, Snowflake trades simplicity for scale. Now you need to worry about worker IDs. Two machines can’t accidentally believe they are the same worker. You also need to worry about the clocks on the machines, because if a server’s clock moves backward, that ID system that should be ordered by time starts having problems, meaning that if the clock moves backward, the generator may try to create an ID “in the past.”

And there’s another thing, Snowflake isn’t particularly difficult to predict. Random UUIDs are very different. A Snowflake, on the other hand, has structure. It carries temporal information. This means that, depending on the system, someone may be able to figure out things that you might not want to expose.

That’s why Snowflake solves uniqueness. It doesn’t solve security. You shouldn’t use an ID as an authorization mechanism just because it seems difficult to guess, and here’s the part that I believe is easiest to misunderstand -> Snowflake is not “the ID system for distributed systems.”

It solves a very specific problem, which is “How do you generate globally unique IDs, at high scale, without needing a central machine responsible for generating all of them?”

Sometimes Snowflake is a great answer, sometimes UUID is better, sometimes ULID makes more sense, sometimes using your database’s AUTO_INCREMENT is all you need.

The fact that you have multiple servers doesn’t automatically mean you need Snowflake. The problem appears when you start having something like this:

And you want all of them to be able to generate IDs independently, that’s the clever part, it’s not the number, it’s the lack of coordination and there’s one last really cool detail about Snowflake, you know that giant number in your database? It’s telling a story.

It can tell you approximately when something was created, it can tell you which worker generated it, and it can tell you what the sequence of that ID was at that moment.

Twitter took a problem that seems extremely mundane, “we need to generate unique IDs”, and ended up turning it into a small piece of infrastructure for distributed systems.

At the end of the day, the result looks ridiculously simple, just a number, just a number, but that’s how good architecture usually works :) until you understand everything that had to happen for that number to exist, the interesting part was never the ID, it was everything you managed to stop coordinating.

Useful links for anyone who wants to dive deeper:

Twitter Announcing Snowflake
https://blog.x.com/engineering/en_us/a/2010/announcing-snowflake

Github Repo of Sonyflake (Sony snowflake implementation)
https://github.com/sony/sonyflake

How Big Tech Generates Unique IDs at Scale
https://guidgenerator.com/engineering-blog/how-big-tech-generates-unique-ids-at-scale-twitter-snowflake-instagram