Md. Fardin Khan

Posted on Jul 14

How We Built Chat Encryption( From First Idea to a Stable Design)

#systemdesign #softwareengineering #encryption #javascript

How We Built Chat Encryption — From First Idea to a Stable Design

A plain-language story of what broke, what we changed, and the architecture we ship today.

When we said “all messaging is encrypted,” we meant something specific: our company should not be able to read your chat text from our servers. Only the people in the conversation on their own phones and browsers should unlock messages.

Getting there was not a single release. It was a series of designs, real product problems, and fixes. This post walks that path from the start to the stable system we use now including the final architecture and the main chat action flows (login, send, receive, encrypt, re-key).

What “end-to-end encryption” means here

Think of a courier delivering a lockbox.

You lock the message on your device before it leaves.
Our servers carry and store the lockbox.
The other person’s device unlocks it.
We never hold the private key.

If the server creates the keys, the server can open every box. That is encryption at rest, not end-to-end. We rejected that early. It would break the privacy promise.

Starting point: plaintext chat

Before encryption, messages were normal text in the database and over the wire. Chat worked well. Anyone with server or database access could read everything.

Why we changed: privacy protocol and user trust. Messaging content should be confidential by default.

Architecture at that time (simple):

Phone / Web  →  Server  →  Database
                (stores readable text)
                →  other devices

Step 1 — First encryption design (one key per person)

What we built

Each user generates a key pair on their device.
Only the public key is uploaded.
Direct messages: derive a shared secret between two people, then lock the text.
Groups: each sender has a sender key for that group. Share it with members, then lock each later message once (same idea WhatsApp uses for groups).
Server role: public-key directory + store/deliver ciphertext. No private keys.

What this looked like

Sender device                Server                    Recipient device
─────────────                ──────                    ────────────────
Create keys locally
Upload public key  ───────►  Key directory
Lock message       ───────►  Store ciphertext  ─────►  Unlock with private key

Issue we hit next

Encryption was treated as required. If the other person (or one group member) had never opened the new app/web, they had no keys. Send failed. Chat looked broken for most existing users.

Lesson: You cannot flip a switch to “everyone must encrypt” in a live product with old clients still in the wild.

Step 2 — Soft rollout: encrypt if possible

Problem

Hard cutover blocked messaging whenever anyone lacked keys. Groups were worse: one member without keys blocked the whole send.

Solution

If everyone in the conversation has keys → encrypt.
If anyone is missing keys → send plaintext and keep chat working.
New clients generate and upload keys on login in the background.
Never put private keys on the server to “migrate” old users (that would not be E2EE).

What users experience

Situation	Behavior
New client ↔ new client	Encrypted
New ↔ old (no keys yet)	Plaintext, still delivers
Later, old user upgrades	Future messages can encrypt

Important boundary we locked later: plaintext is for rollout, not for “this group is too big.” Once keys exist, we encrypt — including large groups.

Step 3 — Early decrypt bugs (still one device per user)

After encryption shipped, several “Unable to decrypt” cases appeared. None were “encryption is fake”; they were edge cases in how keys were derived or loaded.

Issue (user view)	Cause	Fix
DM fails between two people	Both sides didn’t derive the same secret	Align DM key agreement (identity × identity)
Can’t read your own group message after refresh	Own sender key wasn’t resolved from local storage	Fall back to locally stored sender key for self
Some members never get group keys	Member list used for crypto was incomplete (~paginated UI)	Dedicated full member-id API for encryption
Own DM breaks after navigating away	Peer resolution sometimes treated you as the peer	Harden peer id; never fetch your own public bundle as the peer

Lesson: E2EE is as much about product edge cases (reload, pagination, “my own bubble”) as about algorithms.

Step 4 — Multi-device: phone + web for the same account

Problem

Same account in normal browser and Incognito (or phone + web). Send from one → Unable to decrypt on the other.

Why

We stored one public key per user. The second login overwrote the first. Private keys live only in that browser. Device A cannot open a message sealed for Device B’s public key.

Solution (encryption version 2)

Each install gets a stable device id.
Server stores many public bundles per user: (userId, deviceId).
DMs: one locked message body + small envelopes of the message key for:
- each of the peer’s devices
- your other devices
- this device (so your sent messages stay readable after reload)
Groups: share the sender key to each member device, not just each member.
Cap later: max 5 devices so fan-out stays bounded (similar to linked-device limits elsewhere).

One message key
   ├─ lock text once → ciphertext
   ├─ envelope for peer phone
   ├─ envelope for peer web
   ├─ envelope for my phone
   └─ envelope for my web
         ↓
      Server stores ciphertext + envelopes
      (cannot open envelopes)

Lesson: “One account” in the product sense is many devices in the crypto sense.

Step 5 — Scale without dropping encryption (stable design)

Problem

Privacy says all messaging is encrypted. Turning encryption off for “groups over N members” was rejected.

But our group path, while shaped like WhatsApp Sender Keys, still did expensive work on every send:

Re-share sender key wraps to everyone every message
Fetch each member’s keys with one request per user (N+1)
Wasteful database work when listing keys
Unbounded devices → larger and slower fan-out

What WhatsApp taught us (conceptually)

Once (first speak / membership change): share the sender’s group secret with members’ devices.
Every later message: lock once; server copies the same blob to everyone.
Cap linked devices.
Don’t disable encryption because the group is large.

What we changed

Change	Everyday meaning
Distribute only when needed	Heavy work on first send / new device / after join-leave; later sends stay fast
Batch key lookup	One request for many members’ public keys
Short-lived cache	Don’t re-download the same public keys constantly
Stop wasteful key-list side effects	Listing public keys doesn’t burn one-time keys
Max 5 devices	Keeps multi-device cost predictable
Always encrypt when keys exist	Large groups stay encrypted

Cost model (for engineers and curious readers):

Warm group send → cheap (lock once, upload once)
Cold path (first send / rotate / new device) → more work, paid rarely

Final stable architecture

This is the system we run today. Read it as a map of who does what, then follow the action flows for login, send, receive, and re-key.

Big picture

In plain language: your device locks the words, our servers carry and file the lockbox, their device opens it. We never hold the private key.

System architecture — layers

Layer	Job
Web and mobile apps	UI: thread list, message list, composer. Ask crypto to lock/unlock; never invent keys in the UI layer.
On-device crypto service	Create device keys, soft-rollout checks, DM envelopes, group sender keys, decrypt for history and live messages.
Realtime socket + HTTP	Socket for live send/edit/react/typing; HTTP for history, media, and the public key directory.
Backend chat service	Auth, save messages, fan-out to members, unread and conversation previews.
Backend key directory	Store public device keys and wrapped group secrets; signal re-key on membership change. Never private keys.
Database	Conversations, messages, public keys, wraps, read state, media.

Conversation model

Every thread is a Group — a community chat or a one-to-one DM. DMs are the same container with a DM chat type. There is no separate “Conversation” table. That keeps membership, delivery, and unread logic in one place.

An encrypted message stores a sealed body (ciphertext), not readable text. Direct messages also store small device envelopes. Group messages store a sender key id so recipients know which shared secret to use.

Two encryption modes

Direct message — invent a one-time message key, lock the text once, put a sealed copy of that key in an envelope for each relevant device (theirs and yours).

Group — share your sender key when needed; every later message from you is one locked blob for the whole group (WhatsApp-style Sender Keys).

Chat action flows

Flow A — Login / device setup

When someone opens the current app or web build while signed in, encryption sets itself up in the background. No extra button.

What the user notices: nothing dramatic — chat still works. Behind the scenes, this install published its public lock so others can encrypt to it.

Flow B — Send a text message (full path)

This is the main action flow: tap Send → maybe encrypt → server stores → everyone receives → each device unlocks.

Step by step in words:

User taps Send.
Client asks: can everyone who needs to read this unlock a lockbox?
- No → send plaintext (rollout only; chat must not break).
- Yes → DM uses envelopes; group may share a sender key, then lock once.
Transport: prefer realtime socket; fall back to HTTP if needed.
Server saves the message and fans it out to members in the room.
Each peer device unlocks before showing the bubble (or shows plaintext / “unable to decrypt”).

Flow C — Soft-rollout gate (decide encrypt vs plaintext)

Plaintext here means “someone has not upgraded yet,” not “this group is large.”

Flow D — Direct message encrypt / decrypt

Why envelopes: one locked message body; phone and web for the same account each get their own sealed copy of the message key.

Flow E — Group message (cold vs hot)

Sharing a group secret with every member’s devices is expensive. We do it when needed; later messages stay cheap.

Moment	What happens
First encrypted send in a group (from you)	Create your sender key, distribute wraps, then lock this message
Later sends from you	Reuse sender key; lock once; no redistribute
Someone joins or leaves	Server signals re-key; clients drop old local sender key; next send shares a new one
Another member sends	They use their sender key (created on their first encrypted send)

Flow F — Open history or a live message

History and live messages use the same unlock rules. Failure shows a clear placeholder instead of garbage characters.

Flow G — Membership change / re-key

Product meaning: when the roster changes, chat re-locks itself so departed members should not keep reading new messages under the old group secret.

Soft rollout vs privacy promise

Missing keys → plaintext so chat doesn’t die during upgrade.
Everyone has keys → always encrypt, any group size.
Old history before encryption → stays as it was (we don’t rewrite the past on the server).

What is / isn’t encrypted today

Content	Encrypted end-to-end?
Text when all peers have keys	Yes
Text while someone lacks keys	No (temporary rollout)
Photos / files / voice	Not yet
Typing, reactions, presence, group name	No (delivery metadata)

Old groups: do we create sender keys up front?

No mass migration. Existing groups sit unchanged until someone with a modern client sends text and every member has keys.

Then, for that sender only:

Create a sender key (if none locally)
Distribute wraps to member devices
Encrypt that message
Reuse the same sender key on later sends

Other members create their own sender key the first time they send encrypted text. History from before encryption stays plaintext.

Decision log (what we refused and why)

Tempting shortcut	Why we didn’t
Server generates keys for all users	Server could read everything → not E2EE
Block send until everyone encrypts	Breaks production chat during rollout
Plaintext for large groups	Contradicts “all messaging is encrypted”
One public key per account	Phone + web cannot both decrypt
Rebuild full Signal Protocol in one go	Slower path; Sender Keys + device envelopes got us to a stable product shape

Takeaways

True E2EE is a client problem as much as a server problem — private keys must stay on devices.
Ship a bridge (encrypt-if-possible) or you will break existing users.
Multi-device is not optional if people use phone and web.
Scale encryption by doing expensive work rarely, not by turning encryption off.
Product edge cases (own messages, incomplete member lists, account switch) define whether users trust the feature.

Closing

We started with readable messages on the server, then added locks, then learned that locks without a migration story feel like outages, then learned that one lock per account fails on a second browser, then aligned group encryption with the industry pattern: expensive once, cheap always, without abandoning the privacy promise.

The stable system today is: device-scoped keys, soft rollout for missing peers, WhatsApp-style group sender keys, and a server that relays sealed boxes it cannot open.

That is how our chat encryption went from first idea to something we can stand behind in production.