Scaling My Chat App Introduced a Silent Multi-Device Bug

In the initial version of my real-time chat app, Relay, tracking user presence was trivial. Everything ran on a single Node.js instance, so presence was just a local memory fact: an in-memory Map<userId, WebSocket>. If a user connected, I added them to the map. If they disconnected, I removed them.

When I introduced Redis Pub/Sub to scale out across multiple server instances, this simple logic broke down entirely.

The Bug

The issue appeared when a user was logged into Relay on multiple devices simultaneously. Let's say you have your phone connected to Server 1 and your laptop connected to Server 2.

If you close the app on your phone, the WebSocket connection to Server 1 drops. Almost immediately, your laptop—which is still fully connected to Server 2—stops registering as "online" to other users. You appear completely offline despite sitting right in front of an active session.

Why It Happened

The root cause was treating distributed state like local state.

When I moved to Redis, I started using a shared Redis Set (online_users) to track who was online globally. But I kept the old boolean logic for disconnects: when any WebSocket closed, the server just yanked the user out of the Set.

// The naive approach that broke multi-device presence
async markOffline(userId: string) {
  // Wait, the user might still have other active connections!
  await redis.sRem("online_users", userId);
}

The logic didn't account for the fact that a user could have concurrent connections spread across different instances. Disconnecting one device incorrectly wiped their global presence.

The Second-Order Effect

This wasn't just a UI bug where the green "online" dot disappeared. It actively corrupted message delivery statuses in the database.

In Relay, when you send a message, the backend checks the recipient's presence to determine whether to mark the message as "delivered" instantly:

// backend_v1/src/ws/handlers/message.handler.ts
const deliveredAt = (await presence.isOnline(to)) ? new Date() : null;
await saveMessage(to, senderId, content, msg.payload.id, deliveredAt);

Because of the presence bug, presence.isOnline(to) would return false after the phone disconnected. The backend would insert the message into PostgreSQL with deliveredAt = null. However, because the message was also broadcast over Redis Pub/Sub, the laptop connected to Server 2 would still receive the message live.

The DB claimed the message was undelivered, but the recipient was actively reading it on their screen.

The Fix

To fix this, I replaced the naive boolean approach with a distributed connection counter. Now, Redis tracks exactly how many active WebSockets a user has open across the entire cluster.

We only fire an "online" event on a 0 → 1 transition, and an "offline" event on a 1 → 0 transition:

// backend_v1/src/services/presence.service.ts
class PresenceService {
  private static readonly CONN_KEY_PREFIX = "presence:conn_count:";

  async markOnline(userId: string) {
    const key = `${PresenceService.CONN_KEY_PREFIX}${userId}`;
    const newCount = await redis.incr(key);

    // Only mark globally online if this is their first connection
    if (newCount === 1) {
      await redis.expire(key, 86400);
      await redis.sAdd("online_users", userId);
      return true;
    }
    return false;
  }

  async markOffline(userId: string) {
    const key = `${PresenceService.CONN_KEY_PREFIX}${userId}`;
    const newCount = await redis.decr(key);

    // Only mark globally offline if they have 0 connections left
    if (newCount <= 0) {
      await redis.del(key);
      await redis.sRem("online_users", userId);
      return true;
    }
    return false;
  }
}

By relying on redis.incr() and redis.decr()—which are atomic operations—the counter remains perfectly in sync, even when multiple servers process connect and disconnect events for the same user simultaneously.

Wrapping up

Moving from a single-server mental model to a distributed one requires rethinking even the most basic assumptions, like what it actually means to be "online".
I'll be back with more next week. Til then, stay consistent!