DEV Community: Dzaki Amri Zaidaan

Scaling Real-Time APIs to 100k+ Concurrent Connections: WebSockets, SSE, and Redis Pub/Sub

Dzaki Amri Zaidaan — Fri, 28 Aug 2026 07:13:45 +0000

Key Takeaway / TL;DR:

WebSockets are the de facto choice for bidirectional, low-latency communication, but they require sticky sessions or a Redis Pub/Sub layer to scale horizontally.

Server-Sent Events (SSE) offer a simpler, HTTP-based alternative for one-way server-to-client streaming, with automatic reconnection and better HTTP/2 multiplexing.

Redis Pub/Sub acts as a scalable message broker to fan out events across multiple nodes, but you must handle backpressure and connection limits carefully.

The Problem & Industry Shift

Real-time features—live chat, collaborative editing, financial tickers, IoT telemetry—are no longer optional. Users expect sub-100ms updates. Traditional REST polling is wasteful and latency-bound. The industry has shifted toward persistent connections: WebSockets and SSE. However, scaling these to 100k+ concurrent connections on a single server is impossible; you need a distributed architecture.

The core challenge: a single Node.js process can handle ~10k-50k concurrent WebSocket connections (depending on memory and CPU), but beyond that, you must scale out horizontally. This introduces the sticky session problem: a client connected to server A might need to receive events triggered by another client connected to server B. Without a shared message bus, you lose events.

Architecture & Core Mechanics

A robust real-time architecture typically involves three layers:

Client Connection Layer: WebSocket or SSE endpoints terminated by a fleet of stateless API servers.
Message Broker Layer: Redis Pub/Sub (or Kafka, NATS) that relays messages between servers.
Persistence Layer: Optional, for message history or offline delivery.

Here's a high-level flow:

[Client A] <--WebSocket--> [Server 1] <--Redis Pub/Sub--> [Server 2] <--WebSocket--> [Client B]

When Client A sends a message, Server 1 publishes it to a Redis channel. All servers (including Server 1) subscribe to that channel and forward the message to their local clients who are interested. This decouples the connection from the message origin.

Key design decisions:

WebSocket vs SSE: WebSockets are bidirectional and lower overhead per message (after handshake). SSE is unidirectional (server-to-client) but rides on HTTP, which simplifies firewalls and proxies. For chat or collaborative apps, WebSockets are preferred. For live feeds (stock prices, notifications), SSE is simpler and can be more efficient with HTTP/2.
Redis Pub/Sub: It's fast (sub-millisecond) but fire-and-forget. If a subscriber is slow, messages are dropped. For critical messages, consider Redis Streams or a persistent queue.
Connection Pooling: Redis connections are expensive. Use a single Redis client per server, not per WebSocket. Node.js Redis clients (e.g., ioredis) handle connection pooling internally.

Production Code Example

Below is a production-grade Node.js example using ws for WebSockets, ioredis for Redis Pub/Sub, and express for HTTP. It demonstrates horizontal scaling with a Redis adapter.

import express from 'express';
import http from 'http';
import { WebSocketServer, WebSocket } from 'ws';
import Redis from 'ioredis';

const app = express();
const server = http.createServer(app);

// WebSocket server with noServer to handle upgrade manually
const wss = new WebSocketServer({ noServer: true });

// Redis clients: one for publishing, one for subscribing (ioredis recommends separate connections)
const redisPublisher = new Redis({ host: 'redis-1', port: 6379, maxRetriesPerRequest: null });
const redisSubscriber = new Redis({ host: 'redis-1', port: 6379, maxRetriesPerRequest: null });

// Map to track clients per server (in-memory)
const clients = new Map<string, WebSocket>(); // clientId -> socket

// Subscribe to Redis channel for incoming messages from other servers
redisSubscriber.subscribe('chat:global');
redisSubscriber.on('message', (channel, message) => {
  if (channel === 'chat:global') {
    const { clientId, data } = JSON.parse(message);
    // Broadcast to all local clients (or target specific client)
    clients.forEach((ws, id) => {
      if (ws.readyState === WebSocket.OPEN) {
        ws.send(JSON.stringify({ from: clientId, data }));
      }
    });
  }
});

// Handle HTTP upgrade requests
server.on('upgrade', (request, socket, head) => {
  // Authenticate here (e.g., JWT in query string or cookie)
  wss.handleUpgrade(request, socket, head, (ws) => {
    wss.emit('connection', ws, request);
  });
});

wss.on('connection', (ws, request) => {
  // Extract client ID from query params (e.g., ?clientId=abc)
  const clientId = new URL(request.url, 'http://localhost').searchParams.get('clientId');
  if (!clientId) {
    ws.close(4001, 'Missing clientId');
    return;
  }

  clients.set(clientId, ws);
  console.log(`Client ${clientId} connected. Total: ${clients.size}`);

  ws.on('message', (message) => {
    // Publish to Redis so other servers can receive
    redisPublisher.publish('chat:global', JSON.stringify({ clientId, data: message.toString() }));
  });

  ws.on('close', () => {
    clients.delete(clientId);
    console.log(`Client ${clientId} disconnected. Total: ${clients.size}`);
  });

  ws.on('error', (err) => {
    console.error('WebSocket error:', err);
    ws.close(1011, 'Internal error');
  });
});

const PORT = process.env.PORT || 3000;
server.listen(PORT, () => {
  console.log(`Server listening on port ${PORT}`);
});

Critical engineering decisions:

Separate Redis connections for pub/sub: ioredis requires a dedicated connection for subscriptions because it enters a subscriber mode. Mixing pub and sub on one connection will cause errors.
maxRetriesPerRequest: null: Prevents the Redis client from buffering commands when the connection is lost, which could cause memory leaks. Instead, it will emit errors and you can handle reconnection.
Graceful backoff: Implement reconnection logic for Redis and WebSocket clients. For Redis, ioredis has built-in retry strategy; configure it with exponential backoff.
Backpressure: If a client is slow, ws.send can buffer. Use ws.bufferedAmount to monitor and drop or disconnect slow consumers.

Performance, Cost & Trade-offs

Benchmarks (approximate, from real-world deployments):

WebSocket throughput: A single Node.js process can handle ~10k messages/sec with low latency (<5ms) on modest hardware. With Redis Pub/Sub, the bottleneck becomes Redis (can handle 100k+ ops/sec).
Memory: Each WebSocket connection consumes ~20-50KB (including buffers). 100k connections => 2-5GB RAM per server. Plan accordingly.
Latency: Redis Pub/Sub adds ~0.1-0.5ms overhead. End-to-end latency remains under 10ms in a well-configured cluster.

Trade-offs:

Redis Pub/Sub vs. Redis Streams: Pub/Sub is simpler and lower latency, but messages are lost if no subscriber is present. Streams provide persistence and consumer groups but add complexity and higher latency.
Sticky sessions vs. Redis: Sticky sessions (e.g., using a load balancer) avoid Redis but cause uneven load and failover issues. Redis decouples but adds a single point of failure (use Redis Sentinel or Cluster).
SSE over WebSockets: SSE is easier to implement and works over HTTP/2, but it's unidirectional. For bidirectional, you need WebSockets or a combination (SSE + POST).

Security considerations:

Authentication: Validate tokens during the WebSocket handshake. Don't rely on origin headers.
Rate limiting: Implement per-connection message rate limits to prevent abuse.
TLS: Use WSS (WebSocket Secure) to encrypt traffic. Terminate TLS at the load balancer to reduce CPU load on app servers.

Actionable Checklist / Summary

When building a real-time API at scale, follow these steps:

Choose the right protocol: WebSockets for bidirectional, SSE for one-way streaming.
Design for horizontal scaling: Use Redis Pub/Sub (or a similar broker) to sync across nodes.
Manage Redis connections: Use separate pub/sub clients, configure retry strategies with exponential backoff.
Handle backpressure: Monitor bufferedAmount and disconnect slow clients.
Implement graceful degradation: If Redis fails, fall back to local-only broadcast (with a warning).
Monitor metrics: Track connection count, message rates, Redis latency, and error rates.
Load test: Use tools like k6 or wrk to simulate 100k connections and identify bottlenecks.

References

Production RAG in 2026: Hybrid Search with pgvector, Reciprocal Rank Fusion, and Context Caching

Dzaki Amri Zaidaan — Fri, 28 Aug 2026 07:06:12 +0000

Key Takeaway / TL;DR:

Naive vector search alone fails on exact keywords, IDs, and rare terms; hybrid search with BM25 + dense embeddings significantly improves retrieval quality.

Reciprocal Rank Fusion (RRF) is a simple, robust way to merge heterogeneous relevance scores without complex calibration.

Context caching (e.g., Anthropic's prompt caching) can cut RAG latency and cost by up to 90% for repeated system prompts and retrieved contexts.

The Problem & Industry Shift

In 2026, RAG (Retrieval-Augmented Generation) has moved from demos to production, but many pipelines still rely on a single vector index. This approach fails on exact-match queries like product codes, legal citations, or user IDs. Dense embeddings capture semantic similarity but often miss lexical precision. The industry shift is toward hybrid search: combining dense vectors with BM25 full-text search, then fusing results using Reciprocal Rank Fusion (RRF). This is now a standard pattern in production RAG, and PostgreSQL with pgvector is a compelling platform because it supports both vector and full-text indexing in the same database, eliminating the need for a separate vector database.

Architecture & Core Mechanics

A production RAG pipeline involves several stages: ingestion, retrieval, reranking, and generation. Here's the data flow:

[Documents] -> [Chunking] -> [Embedding Model] -> [PostgreSQL: pgvector + tsvector]
                                                          |
                                                          v
[Query] -> [Embedding] -> [Vector Search] -> [RRF Fusion] -> [Reranker] -> [LLM with Context Cache]
                |-> [Full-Text Search] -> [RRF Fusion] -> [Reranker] -> [LLM with Context Cache]

Key components:

Chunking: Overlap and hierarchical chunking to preserve context. For example, chunk size 512 tokens with 50-token overlap.
Dense embeddings: Use a model like text-embedding-3-large to generate vectors.
BM25: PostgreSQL's tsvector/tsquery with websearch_to_tsquery for full-text search.
RRF: Combine ranked lists from both retrievers using the formula: score = Σ 1/(k + rank), where k is typically 60.
Reranking: A cross-encoder model (e.g., cross-encoder/ms-marco-MiniLM-L-6-v2) to refine top-k results.
Context caching: Cache the system prompt and static context to avoid re-encoding them for every request.

Production Code Example

Below is a TypeScript implementation using pg and pgvector for a hybrid search query. It assumes a table documents with columns id, content, embedding vector(1536), and tsv tsvector.

import { Pool } from 'pg';
import { embed, generate } from './ai';

const pool = new Pool({ connectionString: process.env.DATABASE_URL });

// RRF fusion function
function reciprocalRankFusion(results: { id: string; rank: number }[][], k = 60) {
  const scores = new Map<string, number>();
  for (const list of results) {
    list.forEach((item, index) => {
      const score = 1 / (k + index + 1); // rank starts at 1
      scores.set(item.id, (scores.get(item.id) || 0) + score);
    });
  }
  return Array.from(scores.entries())
    .sort((a, b) => b[1] - a[1])
    .map(([id]) => id);
}

async function hybridSearch(query: string, topK = 10) {
  // 1. Generate embedding for query
  const queryEmbedding = await embed(query);

  // 2. Run vector and full-text searches in parallel
  const [vectorResults, ftsResults] = await Promise.all([
    pool.query(
      `SELECT id, embedding <=> $1 AS distance
       FROM documents
       ORDER BY embedding <=> $1
       LIMIT $2`,
      [queryEmbedding, topK]
    ),
    pool.query(
      `SELECT id, ts_rank(tsv, websearch_to_tsquery('english', $1)) AS rank
       FROM documents
       WHERE tsv @@ websearch_to_tsquery('english', $1)
       ORDER BY rank DESC
       LIMIT $2`,
      [query, topK]
    ),
  ]);

  // 3. Convert to ranked lists (lower rank is better)
  const vectorRanked = vectorResults.rows.map((r, i) => ({ id: r.id, rank: i + 1 }));
  const ftsRanked = ftsResults.rows.map((r, i) => ({ id: r.id, rank: i + 1 }));

  // 4. Fuse using RRF
  const fusedIds = reciprocalRankFusion([vectorRanked, ftsRanked]);

  // 5. Fetch full documents for reranking
  const docs = await pool.query(
    `SELECT id, content FROM documents WHERE id = ANY($1)`,
    [fusedIds.slice(0, topK)]
  );

  // 6. Rerank with cross-encoder (pseudo-code)
  const reranked = await rerank(query, docs.rows);

  return reranked;
}

Critical decisions:

Use <=> for cosine distance (pgvector operator).
Use websearch_to_tsquery for robust query parsing.
Run both searches in parallel to minimize latency.
RRF avoids score normalization issues.

Performance, Cost & Trade-offs

Benchmarks (typical):

Vector search alone: Recall@10 ~65% on MS MARCO.
Hybrid (vector + BM25 + RRF): Recall@10 ~82% (+17% relative).
Adding a reranker: +5-10% further, but adds ~50ms latency per query.

Latency breakdown:

Embedding query: ~10ms (local model) or ~100ms (API).
Vector search: ~20ms for 1M vectors with HNSW index.
Full-text search: ~5ms.
RRF: <1ms.
Reranking: ~50ms for top 10 with cross-encoder.
LLM generation: 1-3s (dominates).

Cost:

Context caching can reduce LLM cost by up to 90% for repeated system prompts. For example, Anthropic's prompt caching charges 1.25x for cache writes but 0.1x for cache reads [1].
Storing embeddings in PostgreSQL avoids separate vector DB costs.

Trade-offs:

Hybrid search requires maintaining both tsvector and embedding columns; use triggers to keep them in sync.
RRF is simple but not optimal; weighted RRF can improve results if you have validation data.
Context caching increases memory usage; set appropriate TTLs.

Actionable Checklist / Summary

Start with hybrid search: Combine tsvector and pgvector in PostgreSQL. Use RRF for fusion.
Chunk wisely: Use overlapping chunks (e.g., 512 tokens with 50 overlap) and consider hierarchical chunking for long documents.
Index properly: Create HNSW index on embedding and GIN index on tsv.
Rerank: Add a cross-encoder reranker for top 10 results to improve precision.
Cache context: Use prompt caching (e.g., Anthropic's) for system prompts and static retrieved context to reduce latency/cost.
Monitor and evaluate: Track retrieval metrics (Recall@k, MRR) and end-to-end quality.
Consider weighted RRF: Tune k and weights if you have labeled data.

References

[1] Anthropic Prompt Caching Documentation: https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching
[2] pgvector GitHub Repository: https://github.com/pgvector/pgvector
[3] PostgreSQL Full-Text Search Documentation: https://www.postgresql.org/docs/current/textsearch.html
[4] Reciprocal Rank Fusion paper (Cormack et al.): https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf
[5] Cross-Encoder models (SBERT): https://www.sbert.net/examples/applications/cross-encoder/README.html