Real-Time Event Telemetry: Ingesting 10k Scans/Sec with WebSockets & Redis

During large-scale conferences, tech summits, and government expos, ingress traffic is never evenly distributed. Up to 80% of attendees arrive within a narrow 45-minute window before the opening keynote.

When you have 10,000+ delegates simultaneously passing through turnstiles, badge scanners, and RFID portals, the telemetry ingestion pipeline faces an immediate high-concurrency bottleneck.

Standard relational database writes and synchronous REST endpoints choke under this load. If an API request blocks, turnstiles experience physical delays, gate queues spill out into transit hubs, and live monitoring dashboards desynchronize.

Here is the architectural blueprint for designing a high-throughput event analytics platform capable of processing edge RFID reads, computing dynamic hall capacity, and streaming sub-second visual telemetry to operational dashboards.

The System Requirements

An enterprise-grade event reporting engine must deliver three architectural guarantees:

1. Sub-10ms Gate Ack: Hardware edge controllers cannot wait on network latency.
2. Lossless Telemetry Buffering: Every single entry, exit, and zone transit must be recorded with nanosecond-level sequencing.
3. Sub-Second Dashboard Push: Floor controllers need live gate influx velocity and zone capacity heatmaps pushed instantly, not polled.

┌───────────────────────┐ ┌───────────────────────┐│ Overhead UHF Portal │ │ Turnstile Badge Scan ││ (Passive Telemetry) │ │ (On-Demand Access) │└──────────┬────────────┘ └──────────┬────────────┘│ │▼ ▼[ Edge MQTT Broker / TLS Micro-Payload ]│▼[ Ingestion Gateway (Go / Fastify) ]│▼[ Redis Streams Buffer (Memory) ]│┌─────────────┴─────────────┐▼ ▼[ Worker Consumer Group ] WebSocket Broadcast (Live Ops Dashboard)

1. Edge Communication: Micro-Payloads Over MQTT

Avoid sending full JSON payloads from edge scanners. When hundreds of reader antennas are constantly interrogating passive tags, verbose JSON strings eat up local bandwidth and trigger garbage collection spikes on low-power gateway controllers.

Instead, edge devices pack hardware events into a binary or compact buffer structure before publishing over MQTT:

typescript
// Compact payload schema (16-byte fixed buffer structure)
// [4 bytes: Epoch Timestamp] [6 bytes: UHF EPC Hash] [2 bytes: Portal ID] [1 byte: Direction (0=In, 1=Out)]
interface GateEvent {
  timestamp: number;
  epcHash: string;
  portalId: number;
  direction: 'IN' | 'OUT';
}
The gateway receives these compact events, validates the structure, and immediately pushes them to an in-memory streaming queue without hitting disk storage.2. Ingestion Layer: High-Throughput Buffering with Redis StreamsTo decouple hardware ingestion from analytical aggregation, we feed incoming gateway streams directly into Redis Streams.Redis Streams provides ordered message logging with consumer groups, enabling concurrent workers to read and process telemetry without message duplication.JavaScript// telemetry_gateway.js
import Redis from 'ioredis';
const redis = new Redis(process.env.REDIS_URL);

export async function ingestTagTelemetry(req, res) {
  const { portalId, tagId, timestamp, direction } = req.body;

  try {
    // Non-blocking write to the incoming telemetry stream
    await redis.xadd(
      'stream:event_telemetry',
      '*', // Auto-generated monotonic message ID
      'portal_id', portalId,
      'tag_id', tagId,
      'timestamp', timestamp,
      'direction', direction
    );

    // Fast ACK back to the edge hardware
    return res.status(202).send({ status: 'QUEUED' });
  } catch (err) {
    console.error('Buffer write failure:', err);
    return res.status(500).send({ error: 'BUFFER_UNAVAILABLE' });
  }
}
3. Real-Time Aggregation & WindowingDisplaying raw scan logs is useless to event directors. Ops teams need velocity (attendees entering per minute) and occupancy (current delegates per zone).   A decoupled worker process consumes from the stream, maintains rolling 60-second sliding windows, and tracks active hall density using atomic Redis operations:JavaScript// aggregation_worker.js
import Redis from 'ioredis';
const redis = new Redis(process.env.REDIS_URL);

async function processTelemetryBatch() {
  const entries = await redis.xreadgroup(
    'GROUP', 'analytics_group', 'worker_node_1',
    'COUNT', '500',
    'BLOCK', '2000',
    'STREAMS', 'stream:event_telemetry', '>'
  );

  if (!entries || !entries.length) return;

  const [stream, messages] = entries[0];
  const pipeline = redis.pipeline();

  for (const [id, fields] of messages) {
    const data = Object.fromEntries(
      fields.reduce((acc, cur, i) => (i % 2 === 0 ? acc.push([cur, fields[i + 1]]) : null, acc), [])
    );

    const delta = data.direction === 'IN' ? 1 : -1;
    const zoneKey = `occupancy:zone:${data.portal_id}`;

    // Atomically increment/decrement active zone count
    pipeline.incrby(zoneKey, delta);

    // Record entry frequency in a 60-second sliding time bucket
    const currentMinuteBucket = Math.floor(Date.now() / 60000);
    pipeline.hincrby(`rate:influx:${currentMinuteBucket}`, data.portal_id, 1);
    pipeline.expire(`rate:influx:${currentMinuteBucket}`, 3600); // 1-hour retention

    // Acknowledge processed message in the stream
    pipeline.xack('stream:event_telemetry', 'analytics_group', id);
  }

  await pipeline.exec();
}
4. Live Dashboard Streaming via WebSocketsInstead of browser dashboards polling REST endpoints every 3 seconds—which creates unnecessary database thundering herds during peak traffic—the application tier broadcasts state changes over WebSockets using Pub/Sub:JavaScript// websocket_publisher.js
import { WebSocketServer } from 'ws';
import Redis from 'ioredis';

const redisSub = new Redis(process.env.REDIS_URL);
const wss = new WebSocketServer({ port: 8080 });

// Subscribe to aggregated metric publications
redisSub.subscribe('channel:dashboard_updates', (err) => {
  if (!err) console.log('Subscribed to real-time telemetry stream.');
});

redisSub.on('message', (channel, message) => {
  // Broadcast unified metric frames to all connected monitoring screens
  wss.clients.forEach((client) => {
    if (client.readyState === 1) { // WebSocket.OPEN
      client.send(message);
    }
  });
});
5. Architectural Safeguards for Mission-Critical VenuesBuilding technology for large-scale enterprise summits requires anticipating real-world failure states:   WAN Disconnections: Check-in points must use local SQLite instances with Write-Ahead Logging (WAL) as an edge-tier cache to ensure zero gate downtime if venue fiber is severed, 12].

Passive UHF Multiplexing: Instead of requiring manual scans, passive long-range antennas read smart credentials seamlessly as delegates walk through portal arches.

Sponsor ROI & Dwell Telemetry: Continuous time-in-range tracking allows the aggregation pipeline to calculate verifiable dwell times for partner pavilions without active manual check-ins.

If you are designing high-density event systems or deploying physical credential infrastructure across the GCC, check out how StampIQ architects its localized real-time event analytics dashboard and hardware-integrated RFID attendee tracking stacks to handle peak-scale operations