Ingesting 10,000 RFID Scans/Sec: Edge-First Event Telemetry with Go, Redis Streams, and WebSockets

During high-density international summits and industrial exhibitions in Riyadh and Jeddah, arrival traffic clusters heavily: up to 80% of delegates enter through security turnstiles within a narrow 45-minute window.

When 10,000+ delegates pass through RFID turnstiles and overhead scanning portals simultaneously, traditional architectures fail[cite: 12]:

• Public WAN Degradation: Conventional cellular towers and venue Wi-Fi access points saturate, causing remote cloud REST API latencies to spike from 150ms to over 5,000ms.
• Database Write Contention: Concurrent relational inserts lock tables, stalling turnstile relays and backing up gate lines into exterior corridors.
• Dashboard Desynchronization: Polling-based administration screens crash under HTTP connection overhead, leaving operations teams blind to live hall occupancy.

To maintain an ingress pace below 3 seconds per attendee, telemetry ingestion must be decoupled from remote cloud infrastructure and relational database transactions.

Here is the operational blueprint for an edge-first, high-throughput event analytics pipeline capable of ingesting passive UHF radio telemetry, computing real-time spatial densities, and streaming live updates to operational dashboards.

---

Architecture Topology

┌─────────────────────────┐ ┌─────────────────────────┐│ Overhead UHF Portals │ │ High-Speed Turnstile ││ (Passive EPC Gen 2) │ │ (NFC / Smart Badge) │└────────────┬────────────┘ └────────────┬────────────┘│ │▼ ▼[ Edge TCP/MQTT Broker (Binary Buffer Payloads) ]│▼[ Go High-Throughput Gateway ]│▼[ Redis Streams (In-Memory WAL) ]│┌──────────────┴──────────────┐▼ ▼[ Stream Consumer Group ] Pub/Sub Broadcaster (WebSocket Push Server)│ │▼ ▼[ Cold WAL Storage / TimescaleDB ] [ Operations Command Console ]

1. Edge Gate Protocol: Compact Binary Payloads

Avoid JSON over HTTP at physical entry points. Serializing verbose strings across hundreds of antennas consumes unnecessary gateway compute and introduces garbage collection delays on embedded controllers.

Edge nodes package tag interrogations into fixed 16-byte binary frames sent over persistent TCP sockets:

┌──────────────────┬──────────────────┬──────────────┬──────────────┐│ Epoch Timestamp │ Tag EPC Hash │ Portal ID │ Direction ││ (4 Bytes) │ (8 Bytes) │ (2 Bytes) │ (2 Bytes) │└──────────────────┴──────────────────┴──────────────┴──────────────┘

2. Ingestion Engine: Non-Blocking Gateway in Go

The Go ingestion service handles raw TCP frames from edge readers and appends them directly into Redis Streams via memory pipelining:

go
package main

import (
    "context"
    "encoding/binary"
    "io"
    "log"
    "net"
    "[github.com/redis/go-redis/v9](https://github.com/redis/go-redis/v9)"
)

var ctx = context.Background()
var rdb *redis.Client

type TelemetryFrame struct {
    Timestamp uint32
    TagHash   uint64
    PortalID  uint16
    Direction uint16 // 0x01 = IN, 0x02 = OUT
}

func handleConnection(conn net.Conn) {
    defer conn.Close()
    buf := make([]byte, 16)

    for {
        _, err := io.ReadFull(conn, buf)
        if err != nil {
            return
        }

        frame := TelemetryFrame{
            Timestamp: binary.BigEndian.Uint32(buf[0:4]),
            TagHash:   binary.BigEndian.Uint64(buf),
            PortalID:  binary.BigEndian.Uint16(buf),
            Direction: binary.BigEndian.Uint16(buf),
        }

        // Push to in-memory stream buffer
        err = rdb.XAdd(ctx, &redis.XAddArgs{
            Stream: "stream:event_telemetry",
            Values: map[string]interface{}{
                "ts":       frame.Timestamp,
                "tag":      frame.TagHash,
                "portal":   frame.PortalID,
                "dir":      frame.Direction,
            },
        }).Err()

        if err != nil {
            log.Printf("Stream ingestion failure: %v", err)
        }
    }
}

func main() {
    rdb = redis.NewClient(&redis.Options{
        Addr: "localhost:6379",
    })

    listener, err := net.Listen("tcp", ":9000")
    if err != nil {
        log.Fatalf("Socket binding failed: %v", err)
    }
    defer listener.Close()

    for {
        conn, err := listener.Accept()
        if err != nil {
            continue
        }
        go handleConnection(conn)
    }
}
3. Real-Time Telemetry Processing & AggregationDisplaying raw logs is insufficient for venue directors. Incident response teams require rolling metrics:Gate Influx Velocity: Entries per lane per minute.   Active Room Density: Current occupancy compared against Civil Defense safety thresholds.   A worker pool consumes the stream using Redis Consumer Groups and aggregates metrics within 60-second sliding windows:JavaScript// aggregation_worker.js
import Redis from 'ioredis';
const redis = new Redis(process.env.REDIS_URL);

async function processTelemetry() {
  const entries = await redis.xreadgroup(
    'GROUP', 'telemetry_processors', 'worker_alpha',
    'COUNT', '500',
    'BLOCK', '1000',
    'STREAMS', 'stream:event_telemetry', '>'
  );

  if (!entries || !entries.length) return;

  const [stream, messages] = entries[0];
  const pipe = redis.pipeline();

  for (const [id, fields] of messages) {
    const payload = Object.fromEntries(
      fields.reduce((acc, cur, i) => (i % 2 === 0 ? acc.push([cur, fields[i + 1]]) : null, acc), [])
    );

    const delta = payload.dir === '1' ? 1 : -1;
    const hallKey = `occupancy:hall:${payload.portal}`;
    const rateKey = `rate:gate:${payload.portal}:${Math.floor(Date.now() / 60000)}`;

    // Update spatial totals
    pipe.incrby(hallKey, delta);
    pipe.incr(rateKey);
    pipe.expire(rateKey, 3600); // 1-hour rolling window

    // Acknowledge read in stream
    pipe.xack('stream:event_telemetry', 'telemetry_processors', id);
  }

  await pipe.exec();
}
4. Live Command Broadcasting via WebSocketsTo avoid overwhelming databases with repetitive dashboard polling, metrics are pushed to client sessions through WebSockets using Redis Pub/Sub:JavaScript// websocket_broadcaster.js
import { WebSocketServer } from 'ws';
import Redis from 'ioredis';

const redis = new Redis();
const wss = new WebSocketServer({ port: 8080 });

redis.subscribe('channel:telemetry_updates', (err) => {
  if (!err) console.log('Broadcaster active on telemetry channel.');
});

redis.on('message', (channel, message) => {
  wss.clients.forEach((client) => {
    if (client.readyState === 1) { // WebSocket.OPEN
      client.send(message);
    }
  });
});
Operational Safeguards for GCC VenuesDeploying physical technology in high-profile event environments requires planning for physical and network failures.
Embedded SQLite WAL Buffers: In the event of a total network disconnect, gate kiosks commit transactions to local SQLite instances using Write-Ahead Logging (WAL). 
Scans commit in under 5ms, batching records to cloud storage only after WAN connections recover.
Passive Overhead Portals: Portals running EPC Gen 2 anti-collision protocols resolve up to 400 tag backscatters per second, capturing transit metrics without manual badge taps.

Auditable Dwell Analytics: Spatial telemetry registers entrance and exit timestamps automatically, supplying sponsors with verified engagement and dwell figures.

To see this telemetry architecture in production, review our deployment post-mortem on the Humain Leap Saudi event accreditation case study or inspect StampIQ's real-time event analytics dashboard engineered for regional enterprise venues.