rekuiper 0.500: Moving the Engine Hot Path to RAM and Finding its Exact Physical Limits (up to 200k msg/s)

When we published our earlier benchmarks for rekuiper (our Rust reimplementation of LF Edge eKuiper for edge gateways and IoT hubs), the numbers answered our first question: on five real-world MQTT workloads, memory stayed bounded between 5 and 10 MB while Go-based engines climbed to hundreds of megabytes or failed.

In that test, rekuiper sustained 100,000 messages per second on a single pinned core. At offered 200,000 msg/s, however, it dropped packets or backlogged upstream.

Saying "100k passed and 200k failed" left a massive 100,000 msg/s blind spot. Is the true ceiling 105k? 140k? 195k? And why did it choke at higher rates when the CPU wasn't fully maxed out on every workload?

When we profiled the bottleneck under burst loads, the culprit wasn't stream parsing or window math. It was the disk.

Even when an edge stream engine's data plane runs in memory, typical implementations still touch disk for metadata: checking SQLite tables for stream and rule definitions, resolving auth keys, reading configuration YAMLs, or persisting rule state changes. On a developer NVMe drive, SQLite lookups take microseconds. On an industrial gateway running off slow eMMC flash or a microSD card, a batch of flash writes causes I/O wait spikes that stall the Tokio runtime thread.

For v0.500-beta, we rebuilt the engine's internal catalog and hot paths around an in-memory Redis-style architecture, tuned our storage engine to be zero-contention, and then ran a hierarchical search with 1,000 msg/s resolution under a strictly bounded Mosquitto broker to find the exact physical limit of every workload.

Here is what we built, what we measured, and where single-core stream processing actually hits physical walls.

---

1. The Bottleneck: Why Disk Kills Edge Ingest

In eKuiper and similar edge software, metadata (streams, rules, schemas, auth) is stored in a key-value store or an embedded SQLite database.

Under moderate load (5k–20k msg/s), querying SQLite to validate a rule or look up a schema is unnoticeable. But when 100,000+ messages per second hit an edge machine:

1. Lock Contention: When multiple concurrent connections or rules access SQLite simultaneously, lock acquisition overhead adds tail latency.
2. Flash Storage Stalls: Embedded devices (Raspberry Pi, industrial gateways) do not have enterprise SSD write queues. If the engine writes checkpoint state or logs to flash while simultaneously servicing network buffers, the OS I/O scheduler blocks threads.
3. Queue Backpressure: When a stream processor stalls for even 10 milliseconds waiting on disk, a 150k msg/s MQTT ingress generates a 1,500-message backlog in the socket buffer. Under QoS 0, Mosquitto drops those packets.

If we wanted to push edge ingestion beyond 100k msg/s on a single core, the hot path could not touch the filesystem at all.

---

2. What Changed in v0.500

Redis-Style In-Memory Catalog (MemoryCatalog)

We separated metadata persistence from metadata reads:

// crates/rekuiper-core/src/catalog.rs
pub struct MemoryCatalog {
    streams: parking_lot::RwLock<HashMap<String, StreamDefinition>>,
    tables: parking_lot::RwLock<HashMap<String, TableDefinition>>,
    rules: parking_lot::RwLock<HashMap<String, RuleDefinition>>,
}

• On daemon startup, the catalog hydrates all streams, tables, and active rules from SQLite into memory once.
• During execution, the stream bus, rule evaluators, and REST query endpoints read directly from RwLock<HashMap>, completing lookups in nanoseconds with zero system calls and zero disk I/O.
• Rule mutations update memory first and asynchronously commit to SQLite in the background.

Zero-Disk Hot Path & Caching

• Auth Token Cache: Public RSA keys used for JWT signature verification (KUIPER_AUTH_PUBLIC_KEY_FILE) are parsed and cached in memory. Ingest requests no longer read public keys off disk per request.
• Config & Schema Cache: /etc configuration overlays, source definitions, and JSON descriptors are cached in RAM on first access.
• Connection Pooling: SQL and database sinks now reuse shared connection pools across rule actions rather than acquiring new sockets.
• Multi-Row SQL Batching: For relational sinks (PostgreSQL and SQLite), the sink generates multi-row INSERT INTO ... VALUES (...), (...) statements in parameterized chunks instead of emitting one query per record.

Channel Buffer Sizing

We expanded internal actor queue depths from 1,024 to 32,768 records. On a single pinned core, this provides enough buffer headroom to absorb operating system scheduling jitter without propagating backpressure back into the MQTT network loop.

---

3. Finding the Exact Limits: Hierarchical 1k Peak Search

Rather than testing arbitrary rounded rates, we searched for the exact ceiling of each workload using a hierarchical binary ladder:

• 10,000 msg/s steps to identify the 10k window.
• 2,500 msg/s steps to narrow the bracket.
• 1,000 msg/s steps to find the exact tipping point.

Test Rig (Identical to previous benchmarks)

• Host: 12-core x86-64 machine, Docker on WSL2 (cgroup v2).
• Engine Container: Pinned to 1 CPU core, 1 GiB RAM, --memory-swap=1g, TOKIO_WORKER_THREADS=1.
• Mosquitto Broker: Isolated container on separate cores, with a strict 4,096-message / 1 MiB outgoing queue limit. If the engine falls behind by even a fraction of a second, the broker drops QoS 0 packets immediately.
• Load Generator: mqttgen (our standalone Rust publisher) pushing MQTT 3.1.1 QoS 0 across 8 connections from separate cores.
• Verification: Exact message-by-message sink validation checking count, unique IDs, per-device aggregates, and zero exceptions.

---

4. The Results: Exact Certified Ceilings

Here are the verified limits for all five workloads in rekuiper v0.500-beta:

Workload	Scenario	Certified Ceiling	First Failure	Limiting Factor	Single-Core CPU	Anon RAM
W1	Telemetry Filter (1,000 devices)	150,000 msg/s	151,000 msg/s	CPU saturation (99.4%), broker drops 20.5%	94.5%	17.4 MB
W2	Device 10s Windows (1,000 devices)	200,000 msg/s	210,000 msg/s	Generator schedule (engine lossless to 240k)	94.4%	6.7 MB
W3	ESPHome Topics (10,000 topics, meta(topic))	150,000 msg/s	151,000 msg/s	CPU saturation (99.3%), broker drops 5.0%	97.4%	16.6 MB
W4	Vehicle Windows (10,000 VIN wildcard topics)	200,000 msg/s	210,000 msg/s	Generator schedule (engine lossless to 220k)	97.5%	18.1 MB
W5	EV Charger Sessions (SESSIONWINDOW(10, 2))	126,000 msg/s	127,000 msg/s	Session drain lag (16s exceeds 5s stability limit)	86.7%	6.0 MB

---

5. Workload Deep-Dive: What Broke Where

W1: Telemetry Filter (JSON parsing + condition)

• SQL: SELECT id, device, temp, speed * 3.6 AS speed_kmh FROM telem WHERE temp > 21.0
• At 150,000 msg/s: 0.00% loss, 94.5% CPU, 17.4 MB RAM, 1.0s drain lag.
• At 151,000 msg/s: Single-core CPU hit 99.4%. The network thread could not drain the socket fast enough; Mosquitto's 4,096-message queue overflowed and dropped 20.53% of packets.
• Verdict: 150,000 msg/s is the hard physical CPU ceiling for JSON deserialization, arithmetic projection, and filtering on one core.

W2: Per-Device 10-second Windows (1,000 devices)

• SQL: SELECT device, count(*) AS n, avg(temp), max(speed) FROM telem GROUP BY device, TUMBLINGWINDOW(ss, 10)
• At 200,000 msg/s: 0.00% loss, 94.4% CPU, 6.7 MB RAM, 10.0s drain lag.
• At 210,000–240,000 msg/s: The engine processed all data with 0.00% loss, but the external generator fell off schedule. At 250,000 msg/s, the pipeline collapsed.
• Verdict: 200,000 msg/s is our certified on-schedule ceiling. Memory remained at a tiny 6.7 MB because aggregations (count, avg, max) update in place without buffering raw rows.

W3: ESPHome Fleet (10,000 distinct topics, plain text)

• SQL: SELECT meta(topic) AS topic, self AS state FROM telem
• At 150,000 msg/s: 0.00% loss, 97.4% CPU, 16.6 MB RAM. At the end of sending 4.5 million messages, only 15 messages were in transit.
• At 151,000 msg/s: CPU reached 99.3%, dropping 5.05% at the broker.
• Verdict: 150,000 msg/s is the exact ceiling for routing and extracting MQTT metadata across 10,000 dynamic topics.

W4: Vehicle Wildcard Aggregation (10,000 VIN topics)

• SQL: SELECT device, count(*) AS n, avg(speed), max(temp) FROM telem GROUP BY device, TUMBLINGWINDOW(ss, 10)
• Ingests across bench/vehicles/+/telemetry.
• Sustained 200,000 msg/s with 0.00% loss and 18.1 MB RAM.
• Verdict: Certified at 200,000 msg/s on one core.

W5: EV Charger Sessions (SESSIONWINDOW)

• SQL: SELECT device, count(*) AS n, max(speed) FROM telem GROUP BY device, SESSIONWINDOW(ss, 10, 2)
• 2,000 chargers opening and closing irregular activity sessions.
• Tested in 1,000 msg/s increments:
  • 125,000 msg/s: PASS (0.00% loss, 3.0s lag)
  • 126,000 msg/s: PASS (0.00% loss, 4.0s lag, 86.7% CPU, 6.0 MB RAM)
  • 127,000 msg/s: FAIL (session close lag spiked to 16.0s, exceeding our 5.0s stability limit).
• Verdict: Exact certified limit is 126,000 msg/s.

---

6. Comparison with Other Engines

Here is the updated head-to-head comparison on the common 5k–100k ladder:

Highest Verified Loss-Free Ingest Rate

Workload	rekuiper 0.500	eKuiper 2.4.1	Telegraf 1.40.0	Redpanda Connect 4.109.0
W1: Telemetry filter	150k certified	20k	50k (backlog)	20k
W2: Device windows	200k certified	20k	Lost 6–27%	5k
W3: ESPHome topics	150k certified	20k	50k (backlog)	20k
W4: Vehicle windows	200k certified	20k	Inconsistent	5k
W5: Charger sessions	126k certified	20k	Unsupported	Unsupported

Memory at 20,000 msg/s (Engine Anonymous RAM)

Workload	rekuiper 0.500	eKuiper 2.4.1	Telegraf 1.40.0	Redpanda Connect 4.109.0
W1: Telemetry filter	4.4 MB	15 MB	92 MB	72 MB
W2: Device windows	6.4 MB	536 MB	52 MB (lossy)	1,012 MB (crashed)
W3: ESPHome topics	4.5 MB	43 MB	85 MB	68 MB
W4: Vehicle windows	10.2 MB	886 MB	94 MB (lossy)	993 MB (crashed)
W5: Charger sessions	7.3 MB	832 MB	Unsupported	Unsupported

At 20k msg/s on 10,000 vehicle windows, engines that buffer raw rows in RAM consume 886 MB to 1 GB, risking kernel OOM kills on edge hardware. rekuiper maintains 10.2 MB by computing aggregates incrementally in fixed-size accumulators.

---

7. Real Sustained Score vs. Fake Buffer Backlog

One crucial lesson from this benchmark audit: eventual delivery is not throughput.

If an engine accepts 200,000 msg/s for 30 seconds by buffering everything into a huge internal queue, and then spends the next 45 seconds after the publisher stops draining that backlog, that is not a 200k engine. That is an engine surviving on buffer mercy.

In our benchmark harness:

1. The publisher and broker queues are strictly bounded (Mosquitto queue capped at 4,096 messages).
2. We record the upstream source gap at the exact millisecond publishing finishes.
3. If more than 4,096 messages are backlogged at send-end, or if post-send drain takes longer than 5 seconds, the trial is marked as a failure, even if every single message is eventually written to disk.

When we claim 150,000 msg/s on W1 and W3, it means the engine processed all 4.5 million messages in real-time with an end-of-send backlog of 15 messages and a 1.0s drain time.

---

Trying It Out

rekuiper is free and open source under MIT / Apache-2.0:

• GitHub Repository: github.com/ankur-paan/rekuiper
• Prebuilt Binaries: Available for Linux (x86_64), macOS (Intel & Apple Silicon), and Windows under Releases v0.500-beta.
• Docker:

  docker run -d --name rekuiper \
    -p 9081:9081 -p 20499:20499 \
    -e KUIPER__BASIC__CONSOLELOG=true \
    ankurkrp/rekuiper:0.500-beta

All raw evidence files, harness scripts (mqttgen, iotrunner), and reproduction instructions are published in test/benchmark/iiot-mqtt/.