DEV Community

JhonLiu
JhonLiu

Posted on • Originally published at github.com

How We Built an IoT Platform That Handles 30 Million Concurrent Connections — With a Team of 10

#iot #erlang #opensource #architecture

How We Built an IoT Platform That Handles 30 Million Concurrent Connections — With a Team of 10

DGIOT is an open-source industrial IoT platform. We run 928 gateways across 16 oil fields, process 652 million data points, and maintain 99.9999% uptime. Here's the architecture that makes it possible.

The Problem

In 2021, we got a call from Daqing Oil Field — China's largest oil producer. They had a problem:

  • 928 industrial gateways from different vendors
  • 114,809 sensor points speaking 15 different protocols
  • Data collection every 10 minutes (they needed seconds)
  • 15-30 minute end-to-end latency (they needed <3 seconds)
  • False alarm rate above 20%

The existing system was a patchwork of vendor-specific tools, each with its own database, UI, and authentication. Operators had to log into 8 different systems just to check if a pump was overheating.

They asked: "Can you unify this?"

What We Built

DGIOT is an Erlang/OTP-based platform that acts as a universal translator for industrial protocols. Think of it as a Rosetta Stone for machines.

Modbus ─┐
OPC UA ─┤
MQTT  ──┼──→ Unified Pipeline ──→ TDengine ──→ Dashboard
IEC104 ─┤
A11   ──┘
Enter fullscreen mode Exit fullscreen mode

The key insight: industrial protocols are just state machines. Once you model each protocol as a gen_statem FSM in Erlang, you can handle hundreds of them concurrently with almost zero overhead.

The Architecture: DLAS

We designed a four-layer architecture that separates concerns cleanly:

Layer 1: DATA — Ingestion

  • Parse Server (23 classes) handles device metadata, user auth, tenant isolation
  • TDengine stores 652M time-series data points with 10:1 compression
  • EMQX handles MQTT message routing at 1M+ msg/sec
  • Mnesia/ETS provides in-memory caching for hot data

Layer 2: LOGIC — Ontology Engine

This is our secret weapon. We built a 252-entity OWL ontology that models industrial equipment:

Pump ⊑ Equipment ⊓ ∃hasPart.Bearing ⊓ ∃measures.Pressure
Bearing ⊑ Component ⊓ ∃hasFailureMode.Overheat
Overheat → triggers(Alert) ∧ reduces(RemainingLife, 0.8)
Enter fullscreen mode Exit fullscreen mode

These rules are compiled to Erlang pattern matches. When a pressure sensor reads 2.35 MPa, the platform doesn't just display a number — it knows that pump P-103's bearing is trending toward failure, cross-references the maintenance schedule, and generates a work order.

Layer 3: ACTION — Shadow State Machines

Every physical device has a "shadow" in software — a gen_statem process that mirrors its real-world state:

init()    → auth → online → {normal, alarm, offline}
normal()  → evaluate(Rules) → publish(MQTT)
alarm()   → escalate → notify → acknowledge
offline() → retry(5) → alert
Enter fullscreen mode Exit fullscreen mode

The shadow absorbs network interruptions. If a gateway goes offline for 30 seconds, the shadow holds the last known state and retries. The dashboard never shows "disconnected" unless the device has been unreachable for 2+ minutes.

Layer 4: EDGE — iotStudio

A Python + Vue 3 edge agent that runs on ARM gateways. It does:

  • Protocol adaptation: Modbus RTU → MQTT bridge
  • Stream computation: Moving average, anomaly detection, deadband filtering — 15 algorithms
  • Offline autonomy: 7-day buffer if cloud connection drops
  • OTA updates: Rolling firmware deployment to 928 gateways

The Numbers

Metric Before DGIOT After DGIOT
Collection frequency 10 min 1 second (600×)
End-to-end latency 15-30 min <3 seconds
Storage retention 10 days 2 years
False alarm rate >20% <5%
Concurrent connections ~1,000 30M verified
Protocols supported 3 300+

Why Erlang?

We get this question a lot. The answer is simple: Erlang was built for this.

  • Preemptive scheduling: A slow Modbus device can't block a fast MQTT stream
  • Hot code reloading: We deploy updates without shutting down 928 gateways
  • Let-it-crash philosophy: A failing protocol adapter restarts in milliseconds without affecting others
  • Built-in distribution: Adding a new node to the cluster is one command

The same reasons WhatsApp used Erlang for 2 billion users. Industrial IoT has the same requirements: massive concurrency, high availability, and soft real-time.

Open Source

DGIOT is Apache 2.0 licensed. Full source at github.com/dgiot/dgiot.

We made this choice deliberately. Industrial IoT is too fragmented for any single company to solve. The only way to cover 300+ protocols is to let the community add them. Every new protocol adapter benefits everyone.

git clone https://github.com/dgiot/dgiot.git
cd dgiot && docker-compose up -d
# Open http://localhost:5080
Enter fullscreen mode Exit fullscreen mode

Lessons Learned

  1. Protocols are the moat. Every new protocol adapter opens a new industry. We went from 3 to 300 in 4 years — each one brought new customers.

  2. Ontology beats rules. Hard-coded alarm thresholds break. An ontology that understands "pump→bearing→overheat" generalizes across equipment types.

  3. Open source is the distribution. We don't have a sales team. Our customers find us through GitHub, then ask for enterprise support.

  4. Erlang is a competitive advantage. In a world of Node.js and Python IoT platforms, Erlang's concurrency model lets us handle 100x the load with the same hardware.

  5. Community > code. The 70K developers in our community have built integrations we never would have thought of.

Liu Shouxin is the founder of DGIOT, an open-source industrial IoT platform. He previously worked at EMQ, Huawei, and Kaspersky. He lives in Hangzhou and Dallas.

Top comments (0)